Mitch DeRidder, 1-9-2006

Here, the frequency of an observation is sometimes expressed in this form:  2/7, which means that the observation was recorded for two participants out of seven participants possible.  The denominator of the expression is reduced where a particular observation could occur only in a task which was not attempted by all participants.  In general, however, an observation could occur for all eight participants.

Participants are identified here by sequence number, e.g., P1, P2, etc.

Astra 1.x

The four experienced users of Astra all said that their most common task was adding authorizations.  All of them also delete authorizations, and two participants said they modify authorizations. 

Two of the experienced users have authorized a group of people to use a new application.  Also, at least one of the eight participants mentioned recurring situations in which she needs to give the same authorization to several people at a time. 

Suggestion:  Consider making it easy to grant the same authorization to multiple people.

Information Units

The participants were particularly interested the FAQ link about the difference between an authorizer and a delegator (4/8).  This level of interest suggests that our audience finds this difference confusing.  One user suggested that the article should also distinguish users who are neither authorizers nor delegators.

Two users expressed interest in the FAQ on how to see people in "my unit" and the "What's New" regarding how to modify authorizations for multiple persons at once.

However, the "What's New" section seemed irrelevant to some participants (2/8) because of their lack of Astra experience.

Also, on the Start page, the What's New section mentions "your cart" without any prior explanation, as two participants pointed out.

Two participants felt that there is too much verbiage in the early screens or objected to the need for scrolling, and a third participant found the UI to be cluttered.  Also, at least three participants pointed out or questioned the redundancy of information on the Start and Home pages.    This feedback came from both experienced (2/8) and inexperienced (2/8) users.

Suggestion:  Consider listing out the article links on the Start page only.  On the Home page, provide links that lead to article listings.

Some participants wanted to use the (unavailable) help system during the study or remarked favorably about its availability. 

Start Page

Three participants said they liked the presence of the inline login link here.

Home Page

Here, participants said they liked the links for viewing and editing authorizations (3/8) and creating new authorizations (2/8).  Also, two participants remarked that they were helped by seeing the Step 3 verbiage + link about the need to check out.

General UI

The cart metaphor was readily grasped by all participants, although isolated specific misunderstandings or uncertainties were observed.  Users were typically surprised to see the cart metaphor used in Astra, but no one stated that this usage was objectionable.

Nearly everyone committed (or tried to commit) their pending authorizations on a per-task basis.

Typical overall rating of the UI was 6 out of 10 (5/8).  However, when considering this rating, bear in mind that the participants were not aware of all their errors.

Half of the participants found the aesthetic appearance of the UI to be lacking.  An underlying complaint was lack of color (2/8).

P1:  If there's just one choice in a dropdown list, pre-select it and dim the list.

UI Navigation

Participants remarked favorably about the top navigation bar ("navbar") (4/8), but several participant preferred using equivalent inline links (3/8).

Several participants were inclined to first view a user's current authorizations before adding authorizations (3/8).

In the basic people search facilities (at View & Edit Authorizations, and Create New Authorizations), the user-name links were widely misunderstood (5/8).  Often these links were assumed to lead to directory listings for users (3/8).

Recommendation:  Consider using some other selection method here. 

Upon arrival at the cart, users do not know how to get back to the search results to continue working (at least 4/8, including two observations categorized under Create New Authorization in the compiled results). 

Recommendation:  Provide an obvious way to return to the latest search results.

Going to the Org Code / PUC selection dialog seems to distance users from the task at hand (2/8).

Recommendation:  Use popup windows or a dedicated dialog frame for secondary dialogs.

Also, the left-side navigation options did not seem to draw the level of attention they sometimes deserved.

Recommendation:  Find a way to emphasize these options further.

Create New Authorization

The presence of an inoperable Permissions radio button array was problematic (5/8).

Recommendation:  When the user has no choice of permission type, present the permission alone as text, without an apparent control that might be perceived as interactive.

Several participants changed the start date to the current date (3/8).  I presume we'll use the current date for default start dates in the actual application.

View & Edit Authorizations

Participants ultimately understood the notion of that an Astra edit consists of a deletion plus a creation (7/7).  Users typically found the immediate feedback from an edit to be insufficient (5/7).  The inserted feedback text was not detailed or explanatory enough (3/7), and users expected to see their changes somehow reflected in the results listing (at least 2/7).

Recommendations:  Provide a more explanatory and detailed feedback string.  If possible, show both the original and modified values in the results listing.

Users remarked favorably about the authorization detail, notification information, or history in the authorization listings (4/8).  However, in a task requiring the user to find specific authorization information (a role), two of six users experienced difficulty, even though the search returned only four entries.  I think this is a drawback of the current "stacked" information layout, relative to a completely columnar layout.

Recommendation:  Provide a way to emphasize or isolate a chosen aspect of the stacked data.  For example, we could provide a way to put all roles in bold.

P2:  The typeface is too small in the search filter specification.

Advanced Search

Users were overwhelmed by the Advanced Search facility.  Common problems:

-        The distinction between belonging to a group and administering a group was completely overlooked or unrecognized, even by our most experienced participants.  This distinction was expected to govern whether the group is selected in the People field or the Access Restriction Level 1 field. 

-        Users generally experienced difficulty finding a way to specify that an org code criterion was desired.

-        Users experienced difficulty finding where to enter codes (6/8).

-        The Grantor field was misused due to lack of understanding (3/8).

-        Users set the Permission criterion incorrectly (2/8).  (The tasks did not require changing this setting.)

-        Even the Search Name field was misunderstood (3/8).

Recommendations:  -        Consider expanding the basic person search facility to include common groupings.  This would reduce the need for users to go to Advanced Search.

-        Look for more descriptive field captions in Advanced Search, e.g., "Person or group" in place of "Person".

-        Query automatically for required parameters, e.g., org code or PUC, and provide a more obvious way to (re-)display the query.

-        Consider developing a supplemental wizard that guides construction of queries involving relations more complex than belonging. 

-        Provide field-level help for each field in Advanced Search.

Participants typically saved their search criteria after running the search (5/8).  Searches were also saved at the advanced search page (2/8) and in the org code dialog (2/8). 

Org Code / PUC Selection Dialog

Some users punctuated the org code with dashes or spaces (3/7).

Recommendation:  Ignore any punctuation of a number.

Users did not know what to do after finding an org code or PUC (7/8).  (They just needed to click OK.)

Recommendations:-        Dim any controls that are not usable at the moment.

-        Consider alternate methods of building a list.

-        Better explicate any hierarchy traversal controls.

Cart

Half of the participants struggled with what to do with the checkboxes here.

Recommendation:  Eliminate the checkboxes by using buttons or links for all actions, per entry.

On this page, the view and sort controls were too close to each other, affecting 2/8.

Checkout

This page was especially problematic:

-        Some users believed they had already accomplished checkout (2/8).

-        This page was confused with the cart (1/8).

-        The page does not advertise its primary function, which is adding notifications, making that function more difficult to find (2/6).

Suggestion:  Call the page "Notifications", or turn it into a secondary dialog.  On the cart page, enable users to check out now or add notifications via a pair of buttons.

Users found the Additional Recipients control set unintuitive or confusing (4/6).

Suggestions:-        Improve clarity as to what controls and instructions belong to this set.

-        Consider building the list to the right of, or below, the search controls.

-        P1: Provide a way to remove individual entries from the list.

Two users remarked that they do not know the meaning of PERM.

Checkout Complete

Three users asked to see who was notified.

P2 requested that "Checkout Complete" be more emphasized and that the feedback information be larger or darker.

Wrap-Up

Nearly all of the users (7/8) said they would add notification recipients on an ad hoc basis; these are typically directed to the person who requested the authorization change(s).  Half of the participants would both (a) prepare authorization changes for multiple users before committing and (b) add notification recipients.

Recommendation:  Provide a way to associate additional notifications with authorizations; consider doing this on a per-grantee basis.