Mitch DeRidder, 1-03-2006

This section attempts to group related data from the study as a step towards summarization.  The source of this data is my notes from the usability sessions.   

Participants are identified here by sequence number, e.g., P1, P2.  Since not all participants attempted all tasks, sometimes I note the number of people that could possibly appear in a list, e.g., "out of 6 possible".  In most cases, the number of possible related observations is eight.

Bear in mind that the absence of an event in this section does not necessarily mean that the event did not occur in the study.  Events may be absent here for two reasons:

-        My notes may be incomplete.

-        To improve focus on the more important information, I intentionally exclude some observations.  For example, I exclude individual user comments that do not seem particularly interesting or valuable.

Preliminary Questions

Note:  The preliminary questions were asked only of the experienced Astra users:  P1, P5, P7, and P8.

Most common task was adding authorizations, as for "new" people:  P1, P5, P7, P8

Other Astra tasks:

-        Deleting authorizations:  P1, P5, P7, P8

-        Authorizing a group for a new application:  P5, P7

-        Changing authorizations:  P1, P8

-        Look up people to see who has access to what:  P7

-        Astra web service coding to update our security to use a new role:  P1

Complaints about Astra, v1: 

-        Number of pages (including confirmations) required for a single task:  P1

-        There's a small field into which I sometimes need to enter a series of budget numbers or org codes, separated by commas.  It's hard to check the entry due to the small space (1/2 inch high by 1 inch wide).  I want to see all the entries at once:  P5

Remarked that he/she likes Astra v1:  P5, P8

FAQs

FAQ about authorizer vs. delegator: 

-        Likes this FAQ:  P1, P2, P4, P7

-        Should also discuss users, to help support next FAQ, which regards making someone both an authorizer and user:  P3

-        Was asked by someone else to tell me that this is a point of confusion:  P3

FAQ on how to see people in my unit: 

-        Likes this FAQ:  P6, P7

-        Shows what roles have been assigned?  Would expect to see individual names and authorizations:  P3

Likes FAQs in general:  P3, P6

Would not use FAQs:  P8

Wants a schematic overview of procedures as an FAQ:  P4

What's New

Due to my inexperience, this section would not be relevant:  P2, P3

Does not understand mention of the cart here:  P2, P3

Likes the link about modifying authorizations for multiple people at the same time:  P6, P7

Likes the link about doing things with fewer clicks:  P6

Likes this section:  P4

Start Page (i.e., shown before login)

Likes inline link to log on as authorizer:  P2, P3, P6

Link to log on as user: 

-        Likes this link:  P3

-        Would expect the user to go to his/her Astra administrator for such info:  P5

Likes the "Huh?" link:  P4

Would prefer not to scroll:  P1

Home Page

Likes link to view and edit authorizations:  P2, P5, P7

Likes link to create a new authorization:  P5, P7

Link to review pending authorizations:

-        Likes this as a way to resume after an interruption:  P7

-        Not sure what a pending authorization would be; why would I do half the work and not finish it?  P6

Remarked that seeing the link about checking out helped them understand the UI:  P6, P7

Not sure if news and such should appear here as well as on Start page:  P1, P4

Notes redundancy of statistics with start page:  P8

Would prefer not to scroll:  P1

Likes availability of Help:  P4

Too much to read here; excess text is in the way:  P8

General UI

People generally understood the cart/checkout concept, though they were sometimes surprised to see it in Astra.  Caveats:

-        P3 rarely buys online (her husband usually handles that), so she related our cart to a physical cart in a brick & mortar store.  She understood that our cart was intended to aggregate something.

-        P8 initially supposed that the cart might contain all authorizations she had ever performed.

-        P6 checked out to ensure that a set of search criteria was saved.

Nearly everyone committed (or tried to commit) their pending authorizations on a per-task basis.  The exception was P8, who intentionally chose to accumulate pending authorizations for the first two tasks.

Searched with <lastname>, <truncatedfirstname>, e.g., "trump, d":  P1, P2, P3, P5, P6, P8

Wildcard searches for people:

-        "Smith, J%":  P5, (after trying "Smith, J", which should have worked)

-        "Smith, J*":  P7, (after trying "Smith, J", which should have worked)

If there's just one choice in a dropdown list, it should be pre-selected:  P1, P8

If there are no choices in a dropdown list, dim it:  P1

Would like a way to see past authorizations by me:  P4

UI Navigation

The top navbar was well liked:  P1, P4, P6, P7

Still, equivalent inline links were used:  P3, P5, P6

Chose to view and edit to create new authorizations:  P1, P2, P4

Expectations as to the effect of the links on people's names in the search results:

-        Expected a directory listing:  P1, P4, P5.  (P1 suggested using a rolodex card image next to the name link for access to a directory entry.)

-        Expected a listing of the user's authorizations:  P7

-        Thought it would place the name in the (search-string) entry field:  P2

Difficulty finding where to add notifications (out of 6 possible):  P4, P7

Navigation to or from the Org Code / PUC dialog

-        Upon committing this dialog, felt she should be taken to a new screen that would guide her to the next step, not back to Advanced Search:  P2

-        Remarked that going to a new page for this search made him feel separated from what he had been doing:  P1

Cart-related navigation observations:

-        P7 clicked Checkout in the navbar, expecting to go to the cart; seeing no feedback, she attempted to start the task over.

-        P2 performed just one of two required edits and could not see a good way to resume editing while at cart.

-        P1 said he would rarely go to the cart; confirmatory information received while making changes would suffice to assure him that the desired changes are pending.

-        However, P1 liked that creating an authorization takes you to the cart.

Required prompting to go to Advanced Search for Task 4:  P5

Users often pressed the <Enter> key to initiate searches, e.g., at the person-finder for Create New Authorization.  I did not systematically log those events, but at least the following participants used <Enter>:  P1, P2

Create New Authorization

Permissions radio button array (disabled):

-        Prefers not to see what he can't use:  P1

-        Wanted to click Can Authorize:  P2

-        Tried clicking several times; was unsure if it would allow her to give permissions:  P3

-        Not sure what do here:  P4

-        Are they mine or the person I'm authorizing?  P5

-        Tried clicking to see if the buttons worked:  P5

Changed start date to current date:  P2, P5, P6

Find Person button: 

-        Not sure if she should click Find Person button; does not understand why this is necessary if correct identification has been entered:  P3

-        Clicked button belatedly on Task 7:  P6

Clicked OK when "OK and Add Another" should have been selected:  P8, P3 (wanted to display a clear screen for entering another authorization)

Help: 

-        Supposed that Help for the Application field would show a list of all possible applications, with descriptions:  P1.

-        Would use Help at Role to see definitions of the available roles:  P1

View & Edit Authorizations

Person Search

In the person search control set, there is not a way to remove a person from a list of selected people:  P1

Clicked on user name already in Selected Persons box, supposing the box to be a multi-select list.  Did not understand that the user was already (auto)selected:  P1

For Basic Person Search, "lookup" seems more appropriate than "search":  P1

Editing

Ultimately understood deleting + creating to edit (out of 7 possible):  P1, P2, P4, P5, P6, P7, P8

Feedback text from editing (out of 7 possible):

-        Concerned about the deletion, or the delete + add:  P2, P7, P8 (wants more detail)

-        Understands the feedback:  P1, P4

Authorization display is unchanged due to editing (out of 7 possible):

-        P1 wants to see the changes shown, possibly together with original values:  P1

-        P6 expected to see the dollar limit updated.  She tried to re-edit because of this.

Tried to edit authorizations in order to delete them:  P8

If an error was made when editing, P1 would prefer to re-edit immediately without first going to cart to remove the incorrect edit.

Changed effective date when editing dollar limit (out of 7 possible):  P2, P5

Other

Likes the authorization detail:  P4, P5

Expanded view:

-        Likes the notification info:  P4

-        Likes the history:  P7

Delete button:

-        Tried unchecking authorizations prior to selecting Delete:  P8

-        Would expect the Delete option to be grouped with the Copy and Edit actions:  P1

-        Initially chose Unsubscribe action instead of the (required) Delete button:  P3

Did not readily see role information being sought (out of 6 possible):  P6, P7

Likes the icons for Print and Extract:  P1

Likes the option to assume ownership:  P4

Search filter specification:  The typeface is too small and the pale blue background is just gray to me.  Something bolder is needed here:  P2

Advanced Search

Task 4 implied that the org code should be specified in conjunction with the Person field:

-        Correctly selected org code criterion in Person field, without prompting:  P2, P3, P6, P7

-        Selected an org code criterion in Access Restriction Level 1 field:  P1, P4, P5

-        Selected org code criterion in Grantor field:  P2, P3, P8

-        Remarked about difficult of finding where to put an org code:  P2, P3, P5, P6

-        Chose not to use Advanced Search on task that required it because it was not obvious where to enter an organization code:  P8

Org code entry:

-        Did not go to org code selection dialog without prompting:  P2, P3, P5, P8 (but she came back to this page to do this later)

-        Entered org code in search name field:  P3, P7

-        Hoped to be prompted for an org code upon selecting Search:  P2

Task 5 implied that the PUC should be specified in conjunction with the Access Restriction Level 1 (ARL1) field.  Out of 7 possible observations:

-        Correctly selected PUC criterion in ARL1 field:  P1, P2

-        Selected PUC criterion in Person field:  P4 (who does financial work), P5, P6, P7, P8

-        Selected PUC criterion in Grantor field:  P2

-        Did not use PUC selection dialog without prompting (out of 6 possible observations):  P2, P4, P7 (entered "puc 0331" in search name field)

Search Name field:

-        Entered org code in search name field:  P3, P7  (repeated from above)

-        Thought Search Name field is a place to put a name to search for:  P5

-        Likes that you can name the search immediately:  P1

Permission choices, where the correct choice - Can Use - was pre-selected:

-        Selected Can Authorize instead of Can Use:  P3, P5

Did not select application as required by search task (#4):  P6, P7

Saved a search:

-        In org code dialog:  P5, P6

-        At advanced search page:  P1, P5

-        At search results page:  P2, P4, P6, P7, P8

Likes the more extensive search capabilities:  P8

Prefers to select application first because it seems like the primary field:  P1

Wishes that the required search criteria were indicated with red asterisks:  P2

Org Code / PUC Selection Dialog

Org code punctuation:

-        None:  P2, P4, P5

-        Dashes:  P6, P7 (in search name field)

-        Spaces:  P3

After finding the org code:

-        Tried Select link next to entry found:  P1, P3, P6, P8

-        Tried "Continue at Next Level" link next to entry found:  P1, P2, P3, P5, P6

-        Not sure which button to click after typing org code or PUC:  P3, P4, P5

-        Remarked that there are too many choices of action:  P1, P3

-        Picked OK when Update was required (in early version of mockup):  P1

-        Picked Update unnecessarily:  P2, P4, P5 (trying to save the search), P6

Remarked that if you need to select only one entry, the dialog should not support multiple selection:  P1

Cart

Likes that the cart provides an opportunity to correct errors:  P6, P8

The sort control field was right next to the view options on this page: 

-        Remarked that the sort and view controls are too close:  P1

-        Did not understand "Expanded Sort":  P3

Checkboxes:

-        Marked the checkboxes to prepare for checkout:  P6 (Task 1), P7

-        Marked checkboxes and clicked "Remove from Cart" just to see what the result would be:  P3 (Task 2)

-        P3 checked and unchecked the boxes, wondering what they are for.  Theory:  Could check a checkbox and click Expanded for more information on an item; but parallel logic seems inapplicable to sorting, so the theory seems refuted.  Decided she should check the boxes and click Delete before checking out.  Later said that she would never have supposed that removing items from the cart would undo her actions.  (This participant was unfamiliar with online shopping, and she does not perform E-procurement.)

-        P8 deleted all authorizations from cart in Task 2.

Proceed to checkout link is too small and not located where expected (at bottom):  P2

Prefers sorting by column headers instead of with dropdown list:  P1.

Checkout

What's a PERM?   P1, P4

Committing checkout:

-        P4 repeatedly failed to check out while at Checkout, thinking the process to be complete.  Sometimes she wondered about the lack of feedback.  When I pointed out the problem to her, she asked why the effect of "Checkout" on the navbar is so different than the "Check out now" button.

-        P5 thought checkout was already done at the time of finding "Check out now" button.

-        P7 became disoriented at checkout in Task 1 and complained about this page during later tasks.  In Task 1, she remarked that she does not see the user's name anymore.  "Either I'm in the wrong place or I don't see what this screen is doing."  During wrap-up, she singled out this page as problematic.

-        P8 chose not to check out during first 3 tasks but did so when told (to pretend) that it was time to go home.

Would like to type an address directly into the Additional Recipients box:  P1.

Additional Recipients control set:

-        Says not intuitive:  P1

-        Not clear that selected person will receive notification:  P4

-        Tried to type recipient into notes field:  P7  (said more explanatory text would have helped)

-        Believed that it would allow her to enter more authorizations:  P3

-        Surprised to find this on checkout page:  P4

-        Selected to "Clear selections and start over", intending to empty the cart:  P7

Wanted more information on what she was about to commit:  P7

Need a way to delete individual entries from the notification list:  P1

Would not want to receive notifications himself:  P1

Checkout Complete

Feedback specificity: 

-        Wanted to see who was notified:  P6, P7, P8

-        Wanted specific detail on what authorization changes were performed:  P2.  (For Task 5, P2 wanted to see the $3000 amount specified in the feedback, or at least something about increased limits.)

-        Skipped the cart and wished she had seen more feedback:  P8

Presentation:

-        P2 would like the feedback to be larger or darker; it's too difficult to read.

-        P2 would like "Checkout Complete" to be larger, maybe centered, and with more space around it.

Likes that the number of items in the cart is now zero, confirming that the pending authorizations have been processed:  P2.

Wrap Up

How well liked (bear in mind that users were not aware of all their errors):

-        6 out of 10:  P1, P2, P3, P5, P8

-        7 out of 10:  P7

-        8 out of 10:  P4, P6

-        But Advanced Search is gets just 2 or 3 out of 10:  P5

Appearance:

-        Aesthetics: 

o      3 out of 10:  P1

o      4 out of 10:  P8

-        Needs larger typefaces:  P2

-        Text is too small in secondary navbar:  P7

-        Draw attention to key information:  P2

-        Easy to look at:  P3

-        Cluttered:  P2

-        Not cluttered:  P3

-        Colors are too "flat":  P7

-        Very plain, colorless, blue & white:  P8

Would sometimes prepare authorizations changes for more than person before committing: 

-        Yes:  P1, P4, P5, P6, P7

-        No:  P2, P3, P8

Would add recipients: 

-        Yes:  P2 (spontaneously did so), P3, P4, P5, P6, P7, P8

-        No:  P1 (not typically)

Intersection of participants who (would prepare authorizations for multiple people) and (would add recipients):  P4, P5, P6, P7

Other improvements needed:

-        P1:  Efficiency

-        P2:  I am very interrupt-driven at my job.  It's important that when I go back to a screen after an interruption, I can see where I am.  On the screens we looked at, I would feel ill at ease. 

-        P2:  I don't authorize people much.  I would like to be able to just call an expert to have this done. 

-        P2:  I would like phone support where people would talk me through problems as needed.  It's too much work to enter problem information into an email, and I can't wait 48 hours for a response.

-        P5:  Nomenclature improvements.