Testing to assess ease or efficiency of use through the study of user interaction with a product or product simulation.
Definition of Usability Testing
A usability test is procedure that assesses the ease or efficiency with which a product or feature can be used. A defining characteristic of a usability test is a purposeful interaction between users and a product or product simulation. During the test, this interaction is observed to acquire information about the usability of the product or a product design. The resulting test data are analyzed to identify interaction problems and their frequency of occurrence. Other types of quantitative or qualitative results may also be derived from the data.
Usability testing can be applied to physical products and packaging as well as to software user interfaces. Also, usability tests can be designed to compare products or to compare design options for a given product.
The nature of a usability test is greatly affected by the goals of the test, the types of information sought, and other characteristics, as discussed in the following articles:
Expected Benefits
The main benefit of usability testing (and the consequent UI redesigns) is that your product will be easier and more enjoyable to use. The improved ease of use may be manifested in various ways:
- Reduced or eliminated training time
- Fewer requests for assistance
- Prevention of errors (and thus prevention of the consequences of such errors)
- Improved user performance
- Better acceptance of the product, and fewer cases of people "going around" the product to accomplish particular tasks
For the development organization itself, advantages of usability testing can include the following:
- Reduced risk that the product will be unsuccessful, and a corresponding confidence that the product will be well received
- Avoidance of introducing problematic code into the code base. Once a usability problem has been released, there tends to be more inertia to overcome in correcting the problem.
- Reduced or eliminated uncertainty about how to address specific aspects of the UI design
- Improvements in UI design skills
- Accolades for development of high quality products
When to Run Usability Tests
For the kinds of work done at UW Technology, usability tests are best conducted while a product's user interface is being designed, following the collection and analysis of relevant user requirements. For best effect, usability tests can be performed with a user interface simulation, rather than with actual product code. By the time the actual product UI becomes available, changes to the UI design tend to be more costly and pose more risk to the product stability and schedule. In practice, fewer and smaller UI design changes are made to a product after its actual UI becomes operational, and so problems identified by usability tests in the latter part of a development cycle are often not satisfactorily resolved in that release of the product.
During the period devoted to design and usability testing, minimize effort on product components that make assumptions about the design of the user interface. Insofar as possible, extend the UI design period by structuring the product in such a way that code for the user interface navigation and presentation is separated from the remainder of the product code.
Although UI design and usability testing commonly follow the necessary gathering and analysis of user requirements, these stages can also be successfully overlapped. For example, suppose an additional feature is folded into the product plan -- a feature for which additional data about user requirements is needed. A usability study could be devised that combines requirements-gathering interviews with some usability testing and design review of preliminary paper mockups of the feature's user interface. Results of the study could provide a detailed understanding of how the feature and the proposed user interface would fit (or not fit) the needs of the representative users sampled.
Besides illustrating a possible overlap in development stages, this example shows how a usability study may include activities besides usability testing. The example also illustrates the benefit of planning usability studies in a flexible manner and in response to current needs for design-related information.
Further more about planning for usability testing in your project, see the following articles:
Examples and Ideas
Steps for conducting a typical software usability test
Here is a set of steps for setting up and running a typical usability test that is aimed at identifying usability problems in a proposed software user interface.
1. Establish the purpose of the test and resources available
- Identify the goals of the study and the user population(s) from which participants should be drawn.
- Allocate or identify the personnel needed to plan and carry out the study.
- Think about how you will use the resulting data to improve the product.
2. Design the usability test
Work out a more detailed proposal for the study:
- Identify the specific UI features or capabilities that should be exercised by the test.
- Outline the tasks and interview questions, devoting just a sentence or two to each item.
- Plan the number and type of participants, and identify any recruitment restrictions.
For a typical usability study, plan on three to ten participants. If you need different types of participants (e.g., some experienced, and some inexperienced), include at least three participants of each type.
As the number of participants increases, the fewer new usability problems are discovered per participant. However, with a greater number of participants, the prevalence of each problem in your sample more closely resembles what you would see in the entire population from which the sample was drawn; thus you can better discern the relative importance of each problem.
Review this proposal with the design team, and work out a general plan for building a UI simulation. Construction of a UI simulation can now begin.
3. Schedule the test and recruit participants
It is very important that UI simulation be ready at the time the usability test begins. Wait until you have a confident estimate of prototype availability before scheduling test participants. In the university context, staff participants can be readily recruited during the week prior to the test.
Consider designating the first usability test session as the "pilot" session, which serves as an opportunity to try out the UI prototype and procedure. Between the pilot session and the first regular session, allow enough time for correcting the prototype and procedure.
If you plan to use a usability lab, make arrangements to use the lab before scheduling participants.
On a calendar or worksheet, lay out a set of time slots, allowing for about 50% more slots than participants. This makes it easier for willing recruits to find a time that they can participate. Try to schedule the sessions at times that will be convenient for observers to attend, also, i.e., not too early or too late. Be sure to allow enough time between sessions to perform any setup necessary for the next participant. Also, allow some time for discussion after each session.
When recruiting (and when working with participants), use the word "study" rather than "test", to avoid an implication that the participant will be tested.
On the day before a test session, discreetly remind the participant about the test session.
4. Finalize the tasks and procedure
Prepare the final task statements, ensuring that the details of those statements are consistent with the UI simulation. Provide just enough information to explain the task, and avoid verbiage that unduly corresponds to that of the UI design.
To keep the participant focused on the current task, print a version of the tasks that shows just one task per page. Also prepare a task list for observers, so they can follow along more easily; this version may also include the goals of each task and steps to a correct completion.
A final plan for your procedure might include such details as the following:
- A specification of the initial state of the test system(s)
- How the participant will be oriented to the test setting and his or her role in the study. See example of participant orientation.
- When and how participants should receive assistance with tasks
- When any cameras and recording equipment should be turned on
- Where the test administrator should sit or stand during the test
- Who, if anyone, will take detailed notes of observations during the study; this activity is commonly referred to as "logging", and the notes are often referred to as the "log".
5. Conduct a pilot session
Set up the lab as necessary, and then run the pilot session. Since this session may or may not go well, consider limiting the list of invited observers to those people most involved in preparing for the test. After the session, discuss with the team what needs to be corrected or changed before the next session. If the session went well and the pilot participant was representative of your product's audience, you may be able to include the data from the pilot session in the results of your study.
6. Conduct the usability test
a. Orient the participant
- Review the general purpose of the study.
- Briefly describe any cameras or other data-capture facilities.
- To put the participant at ease, say that he or she may take a break or end the session at any time.
- Emphasize that the purpose is to evaluate a software design, not to evaluate the participant.
- Briefly explain any consent forms or nondisclosure agreements that must be signed.
b. Obtain written consent
For formal usability studies at the University of Washington, both the test administrator and participant must sign a standard consent form for human subjects compliance before starting a test session. (Contact usable@u.washington.edu for the current form.) With the participant's permission, you may want to start a videotape recording of the session at this time.
c. Conduct a preliminary interview (optional)
A preliminary interview is optional and may not be entirely related to the actual test. Examples of topics you may want to cover in a preliminary interview include the following:
- The participant's requirements for the product: Requirements gathered at this time can be unaffected by any participant knowledge of your proposed design.
- Terminology: This is an opportunity to discuss relevant terminology before the participant sees the terminology that has been used in the UI simulation.
- The participant's job setting, materials, or procedure: Such information can indirectly supplement your requirements data or inform the design process of potential metaphors or terminology.
d. Provide explicit instructions
Here, you may want to discuss the general task scenario or relevant aspects of the hardware setup. This is also a good time to encourage resourcefulness while assuring the participant that he or she is not being tested. Finally, this is the best time to introduce a think-aloud protocol, wherein the participant endeavors to think aloud while working on the tasks. This protocol can be provide much valuable qualitative data and can aid in monitoring the participant's stress level.
e. Administer the tasks
Give the user tasks to complete, one at time. Task completion should include an evident awareness or declaration, on the part of the participant, that the task is indeed complete.
You or someone else should log observations as the participant works. For anonymity of user performance, use a symbol (e.g., "Pilot", "P1", etc.) in place of the participant's name on all notes, tapes, or other records of the session. Keep any key to participant identities separate from the session data. Compliance with this guideline on anonymity is required when using human subjects.
Sometimes (or often), a participant will experience uncertainty or difficulty in carrying out a task. Insofar as reasonable and practicable, respectfully encourage the participant to draw upon his or her own resources to overcome such obstacles. Once it becomes clear that task completion is blocked by a problem, the test administrator typically provides the participant with just enough information to proceed further. Thus, further usability problems, encountered at later steps of the task, may also observed.
f. Conduct a closing interview (optional)
After the participant has attempted as many tasks as possible in the available time, ask any further questions you have planned. The participant is often queried for ratings or general opinions at this time. Realize, however, that the actual task attempts constitute a more direct -- and thus more valid -- source of information than do verbal reports and ratings at the close of a test session.
g. Thank the participant
Make it clear that the user's participation and feedback has been very helpful and will have an important influence on the product design.
7. Analyze the results
Between test sessions or after the test is complete, study the session logs to compile a list of the usability problems observed. Notes about why or in what manner the problems occurred may be very helpful in designing effective solutions. When prioritizing problems, it is helpful to know how many of the participants were affected by each problem, and whether the problem blocked task completions. As you work through the logs, also note any interesting comments or suggestions from the participants. Substitute symbols for participant names in your compilation and analysis to preserve anonymity of user performance; this is required for compliance with guidelines on the use of human subjects. See sample compilation and summary of results.
If a related study was run previously for the product, compare the findings to identify areas of improvement or deterioration in usability. This is a great way to learn what changes to a UI are likely to be most effective. Be aware that small differences in measured performance should generally be attributed to chance rather due to differences in the designs tested.
8. Discuss the findings
When your compilation is ready, discuss the findings with the product design team. Work through the items systematically, i.e., in order of importance, or by task, or by screen. Together, devise solutions to the problems and address participants' comments and suggestions.
As you proceed, begin thinking about what to test or re-test in the next iteration. In particular, if you are unsure about the effectiveness of any solution to a usability problem, you should plan to test that solution.
Tips on usability testing
*Ensure that the participant feels secure.*Provide a safe, nonthreatening environment for the participant. Manage the participant's stress level throughout the study. Also monitor the participant for possible exhaustion. Provide opportunities for breaks between tasks. At the close of the session, you'll know you handled the emotional environment well if the participant remarks that the session was fun.
Also help the designer(s) feel secure. Usability tests are intended to identify design problems, and so the designer(s) of the user interface may feel personally threatened by the test and the results. If you sense these feelings, assure the designer that usability tests almost always find problems. Emphasize that the goal is to make the product better, not to evaluate the designer. Engage the designer in working out solutions to the problems. This will keep him/her constructively involved and will provide assurance as to continued ownership of the design.
Avoid running the usability test as a seminar or demonstration. People have a natural tendency to want to help. The test administrator must manage this tendency in himself or herself so as not to affect the test results unnecessarily. Be aware that even nonverbal cues can be highly informative to the participant. Having an impartial person run the usability test is helpful in preventing this contamination of your results. Remember: Even though the participant knows less about the UI simulation than anyone else involved, you are there to learn from the participant, not the other way around.
*Record observations during each session.*Instead of relying on your memory, log observations as they occur. Though a videotape of the session is more detailed complete than your notes, you can review a written or typed log more quickly than you can review a videotape. In general, you will refer to the videotape only for information about specific events that were not satisfactorily logged.
*Feed your design process with usability tests.*Usability testing provides the design team with fresh perspectives on the current design -- thanks mostly to the test participants involved. These perspectives, along with a summary of the test results, can energize and focus a group process that dramatically improves the UI design. That process is no less important than the usability test itself.
Tools and Reviews
The following classic text on usability testing is thorough and easy to understand:
A Practical Guide to Usability Testing, by Joseph S. Dumas and Janice C. Redish
Another resource is the Usability Professionals' Association (UPA), an organization for professionals who specialize in user studies, particularly usability tests. The UPA offers publications, conferences, etc. for sharing knowledge about user studies and their role in product development.
Local Resources
UW Technology's usability specialist is Mitch DeRidder, in AIS-ASM's department of Usability, Research, and Tools. For usability testing and other usability support, you may contact Mitch or his manager, Alexis Raphael, or email usable@u.washington.edu.
UW Technology maintains a usability lab in the basement of Mary Gates Hall. The lab consists of two adjacent rooms, secured by key locks. A video camera in the test room can transmit to large television monitor in the observation room. If you wish to use the lab for your own usability study, make arrangements with Mitch or Alexis. At that time, also ask them for current participant consent forms and current policies regarding storage of data from usability tests.
Expected Costs
The primary cost to consider in usability testing is the time of the staff involved. Considerable time and effort is required to prepare UI simulations, to recruit participants, to plan and conduct the tests, and to analyze and discuss the results. Also, for best effect, members of the design team should personally observe as many of the test sessions as possible. The time and effort required for designing the UI can also be expected to increase because usability tests often uncover problems in the design, which should then be reworked. Also, bear in mind that the later you wait to perform usability testing, the more it costs to redesign the UI.
Page Operations
Browse Space