User Acceptance testing
Testing to enable the 'owners' (clients) of the product to finally accept the product deliverables or application.
Definition
User Acceptance Testing (UAT) is the process where the 'owners' (clients) of the product finally accept (and in some organizations, sign off on) the product deliverables (referred to as application here). While functional testing during earlier test phases (e.g., System, Integration) verifies that the application does what we told it to, User Acceptance testing verifies that we did what was requested by the user. This testing focuses on how (and how well) the application meets the requirements. User Acceptance Testing examines the application from the user's perspective, and is performed by a client Acceptance Test Team (e.g., HR, Budget, Payroll), with assistance from the technical team. This testing provides assurance that the application meets the requirements defined by the clients and is ready for production. Where possible, this is done before the application is actually deployed to production.
During User Acceptance tests, the client evaluates the application to ensure it meets the documented acceptance criteria and determines if the application is ready to be implemented in production. The client evaluates a variety of areas of the application, typically using business scenarios, but exception tests may also be run.
This testing, from the user's point of view, is 'platform agnostic'. Except for navigating to or initiating the application (e.g., web page, mainframe driver screen, desktop client), most aspects of the User Acceptance testing are the same regardless of the underlying system.
At the end of this testing, the results are reviewed and the client / development teams determine whether to move ahead and deploy the application into production. Outstanding issues are documented and prioritized for future releases. (Test results may be reviewed iteratively during the testing phase, but a final review is needed for the deployment decision.)
Note: Besides User Acceptance Testing, another form of acceptance testing is the Configuration Acceptance Test, a quick system configuration and application test to determine if the application is ready to run in a new environment. This test (sometimes called a 'Smoke Test') is used to verify the correct hardware and software configurations, including databases and tools. This is a simple end-to-end test to verify that the basic system functionality is working (e.g., logon, execute basic procedures, logoff); this test should uncover any major deficiencies caused by the installation and setup processes, but is not intended to provide overall test coverage. This set of tests can be performed for each cycle of testing, immediately after the system conversion and data setup, and before most testers (or any production users, for an acceptance test) have access to the modified components of the application. This test can also be run during a test cycle after configuration or application changes have been made.
A Pilot Test can be considered a sub-area of User Acceptance Testing. The Pilot Test validates the product in a production environment, allowing the system to be exercised on a configuration and with data that may not be available in a 'lab' setting. The Pilot also allows the true end-users to assess the product prior to roll out to multiple sites or users. A Pilot is not actually a 'test' except in the sense that the product is being evaluated in a production environment and assessed for its readiness for roll out to more than one set of users. In order to reduce impact to business processes and users, Pilot Verification does not generally use specific test cases (or 'scripts'), but instead uses some other means to track coverage of the requirements. For example, users may be asked to perform their normal job functions using both the existing and the piloted systems (the 'parallel' methodology) and may be given checklists which will identify a job function and some specific system functionality that requires verification. User responses to the checklists would be tracked to verify assessment coverage against the requirements.
When to Start
Physically executing the user acceptance tests usually happens near the end of the development life cycle, just prior to deployment. Planning for this task, however, should start as soon as requirements begin to be identified. The specific User Acceptance tests correspond to the business, functional, and or interface requirements defined by the users.
The plan is prepared early so that the client can gain a better understanding of the testing concepts, the content and structure of the tests, and the testing mechanics. Early preparation also helps the developers know what will be expected of the application at acceptance time.
The User Acceptance Test plan is prepared in enough detail to ensure that the testing verifies that the application meets requirements, is stable, and that new or modified functionality does not adversely affect existing functionality. The UAT plan should document the test environment and any physical set-up required, and contain procedures for both testing and problem reporting.
The plan should document the UAT entrance and exit criteria. Entrance Criteria must be met in order to start the UAT phase; Exit Criteria must be met prior to declaring the UAT phase complete. Typically, one of the UAT Entrance Criteria is complete and successful System and Integration (Functional) Testing (e.g., no show stopper issues prior to beginning the tests), so it should not be necessary to re-test all of the technical elements verified by the previous phase.
Either the client or the development team can prepare the user acceptance test plan, but both organizations must work together to ensure a complete test. During the execution of the tests, it can be helpful for someone from the development team to sit with the user at the beginning of the tests, to explain procedures for capturing problem information and to provide a sense of support for the user. The time it takes to run the acceptance tests can vary, from a portion of a day for small applications with one user to up to a month, for larger applications or for users with over-extended schedules.
Examples and Ideas
User Acceptance Testing is done by the user, but often help is needed from the technical team. Checklists can be useful, indicating to the client that the tests should as closely approximate 'real work' as possible. Checklists can be used to confirm that a specific activity or function is done at least once, but mostly clients should just 'do their work' as they would in production. Checklists should include instructions on how to record status and report (and escalate) any problems.
During the execution of the tests, it can be helpful for someone from the development team to sit with the user at the beginning of the tests, to explain procedures for capturing problem information and to provide a sense of support for the user.
The time it takes to run the acceptance tests can vary, from a portion of a day for small applications with one user to up to a month, for larger applications or for users with over-extended schedules.
Specific examples will be provided in the future. A generic user test checklist can be used as a starting point.
Tools and Reviews
Requirements mapping
A Baseline Traceability Matrix is a useful tool to determine if a test has been designed for every requirement (it documents the relationships between Requirements and Acceptance Tests). This mapping can be useful for other tests, as well, particularly functional testing.
Examples of a Traceability Matrix.
Other tools
Automated tools may be useful for projects where the acceptance includes specific technical criteria, like performance parameters. Some of the same automated testing tools used for other test types can be used for User Acceptance Testing, depending on the applications. For most small applications, however, the most valuable testing tools are most likely to be in the form of checklists and problem reporting mechanisms.
Local Resources
The CSR project had users of the new Job Class Maintenance web application (HR, Academic HR, Budget Office, Equal Opportunity Office) and the new Union and Bargaining Unit Maintenance web application (Labor Relations) perform acceptance testing prior to the release of the applications into production. Checklists were prepared for the users:
Additional project links will be provided in the future.
Expected Costs
User Acceptance Testing requires a stable test environment that mirrors the production environment as much as possible. This testing requires a commitment of time from both the client and the development team to plan and execute the tests, and evaluate and resolve issues. The clients, as testers, must take the time to exercise most (if not all) aspects of the application and record discrepancies/defects in such a way that they are reproducible. Also, the right users should be chosen to perform the tests - including people who are very familiar with the business processes, whether or not they are familiar with the application.
Expected Benefits
Actively involving the client in the acceptance test planning and execution process should bring any issues to the surface early. The process gives the client a chance to find problems before the applications are moved into production, when finding and fixing defects becomes much more costly. In some instances, the User Acceptance Test Plan may act as a 'contract' between the client and the development team, especially when other documentation is insubstantial or missing.
Page Operations
Browse Space