Testing activities to identify test coverage (which tests have been run, passed, and failed), and document the status of issues which have been discovered during testing and other development processes.
On this page:
Definition
Both the positive and negative results of testing can be tracked in order to provide a clear status to the project manager. As tests are run, the tester notes which tests passed and which failed; the failed tests result in documented defects. As defects are fixed and those fixes promoted into the test environment, tests are rerun. Test results tracking can let the tester see at a glance if previously failed tests now pass, or if previously passed tests now fail. In addition to tracing test cases back to requirements, a Baseline Traceability Matrix is a useful tool to track the status of tests. See Examples and Ideas for examples of a Traceability Matrix.
Testing can result in different kinds of issues, including defects, questions, actions and enhancements (feature requests). Although this article deals primarily with defects, all these types of issues can be tracked in the same way, as long as there is some way to 'tag' the category. (For example, a work planning report might contain all categories of issues, whereas a program quality report might contain only defects found in a given program.)
Defects are logged and monitored from discovery to resolution (or stagnation). Once defects have been logged, and depending on the data captured, statistics can be gathered to track metrics such as increasing or decreasing defect counts, peak trouble times in a product's cycle, or volatile modules. For example, once you've captured details about a problem, if you see the same problem arise in production after code fixes, your regression tests need to be tweaked. Tracking defects arising out of tests is usually done with some kind of tool (e.g., TestTrack, Request Tracker) but a simple Excel spreadsheet can be used as well (see Tools and Reviews and Examples and Ideas for more info).
The process may be different for tracking defects during the test phase than for tracking them once the product is in production. For example, you may want to try to capture the 'root cause' of all defects in production to locate production process quality issues, while streamlining the tracking during testing.
In general, however, the following is true for both production and non-production change requests:
A newly entered defect
At a minimum, you should capture a Summary, Found By and Date, and a Description:
- Summary: a short summary of the Item - be as clear and concise as possible - this is the main 'title' of the Item which will appear on most lists
- Found By: Name of the person who first spotted the issue
- Date Found: Date when the issue was found
- Description: The details of the problem (including what was expected, not just what happened), with as much detail as possible - try to picture someone else reading and resolving this in 6 months! Include the steps needed to recreate the problem (typically a separate section called Steps to Reproduce) where at all possible.
Other data that can be useful to capture:
- Issue Type: Is this a bug or an enhancement or a question? Tracking example: If you get a lot of questions about how to use a given page, the usability may need to be reviewed.
- Priority: How important is this to fix? Tracking example: Having a disproportionate number of legitimately high priority defects can indicate that the code is not ready for testing or that the design does not meet the business requirements.
- Severity: How bad is the problem (e.g., is there a workaround?). Many teams combine priority and severity to triage bugs. Simpler tracking systems use only priority. See Articles for more discussion of priority and severity.
- System/Subsystem/Module/Deliverable: Many teams work on more than one system, area of a system, or module. Tagging the item with this data allows you to refine searches and identify areas of the system that may need attention. Problems may also be found in other deliverables, such as documentation or design.
- Phase: Where in the life cycle (e.g., Requirements, Design, Unit Testing, System Testing, User Acceptance) was this issue raised? Probably only needed for large projects where issues are tracked throughout the life cycle. If you are tracking only software problems, this is probably not needed.
- Test Case: The test that detected the defect (which should be rerun when the issue is resolved).
- Instructions for Verification - the expected results of a successful test once the defect has been fixed.
- Root Cause: Although typically not recorded until after a fix is found and analysis done, an initial 'root cause' can be noted when the defect is entered. (e.g., Missing Code Review).
- Found-in Version: What version of the software (current released feature set) was running when the defect was found?
- Attachments: Any files that pertain to the defect, such as screen captures or log files. Some tools store these; the path to the file could be sufficient.
A defect 'in progress'
After a defect is entered into a tracking system, it can collect more data:
- Who is it assigned to now?
- What program was the bug found in?
- Are there related bugs? Did this bug cause other bugs? Did another bug cause this bug? Is it similar to other bugs?
- Estimated Fix Date/Release Date: When should this be fixed? Moved to production?
- Estimated Fix Version: What version will have this fix (not all teams use versions).
- Number of times encountered: How often does this defect occur? (Particularly useful with intermittent problems that are hard to isolate but continue to occur - the defect can be recorded once and a 'counter' used to track each occurrence.)
- Who fixed it?
- What program changed when it was fixed? This is useful when it was fixed in a different program than where it was found.
- What were the specific changes? Brief description of the fix, useful for both the tester and future production support research.
- When did it get fixed? Date/Program bundle/Package/Release.
- When did the code changes go to the test environment?
- When was it tested?
- Who tested it?
- Verification Test Cases: What tests (besides the one that found it) should be run to verify the fix?
- When was it moved to production?
- Who verified it?
- Are there release notes?
- Did any documentation need to be modified because of the changes?
Defect work flow (life cycle)
Depending on the project, the work flow can be as simple or as complex as you want. Here are some examples:
Simplest:
The simplest work flow includes just developers and testers, leaving the relationship with the test environment outside the defect tracking process: The location (development? test?) of the code changes can be determined by the status:
- 1. Open, unassigned
- 2. Reviewed and (if not deferred) Assigned to a developer
- 3. Fixed, assigned to a tester (could be another developer; in rare cases the person who fixed it, although this is not recommended)
- 4a. Verified but Failed (goes back to the developer)
- 4b. Verified and Passed (Closed)
Medium Complexity:
The medium work flow includes developers, testers, and maybe configuration management and customers. The relationship of the defect with the test environment can be determined from the status (fixed = development, ready for test = system test):
- 1. Open, unassigned
- 2. Reviewed and (if not deferred) Assigned to a developer
- 3. Fixed (code is changed but not in the test environment yet)
- 4. Ready to Test (changed code has been moved to the test environment, assigned to a tester)
- 5a. Verified but Failed (changed code failed the test - goes back to developer)
- 5b. Verified and Passed (Closed)
Most Complex:
The complex work flow includes developers, testers, configuration management and customers. This is not often used within UW Technology, but provides a look into the state of the development, test and acceptance environments using the defect tracking system:
- 1. Open, unassigned
- 2. Reviewed and (if not deferred) Assigned to a developer
- 3. Fixed (code is changed but not in the test environment yet)
- 4. Ready to Test (changed code has been moved to the test environment)
- 5a. Verified but Failed (changed code failed the test - goes back to developer)
- 5b. Verified and Passed (but needs customer verification)
- 6. Ready for Customer Verification (changed code is in the acceptance test environment)
- 7a. Verified by the Customer, but failed (changed code failed the test - goes back to developer)
- 7b. Verified by the Customer, and Passed (Closed)
In all of these cases, if a problem can't be reproduced by the developer, it may either be returned to the originator for more information, or closed by the project manager, to be reopened if the problem can later be reproduced.
Top of Page
When to Start
Identifying bugs found and fixed during unit testing is important to the developer's task management, but the overhead of formally tracking those bugs doesn't usually pay off until the more controlled and rigorous system test phase begins. Any unresolved defects found during unit testing should be logged and then tracked once system testing starts.
Defect tracking continues through the life of the product. In most cases it makes sense to track bugs found and fixed during the test cycle separately from those found in production. You may use the same 'database' but the test-cycle bugs should be tagged as such to avoid clouding production quality metrics. By keeping them separate, you might be able to show a correlation between a higher rate of defects found during test and a lower rate found in production.
Examples and Ideas
Examples
Test Results Tracking
Defect Tracking
Articles
Tools and Reviews
TestTrack Pro from Seapine Software is used by several teams within AIS. The tool is configurable by 'project' (by team and project), so that differences in development processes (e.g., between a large development project team and a small production support team) can be accommodated. All the items in the Definition section can be tracked and searched using TestTrack (through provided or custom fields).
Request Tracker (from Best Practical Solutions) is in use as a help-desk tool for UW Technology. It can be used to track bugs during testing, but is engineered for hefty production support trouble-ticket tracking so may be overkill. We haven't evaluated it yet. See https://www.washington.edu/cactools/projects/cac-tt/.
Local Resources
How to use TestTrack for the CSR Project is a TestTrack Pro "Cheat Sheet' created for the CSR project.
Currently, there are two instances of TestTrack Pro in use by AIS teams. Each instance takes a slightly different usage and maintenance approach. One resides on the AIS departmental server (\\nebula2.washington.edu\uw\cac\is\ais\shareserver), and one resides on the ucsdocs server.
ais-server:* This instance was originally set up by the HEPPS team in 2001, has 10 floating and 2 named (static) licenses, and is maintained by AIS team members.
ucsdocs:* This instance was originally set up by the SAGE team, has 10 floating licenses and is maintained by TEG Distributed Systems Administrative Platforms (DSAP).
Expected Costs
The cost of tracking test results is basically time. The costs of tracking defects include time (logging decently detailed descriptions and keeping track of fixes) and the purchase of any tools used.
Expected Benefits
- Electronically tracking defects prevents them from 'falling through the cracks'.
- Metrics from test results and defect tracking can be used to monitor trends in areas of the application that may need more work before deployment, or development processes that can be improved. Having data about defects is a prerequisite for Root Cause Analysis.
- Having a standard set of tools, and a common tracking glossary can reduce ramp-up time for developers moving between projects.
- Having a list of all open defects and feature requests aids work prioritization. (You can ask the client, "What do you want us to work on first?")
- Formally logging a defect enforces thinking about what the problem really is, especially if using a tool with required fields.
Top of Page
Page Operations
Browse Space