Testing to ensure that previously-working software continues to work as intended, and to detect unintended consequences resulting from program changes.

Definition

Wikipedia defines Regression Testing as "any type of software testing which seeks to uncover regression bugs. Regression bugs occur whenever software functionality that previously worked as desired stops working or no longer works in the same way that it was working previously. Typically regression bugs occur as an unintended consequence of program changes."

Common methods of regression testing include re-running previously run tests and checking whether previously-fixed faults have reemerged.

Experience has shown that as software is developed, this kind of reemergence of faults is quite common. Sometimes it occurs because a fix gets lost through poor revision control practices (or simple human error in revision control), but just as often a fix for a problem will be "fragile" - if some other change is made to the program, the fix no longer works. Finally, it has often been the case that when some feature is redesigned, the same mistakes will be made in the redesign that were made in the original implementation of the feature.

During the update of any program, the programmer should ask two questions:

1) "Did the change work?"

2) "Does the old stuff still work?"

When to Start

Regression tests begin as soon as there is anything to test at all. The regression test library grows as the project moves ahead and acquires new or rewritten code.

Any time an implementation within a program is modified, regression testing should be done. This can be done by rerunning existing tests against the modified code to determine whether the changes break anything that worked prior to the change and by writing new tests where necessary. Adequate coverage without wasting time should be a primary consideration when conducting regression tests. Attempt to spend as little time as possible doing regression testing without reducing the probability that new failures in old, already tested code are detected.

Examples and Ideas

Some strategies and factors to consider during this process include the following:

• Test fixed bugs promptly. The programmer might have handled the symptoms but not have gotten to the underlying cause.

• Watch for side effects of fixes. The bug itself might be fixed but the fix might create other bugs.

• Write a regression test for each bug fixed.

• If two or more tests are similar, determine which is less effective and get rid of it.

• Identify tests that the program consistently passes and archive them.

• Focus on functional issues, not those related to design.

• Make changes (small and large) to data and find any resulting corruption.

• Trace the effects of the changes on program memory. For example, checking how many/how much resources a program is using, particularly where tables and indexes are involved. A C programmer, as another example, may want to determine if a memory leak has developed.

BUILDING A LIBRARY

The most effective approach to regression testing is based on developing a library of tests made up of a standard battery of test cases that can be run every time you build a new version of the program. The most difficult aspect involved in building a library of test cases is determining which test cases to include. The most common suggestion from authorities in the field of software testing is to avoid spending excessive amounts of time trying to decide and err on the side of caution. Automated tests, as well as test cases involving boundary conditions and timing almost definitely belong in your library. Some software development companies include only tests that have actually found bugs. The problem with that rationale is that the particular bug may have been found and fixed in the distant past.

Periodically review the regression test library to eliminate redundant or unnecessary tests. Microsoft, for example, suggests that this be done about every third testing cycle. Duplication is quite common when more than one person is writing test code. An example that causes this problem is the concentration of tests that often develop when a bug or variants of it are particularly persistent and are present across many cycles of testing. Numerous tests might be written and added to the regression test library. These multiple tests are useful for fixing the bug, but when all traces of the bug and its variants are eliminated from the program, select the best of the tests associated with the bug and remove the rest from the library.

Tools and Reviews

In most software development situations it is considered good practice that when a bug is located and fixed, a test that exposes the bug is recorded and regularly retested after subsequent changes to the program. Although this may be done through manual testing procedures using programming techniques, it is often done using automated testing tools. Such a 'test suite' contains software tools that allows the testing environment to execute all the regression test cases automatically; some projects even set up automated systems to automatically re-run all regression tests at specified intervals and report any regressions. Common strategies are to run such a system after every successful compile (for small projects), every night, or once a week.

Regression Testing is an integral part of the extreme programming software development methodology. In this methodology, design documents are replaced by extensive, repeatable, and automated testing of the entire software package at every stage in the software development cycle.

Regression Testing must also test the revised software by simulating its operational environment to ensure that all systems and interfaces operate as expected.

Interfaces facilitate communication between different computer systems or allow people to communicate with machines (and vice versa). Interfaces can be software, such as the Graphical User Interface (GUI) of Microsoft Windows, or hardware, e.g. the physical connections between, say, a simple terminal and a host computer. Interfaces use an agreed protocol ('language') to send and receive information from one machine to another.

Regression Testing should be conducted as per any system testing as proceeding in accordance with a test plan.

The key point about a test plan is that it not only documents what will be tested, but also the expected results. In addition, a test plan can identify additional areas which should be tested in order to make the resultant plan more comprehensive.

Having completed the tests, the results need to be considered and a determination of whether or not any results have failed to meet an acceptable standard. In particular, each failure should be allocated a 'severity level'. Without this gradation, an objective view cannot be taken. The sections on System Testing and User Acceptance Testing should be consulted as well.

Examples of automated resources would be "Empirix e-Test Suite" (.NET applications) by Empirix, Inc., "QAWizard" by Seapine Software and "JEvolve" by Man Machine Systems.

Local Resources

"Home grown Resources " such as DFDS accumulator stuff as well as comparison tools between COBOL 68 and COBOL 85 are available on the mainframe. (More details will be provided later.)

Expected Costs

If Regression Testing is not performed, a strong possibility exists that the system could fail upon upgrade.

Upon each iteration of true regression testing, all existing, validated tests are run, and the new results are compared to the already-achieved standards. And normally, one or more additional tests are run, debugged and rerun until the project successfully passes the test.

Expected Benefits

Easier transitions for new staff.

Increased user confidence.

"The more you do this, the more you can do this". That is, things become easier with repetition. When you work with something at first, then it should already be there for you later. Set up, not necessarily completely automated, should be made easy as possible - making many procedures rote.