Guideline: Maintaining Automated Test Suites
This guideline explains ways to maintain automated test suites - collection of tests performed together for breadth and depth coverage.

Leverage design and management principles that facilitate the maintenance of test suites.

Relationships
Related Elements
Main Description

Introduction

At some point in your test effort, you may find it necessary to manage your test effort by creating test suites for your test assets. Maintaining test suites can take many different forms. To facilitate your testing, you may want to introduce some level of automation of your test suites. The fact that you've automated your test suites does not necessarily make your testing easier however. It may actually increase the maintenance burden of your suites.

This guideline introduces you to useful heuristics on how to facilitate the maintenance of your automated test suites.

Plan your test suites

Automating your testing without planning increases the chances that testing will be ineffective and inefficient. Some level of planning should take place whether implicit or explicit. An essential part of any test plan is the definition of a strategy for test automation. Use your plan to articulate to the development team how you plan to maintain your test assets. In many cases, this is never done. The rest of the development team may be unaware of how you intend to maintain your tests. It is also a good practice to get the rest of the development team to understand that this maintenance can be a substantial part of the overall development effort. Use your test tooling to capture this information and treat this plan just like you would treat any other test asset in your test repository.

Centrally locate your test assets

To facilitate the maintenance of your automated test suites, locate your test assets in a repository that can be accessed by the development team. Many test automation environments provide test management tools that make it easier to organize and access your test assets by maintaining the test assets (test cases, test scripts, and test suites) in a common repository.

In addition, some form of access control is enforced by the automation test tool. This eases the maintenance burden by ensuring the integrity of your test suites. You may choose to grant stakeholders and managers read-only access, whereas developers and testers at the practitioner level may have read/write access.

Treat your test assets like any other software

Software must be maintained. This also applies to the software in your test suites. Test cases and their associated test scripts, whether recorded or programmed, should be maintained. And just as software has different kinds of maintenance (e.g., corrective, preventative, or adaptive) so too do the assets in your automated test suites. As you lifecycle your test suites, identify, if only informally, how you plan to disposition the test suite corrective maintenance (e.g., syntactical errors in your scripts), preventative maintenance (e.g., where possible to write generalized test scripts), and adaptive maintenance (e.g., how you can use your test tooling to re-assign test assets within one suite to another suite or suites). This can be captured, as described in the section Plan Your Test Suites above, in your test plan.

Improve the testability of your test suites through collaboration with developers

It's one thing to say that your test suites will need to be maintained due to changes in the application, changes in the testing target, etc. It's quite another thing to actually determine whether a test suite needs to be revamped and, if it does, what test assets within it need to be addressed.

One way to facilitate this is to use test suites as a way to communicate test decision to the developers. One way to perform continuous perfective maintenance of test suites is to think of your test suites as assets that belong to the development team rather than just the testers.  You can perform a kind of perfective maintenance on test in the following ways:

  • use test suites to raise the level of abstraction
  • use test suites to provide focus for the developer
  • use test suites to articulate areas that the developers would like testers to focus on
  • make the construction and maintenance of test suites more efficient by understanding what area(s) developers want to focus on
  • use test suites to clarify test targets with developers

Don't be afraid to clean up your suites

Your test assets will evolve just as the application under test will. As requirements to the system change, the application will change as well. To maintain your test suites, you should continually check whether test assets are valid. If possible, validity checks should be performed after each new release of the software, preferably more frequently. Keeping your test suites relevant is a full-time job. Assume that changes in the software will lead to some degree of invalid tests within your test suites. Once these test assets have been identified as invalid, get rid of them. This will make the maintenance burden much more tolerable. Some automated test tooling environments make this task easier by providing ways to package outdated or invalid tests. In some cases, you may not be absolutely sure whether you want to completely get rid of tests within your test suite or even of getting rid of test suites altogether.  To alleviate this burden, you can create packages for obsolete tests or test suites and dispose of tests or test suites by putting them in packages labeled for this purpose.

Like physical objects, tests can break. It's not that they wear down, it's that something's changed in their environment. Perhaps they've been ported to a new operating system. Or-more likely-the code they exercise has changed in a way that correctly causes the test to fail. Suppose you're working on version 2.0 of an e-banking application. In version 1.0, this method was used to log in:


public boolean login (String username);


In version 2.0, the marketing department has realized that password protection might be a good idea. So the method is changed to this:

public boolean login (String username, String password);

Any test that uses login will fail It won't even compile. Since not much useful work is possible at this point, not many useful tests can be written without login. You might be faced with hundreds or thousands of failing tests.

These tests can be fixed by using a global search-and-replace tool that finds every instance of login(something) and replaces it with login(something, "dummy password"). Then arrange for all the testing accounts to use that password, and you're on your way. Then, when marketing decides that passwords should not be allowed to contain spaces, you get to do it all over again. This kind of thing is a wasteful burden, especially when-as is often the case-the test changes aren't so easily made. There is a better way.

Suppose that the tests originally did not call the product's login method. Rather, they called a library method that does whatever it takes to get the test logged in and ready to proceed. Initially, that method might look like this:

public boolean testLogin (String username) {
return product.login(username);
}

When the version 2.0 change happens, the utility library is changed to match:

public Boolean testLogin (String username) {
return product.login(username, "dummy password");
}


Instead of a changing a thousand tests, you change one method.

Ideally , all the needed library methods would be available at the beginning of the testing effort. In practice, they can't all be anticipated-you might not realize you need a testLogin utility method until the first time the product login changes. So test utility methods are often "factored out" of existing tests as needed. It is very important that you perform this ongoing test repair, even under schedule pressure. If you do not, you will waste much time dealing with an ugly and un-maintainable test suite. You might well find yourself throwing it away, or being unable to write the needed numbers of new tests because all your available testing time is spent maintaining old ones.


Note: the tests of the product's login method will still call it directly. If its behavior changes, some or all of those tests will need to be updated. (If none of the login tests fail when its behavior changes, they're probably not very good at detecting defects.)


Another example
Suppose you're testing a compiler. Some of the first classes written define the compiler's internal parse tree and the transformations made upon it. You have a number of tests that construct parse trees and test the transformations. One such test might look like this:

/*
* Given
* while (i<0) { f(a+i); i++;}
* "a+i" cannot be hoisted from the loop because
* it contains a variable changed in the loop.
*/
loopTest = new LessOp(new Token("i"), new Token("0"));
aPlusI = new PlusOp(new Token("a"), new Token("i"));
statement1 = new Statement(new Funcall(new Token("f"), aPlusI));
statement2 = new Statement(new PostIncr(new Token("i"));
loop = new While(loopTest, new Block(statement1, statement2));
expect(false, loop.canHoist(aPlusI))


This is a difficult test to read. Suppose that time passes. Something changes that requires you to update the tests. At this point, you have more product infrastructure to draw upon. In particular, you might have a parsing routine that turns strings into parse trees. It would be better at this point to completely rewrite the tests to use it:

loop=Parser.parse("while (i<0) { f(a+i); i++; }");
// Get a pointer to the "a+i" part of the loop.
aPlusI = loop.body.statements[0].args[0];
expect(false, loop.canHoist(aPlusI));

Such tests will be much easier to understand, which will save time immediately and in the future. In fact, their maintenance costs are so much lower that it might make sense to defer most of them until the parser is available.

There's a slight downside to this approach: such tests might discover a defect in either the transformation code (as intended) or in the parser (by accident). So problem isolation and debugging may be somewhat more difficult. On the other hand, finding a problem that the parser tests miss isn't such a bad thing.
There is also a chance that a defect in the parser might mask a defect in the transformation code. The chance of this is rather small, and the cost from it is almost certainly less than the cost of maintaining the more complicated tests.

Focusing test improvement
A large test suite will contain some blocks of tests that don't change. They correspond to stable areas in the application. Other blocks of tests will change often. They correspond to areas in the application where behavior is changing often. These latter blocks of test will tend to make heavier use of utility libraries. Each test will test specific behaviors in the changeable area. The utility libraries are designed to allow such a test to check its targeted behaviors while remaining relatively immune to changes in untested behaviors.

For example, the "loop hoisting" test shown above is now immune to the details of how parse trees are built. It is still sensitive to the structure of a while loop's parse tree (because of the sequences of accesses required to fetch the sub-tree for a+i). If that structure proves changeable, the test can be made more abstract by creating a fetchSubtree utility method:

loop=Parser.parse("while (i<0) { f(a+i); i++; }");
aPlusI = fetchSubtree(loop, "a+i");
expect(false, loop.canHoist(aPlusI));

The test is now sensitive only to two things: the definition of the language (for example, that integers can be incremented with ++), and the rules governing loop hoisting (the behavior whose correctness it's checking).


Throwing away tests
Even with utility libraries, a test might periodically be broken by behavior changes that have nothing to do with what it checks. Fixing the test doesn't stand much of a chance of finding a defect due to the change; it's something you do to preserve the test's chance of finding some other defect someday. But the cost of such a series of fixes might exceed the value of the test's hypothetically finding a defect. It might be better to simply throw the test away and devote the effort to creating new tests with greater value.
Most people resist the notion of throwing away a test-at least until they're so overwhelmed by the maintenance burden that they throw all the tests away. It is better to make the decision carefully and continuously, test by test, asking:


1. How much work will it be to fix this test well, perhaps adding to the utility library?
2. How else might the time be used?
3. How likely is it that the test will find serious defects in the future? What's been the track record of it and related tests?
4. How long will it be before the test breaks again?


The answers to these questions will be rough estimates or even guesses. But asking them will yield better results than simply having a policy of fixing all tests.


Another reason to throw away tests is that they've become redundant. For example, early in development, there might be a multitude of simple tests of basic parse-tree construction methods (the LessOp constructor and the like). Later, during the writing of the parser, there will be a number of parser tests. Since the parser uses the construction methods, the parser tests will also indirectly test them. As code changes break the construction tests, it's reasonable to discard some of them as being redundant. Of course, any new or changed construction behavior will need new tests. They might be implemented directly (if they're hard to test thoroughly through the parser) or indirectly (if tests through the parser are adequate and more maintainable).