CPSC 333: Testing the Tests

Location: [CPSC 333] [Listing by Topic] [Listing by Date] [Previous Topic] [Next Topic] Testing the Tests


This material was not covered in lectures in 1997, and can considered to be optional reading


Introduction

These ideas can be applied to unit testing, or to integration testing (which will follow).

Since ``exhaustive testing is impossible,'' we can only hope to use a relatively small number of tests that will detect as many of the errors in the system as possible. Under these circumstances it's useful to have a way to estimate the likelihood that a set of tests will catch most of the errors in code - or, to estimate the proportion of the errors in a program that will be detected by a given set of tests.

The following two, similar, techniques can be used to try to assess (or ``test'') a given set of tests in this way. In both cases, a set of programming errors that could plausibly have been made, for the code to be tested, is selected.

Error Seeding

For this method, all of these chosen errors are added into (or "seeded" into) the code. That is, a new version of the program is created by introducing these errors.

The tests are then executed on the new ``seeded'' program. The number of ``seeded'' errors that were caught by the tests is counted. Clearly, the higher the number of these errors that were caught, the better (and, ideally, you'd like all the seeded errors to have been detected).

One might use the proportion of ``seeded errors that were detected'' to ``seeded errors,'' as an estimate of the proportion of the unseeded (that is, ``real'') errors in the program that were ``caught'' by the tests as well.

Here's some ``common-sense'' advice that hasn't always been followed in practice: If you use this technique, don't forget to take all the seeded errors out of your source code, before producing executable code for the system you're to deliver, using it!

Mutation Testing

In this case, instead of having one new version of the program, many new versions - ``mutants'' - are created. Each is created by seeding exactly one of the chosen errors into the program - so that there are as many ``mutants'' as there are chosen errors, and each is very similar to the original program.

Now, the entire set of tests is run on the original program and on the mutants. A mutant ``dies'' if at least one of the tests produces different (incorrect!) output when run on the mutant than it does on the original, and a mutant ``survives'' otherwise. Now, the ideal situation is that all the mutants ``die.'' The proportion of the number of mutants that die, to the number mutants that were created, could be taken as an estimate of the proportion of ``real'' errors in the code that these tests will catch.

Location: [CPSC 333] [Listing by Topic] [Listing by Date] [Previous Topic] [Next Topic] Testing the Tests


Department of Computer Science
University of Calgary

Office: (403) 220-5073
Fax: (403) 284-4707

eberly@cpsc.ucalgary.ca