Location: [CPSC 333] [Listing by Topic] [Listing by Date] [Previous Topic] [Next Topic] Integration Testing
This material was covered during lectures on April 2-4, 1997.
System integration follows unit testing, and it's assumed that all modules have ``passed'' unit tests before they're combined with any others. During system integration, modules are combined together to form progressively larger subsystems, starting with individual modules and ending with the complete system.
A small amount of additional information about system integration and integration testing is included in recent editions of Pressman's Software Engineering: A Practitioner's Approach. Marick's book is devoted to ``testing in the medium,'' and contains quite a bit more information about this:
B. Marick
The Craft of Software Testing. Subsystem Testing Including Object-Based and Object-Oriented Testing
Prentice Hall, 1995
This is (sometimes) also called ``Big Bang Integration.''
This method goes straight from unit testing of integration modules, to assembly and testing of the entire system - with no intermediate subsystems tested.
It's never advisable (for anything but a "completely trivial" system), because
For the remainder of notes on system integration, we will use a system with six modules in examples:
As the above picture shows,
When this kind of strategy is used, there is aways one (connected) subset of modules that forms a subsystem that is being tested.
You can think of first subsystem that is tested by this process as being the main module by itself. However, there probably won't be much to do at this point, since the main module will have been unit tested already, at this point. (On the other hand, you will need to run additional tests for all the subsystems that follow this one.)
After the testing of a subsystem is completed, the next subsystem (that is to be tested) is obtained by choosing a module that isn't part of the current subsystem but that is called by one of the modules already in the current system, and adding that new module to the current subsystem.
Since we start with the main module and grow the subsystems ``from the top down,'' it's always easy (at least, in principle) to ``activate'' the subsystem that is being tested. However, the subsystem will generally call other modules that haven't been included (or ``integrated'') yet.
This process would continue until all the modules had been integrated - that is, until the current ``subsystem'' consisted of the entire system (and, this system had passed all tests).
Since the main module will always be included in a subsystem that is being tested, if you're using top-down integration, it won't be necessary to use drivers for any modules. However, you'll need to write stubs for the modules that are called by the subsystem that is being tested.
You could perform top down integration in either a breadth first manner or in a depth first manner.
If a breadth first approach is used then you should think of the modules in the structure chart as sitting at various ``levels'' in the chart. You can think of the main module as being the only module at level 1. Its children are all at level 2, the children of these modules are at level 2, and so on. Thus, in the above example, module I is at level 1, modules II and III are at level 2, and modules IV, V, and VI are at level 3. When using a ``breadth first'' strategy, you include all the modules at a given level (when constructing subsystems) before you include any of the modules that are at lower levels in the structure chart.
For example, if a breadth first strategy is applied to the example, then you might consider subsystems that include the modules, and need the stubs, listed below (in the order in which they're listed):
Modules Included: I
Stubs Required for Modules: II, III
Modules Included: I, II
Stubs Required for Modules: III, IV, V
Modules Included: I, II, III
Stubs Required for Modules: IV, V, VI
Modules Included: I, II, III, IV
Stubs Required for Modules: V, VI
Modules Included: I, II, III, IV, V
Stubs Required for Modules: VI
Modules Included: I, II, III, IV, V, VI
No Stubs Required
If a depth first approach is used then you should think of the modules as lying along paths that all start from the main module, and follow the (control) connections shown by arrows on the structure chart. You add modules by following a path as far as you can go, until a utility module (or some other module whose children have all already been included) is reached. At that point you ``backtrack'' until you can find another module to include.
For example, if a depth first strategy is applied to the example, then you might consider subsystems that include the modules, and need the stubs, that are listed below. (Note that the first two subsystems are the same as above, but the third is different.)
Modules Included: I
Stubs Required for Modules: II, III
Modules Included: I, II
Stubs Required for Modules: III, IV, V
Modules Included: I, II, IV
Stubs Required for Modules: III, V
Modules Included: I, II, IV, V
Stubs Required for Modules: III
Modules Included: I, II, III, IV, V
Stubs Required for Modules: VI
Modules Included: I, II, III, IV, V, VI
No Stubs Required
When this kind of strategy is used, one starts with the utility modules at the bottom of the structure chart. Again, after testing one subsystem you generally obtain by adding one more module that hasn't yet been included. In particular, for this method, you either add
or, if no such module X exists,
Unlike top down testing, we don't always have a single ``connected'' system being tested by this method. Instead, we may have several subsystems, for which integration testing has been performed. The addition of a new module X might cause several of these to be merged together, because X might call modules in several of these subsystems.
Since we only include a module X if it's a utility module or if all the modules it calls have been integrated already, stubs aren't required when ``bottom-up'' integration is used.
However, the system's main module won't be part of any subsystem that's being tested until the end of the process. More generally, there will always be at least one module that calls the subsystem(s) being tested, that hasn't yet been included. Thus, we still need extra code - this time, to replace one or more of the system's modules that call modules in the current subsystem(s) and that haven't been integrated yet. Thus, you'll need to write a driver that calls the (top module(s) in the) subsystem being tested.
If ``bottom-up'' integration is applied to the above example, then we might integrate and test subsystems in the following order.
Modules Included IV
Drivers Required: A driver to call module IV
(replacing module II)
Modules Included: V
Drivers Required: A driver to call module V
(replacing modules II and III)
Modules Included: II, IV, V
Drivers Required: A driver to call module II
(replacing module I)
Modules Included: VI
Drivers Required: A driver to call module VI
(replacing module III)
Modules Included: III, V, VI
Drivers Required: A driver to call module III
(replacing module I)
Modules Included: I, II, III, IV, V, VI
No Drivers Required
Top-down integration has the ``advantage'' (over bottom-up integration) that it starts with the main module, and continues by including top level modules - so that a ``global view'' of the system is available early on.
It has the disadvantage that it leaves ``utility modules'' until the end, so that problems with these won't be detected early. If ``bottom-up'' integration is used then these might be found soon after integration testing begins.
Top-down integration also has the disadvantage or requiring stubs - which can sometimes be more difficult to write than drivers, since they must simulate computation of outputs, instead of their validation.
Of course, there is no rule that says that you must use either a strictly top-down or a strictly bottom-up method for system integration.
An alternative is to consider the functions that modules are required to perform, and to try to cluster modules into major (functional) subsystems, rather than simply working from the structure chart.
Taking this approach, you might to decide to assemble each of the following subsystems separately (perhaps using top-down integration, or bottom-up integration, or some combination of the two, when assembling each subsystem):
Early on during system integration and integration testing, these subsystems could be formed and tested independently; in later stages, you'd combine the systems together.
Each of the subsystems assembled during ``system integration'' should be tested.
When performing integration testing on a subsystem, you can (and probably should) repeat the tests that were developed in order to conduct ``unit testing'' for the top module(s) in the subsystem that is being tested.
You might (possibly) be able to form additional (useful) tests by applying some of the methods for structural unit testing - path testing, condition testing, and loop testing - but working with the structure chart for the subsystem being tested, instead of a flow graph for a single module. Now, instead of trying to cover edges in a flow graph (during path testing), you would try to cover the control connections between modules that are drawn as arrows on the structure chart. You might (or might not) be able to add tests that increase coverage of the conditions (inside modules) used to decide whether other modules are called, and you might also be able to add tests ensuring that the number of times that a module is called is at, above, and below, the minimum - and the maximum - number of times that is allowed (or expected).
As mentioned above, Marick's book is concerned with ``testing in the medium.'' The book includes two versions of a ``Test Catalog,'' in its appendices, which suggest additional test cases that one might try to cover during integration testing (and, this book has been replaced on reserve in the library, for use by CPSC 333 students in Winter 1997).
When process specifications were discussed, it was noted that one could document ``assumptions'' that one could assume to be satisfied whenever a given process was executed. A process isn't generally required to confirm that its assumptions are satisfied, and ``any output at all'' is generally considered to be acceptable, if the process is called when the assumptions are false.
In fact, a function's assumptions often relate to requirements for the functions that call the function, rather than requirements for the function itself.
Thus, while it isn't a good idea to include code that checks assumptions in production code (that is, in the system that's to be delivered and used after testing has been completed), and this code also isn't very useful during unit testing,, this code can be extremely useful during integration testing, because the discovery that a function has been called when its assumptions aren't satisfied can provide evidence of a problem elsewhere in the system.
If you are using C++ (or C) then it is possible to include code for assumption checking in (at least) two ways:
If the assumption for a function can be coded as a single Boolean expression e, then an assert statement can be used to check it.
In particular, after making sure that you've included the necessary library, by including the statement
#include <assert.h>
along with the other #include statements in your source code, you should include the statement
assert(e)
at the beginning of the code for the function whose assumption you are checking.
It isn't necessary (or, necessarily, a good idea) to remove the assert statements from source code after testing is completed - after all, it may be necessary to test again after changes are made, and these statements provide inline documentation of assumptions that may also be valuable later on.
On the other hand, assumption checking is costly, and you shouldn't include it in production code.
In order to disable assumption checking (for production code) without removing the assert statements from the source, add the statement
#define NDEBUG
before the ``#include <assert.h>'' statement mentioned above.
The man page for assert is available on our Unix system. It's the source for the above information and includes (a very small amount of) additional information about this.
This won't work if the assumption you wish to check is effectively checkable, but can't be checked by a single (readable) Boolean expression.
For example, if you are using a function for ``binary search'' then this may accept an array A of integers (or, elements from some other ordered type) as one of its inputs, and you may wish to include the ``assumption'' that A is sorted. It's easy to write a simple while loop to check this, but it isn't so easy to write a single Boolean expression for it.
Preprocessor directives can be used to include more complex code for assumption checking, that can also be disabled in production code.
Suppose, in particular, that you'd like to include a block of code to check an assumption, and that you'd like this to be disabled in the same way that assert statements are disabled - that is, through the use of the statement ``#define NDEBUG'' as above.
Then it suffices to include the code in a (kind of a) ``#if-#endif'' statement:
#ifndef NDEBUG
(code for assumption checking)
#endif
For additional information, read the man page for the C language preprocessor, cpp.
When tests of a subsystem fail, and errors are identified, it's almost certain that this will lead to changing one or more of the subsystem's modules. At this point, the modules that have been changed must be ``unit tested'' again - and the (smaller) subsystems that contain those new modules should be ``integrated'' and tested again as well, until either more errors are found (and you have to ``roll back'' to unit testing again) or you reach the subsystem that was being tested when errors were first detected.
This process of ``rolling back'' to unit testing, and repeating unit and integration tests for modules and subsystems that have been changed, is called regression testing.
In an even worse case, a test may uncover an error that must be corrected by changing part of the system design, or even a part of the requirements specification. Correcting the error will (clearly) be more expensive if this is the case; among other things, it will probably require the design of new tests, as well as (modification and) repetition of (some of) the tests that have already been developed.
A lab exercise based on this material is also available.
Location: [CPSC 333] [Listing by Topic] [Listing by Date] [Previous Topic] [Next Topic] Integration Testing