CPSC 333 --- Lecture 25 --- Friday, March 15, 1996 System Integration (and Integration Testing) System integration follows unit testing (and it's assumed that all modules have "passed" the unit testing phase before they're combined with any others). During system integration, modules are combined together to form progressively larger subsystems, starting with individual modules and ending with the complete system. Strategies for System Integration 1) "How Not to Do It": Non-Incremental Integration ... also called "Big Bang Integration" This method goes straight from unit testing of integration modules, to assembly and testing of the entire system --- with no intermediate subsystems tested. It's *never* advisable (for anything but a "completely trivial" system) because - the "complete system" will inevitably contain faults that should be detected and fixed during integration testing - these faults *will* probably cause some tests of the system to fail --- but, when tests fail, developer will have virtually no idea of *where* to look in the system for the problem that caused the failure 2) How to Do It: Incremental Integration For the rest of this lecture we will use a system with twelve modules in examples: Module I / | \ / | \ / | \ Module Module Module II III IV / / \ / / \ / / Module Module / V / VI \ / / | | \ / / | | \ | / | | \ | / | | \ | | Module Module Module Module | VII VIII IX X | / \ / | / | / | / | / / \ | / | / / \ | / | / / \ | / Module XI Module XII The main module, Module I, calls Modules II, III, and IV Module II calls Module XI Module III calls Modules V and VI Module VI calls Modules VII, VIII, IX, and X Module VII calls Modules XI and XII Module VIII calls Module XI Module IX calls Module XII Module X calls Module XII. A) Top Down Integration Using this method, one starts with the main module as the first subsystem to be tested. We always have once (connected) subsystem that is "currently" being tested. After testing of a subsystem is completed, the next subsystem is obtained by choosing a module that isn't part of the current subsystem but that *is* called by one of the modules already in the current system, and adding that new module to the current subsystem. Because we start with the main module and grow the subsystems "from the top down," it's always easy (at least, in principle) to "start up" the subsystem. However, the subsystem will generally call other modules that haven't been "integrated" yet. In order to test the current subsystem we must provide something to replace the lower level modules that it calls. The replacements for lower level modules that one provides for use in (top down) integration are called *stubs*. Each stub should should have the same name and parameters as the lower level module it replaces. The stub might be - a "hard coded" table of possible inputs and outputs --- including all the inputs we'd expected the lower level modules to be given when it's called by the subsystem being tested, for whatever tests have been designed; - a program that displays its inputs to a user and prompts the user for the outputs it should send back to the subsystem - something that does the same work as the "lower level module" it replaces --- but that does the work more slowly, or using more storage space, or on a more powerful system than the "lower level module" would be allowed to use - or (as Assignment 4 might suggest) something that displays a "not implemented yet" message and terminates --- *if* the subsystem can be tested adequately with this simple a "stub." We continue to add lower level modules in order to build larger and larger systems, until the entire system has been assembled. You could perform top down integration in a "breadth first" manner or in a "depth first manner" Example: "Breadth First" Top Down Integration Using "Breadth First Top Down" Integration, the following subsystems might be obtained from the above system. Modules Included Stubs Required for Modules... #1 I II, III, IV #2 I, II III, IV, XI #3 I, II, III IV, V, VI, XI #4 I, II, III, IV V, VI, XI #5 I, II, III, IV, V VI, XI #6 I, II, III, IV, V, VI VII, VIII, IX, X, XI #7 I, II, III, IV, V, VI, VII VIII, IX, X, XI, XII #8 I, II, III, IV, V, VI, VII, VIII IX, X, XI, XII #9 I, II, III, IV, V, VI, VII, VIII, IX X, XI, XII #10 I, II, III, IV, V, VI, VII, VIII, IX, X XI, XII #11 I, II, III, IV, V, VI, VII, VIII, IX, X, XI XII #12 I, II, III, IV, V, VI, VII, VIII, IX, X, XI, XII No stubs required. Example: "Depth First" Top Down Integration Using "Depth First Top Down" Integration, the following subsystems might be obtained from the above system. Modules Included Stubs Required for Modules... #1 I II, III, IV #2 I, II III, IV, XI #3 I, II, XI III, IV #4 I, II, III, XI IV, V, VI #5 I, II, III, V, XI IV, VI #6 I, II, III, V, VI, XI IV, VII, VIII, IX, X #7 I, II, III, V, VI, VII, XI IV, VIII, IX, X, XII #8 I, II, III, V, VI, VII, XI, XII IV, VIII, IX, X #9 I, II, III, V, VI, VII, VIII, XI, XII IV, IX, X #10 I, II, III, V, VI, VII, VIII, IX, XI, XII IV, X #11 I, II, III, V, VI, VII, VIII, IX, X, XI, XII IV #12 I, II, III, IV, V, VI, VII, VIII, IX, X, XI, XII No stubs required. B) Bottom Up Integration Using this method, one starts with the utility modules at the bottom of the structure chart. Again, after testing one subsystem you generally obtain by adding one more module to it. For this method, you add - a new module X that *calls* one of the modules in the current subsystem --- and such that *all* the modules X calls are already in the current system; or, if no such module X exists, - a new "utility" module at the bottom of the structure chart. Unlike top down testing, we *don't* always have a single "connected" system being tested by this method. Instead, we may have a "subsystem" that contains several unconnected (and independent) components. Now, if a module X is added to the subsystem we're testing, then every module that X calls is already part of the subsystem --- so *stubs* aren't required. However, the "main module" isn't part of the system. We still need extra code --- this time, to replace one or more of the system's modules that *call* modules in the current subsystem and that haven't been integrated yet. A "driver" could prompt a user for inputs, read from them from a "lookup table," or generate them randomly. After calling the subsystem to be tested, it might simply report the outputs it received from the subsystem --- or it might check them using a lookup table, or run some sort of "validation" procedure to confirm that the subsystem functioned correctly. (The latter is feasible, and might actually be simpler than the system being tested, because it's often easier to *check* whether an output is correct *when given that output,* than it is to *compute* the output from the input.) Example: Using "Bottom Up Integration," the following subsystems might be obtained from the above system. Modules Included Drivers could be Used to Call ... #1 XI XI (replacing Modules II, VII, VIII) #2 II, XI II (replacing Module I) XI (replacing Module VII, VIII) #3 II, XI, XII II, XI (as above) XII (replacing Modules VII -- X) #4 II, VII, XI, XII II, VII, XI (replacing Module VIII), XII #5 II, VII, VIII, XI, XII II, VII, VIII, XII #6 II, VII, VIII, IX, XI, XII II, VII, VIII, IX, XII #7 II, VII, VIII, IX, X, XI, XII II, VII, VIII, IX, X #8 II, VI, VII, VIII IX, X, XI, XII II, VI #9 II, V, VI, VII, VIII, IX, X, XI, XII II, V, VI #10 II, III, V, VI, VII, VIII, IX, X, XI, XII II, III #11 II, III, IV, V, VI, VII, VIII, IX, X, XI, XII II, III, IV #12 I, II, III, IV, V, VI, VII, VIII, IX, X, XII, XIII No drivers required. Top Down Integration Versus Bottom Up Integration Top down integration has the "advantage" that it starts with the main module, and continues by including top level modules --- so that a "global view" of the system is available early on. It has the disadvantage that it leaves "utility modules" until the end, so that problems with these won't be detected early. It also has the disadvantage or requiring "stubs". Bottom up integration fails to provide a "global view" of the system early on --- though it does test utility modules early. While it does require some extra code --- drivers --- these are frequently simpler than stubs (so the fact that "stubs aren't required" might be viewed as an "advantage" of bottom up integration). An Alternative to Top Down and Bottom Up Integration: An alternative is to consider the *functions* modules are required to perform, and try to cluster modules into "major (functional) subsystems," rather than simply working from the structure chart. Taking this approach, you might to decide to assemble each of the following subsystems separately (perhaps using top down integration, or bottom up integration, or some combination of the two, when assembling each subsystem) --- the set of "interface modules" for each data store --- the set of modules handling I/O from/to each external person or system that the system communicates with --- for a transaction system: the subsystem handling each of the different commands the system is expected to recognize and deal with ... and then combine these together near the end of the "system integration" process. Integration Testing Each of the subsystems assembled during "system integration" should be tested. Guidelines for "integration testing" are similar to guidelines for "unit testing" (when they're given at all): White box testing methods may, for example, ensure that every module in a subsystem is called by at least one of the tests for that subsystem. Black box testing methods work from the specification of the subsystem's requirements. This may resemble the functional requirements for a single module (namely, a "top module" for the subsystem") extremely closely, so "black box" tests for a subsystem might resemble "black box" tests for a module. When tests of a subsystem fail, and errors are identified, it's almost certain that this will lead to changing one or more of the subsystem's modules. At this point, the modules that have been changed must be "unit tested" again --- and the (smaller) subsystems that contain those new modules should be "integrated" and tested again, until either more errors are found (and you have "roll back" to unit testing again) or you reach the subsystem that was being tested when errors were first detected. This process of "rolling back" to unit testing, and repeating unit and integration tests for modules and subsystems that have been changed, is called *regression testing.* In an even worse case, a test may uncover an error that must be corrected by changing part of the system design, or even a part of the requirements specification. Correcting the error will (clearly) be more expensive if this is the case. Reference: See Section 19.3 of Pressman's "Practitioner's Guide" for a bit more information about this topic. Validation Testing Reference: Pressman's Practitioner's Guide, Section 19.4 Validation testing follows system integration and integration testing. The goals of validation testing are - to check that the system satisfies the functions specified in (that is, "agrees with") the requirements specification and user documentation; - to check that the system meets users' expectations. Ideally, these are the same thing! However, it's possible that validation testing might reveal that the system agrees with the requirements specification, but that the requirements specification itself is invalid. Validation testing is the first testing stage in which tests aren't performed entirely by developers. Instead, many validation tests are conducted by the potential *users* of the system being developed. Some validation tests are carefully planned black box tests based on the requirements specification. Others aren't planned at all --- instead, they consist of a group of people who will be using the system "taking it for a test drive," trying to use it to solve the problems they'll encounter in practice. Two Scenarios for Validation Testing I) The proposed system will have only one user, or only a small group of users. This is plausible if the system is a "special purpose" or "customized" system being developed for one organization by a contractor. In this case, "acceptance tests" are used for validation testing. These are tests performed (ideally) by *all* the potential users of the system. Tests are generally performed at the developers' site, with system developers present when tests are performed. Developers won't try to *solve* problems immediately, but they will note them as problems are detected. II) The proposed system will have a much larger group of users --- far too many for all of them to be involved in "acceptance tests." This scenario is likely if general purpose commercial software is being developed. In this case, two testing stages will be used --- so that developers can be closely involved for at least part of the testing, but so that a large group of potential users can be involved as well. A) Alpha Testing Initially, a *small* group of users are involved in testing of the system. As for acceptance testing, tests are generally conducted at the developers' site, in the presence of developers, so that developers are available to suggest tests, and answer questions, and so that developers are aware of problems as soon as they are discovered. B) Beta Testing Following alpha testing, the software is released for trial use to a much larger group of users. During beta testing, the software is used at users' sites, without developers present. It is up to users to log and report problems as they are encountered; error reports are sent to developers on a regular basis. Changing of the language ... recently, "alpha releases" of software have appeared on FTP archives. Since these are being made widely available, they clearly aren't to be tested under the conditions described above for "alpha testing." It *seems* that an "alpha release" of software now means that major subsystems might be missing completely (so that the software provides only some of the functions planned for it). "Beta releases" of software still typically follow alpha releases; these are generally functionally complete, but (as in software under "beta testing") they aren't considered ready for "delivery."