CPSC 333 --- Lecture 21 --- Wednesday, March 6, 1996 Unit Testing: Each module in the system is individually tested. Some White Box (Unit) Tests: - (Basis) Path Testing - Control Structure Testing: - Condition Testing - Data Flow Testing - Loop Testing Note: These assume that coding has been performed using a high level procedural programming --- like Pascal, Modula, Ada, or (arguably) C. Path Testing: "Caveat:" Both versions of Pressman's book include material on "basis path testing." After teaching Pressman's (and presumably the literature's) version of this for several years, I've concluded that it's unnecessarily complicated. I'll present a "simplified" method and will then briefly discuss the more complicated one described in Pressman's books. Goal of Path Testing: Design a *small* number of test cases, while ensuring that every statement in the program (and both sides of every test) is exercised by at least one of the tests. Basis path testing is most easily described using *logic flow graphs.* These are similar to, but simpler than, (structured) flowcharts. Program statements and simple predicates are represented by nodes; edges indicate possible flow of control. Flow graphs can be defined for a number of control structures used in "structure programming: - a sequence of two statements (s1, followed by s2) so that whenever statement s1 is executed, s2 is always executed in the next step, is represented by having a directed edge from the node for s1 to the node for s2: -> s1 -> s2 This is usually *drawn* so that s2 appears *below* (instead of to the right of) s2: | V s1 | V s2 - an if-then statement, "if c then s1 end if; s2" Here, condition c is tested. If its value is "true" then s1 is executed and s2 is executed after that. If, instead, its value is "false," then s1 is skipped (and s2 is executed right away). In this case there are *two* directed edges out of the node for c, and each is labelled with one of c's possible values, (or, possibly by "Yes" and "No," where "Yes" is generally understood to mean that c has value "true.") | V c /| / | Y / | N / | / | | | V | | s1 | | | | | V | -> s2 - an if-then-else statement, "if c then s1 else s2 end if; s3" Here, condition c is tested. If its value is "true" then s1 is executed, and s2 is executed otherwise. In both cases, s3 is executed after that | V c / \ / \ Y / \ N / \ / \ / \ | | V V s1 s2 | | ---> s3 <---- - a while-do loop, "while c do s1 end while; s2" Here, condition c is tested. If c is satisfied then s1 is executed and the process repeated. This continues until it's discovered that c is false when tested. At that point, s2 is executed: | V ------> c ----- | | | | | N | Y | | | | V | | -- s1 <-- s2 - a repeat-until loop, "repeat s1 until c; s2" is similar, but s1 is always executed at least once (before c is ever checked) --- and you exit from the loop when c has value *true* (instead of false): | V -----> s1 | | | | | | V | N Y -------- c -----> s2 In order to obtain flow graphs for tests and loops with loop (or test) bodies that consist of more than one statement, you could work from the top down (or from the outside in), using whichever of the above flow graphs as a "template," and including the *flow graph* for the body instead of including just a single vertex (as is given above, for the case when the body includes only one statement). Using this technique, you can derive the "flow graph" for some other control structures by writing equivalent programs that use the control structures discussed above, then producing the flow graphs for these programs. For example, to derive a flow graph for case variable of value1: s1; value2: s2; value3: s3; . . . valueN: sN; default: sN+1 end case you could write an equivalent program if (variable = value1) then s1 else if (variable = value2) then s2 else if (variable = value3) then . . . else if (variable = valueN) then sN else sN+1 end if end if . . . end if end if end if and generate the flow graph for that, using the "templates" given above. I claim that a "for"-loop can be handled in the same way (Exercise!) Finally, for some control structures, it seems easiest just to think about what execution paths through the program are possible, and then try to generate a flow graph that corresponds to that. Assuming that all "tests" are two-sided (they can only result in values "true" or "false") all tests (or "conditions") will correspond to vertices with two outgoing edges. All other statements (except a "stop" statement) will have exactly one outgoing edge (and the "stop" statement has no outgoing edges at all). Each program should have exactly one entry point (first statement executed), there will always be one vertex "labelled" as the start vertex --- by having an edge coming in "from nowhere" and pointing to that vertex, as in the above examples. It's probably easiest to construct the flow graph for a general "loop" (as included in the programming language "Ada") using this technique. This control structure doesn't necessarily have a test at the beginning of the loop *or* at the end. Instead, it includes one or more "exit" statements that trigger exit from the innermost loop, inside the loop body (usually inside if-then-else tests, etc.)). It's not clear (to the instructor) how you could use "repeat until" or "while do" loops to write a program that's equivalent to this kind of a more general "loop." (Actually, there *is* a way to do it... but it's ugly.) Example: Consider a simple "sorting" program Input: Array a of integers a[1], a[2], a[3], ..., a[n] Output: Same array, with contents rearranged in sorted order i := 2 while i <= n do { Move a[i] forward so that the sublist a[1], a[2], ..., a[i] is sorted } j := i-1 while ((j >= 1) and (A[j] > A[j+1])) do temp := A[j] A[j] := A[j+1] A[j+1] := temp j := j-1 end while i := i+1 end while Label statements and conditions as follows: s1: i := 2 c1: i <= n ? s2: j := i-1 c2: ((j >= 1) and (A[j] > A[j+1])) ? s3: temp := A[j] s4: A[j] := A[j+1] s5: A[j+1] := temp s6: j := j-1 s7: i := i+1 s8: "stop" Then the flow graph for this program looks like the following. | V s1 | e1 V e10 -----------------------> c1? | | / \ | Y / \ N | / \ | / \ | e2 | | e11 | V V | | s2 s8 | | | e3 | V | e8 | ---------> c2? | | | | / \ | | Y / \ N | | / \ | | e4 / \ e9 | | | | | | V V | | | | s3 s7 | | | | e5 | | | | V | | | | | | s4 | | | | | | e6 | | | | V | | | | | | s5 | | | | | | e7 | | | | V | | | | | | s6 | | | | | | | | | |_______| | | | |_________________________| As the numbering of edges may suggest, this graph includes ten nodes (eight for statments --- including a single "stop" statement added at the end of the program --- and three for tests) and eleven edges (not counting the arrow pointing to the start node at the top of the graph). In general --- if a program has been written using a "structured" programming language that includes assignment statements, if-then and if-then-else statements, and while-do and repeat-until loops --- then (since every node except the "stop" node corresponds to a statement in the program, and every node has "fan-out" at most two) the number of edges will be at most twice the number of statements in the program. If "case" (or "switch") statements are added as well, then this remains true provided that you pretend that a "case" statement counts for k statements, if k is the number of cases included in the statement. Now, consider the following three test cases for this program: Test #1: Input: n=1; a[1] = 1 Expected Output: a[1] = 1 Test #2: Input: n=2; a[1] = 1, a[2] = 2 Expected Output: a[1] = 1, a[2] = 2 Test #3: Input: n=2, a[1] = 2, a[2] = 1 Expected Output: a[1] = 1, a[2] = 2 If you trace through the program using these three tests, you will find that the following paths (from the start node to the stop node) are followed in the flow graph as the program is executed. Test #1: e1 e11 s1 -----> c1 -----> s8 Test #2: e1 e2 e3 e9 e10 e11 s1 -----> c1 -----> s2 -----> c2 -----> s7 -----> c1 -----> s8 Test #3: e1 e2 e3 e4 e5 s1 -----> c1 -----> s2 -----> c2 -----> s3 -----> s4 ... e6 e7 e8 e9 e10 e11 ... s4 -----> s5 -----> s6 -----> c2 -----> s7 -----> c1 -----> s8 These three tests have the property that every edge in the flow graph is "used" or "exercised" at least once when all these tests are executed. The goal of "path testing" is to find a set of tests with this property. One way to try to find such a set of tests is to think about what the program does, and then repeatedly look for *simple* tests, such that each new test uses at least one edge of the flow graph that hasn't been used by any previous test. Notes: - The third test satisfies this requirement all by itself --- so that this one test is "sufficient" (in some sense). However, it is useful to include a few more tests --- because, if execution of one (or more) of the tests leads to an incorrect output, then it's easier to decide whether the error in the program's source code might be found, than it would be if a smaller number of (more complicated) tests had been used instead. In particular: If test #3 had been the only one executed for this program, and the program had failed this test, then (in principle) you'd need to look at the entire program when looking for the error that had caused the problem. On the other hand, if tests #1 and #2 had been executed first, and the program had passed these, then it would make sense to concentrate (first) on the part of the program used by test #3 but not used by tests #1 and #2 --- namely, the inner "while" loop body. - Since each test "exercises" at least one new edge, and the number of edges is at most twice the length of the program, the number of tests is also at most twice the length of the program. If you're careful, you can prove something even better: the number of tests will never be more than (one or two more than) the number of statements in the program. As suggested above, you can frequently use even fewer tests than this, but the tests likely be more complicated and less useful if you try to get by with substantially fewer tests than this bound. - It should be clear that this set of test cases *won't* include every kind of test you might want. For example, note that in the above example, none of the test cases requires more than one execution of the body of either of the while loops. Because of this, you should expect that you'll need to use several *other* methods to design test cases, along with "path testing," in order to generate a reasonably "comprehensive" set of tests. Cyclomatic Complexity The "cyclomatic complexity" number of a flow graph can be defined in three different ways: I) If you think of the edge flow graph of as partitioning (or separating) connected components in the plane (or, on your computer screen, or on the blackboard), then the cyclomatic complexity of the flow graph is equal to the number of regions that are separated by these edges ---- including "the rest of the plane" that surrounds the flow graph as one region II) The a flow graph has "e" edges and "n" nodes then the cyclomatic complexity of the flow graph equals e-n+2. Note that "the edge coming in from nowhere" in order to label the start node is *not* counted as a vertex here (in order to figure out what value "e" has) III) If the program for the flow graph has p (two-sided) tests, so that there are exactly p nodes that have "fan-out" 2 (and all the rest of the nodes have fan-out 1 or 0) then the cyclomatic complexity of the flow graph equals p+1 *If* you only have two-sided tests in your program (and not three-way tests, etc) *and* you constructed your flow graph correctly, then all three definitions of "cyclomatic complexity" should give the same value when applied to your flow graph. The flow graph in the above above example - partitions 3 regions of the plane - has 10 nodes and 11 edges... and 11 - 10 + 2 = 3 - represents a program with 2 tests ... and 2 + 1 = 3 ... so that computation of the cyclomatic complexity does *not* suggest that there was an error when the flow graph was constructed. One common mistake made when constructing a flow graph is to add an extra edge by accident, when building the flow graph for a "while" loop. In particular, a common error is to add the edge that would correspond to a connection from s1 straight to s2, in the "template" given above for this loop. Note that this mistake would increase the number of regions for the graph by one, and add an edge --- so it would increase the value of "cyclomatic complexity number" for the first two definitions given above, but it wouldn't increase the number of tests in the *code* --- so that the value of "cyclomatic complexity number" given by the third definition wouldn't be changed. That is, checking the flow graph by computing the cyclomatic complexity all three ways *would* indicate that a mistake had been made in this case. The cyclomatic complexity number is also interesting because it gives an upper bound on the number of tests that you could possibly end up using, when applying "path testing" to the corresponding program. By the third definition, this is only one more than the number of tests in the code --- so it's certainly linear in the length of the program, and can be substantially smaller than that for some programs. I won't *prove* that the cyclomatic complexity number forms an upper bound on the number of tests required (and won't ask you to, either).