TSV3561/TSE3251 Software Verification & Validation
Dynamic White-box Testing
Highlights of this chapter include
What is dynamic white-box testing
The difference between debugging and
dynamic white-box testing
Unit, integration and system testing
Data Coverage
Code Coverage
TSV3561/TSE3251 Software Verification & Validation
Dynamic White-box Testing
Dynamic white-box testing is using information you gain
from seeing what the code does and how it works to
determine what to test, what not to test, and how to approach
the testing.
It is also known as structural testing because you can see
and use the underlying structure of the code to design and run
your tests.
TSV3561/TSE3251 Software Verification & Validation
Dynamic White-Box Testing
The four areas that dynamic white-box testing encompasses are
1. Directly testing low-level functions, procedures, subroutines, or
libraries. In Microsoft Windows, these are called Application
Programming Interfaces (APIs).
2. Testing the software at the top level, as a completed program, but
adjusting your test cases based on what you know about software's
operation.
3. Gaining access to read variables and state information from the
software to help you determine whether your tests are doing what you
thought. And, being able to force the software to do things that would
be difficult if you tested it normally.
4. Measuring how much of the code and specifically what code you "hit"
when you run your tests and then adjusting your tests to remove
redundant test cases and add missing ones.
TSV3561/TSE3251 Software Verification & Validation
Dynamic White-Box Testing Versus Debugging
1. The goal of dynamic white-box testing is
to find bugs.
2. The goal of debugging is to fix them.
3. Grey area: (overlapped) in the area of
isolating where and why the bug occurs.
TSV3561/TSE3251 Software Verification & Validation
Unit, Integration, and System Testing
Unit Testing or Module Testing occurs at the lowest level. As the
units are tested and the low-level bugs are found and fixed, they are
integrated and integration testing is performed.
System Testing involves the process of incremental testing
continues, putting together more and more pieces of the software
until the entire product - or at least a major portion of it - is tested at
once in a process.
There are two approaches to this incremental testing: Bottom-up
and top-down.
In bottom-up testing, you write your own modules, called test
drivers, that exercise the module you're testing. They hook in
exactly the same way that the future real modules will. These
drivers send test-case data to the modules under test, read back the
results, and verify that they're correct.
TSV3561/TSE3251 Software Verification & Validation
Unit, Integration, and System Testing
You can very thoroughly test the software this way, feeding it all types
and quantities of data, even ones that might be difficult to send if
done at higher level.
Top down testing may sound like a big band testing on a smaller
scale. After all, if the higher-level software is complete, it must be too
late to test the lower modules, right?
A test stub sends test data up to the module being tested.
Creating your black-box testing cases based on the specification,
before your white-box cases, is important. That way, you are truly
testing what the module is intended to do. If you first create your test
cases based on a white-box view of modules, by examining the code,
you will be biased into creating test cases based on how the modules
works. The programmer could have misinterpreted the specification
and your test cases would then be wrong. They would be precise,
perfectly testing the module, but, they wouldn't be accurate because
they wouldn't be testing the intended operation.
TSV3561/TSE3251 Software Verification & Validation
Data Coverage
The logical approach is to divide the code
into its data and its states (or program flow).
Consider the data first. Data includes all
variables, constants, arrays, data structures,
keyboard and mouse inputs, files and screen
input/ouput, I/O to other devices such as
modems, networks, and so on.
TSV3561/TSE3251 Software Verification & Validation
Data Flow Coverage
Data Flow coverage involves tracking a piece
of data completely through the software.
At the unit test level this would just be
through an individual module or function.
The same tracking could be done through
several integrated modules or even though
the entire software product.
TSV3561/TSE3251 Software Verification & Validation
Sub-Boundaries
Examples of sub-boundaries:
-A module that computes taxes might switch from using a data
table to using a formula at a certain financial cut-off point.
-An operating system running low on RAM may start moving
data to temporary storage on the hard drive. This sub-boundary
may not even be fixed. It may change depending on how much
space remains on the disk.
-To gain better precision, a complex numerical analysis program
may switch to a different equation for solving the problem
depending on the size of the number.
TSV3561/TSE3251 Software Verification & Validation
Formulas and Equations
Given this formula:
A = P(1+r/n)nt
A good black-box tester would hopefully choose a
test case of n=0.
A white-box tester, after seeing the formula in the
code, would know to try n=0 because that would
cause the formula to blow up with a divide-by-zero
error.
TSV3561/TSE3251 Software Verification & Validation
Error Forcing
The last type of data testing is error forcing.
If you're running the software that you're
testing in a debugger, you don't just have the
ability to watch the variables and see what
values they hold, you can also force them to
specific values.
TSV3561/TSE3251 Software Verification & Validation
Code Coverage
You must attempt to enter and exit every module,
execute every line of code, and follow every logic and
decision path through the software.
The simplest form of code coverage testing is using
your compiler's debugger to view the lines of code
you visit as you single-step through the program.
Code coverage analyzers hook into the software
you're testing and run transparently in the
background while you run your test cases.
Each time a function, a line of code, or a logic
decision is executed, the analyzer records the
information.
TSV3561/TSE3251 Software Verification & Validation
Program Statement and Line Coverage
The most straight forward form of code
coverage is called statement coverage or line
coverage.
If you're monitoring statement coverage
while you test your software, your goal is to
make sure that you execute every statement
in the program at least once.
TSV3561/TSE3251 Software Verification & Validation
Program Statement and Line Coverage
It is very easy to test every line of this simple program.
1:
2:
3:
4:
PRINT Hello World
PRINT The date is: ; Date$
PRINT The time is: ; Time$
END
100% statement coverage would be the execution of lines 1
through 4.
However, statement coverage is misleading, it can tell you if every
statement is executed, but, it cant tell you if youve taken all the
paths through the software.
TSV3561/TSE3251 Software Verification & Validation
Branch Coverage
Attempting to cover all the paths in the software is called path
testing.
The simplest form of path testing is called branch coverage
testing.
1:
2:
3:
4:
5:
6:
7:
PRINT Hello World
IF Date$ = 01-01-2010 THEN
PRINT Happy New Year
END IF
PRINT The date is: ; Date$
PRINT The time is: ; Time$
END
-If you test this program with the goal of 100%
statement coverage, you would need to run only a
single test case with the Date$ variable set to
January 1, 2010. The program Would then execute
the following path: Line 1,2,3,4,5,6,7
-You may have tested every statement, but you
didnt test every branch.
-You still need to test for a date that is not January
1, 2010. The program would execute the other path
through Lines 1,2,5,6,7
TSV3561/TSE3251 Software Verification & Validation
Condition Coverage
Just when you thought you had it all figured out, there's yet another
complication to path testing.
An extra condition is added to the IF statement in line 2 that checks
the time as well as the date.
1: PRINT Hello World
2: IF Date$ = 01-01-2010 AND
Time$ = 00:00:00 THEN
3:
PRINT Happy New Year
4: END IF
5: PRINT The date is: ; Date$
6: PRINT The time is: ; Time$
7: END
-to have full condition coverage testing, you
need to have 4 sets of test cases:
Date$
Time$
Valid date
Valid time
Valid date
Invalid time
Invalid date Valid time
Invalid date Invalid time
Line# execution
1,2,3,4,5,6,7
1,2,5,6,7
1,2,5,6,7
1,2,5,6,7
Condition coverage testing takes the extra conditions on the branch
statement into account.
TSV3561/TSE3251 Software Verification & Validation
Code coverage example
What test values for n do we need to
cover all the statements?
n < 0, n > 0
Does that get us branch coverage ?
No, we also need n = 0
Does that get us condition
coverage?
Yes: no compound conditions
How about loop coverage?
Need to cover n=1 and n = max
1. #include <stdio.h>
2. main()
3. {
4. int i, n, f;
5. printf(n = );
6. scanf(%d, &n);
7. if (n<0) {
8.
printf(Invalid: %d\n, n);
9.
n = -1;
10. } else {
11.
f= 1;
12.
for (i=1;i<=n;i++) {
13.
f *= i;
14.
}
15.
printf(%d! = %d\n, n, f);
16. }
17. return n;
18. }
TSV3561/TSE3251 Software Verification & Validation
Summary
Static black-box testing involves examining the
specification and looking for problems before they
get written into the software.
Dynamic black-box testing involves testing the
software without knowing how it works.
Static white-box testing involves examining the
details of the written code through formal reviews
and inspections.
Dynamic white-box testing involves testing the
software when you can see how it works and basing
your tests on that information.
TSV3561/TSE3251 Software Verification & Validation
Reference:
TSV3561/TSE3251 Software Verification & Validation
TSV3561/TSE3251 Software Verification & Validation
TSV3561/TSE3251 Software Verification & Validation
TSV3561/TSE3251 Software Verification & Validation
LCSAJ Linear Code Sequence And Jump Coverage
TSV3561/TSE3251 Software Verification & Validation
TSV3561/TSE3251 Software Verification & Validation
TSV3561/TSE3251 Software Verification & Validation
TSV3561/TSE3251 Software Verification & Validation
