This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine.

Testing
How to automate tests for smooth
system installation by Christopher Agruss

Software
Installation
Testing
I
deally, a software program’s installer smooth-
ly integrates users’ new software into their
existing systems, like a new guest being intro-
duced around at a party. The appropriate
greetings are exchanged, a few dialog screens
offer easily understandable installation options and
encouraging words, and the process is done. In
worst-case scenarios, however, installers can go se-
riously awry. New programs can be unworkable, ex-
isting functions can be compromised, and the
process can leave the user’s system badly damaged.
I know, because I’ve had the bad fortune of purchas-
ing software that left my machine completely unus-
able after installation, requiring a full re-install of
the operating system.

QUICK LOOK
■ Determining what tests to automate
■ Using diagrams to map your tests
■ Tips for testing the uninstallation process
32
www.stqemagazine.com Software Testing & Quality Engineering
 This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine.

This is clearly not the kind of first im- this article’s Webinfolink for more in- compatible with the new software,
pression that builds a loyal customer formation on installer testing.) Most that it has enough free disk space,
base! In order to be confident about of the examples I’ll use are from do- and the like. In this light, a software
your users’ initial experience with ing installations on x86 machines, installer is already an automated
your software, your installer needs to and they will illustrate some of the testing routine of sorts.
be of the highest possible quality. One thought process my teams have gone Viewed this way, explicit automat-
way to ascertain the quality of your in- through in prototyping tests. I’ll also ed installer testing takes the form of
staller is to design a sensible balance touch on some related issues, such as an additional layer of automated test-
of manual and automated tests that automated testing of the complemen- ing (or meta-automation), sitting on
can be run across a range of different tary operation: uninstallation. top of the built-in layer. For example,
most installers check for sufficient
disk space before installing, but your
One difficult aspect of installer testing is that automated test routines can assess
whether the installer is checking for
it often takes a long time to run a single the correct amount of free disk space.
If the installer is assuming that you
need 1,000,000 bytes of free space
installation test case, especially for larger software but you really need 1,000,001 bytes,
an automated test could flag this as a
packages. Multiply this by the number of machine configurations defect.
It’s much easier to design this
that you need to cover, and the time required for testing meta-automation when you under-
stand the assumptions of the in-
staller’s own script. For instance, the
becomes daunting. This is one of the primary reasons that installer’s disk-space-checking logic
might intentionally be assuming that
automated installer testing is such an appealing option. the file system for the disk being in-
stalled upon is the relatively ineffi-
cient FAT16 system; if so, then by de-
machine configurations. In theory, We’ll look at the following areas: sign the installer will overestimate the
this improves the likelihood that im- amount of free space needed for the
portant defects will be found and ■ Recognizing installers’ built-in layer of auto- more efficient NTFS or FAT32 sys-
fixed before the installer arrives in mated tests tems. Knowing this, you could then
your customer’s hands. design your installer tests to flag only
In reality, installer testing—espe- ■ Controlling the machine base states underestimation errors and to ignore
cially manual testing—can sometimes overestimation errors.
be grueling. My friend Alexei was ■ Driving the installer using testing tools Another aspect of installer testing
managing a release testing effort as meta-automation is that your suite
some years ago, and on the last day ■ Using flow diagrams to design your auto- of tests can span a much broader se-
before release he asked a tester to run mated tests ries of related tasks than simply run-
one more installation, to make sure it ning the installer. For example, your
worked on a particular machine con- ■ Using more than one machine to run the test suite might include setting a ma-
figuration. “Okay,” growled the tester, tests chine base state, running an installa-
“I’ll run this last one, but if you ask tion, rebooting, launching the pro-
me to run one more installation test, I ■ Verifying installer test results gram, and then running the uninstaller.
quit!” And he may not have been ex- Along the way, it will probably be veri-
aggerating. ■ Calculating needed disk space fying a variety of critical items too.
One difficult aspect of installer First, let’s take a look at some
testing is that it often takes a long ■ Automating tests for the uninstaller ideas for setting up machines in your
time to run a single installation test installation testing lab.
case, especially for larger software Installers Have Built-In
packages. Multiply this by the number
of machine configurations that you Automated Tests Controlling the
need to cover, and the time required Most installers follow scripts that de- Machine Base States
for testing becomes daunting. This is termine how they write files to disk, A fundamental first step for any instal-
one of the primary reasons that auto- configure your system settings, and lation testing is to establish a baseline
mated installer testing is such an ap- make any other necessary changes so machine state. Disk imaging pro-
pealing option. that you can run the software being grams such as Ghost and IC3 are
In this article, we’ll cover several installed. On top of this, many in- widely used now to return a machine’s
aspects of installation testing best stallers do some type of checking to disk to a base state. (See this article’s
suited to automated methods. (See verify that the machine being used is Webinfolink for a listing of several
33
July/August 2000 Software Testing & Quality Engineering www.stqemagazine.com
 This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine.

Run setup.exe
commercial tool options.) You’ll prob- the installer, followed by launching
ably need an array of machine base the program to run some tests, and
state images, depending on how many then running the uninstaller, also re-
configurations you want to test. One turn your machine to the base state? Yes Previous No
way to approach this is to have one The answers to these questions will installation
image for every combination of oper- help you determine how often you detected?
ating system and disk file format that need to restore the machine base
you plan to test. state from the image file.
If you’ve never done this before, Select destination path
you could start by reformatting a ma-
chine’s disk drive to the most com- Will You Need
Choose installation options
monly used file format, and follow
that by freshly installing the most
a Testing Tool to Drive
commonly used version of the operat- the Installer? Begin installing
ing system. Create the image for that The next question you’re faced with is
simple base state, and then move on how to drive your automated installer
to create images for the other file for- tests. One option is to use the in-
Yes Reboot No
mats and operating system versions staller’s own scripting capability, if it from
that you need to test. has one built in. Some commercial in- installer?
For optimum efficiency you stallers provide command line switch-
should assemble some dedicated lab es for recording a given installation,
machines for installation testing, so and for playing it back in “silent” Quit setup.exe and
that they can be set to the base state mode—meaning that the installer runs reboot manually
when needed, without interrupting without prompting the user for inter-
other activities. The process of imag- actions via the usual dialog screens.
Complete any remaining
ing and restoring an array of ma- Using such an internal silent in- installer tasks
chines is commonly automated, and stall offers several advantages: you
will reap you significant time sav- don’t have to purchase a test tool to
ings—regardless of whether you will drive your installer’s user interface Installation
be doing manual or automated in- (UI), and you’re saved the expense of complete
staller testing. writing test code to drive that UI. An-
Should you restore your baseline other benefit is that the tests may run
images in between each installation faster than they would have had you FIGURE 1 Flow diagram for the
test, or might it be valid to do it less driven them through a UI testing tool. installer
often? If your uninstall program is There are, on the other hand,
reasonably free of defects, then one many advantages to using an industrial- Using Flow Diagrams
alternative is to reset to the full base strength UI testing tool. To begin with,
state only when receiving a new build. a good tool will provide a recovery sys- to Design Your
If this is the case, you should be able
to rely on the uninstallation program
tem in case you wind up in an error
condition. Moreover, these tools al-
Automated Tests
to restore your base state; if not, then ready have extensive testing libraries, A good starting point for designing
you may have found an important de- helping you to verify the results in an the flow of your tests is a simple flow
fect in your uninstaller. automated manner. Of course, if you diagram of your installer’s behavior.
To answer that question more ful- want your automated installation tests The behavior of most installers is
ly, you can make this your first test- to include cases that cover your in- constrained enough to allow them to
case: Does running the installer, and staller’s user interface, then you will be diagrammed without difficulty.
then running the uninstaller, truly re- have to use a UI tool—if, for example, Consider the simplified example of a
turn the machine to the base state? If you wish to automate the testing of flow diagram for a generic installer,
that test is successful, consider an ex- navigating backwards in the installer, or shown in Figure 1.
panded follow-up test: Does running of canceling at various stages. Once you’ve drafted your initial
flow diagram, it can then become the
template on which you superimpose
A good starting point for designing the flow of your the flow of your automated tests. In do-
ing this, be sure to indicate on the dia-
tests is a simple flow diagram of your installer’s gram the points at which you need to
verify the results of your tests. Figure
2, for example, expands on the previ-
behavior. The behavior of most installers is con- ous diagram by placing the installer
ANNIE BISSETT

within an automated test wrapper

strained enough to allow them to be diagrammed without difficulty. (shown in blue). Notice that in this ver-
sion all of the test-related items occur
34
www.stqemagazine.com Software Testing & Quality Engineering July/August 2000
 This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine.

Launch automated Launch automated

before the installer is launched, and af- installation tests installation tests
ter it has completed—if this is accept-
able for your testing purposes, then it
might suffice to use the installer’s own Establish all base Establish all base
states and baselines states and baselines
scripting capability to drive the in-
staller during your automated routines.
The next diagram, Figure 3, is Run setup.exe Run setup.exe
based on the previous ones but has
some testing-related items (shown in
green) added in the midst of the in- Yes Previous No Yes Previous No
stallation procedure. To drive your in- installation installation
staller through these types of inter- detected? detected?
vention tests, you’ll need a UI testing
tool. In this example, one of the
checkpoints we’ve included is for disk Select destination path Select destination path
space calculations at the end of the in-
stallation process (before quitting the
installer) in order to get a direct read- Choose installation options Choose installation options
ing on how much space is required for
the temporary files used. Navigate backwards and
Begin installing
These examples of flow diagrams modify options
are vastly oversimplified. When ex-
panding them into more realistic ver-
Yes Reboot No Begin installing
sions, here are some things to consid- from
er adding. If this is not the first release installer?
of the software being installed, and Alter the amount of
your installer supports upgrades, be free disk space
sure to include different logical paths, Quit setup.exe and
depending on whether a previous ver- reboot manually
sion of the program is installed. Simi- Yes Reboot No
larly, you might include paths for re-in- from
Complete any remaining installer?
stalling the same version on top of installer tasks
itself, and for installing a full version
on top of a compact version.
Verify diffs from baselines Verify disk space requirements
Check to make sure that your de- against expected changes before temp files are removed
sign is flexible before actually translat-
ing your flow diagram into test rou-
tines. Ideally, the same routine you use Exit automated Quit setup.exe and
testing reboot manually
to verify that the correct files are writ-
ten to disk during an installation can
also be used to verify that those same Complete any remaining
files are removed after running the FIGURE 2 Flow diagram for the installer tasks
uninstaller—and you can apply the installer, with an automated test wrapper
same sort of routine for changes to the Verify diffs from baselines
registry. A little planning like this can staller, I recommend using the mas- against expected changes
save you major rewrites later on. ter/slave configuration—in which
When your diagrams are com- one master machine drives the tests
plete, they should make the task of on one or more slave machines. Exit automated
testing
automating installation testing more Some test driving tools include mas-
straightforward. If the diagrams con- ter/slave functionality via an agent
tain sufficient detail, you may be able running on the slave machine. This FIGURE 3 Flow diagram for the
to translate them directly into auto- method affords a number of bene- installer, with automated test wrapper,
mated testing code. fits—allowing you to trigger installa- plus “intervention” tests
tions simultaneously on as many ma-
chines as you have available, and
Using More than consolidate all the result logging eas- the same thing using a single ma-
One Machine to Run ily into a single file.
The master/slave approach also
chine, but picking up the flow of
events after a reboot becomes a real
ANNIE BISSETT

the Tests makes rebooting easier—and allows challenge. It’s possible, but it’s much
Of the many ways to build a distrib- for easier control flow after the reboot more complicated. With a master ma-
uted testing environment for your in- has completed. You could accomplish chine controlling the slave, the master
35
July/August 2000 Software Testing & Quality Engineering www.stqemagazine.com
 This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine.

simply needs to wait until the slave tools that will report changes made to what’s sufficient space for NTFS or
finishes rebooting; the master ma- the registry. (See this article’s Web- FAT32 may not be enough for a disk
chine can then continue with the next infolink for more information.) using FAT16. Non-standard disk for-
routine in the test suite. How well does the installer be- mat settings, such as changes in the
have if you cancel an installation in number of bytes per sector, can also
Automated Verification the middle of the process? This is an-
other test you should consider au-
introduce anomalies in an installer’s
disk space computations. As you can
of Installer Test Results tomating. You can design your auto- see, rigorous testing in this area will
One of the most fundamental things mated routines to include one require a wide variety of machine con-
you’ll want to verify is that the in- “cancel” operation at every step of figurations.
staller writes the expected files to the way. The expected behavior is When automating tests for disk
disk. This is a dynamic element—the usually that everything the installer space requirements, I’ve used the test
number of files written typically has done up to that point is complete- tool’s ability to capture the free disk
grows with the size and complexity of ly undone. space at strategic points in time.
the program being installed. Further, Here’s the basic scheme:
the specific files required often
change during the project’s lifecycle,
How Much Disk Space ■ Capture the baseline free space before
creating the potential for a lot of Do You Really Need? launching the installer.
maintenance work on your verifica- Closely associated with the disk spec
tion routines. is the related issue of how much disk ■ Record the amount of space the installer
This problem can be minimized space is required for a given installa- claims it needs for the various type of in-
by creating data-driven tests, which tion. Fortunately, this is one aspect of stallation.
read the list of files to be verified from installation testing that can be auto-
an external table. This has two advan- mated handily. Some test tools pro- ■ Run each type of installation (beginning
tages. First, it’s often easier to main- vide a library function that will report with the Typical default option), recording
tain a table using a spreadsheet pro-
gram than to edit hard-coded lists of
files inside a test routine. Secondly, Many commercial testing tools allow you to
you might be able to get the people
who wrote your installer program to
maintain the list for you. In fact, in-
test for specific expected changes to the
staller programs for major software
companies are usually written using registry. There are also some free tools that
something called the disk specifica-
tion (or disk spec), which lists all of will report changes made to the registry.
the files that are supposed to be in-
stalled. A more sophisticated ap-
proach would be to make your verifi- the current free disk space, which the free disk space after installing, but while
cation routines intelligent enough to makes these computations especially the installer is still running.
parse the disk spec, and to determine easy.
on that basis which files to verify on Most commercial installers will ■ Quit the installer (don’t choose a restart op-
disk. check that there is sufficient free disk tion), and record disk space after that,
You can begin by simply verifying space, and then will either prevent the which will reflect temp files that were delet-
that the expected files exist in the cor- installation if there isn’t enough ed on quitting.
rect location after installing, and that space, or else will warn you first that
they don’t exist after uninstalling. Fol- you don’t have enough, allowing you ■ Restart, and allow any final setup to occur
lowing on that, you may want to de- to free up disk space before continu- during the boot process, then record disk
velop routines to verify deeper levels ing. The amount of total space re- space afterwards.
of complexity: Is the file the correct quired is usually the sum of the
version? Can a newer version be in- amount of permanent disk space ■ Subtract the baseline from each of the sub-
stalled over an older one, and are old- needed, added to the amount of short- sequent measures, to see how much space
er versions prevented from replacing term storage space for temporary was actually required at each juncture.
newer ones? files that are used during the installa-
If you’re testing on x86 ma- tion. The amount of total disk space This method will give you direct infor-
chines, you’ll probably need to verify needed is often hard-coded into the mation about how accurate your in-
the changes to the system registry af- installer by the developer—and even staller’s disk space requirements are.
ter installing; this can be automated if it is accurate to start with, it is sub- As a follow-up to the procedure
without much difficulty. Many com- ject to error as soon as the disk spec above, we sometimes use Stress Utili-
mercial testing tools allow you to test changes. ty, distributed by Microsoft, to create
for specific expected changes to the Another source of inaccuracy is low-disk-space conditions. It can cre-
registry. There are also some free the file system in use. For example, ate files of any size you choose with
36
www.stqemagazine.com Software Testing & Quality Engineering July/August 2000
 This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine.

which to consume disk space, which

will let you test your installer’s error
Finally worthwhile, given the apparent growth
in the use of shared components.
handling when it finds itself running I’ve tried to highlight some of the rea- I hope that something analogous
low on available space. sons our team found it worthwhile to to Newton’s Third Law will prove true
automate certain aspects of installation for your products: that for every in-
Automating Tests testing. In writing this article, however,
I realized that I’d only scratched the
stallation, there is an equal and oppo-
site uninstallation.
for the Uninstaller surface with respect to the many other I know that by using the tech-
The installer’s complementary opera- dimensions of installer testing that niques I’ve described here—such as
tion—uninstallation—also needs to be would be useful to automate: establishing machine base states, de-
fully tested. One fundamental issue signing automated tests with flow dia-
with some popular installers is that ■ Network installations grams, and rigorously verifying the
they will only uninstall the last opera- test results—my team has been able
tion run, even if the last operation was ■ Installations over the Web to cut back on the amount of manual
a small addition to a large existing in- drudgery normally involved with test-
stallation. In this sense, the uninstaller ■ Generic installer test drivers ing installers. We’ve also been able to
resembles an “undo” function. Know- get more rigorous results from these
ing this, it’s relatively straightforward ■ Installation of third-party software tests by paying special attention to
to automate tests for the uninstaller. the automated verification routines. If
One approach is to simply follow each With third-party software, for exam- you, too, are already using some of
installation test with its equal and op- ple, consider the impact of installing these methods, then you’re already
posite uninstallation. shared DLLs on x86 platforms. As well on your way. STQE
Note that before going through you may have noticed if you’ve run
the uninstall process you’ll probably many uninstallations for different pro- Chris Agruss (chris.agruss@auto-
want to launch the installed program grams, you will sometimes receive an desk.com) has been testing software
at least once to confirm that it was re- alert asking whether you’re sure you since 1984, with a focus on test au-
ally installed successfully. One side ef- want to uninstall a shared DLL. In a tomation since 1994. He manages
fect of this is that launching a pro- nutshell, a counter (refcount) is incre- test automation and installer-
gram will often introduce some mented in the system registry when development teams for the Discreet
configuration files that will not be re- you install a program that installs a division (www.discreet.com) of Auto-
moved by a subsequent uninstallation. shared DLL. If you later uninstall that desk, an international design soft-
For these cases, you could write a program, then the counter is decre- ware company.
garbage collection routine to delete mented, but the DLL won’t be unin-
the user files prior to the next auto- stalled if the counter indicates that
mated installation. some other program that uses the

The installer’s complementary operation—

uninstallation—also needs to be fully tested.
One fundamental issue with some popular installers
is that they will only uninstall the last operation run.
As with many installations, it’s DLL is still installed. However, if the
usually safest to reboot after running program you’re uninstalling is the last
the uninstaller. (Some uninstallers one to be using the DLL, then you’ll
prompt with a suggested reboot at the get the alert message asking whether
end, and others don’t.) My experience you really want to remove the DLL.
is that uninstalling DLL’s without an This is a more complex aspect of
immediate reboot may lead to system installation testing—it requires you to
instability. When manually testing, install two or more programs that
you’ll probably want to check the be- share a component, and then to ob-
havior with and without a reboot (es- serve the behavior during uninstalla-
pecially if your uninstaller doesn’t tion. I haven’t tried to automate the
prompt for it), but for the purposes of testing of this area yet, but it doesn’t
automated testing, I recommend a appear to be prohibitively difficult to
routine reboot after uninstalling. do. I believe this approach would be
37
July/August 2000 Software Testing & Quality Engineering www.stqemagazine.com