09/09/2020

Objectives

Software Testing  Configuration testing and compatibility testing are two

testing processes that are often ignored in many project
Practical Skills for Testing Practitioners © 2009
teams. In this lecture we will explore:
• What is configuration testing
Configuration Testing and Compatibility • Configuration testing steps
Testing • What is compatibility testing
• Compatibility testing steps

Instructor: Tran Kim Sanh

© 2009, CMU-ISR 2

What is Configuration Testing(1) What is Configuration Testing(2)

 Definition  Why configuration testing is necessary.

• The process of checking the operation of the tested software • Different software may tie to different configuration areas,
with all various types of hardware. for example:
 Why configuration testing is necessary 1. A highly graphical computer game requires lots of
attention to the video and sound areas.
• There are various configuration possibilities provided by
different manufacturers that your software may run on. The 2. A greeting card program will be especially vulnerable to
important thing to remember is your software may run printer issues.
smoothly on one kind of configuration, but often crashes on 3. A communication program will need to be tested with
another. numerous modems and network configuration.

© 2009, CMU-ISR 3 © 2009, CMU-ISR 4

1
 09/09/2020

Configuration Testing Steps Identify Hardware Configuration(1)

 When to test configuration:  The core task of configuration testing is to reduce hardware
• In general, we schedule configuration testing after the configuration combinations because:
baseline of function testing in the system testing phase. • It is not practical to perform a complete configuration testing
Otherwise, it is much harder to determine whether the • A windows online game program support
defect lies in software or hardware.
• 350 possible display card
 Steps • 200 sound card, 800 modems
• Identify hardware combination • 1200 printers
• Design the test cases to run on each configuration there should be 350*200*800*1200 combinations
• Conduct the tests on each configuration • You may conduct a certain number of test cases on these
• Fix defects and verify modifications different configurations. The number of total test cases could be
• Report results larger.

© 2009, CMU-ISR 5 © 2009, CMU-ISR 6

Identify Hardware Configuration(2) Identify Hardware Configuration(3)

 Steps to size up your configuration testing:  Determine the types of hardware you need by:
• Determine the types of hardware you need. • Asking yourself what kind of hardware you need.
• Determine what hardware brands, models and device drivers • Identifying the critical hardware to your software.
are available. • Examining your software feature carefully to ensure
• Determine which hardware features, modes and options are everything is included.
possible.
 Determine what hardware brands, models and device
• Write down the identified hardware configuration. drivers are available.
• Identify your unique software features that work with the • Determine proper devices
hardware configuration.
• Identify equivalent devices
• Determine device drivers included with:
1. Operation system
2. The device
3. Latest drivers available for the hardware or operating system
• Hardware that your customers have

© 2009, CMU-ISR 7 © 2009, CMU-ISR 8

2
 09/09/2020

Identify Hardware Configuration(4) Identify Hardware Configuration(5)

 Determine which hardware features, modes and options are  Identify your unique software features that work with the
possible. hardware configuration.
• Each device has different options. • Don’t try to test every software feature on each hardware
• Not all the device options should be supported by your software. configuration.
• Minimum configuration should be tested. • In many cases it is a hard work to identify unique features,
you may:
 Write down the identified hardware configuration.
1. Walk through the product using the black-box method
1. Classify the identified hardware by applying equivalence and pull out obvious ones.
partitions.
2. Talk with project members, especially, the programmer.
2. Write down configuration information into a table, such as
popularity, type, age, manufacturer, model, device driver
version, options and so on.
3. Fill in the table with the hardware you get in step 1.
4. If needed, use decision-making processes to get a smaller set.
5. Review the table and decide which configuration best fits your
project.

© 2009, CMU-ISR 9 © 2009, CMU-ISR 10

How to Obtain the Hardware Suggestions

 Getting all the identified hardware is hard work  You may adopt the following:
• It is expensive to buy everything. • Identifying hardware standards to help you get deep insight
• Sometimes the hardware will be used only once. about the hardware so that you can make more informed
equivalence partition decisions.
• It is hard to maintain different models, versions, drivers of
hardware. • Answering the following questions may help you when
performing configuration testing.
 Approach 1. What external hardware will operate with this software?
• Only buy the most frequently used configuration. 2. What models and versions of that hardware are available?
• Borrow or rent from the hardware manufacturers. 3. What features or options does that hardware support?
• Ask other project teams in your company for the needed
hardware.
• If it is possible, build a configuration and a compatibility
testing lab.

© 2009, CMU-ISR 11 © 2009, CMU-ISR 12

3
 09/09/2020

Design Test Cases Conduct the Configuration Tests

 Design test cases to run on each configuration.  Identify configuration defects

• According to previous steps, you may design the details of • Configuration defects are hard to identify because:
test cases: 1. As a software engineer, you may believe your hardware
1. Configuration and features to be tested vendor provides perfect products and the defect only lies
in your program.
2. Preconditions
2. In most cases, it is hard for a programmer to find
3. Operation steps
hardware problems because of the lack of domain
4. Expected result knowledge.
3. You may be disappointed by huge amounts of
configuration possibility combinations when trying to
identify configuration defects.

 Approach
• Perform the exact same operation that caused the problem,
step by step on another computer with a different hardware
setup. If the defect doesn’t occur, it may be that you have a
configuration problem.

© 2009, CMU-ISR 13 © 2009, CMU-ISR 14

Fix Defects And Verify Modifications What is Compatibility Testing

 Who fixes the configuration defects  Definition
• The configuration problems may occur because: • The process of checking how the software interacts and
shares information correctly with other software.
1. Your software may have a defect that appears under a
broad class of configurations or only to one particular • The interaction could occur between:
configuration. 1. Two programs simultaneously running on the same computer.
2. The hardware device or its device drivers may have a 2. Two programs running on different computers connected through
defect that only your software reveals or can be seen a network.
with lots of other software. 3. Simple as saving data to a floppy disk and hand-carrying it to
another computer.
 Solution
• Compatibility testing is a test process much like configuration
• Under the first situation, it is surely your responsibility to fix testing. It takes care of interactions with compatible software
the defect. which should be defined in the specification.
• Under the second situation:
 Why compatibility testing is necessary:
• The hardware vendor fixes the defect, or
• The software may interact with each other. If the interaction
• Provides a software version to get around the hardware fails, of course, the function will also fail.
defect.

© 2009, CMU-ISR 15 © 2009, CMU-ISR 16

4
 09/09/2020

Compatibility Testing Steps Identify Compatible Software (1)

 When to test compatibility:  Approach

• Like configuration testing, in general, we schedule • Analyze the specifications.
compatibility testing after the baseline of function testing in
• Discuss with the technical stakeholders.
the system testing phase. Otherwise, it is much harder to
identify the compatibility defects. • Consider the following:
• Most project teams combine configuration testing with 1. Platform and application versions: What platform, such
compatibility testing and call it compatibility testing. as operating system, web browser, and other application
is the software designed to be compatible with?
 Steps
2. Standards and guidelines: What compatibility standards
• Identify compatible software or guidelines should be followed that define how your
• Design test cases for each compatible software software should interact with other software?
• Execute the tests 3. Data sharing: What types of data will your software use
• Fix defects and verify modifications to interact and share information with other platforms
and software?
• Report results

© 2009, CMU-ISR 17 © 2009, CMU-ISR 18

Identify Compatible Software (2) Identify Compatible Software (3)

 Platform and Versions  Platform and Versions

• The compatible applications and their versions can be found 2. Write down compatible information into a table, such as
in the specifications and should be defined by managers or popularity, type, age, manufacturer, version, options and so on.
marketing members. 3. Fill in the table with the software you get in step 1.
• Back and Forward compatibility: Don’t forget the software 4. If needed, use a decision-making process to get a smaller set.
itself may be back compatible or forward compatible or both.
5. Review the table and decide which compatible software fits
1. Back compatible means the software will work with its your project best.
previous versions.
 Standards and Guidelines
2. Forward compatible means the software will work with its
future versions. • There are two levels:
• Multiple version: Like configuration testing, in order to 1. High level standards and guidelines: the ones that guide
reduce the number of compatible software to test, you must your product’s general operation such as its look and feel,
determine which ones are the most important to test: supported features and so on.

1. Classify the identified software by applying an 2. Low-Level standards and guidelines: details of your
equivalence partition. products such as file formats, communication protocols
and so on.
(to be continued)

© 2009, CMU-ISR 19 © 2009, CMU-ISR 20

5
 09/09/2020

Identify Compatible Software (4) Other Steps

 Data Sharing Compatibility  About the remaining steps:
• If data is shared among applications, you should test the • Design test cases for each compatible software.
data format according to the standards and protocols, and
• Conduct the tests.
the following should also be taken into account:
• Fix defects and verify modifications.
1. File save and file load: data format of the files needs to
meet standards. • Report results.
2. File export and file import: use equivalence partition to • These steps are much like that of configuration testing. You
create test documents in each compatible file format. can refer to previous slides.
3. Cut, copy and paste: you should make sure that the
application data can be properly copied in and out of
clipboard to other programs.
4. DDE, COM and OLE are the windows methods to transfer
data between two applications. These methods transfer
data in real time and can occur automatically.

© 2009, CMU-ISR 21 © 2009, CMU-ISR 22

Summary
 Configuration testing and compatibility testing are two similar
testing processes. Configuration testing supports hardware while
compatibility testing supports software.
 Many project teams combine configuration testing with
compatibility testing.
 The common steps of the two testing processes are:
1. Identify needed hardware/software
2. Design test cases
3. Run test cases
4. Fix defects and verify modifications
5. Report results

© 2009, CMU-ISR 23