09/09/2020

Objectives
 Explore the knowledge of performance testing, and help
Software Testing people understand:
Practical Skills for Testing Practitioners © 2009
• The definition of performance
• The definition of performance testing
• Performance testing lifecycle, including:
Performance Testing 1. Planning for performance testing
2. Creating scripts and conduct testing
3. Analyzing results and tune system
Instructor: Tran Kim Sanh

© 2009, CMU-ISR 2

What is Performance What is Performance Testing

 Definition  Definition
• Different projects may have different performance targets. It • Performance testing is the process of analyzing whether or not
is hard to give an exact definition of performance. But the system performance achieves expected targets. This is done by
following are the most important items that almost every simulating typical user activities with the help of performance
project should take into account: testing tools or manual test scripts while monitoring the key
1. Response time: the length of time between sending of an performance metrics.
inquiry from the client, to the display of the last byte of • It gathers information about current performance, but places
the response at a user terminal. no value judgments on the findings even though the findings
2. Throughput: The amount of work that can be performed sometimes are very important.
by a computer system or component in a given period of • Performance testing is so important that some professionals
time. regard it as an engineering, that is, performance engineering.

© 2009, CMU-ISR 3 © 2009, CMU-ISR 4

1
 09/09/2020

What is Stress Testing Performance Testing Lifecycle(1)

 Definition  A simple performance testing lifecycle,
• Testing conducted to evaluate a system or component at or including:
beyond the limits of its specified requirements, for example, • Plan for Performance Test
response time, user volume etc. • Create Scripts and Conduct Test
• The main purpose of stress testing is to determine the Plan for Create Scripts • Analyze Results and Tune System
Performance Test and Execute Test
maximum load under which the specific critical performance
 Plan for performance test
targets cannot be met.
• This stage is the most important of
• Stress Testing is a part of performance testing that usually is performance testing. It may take
the last step when executing performance testing, as you will about 50% effort to do the job. In this
see later in the performance testing lifecycle. stage, performance testers gather
Analyze Results
and Tune System information that would be used in
performance tests, such as system
Simple Performance Testing Lifecycle
architecture, deployment
environment, the expected user
activities, performance targets etc.
and putting it together in a document
named performance test plan.

© 2009, CMU-ISR 5 © 2009, CMU-ISR 6

Performance Testing Lifecycle(2) Performance Testing Lifecycle(3)

 Create Scripts and Conduct Testing  Analyze Results and Tune System
• Develop performance test scripts
• Use the help of performance tools or • Consolidate, analyze and share
by programming test results.
Plan for Create Scripts • Run baseline test Plan for Create Scripts • Then fine tune and test again to
Performance Test and Execute Test Performance Test and Execute Test see if there is an improvement or
• As a basis of comparison by all future decrease in performance.
testing
• Since improvements generally
• Tester will increase system load step
grow smaller with each retest,
by step to gather performance metrics
stop when bottlenecking is
Analyze Results Analyze Results caused by the CPU.
and Tune System and Tune System
• Then you may have the consider
option of increasing CPU power
Simple Performance Testing Lifecycle Simple Performance Testing Lifecycle

© 2009, CMU-ISR 7 © 2009, CMU-ISR 8

2
 09/09/2020

Planning for the Performance Test Create Performance Test Environment(1)

 In this step, with the help of technical stakeholders,

 Steps performance testers will:
• Create performance test • Be aware of system functions:
environment. 1. Performance testers must first understand what the system
Plan for Create Scripts
• Identify performance goals. is capable of doing.
performance test and Execute Test
• Develop usage model. 2. It is much better if the performance tester takes part in
the project as early as possible.
• Define the key metrics to monitor
and analyze when creating a 3. Performance testers can get the information from
performance test plan. specifications and can talk with the technical stakeholders.

Analyze Results
• Identify user activities: This information will be included in the
and Tune System performance test scripts to simulate user action, including:
1. All types of users and activities.
2. The frequency with which users perform the activity.
3. Which activities are most performance critical.
(to be continued)

© 2009, CMU-ISR 9 © 2009, CMU-ISR 10

Create Performance Test Environment(2) Identifying Performance Goals

4. Information from historical data or marketing survey.  Approach
• Discussion with your stakeholders
• Keep system architecture in mind to identify performance • Use industry standards and guidelines
bottlenecks: • Use historical data
• From deployment point of view • Extensive marketing research
• From code point of view
 Basic categories of high-level performance requirements
• From process point of view
• Response time acceptability
• Throughput and concurrent user goals
• Future performance growth requirements
 Remark
• Surely, there are other important performance metrics in
your application, but those three, especially the first two, are
the most critical and are exposed directly to end user.

© 2009, CMU-ISR 11 © 2009, CMU-ISR 12

3
 09/09/2020

What is Response Time(1) What is Response Time (2)

 Definition  Commonly used types of response time:
• The length of time between the sending of an inquiry from • Average Response Time: arithmetic mean of all user’s
the client and the display of the last byte of the response at response time for a specific inquiry.
a user terminal. • Total Average Response Time: the sum of all single average
• Ingredients response time in a test script divided by the request number.
1. Process time: the length of time between the receiving • Maximum Average Response Time: the maximum value of all
of an inquiry by the server and the sending of the single average response time in a test script.
response to the user.
2. Transportation time: the length of the time between
leaving of an inquiry from the client side and receiving
of the server plus the return of the response.
3. Drawing time: the length of the time it takes to display
the results to the user by the client.

© 2009, CMU-ISR 13 © 2009, CMU-ISR 14

Define Response Time

What is Response Time (3)
Acceptability (1)
• There are different definitions of response time. For example:  Steps
1. Another definition of response time: the length of time • Identify the ways your customers and potential customers
between an indication of the end of an inquiry and the access your application to determine an acceptable amount
display of the first character of the response at a user of time, you can:
terminal.
1. Discuss with your stakeholders
2. IEEE: The elapsed time between the end of an inquiry or
2. Compare with similar systems
command to an interactive computer system and the
beginning of the system’s response. 3. Use standards and guidelines
• Keep it consistent. 4. Use historical data
5. Do an extensive marketing survey
• Determine your response time acceptability targets based
on the results of step 1.

© 2009, CMU-ISR 15 © 2009, CMU-ISR 16

4
 09/09/2020

Define Response Time Throughput and Concurrent User

Acceptability(2) Goals
 Example  Approach
• Take your website as an example, if you identify the ways • Ask yourself the following questions:
your users access your site as shown in the following table, 1. How many concurrent users do we currently sustain
you may define response time acceptability. That is, with all or expect in a given time period?
things being the same for the same pages to load on a
1.5Mbps T1 connection, will obviously be faster than a 28.8- 2. What actions does a typical user perform on our
kbps modem connection. The response time acceptability for application and which functions are the most used in
the 28.8-kbps modem might be 20 seconds, while the T1 a given time period?
connection might be much less, perhaps 2 seconds. 3. How many user scenarios will my application process
in a given time period?
• You can:
1. Discuss with your stakeholders.
2. Find your answer from historical data.
3. Refer to similar products or business standards and
guidelines.

© 2009, CMU-ISR 17 © 2009, CMU-ISR 18

Future Performance Growth

Develop Usage Model
Requirements
 Approach  Steps
• The easiest way to determine the growth capacity of your 1. Describe a usage model that would be used in
application is to calculate the increase in volume you are performance testing, including:
currently experiencing over a specific period of time. • User activities
• For example, the table below shows an anticipated growth • System activities
plan. It assumes your application is currently seeing 2,000
• How often these activities occur by both time and
users per day and will grow at a rate of 50% per month.
relative percentage.
You may use this information in your stress testing.
• Notice: You can’t develop all of the usage model. A
good target is to model 20% of the user activity that
accounts for 80% of the total volume of activities.
2. Verify usage model: present the critical usage model to
the project stakeholders to review.

© 2009, CMU-ISR 19 © 2009, CMU-ISR 20

5
 09/09/2020

Define Key Metrics(1) Define Key Metrics(2)

 Definition  Example
• Key metrics are those metrics that would be monitored • The following metrics may be critical for a web application:
and analyzed during the actual performance test in order 1. Server error acceptability
to validate whether the defining performance targets are
met or not. 2. Server utilization acceptability

• Different projects may define different key metrics, the 3. Memory leaks or other stability issues
key point is to come up with a set of requirements that 4. Processing delays
make sense for your application. • And you may have the performance metric acceptability table
• You can get your key metrics by: below:
1. Consulting the domain expert
2. Referring to similar project and business standards

© 2009, CMU-ISR 21 © 2009, CMU-ISR 22

Performance Test Plan Create Scripts and Execute Tests

 Performance testers may determine the performance test
strategy based on what he gets during this stage and output a  Steps
document named “Performance Test Plan.”
• Create performance test
 Synopsis scripts
• Introduction: Purpose, Scope, Definitions Acronyms and Plan for Create Scripts • Configure performance
Abbreviations, Reference, Overview performance test and Execute Tests
monitor
• System Architecture Description • Conduct testing:
• Test Architecture Description 1. One user
• Performance Acceptance Criteria 2. Multiple users
Analyze Results
• Performance Test Process and Tune System 3. Stress testing
• Performance Test Script Description 4. Long sequence testing
• Performance Test Schedule

© 2009, CMU-ISR 23 © 2009, CMU-ISR 24

6
 09/09/2020

Create Performance Test Scripts Sleep Time

 Approach  Definition
• Programming the scripts yourself • Also called thinking time, it is the elapsed time between user
• Applying load generator tools operations.

 Steps  Why sleep time

• Before creating performance test scripts, performance • To simulate user actions closely to reality in test scripts.
testers may develop or collect test data to reflect the real  Approach
users. If the data used in a real application is created
dynamically, the test data should be created dynamically. • Zero sleep time: the simplest method, but not accurate, as it
may cause performance deviation. It can be used to verify
• Develop scripts test script functions.
1. The scripts should be developed strictly to represent • Sleep time in reality: this method may be time consuming
the usage model and all other details related to the when executing a performance test within time schedules.
performance testing strategy. This is very important!
• Fixed sleep time: like zero sleep time, this method may also
2. It is much better if script development processes are cause performance deviation.
treated like any other software development process,
applying appropriate practices.

© 2009, CMU-ISR 25 © 2009, CMU-ISR 26

Configure Performance Monitor(1) Configure Performance Monitor(2)

 Approach 1. In order to determine process bottlenecks, you can monitor:

 % Processor Time
• Using system monitor
 % Privileged Time
• Using specific tools
 % User Time
• Configure performance test tools  % Interrupt Time
 Using System Monitor  Interrupts/sec
• The simplest way  Processor Queue Length
 Context Switches/sec
• Consumes a small amount of CPU and disk resources on the
system it is monitoring, which you should keep in mind when 2. In order to determine disk bottlenecks, you can monitor:
measuring and determining the performance of systems and  Average Disk Queue Length
applications.  Average Disk Read Queue Length
• Several commonly used performance counters  Average Disk Write Queue Length
(to be continued)  Average Disk sec/Read
 Average Disk sec/Transfer
 Disk Reads/sec
 Disk Writes/sec

© 2009, CMU-ISR 27 © 2009, CMU-ISR 28

7
 09/09/2020

Configure Performance Monitor(3) Conduct Tests(1)

3. In order to determine memory bottlenecks, you can monitor:  Including
 Page faults/sec
• Conduct baseline test
 Available Bytes
• Conduct stress test
 Page Reads /sec
 Page writes /sec Pages/sec • Conduct long sequence test

 Using Specific tools  Steps

• Tools include: memory detection tool, transaction • Conducting baseline tests with a single user over multiple
analyzer, log viewer etc. iterations to validate whether the test script is developed
properly and accurately represents the usage model.
• For example, if you want to monitor the network tie,
you may use specific network monitoring tools. • Conducting approximately 15% of the expected user load test
against the system.
 Monitors of performance test tools • Increase the user load step by step until reaching the
• In general, every load test tool integrates the function expected value according to your performance test strategy
of a configuring monitor of its own. and usage model.
(to be continued)

© 2009, CMU-ISR 29 © 2009, CMU-ISR 30

Conduct Tests(2) Analyze Results and Tune System

 Steps
• Increase the user load until reaching the limits of your  Steps
performance targets to conduct stress testing. • Analyze Results
• Conduct long sequence tests to ensure there are no memory • Tune System
leaks or other problems occur after long time run. Plan for
performance test
Create Scripts
and Execute Tests
• Validate Modification
 Notice • Report
• If performance issues are detected during any step of the
testing, tuning should begin immediately and baseline the
new result after verification.
• When conducting performance tests, the most efficient way Analyze Results
and Tune System
to increase your user load is step by step, not all at once.

© 2009, CMU-ISR 31 © 2009, CMU-ISR 32

8
 09/09/2020

Analyze Results Analyze Results

 Processor bottleneck
 When analyzing results, one of the most important tasks is to • Occurs when the demand is overshooting the supply of
identify bottlenecks which will lead to performance degrading. processor threads of the system or applications being
 It is important to know what and how to analyze results based deployed.
on the metrics you defined in your performance testing strategy • If you use system monitors, one way to determine if a
and your system architecture. processor bottleneck exists is to monitor the System/
Processor Queue Length counter. A sustained queue
 When analyzing the performance metrics of a server, you may
length along with an over-utilized processor (90 percent
take care of the following:
and above) is a strong indicator of a processor bottleneck.
• Processor bottleneck
• Methods for correcting most processor bottlenecks:
• Disk bottleneck
1. Add faster or additional processors to your system.
• Memory bottleneck
2. Analyze the software to see which specific process or
portion of the application caused this bottleneck.

© 2009, CMU-ISR 33 © 2009, CMU-ISR 34

Analyze Results Analyze Results

 Disk bottleneck  Memory bottlenecks
• Disk bottleneck problems are related to time, not disk • A memory leak would exist when Process/Private Bytes
space. and Process/Working Sets are increasing while
• The reason is that the components involved in reading Memory/Available bytes would be decreasing.
from and writing to the disk cannot keep pace with the • Memory leaks can be detected by performing stress
rest of the system. These components include: testing and long sequence testing.
1. I/O bus • There are other memory leak detecting tools you can
2. Device bus apply.
3. disk controller
4. Head stack assembly
• If you find the actual I/O per disk value is more than the
Disk I/O capability, there may be a disk bottleneck.

© 2009, CMU-ISR 35 © 2009, CMU-ISR 36

9
 09/09/2020

Performance Tools Summary

 Why use tools  Performance is hard to define. Response time and throughput are
• As you can see, performance testing is to simulate the most critical performance targets because they are exposed
thousands of millions of user actions. It is really hard directly to end users.
work by manual testing. There are many testing tools  Performance testing verifies whether the system achieves the
that can be used in performance testing, such as:
expected performance targets or not by simulating typical user
1. LoadRunner activities.
2. IBM Performance Tester  A simple performance test process has three steps: plan for the
3. Open source tools: Jmeter, Opensta. Grinder etc. performance test, create scripts and execute tests, analyze
results and tune system.
 Performance test tools can help greatly.

© 2009, CMU-ISR 37 © 2009, CMU-ISR 38

10