1 Performance Testing

1.a Difference between Functional & Performance

1.b Performance Testing life cycle
1.c Performance Risk
1.d Challenges of the Performance testing
1.e Benefits of Performance Testing
1.f Criteria to select a performance-testing tool

2 Performance Testing Types

2.a Types of Performance testing
2.b Quality of Service
2.c Customer Behavior Model

1 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Contents

3 Client-side emulation and Workloads

3.a Client-side emulations for Load tests
3.b Workload specification
3.c Workload specification types
3.d Load test designs

4 Load Metrics
4.a HTTP Status Codes – Types
4.b HTTP Status Codes – explained
4.c Typical Load test metrics – server based
4.d Typical Load test metrics – user based
4.e Results Evaluation
4.f Performance Monitoring Counters
4.g Provisions for Performance Test Engineer
4.h Tools for Performance Testing

2 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 1. Performance Testing

3 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 1. Overview

Performance of a system is generally measured in

terms of response time for the user activity.
To illustrate:
a customer likes to withdraw money from an Automated Teller Machine (ATM) counter of a
specific bank. Customer inserts debit/credit card and waits for the response. If the system
takes more than 5 minutes (say) to deliver the cash, customer may not appreciate the
system. In this situation, though, the system is functioning (delivering cash) according to
requirements. It fails in terms of performance (takes more than 5 minutes to complete the
transaction).

 Objectives of Performance Testing

o To enhance the responsiveness and usability of systems while
preserving quality
o To improve end-user's experience
o Ensure that system can process transactions as per defined KPI’s

4 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 1.a Difference between Functional & Performance

Functional Testing Vs Performance Testing

Functional Testing:
 to verify the correctness of the operations of the software.
 features and functions are tested
 verify that the internal operations of the product are working according to
desired specifications.

Performance Testing:
 designed to test the overall performance of the system at high load and
stress conditions
 occurs through out the software development life cycle to isolate
performance related constraints
 even at unit level, the performance of the individual module may be
assessed as the internal logic of software that is being tested.

5 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 1.b Performance Testing life cycle

Approved Performance
Requirements requirement document

R
R
Approved Performance

a
Forecast Plan
/ Design
Assurance Test Plan

Validation of scripts
Performa Develop
and data.
Modeling
nce Scripts
Assurance
R
Lifecycle
Execute Smoke, Baseline and all
Tuning planned test are executed
/Monitor and monitored.

R Indicates Review Process

6 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 How users react on poor performance

7 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 1.c Performance Risk
Risk = Probability * Impact
 Analyze most probable scenarios/loads and define impact in terms of acceptable
response times
 Identifying project-related risks and the associated mitigation strategies

 Identification of risks in initial stages of project reveals information relevant to

usability, functionality, security, and corporate image that is difficult to obtain in
other ways.

 Three major Business Risks are:

 Speed – To check whether the system responds within the expected
time Speed Risk are typically End User Concern
 Scalability – To check the number of users system can support, the
volume of data it can process and to identify the capacity of the system
(Scalability is a business concern)
 Stability - Stability encompasses reliability, uptime and recoverability.
Stability risks are addressed with high load/ Endurance/Stress tests
(Stability is a technical or maintenance concern)

8 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 1.d Challenges of the Performance testing

• Performance Testing is a complex and time consuming activity.

• The testing process should start from the requirements collection phase itself.

• Performance Testing requires simulating several hundred concurrent users. This

requires automated tools that are expensive.

• A proper environment like bandwidth, system configuration, and concurrent users

hinders in providing adequate performance results.

• Production environment cannot be simulated as it requires investments. A subdued

environment may not produce the relevant results.

• Skill set to plan and conduct Performance Testing is not adequately available. It is a
late activity, and duration available for testing is not adequate. However, testing the
system for an optimal performance is a challenge and an opportunity.

9 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 1.e Benefits of Performance Testing

 Improved quality from a user’s perspective;

 Reduced cost of change;
 Reduced system costs;
 Increased profits;
 Early identification of major application defects and architectural
issues;
 Guaranteed customer satisfaction;
 Clarity on resource utilization;
 Performance Testing also removes many myths associated with the
users and builds confidence

10 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 1.f Criteria to select a performance-testing tool

• Support for your application environment

• Intuitive test case design and execution

• Accurate emulation of real users

• Power and scalability to generate required system load

levels

• Accurate error detection and performance measurements

• Powerful root cause analysis capabilities

11 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 2. Performance Testing Types

12 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 2.a Types of Performance testing
(1) Load Testing

 A performance test which subjects the System-under-

test(SUT) to varying workloads to measure and evaluate
the performance behaviors and ability of the target-of-
test to continue to function properly under these
different workloads.

13 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 (2) Stress Testing

 A performance test wherein the load on SUT keeps increasing

and find when the system breaks.
 Stress testing shall be executed to find errors due to low
resources or competition for resources. Low memory or disk
space may reveal defects in the SUT that are not apparent
under normal conditions.
 Also called as Overload / Sustainability Testing

14 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 (3) Endurance / Soak / Stability Testing

 A performance carried out for a longer duration(s) with a

constant load in the objective to find errors due to low
resources or competition for resources. Low memory or disk
space may reveal defects in the SUT that are not apparent
under normal conditions.
 Other defects might result from competition for shared
resources like database locks or network bandwidth. Stress
testing can also be used to identify the peak workload the
target-of-test can handle

15 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 (4) Scalability Testing

 The purpose of scalability testing is to determine whether the

application scales to accommodate the expected workload
growth.
• Increase in number of online transactions
• Increase in the database size (Volume Testing)

16 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 2.b Quality of Service

 Key concept in assessing how well Web-based

applications deliver what customers expect in terms of
**availability** and **response time**

17 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 QoS (1) Availability

The Percentage of time Website is available

to customers
 ranging from Brokerage Websites to Online Travel Booking

and more importantly, required at particular

time
 Brokerage site to be available when more volatility in market
 Travel Booking required during festival times – high demand times

18 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 QoS (2) Response Time

End-end response time – time taken to see a response for

any request that is made to website

Factors that contribute

 Website ISP
 Customer ISP
 Bandwidth
 Network

19 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 QoS (2) Response Time contd…

Illustration
Response time for a request made on http://abc.com website takes 4.03
seconds

Components that contribute to this

 DNS Lookup time (Human-readable to IP address)
 Time for Browser to make initial TCP con
 Re-direction, if any to mirror site
 Time to download <first byte> of **base page**
 Time to download entire base page
 Time to download entire miscellaneous content (images, flash,
applets, if any)

20 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 2.c Customer Behavior Model

 How customers navigate in the site?

 Home -> Search Flight -> View Flight / Reserve Flight / Cancel
Flight -> Exit

 Conversion Rate from one action to other action (From home,

30% go for Viewing Reservations, 70% for Searching the flight)

 How much time people spend at every action (3-5 s) ==> called
as think time

Research shows that 8 seconds is the threshold after which customer gets
Frustrated!!!

21 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 CBMG

 Simplified Customer Behavior Model Graph (CBMG) for a

Travel site
22 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL
 3. Client-side emulation and Workloads

23 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 3.a Client-side emulations for Load tests

Browser threads:
 A thread is a path of execution through an application. In the
case of web browsers, this path consists of sending a request,
receiving the response and displaying the object. Each thread
operates independently.

• Single-threaded mode is used when a web browser executes a

Java script that communicates continually with a web server,
or more generally, when a client application other than a web
browser is employed.
• Load testing tools need to have the ability to accurately
emulate web browsers in both single and multi threaded
modes in order to apply realistic workloads to the web
applications being tested.

24 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Client-side emulations for Load tests (2)
Redirections:
 Redirection refers to the process by which a page is sent to the browser with
the sole purpose of forwarding the user to another page or server. This
technique is usually used to handle load-balancing or dynamic page
generation, or simply to transfer users from an old server to a new one.

 Browsers usually do not follow redirections more than five times, since a high
number of redirections could indicate a infinite loop.

A redirection from one page to another can occur in various way:

• Location field: The header of the HTTP response may contain a “location” or a “refresh” field that
orders the browser to redirect the user to a different page
• Refresh Header: A web page may invoke a JavaScript the leads the user to another page
• Refresh Tag: A web page may contain an HTML tag that indicates a redirection
• Load testing tools must be able to emulate redirection as it is frequently used technique in web
applications.

25 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Client-side emulations for Load tests (3)

Document Cache:
 Document cache is a location on the user’s computer where a web
browser temporarily stores downloaded web pages. This is a way to
speed up page downloading times and to reduce traffic on the network

• Using the cache saves time. However, documents stored persistently

will become outdated if the original documents on the web sever
changes. To guard against this, web browser checks whether a
document in the cache is an up-to-date copy of the original on the
server. This is a user specified option, and the most popular setting
for web browser today is set to check once per session whether a
document on the server has changed since the user last viewed it.

• A load testing tool must emulate web browser’s document cache.

Browsers usually do not follow redirections more than five times,
since a high number of redirections could indicate a infinite loop.

26 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Client-side emulations for Load tests (4)

Cookie database emulation

A cookie is simply a piece of text stored on a client computer by a web
application.

• Persistent cookies. Also called a permanent cookie, or a stored

cookie, a cookie that is stored on a user’s hard drive until it expires
(persistent cookies are set with expiration dates) or until the user
deletes the cookie. Persistent cookies are used to collect identifying
information about the user, such as Web surfing behavior or user
preferences for a specific Web site
• Non-persistent cookies (also known as session cookies): Used by
Commerce Server to track authenticated users who visit your site?
When the session ends, the cookies are deleted. Non-persistent
cookies store MSCSAuth tickets.

• Both persistent cookies and non-persistent cookies can be

encrypted. The non-persistent cookies contain the fields like User ID
, Time of last login, Time Window

27 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 3.b Workload specification

 The workload specification requires some analysis of what

we hope to implement. To the extent that the new system
has roots in current systems, the specification will have
some real numbers. If the system is completely new, some
effort must go into database and application design before
any work specification can be performed

28 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 3.c Workload specification types

• A steady-state workload is the simplest workload module used in load

testing. Here a constant number of simulated users are run against the
web application for the entire duration of the test. The use of a steady-
state workload provides exact response time and throughput
measurements for a given number of users. This type of workload is
often used for stress and stability tests.

• An increasing workload helps testers to find the limit of a web

application’s work capacity, and is crucial for establishing fail-safe
conditions. Load tests using this workload model run until the web
application response times exceed the permitted limit, or, in a worst case,
the application crashes. Increasing workload tests identify the maximum
number of simultaneous users the web application can support.

• A scenario-based workload is the most complex workload model. The

object is to achieve real world conditions by varying the number of
simulated users run during a test, depending on the time of the day. This
workload type is used for stability tests that last a long time – from a few
days up to weeks.

29 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 3.d Load test design (1)

• Concurrent Users: Although your site may be handling x number of

users per day, only a small percentage of these users would be hitting
your site at the same time.
Ex: if you have 3000 unique users hitting your site on one day, all
3000 are not going to be using the site between 11.01 and 11.05 am.

• Estimating Target Load Levels: Once you have identified the current
load levels, the next step is to understand as accurately and as
objectively as possible the nature of the load that must be generated
during the testing.

• how the overall amount of traffic to your Web site is expected to grow
• the peak load level which might occur within the overall traffic
• how quickly the number of users might ramp up to that peak load
level
• how long that peak load level is expected to last

30 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Load test design (2)

• Estimating Test Duration: The duration of the peak is also very

important-a Web site that may deal very well with a peak level for five
or ten minutes may crumble if that same load level is sustained longer
than that. You should use the length of the average user session as a
base for determining the load test duration.

• Ramp-up rate: As mentioned earlier, although your site may be

handling x number of users per day, only a small percentage of these
users would be hitting your site at the same time.

 Therefore, when preparing your load test scenario, you should

take into account the fact that users will hit the website at different
times, and that during your peak hour the number of concurrent
users will likely gradually build up to reach the peak number of
users, befor tailing off as the peak hour comes to a close.

31 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Load test design (3)

3 Basic Elements
 All performance measurement experiments require three basic elements:
• the system under test (SUT),
• load generators (driver) that apply a workload to the SUT, and
• instrumentation to collect performance data.
 Depending on the experiment, load generation could be implemented by
special-purpose hardware, software running on separate system, or even
a
 Process running on the SUT. Care should be taken to ensure the
performance bottleneck lies in the SUT, not the load generators!

• Test Servers
Customers will not allow load tests to be performed on
productions systems since the high load levels would interfere
with production activities. Also, production activities may skew
the load test results.

32 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Load test design (4)

Databases
 The database(s) in the test lab must be pre-loaded with either a
copy of production data or dummy data that is similar in size
and content to the production data. Databases that are too
small will tend to give erroneously fast results and can obscure
table scan and index problems

Don't Use Your Dev Environment for Load Testing

• First, even if the hardware is the same as production, the
chance that you control the software installed and the
configurations is small.
• Second, load test scripts require a stable environment, as
you'll want to minimize the amount of time spent rerecording
scripts between tests. Some products allow you to ignore
HTML or code changes to a Web page

33 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 4. Load Metrics

34 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 4.a HTTP Status Codes - Types

• 1xx Informational
This class of status code indicates a provisional response, consisting only of the Status-Line and optional headers,
and is terminated by an empty line. Since HTTP/1.0 did not define any 1xx status codes, servers must not send a 1xx
response to an HTTP/1.0 client except under experimental conditions.

• 2xx Success
This class of status codes indicates the action requested by the client was received, understood, accepted and
processed successfully.

• 3xx Redirection
This class of status code indicates that further action needs to be taken by the user agent in order to fulfil the request.
The action required may be carried out by the user agent without interaction with the user if and only if the method
used in the second request is GET or HEAD. A user agent should not automatically redirect a request more than five
times, since such redirections usually indicate an infinite loop.

• 4xx Client Error

The 4xx class of status code is intended for cases in which the client seems to have erred. Except when responding
to a HEAD request, the server should include an entity containing an explanation of the error situation, and whether it
is a temporary or permanent condition. These status codes are applicable to any request method. User agents should
display any included entity to the user.

• 5xx Server Error

Response status codes beginning with the digit "5" indicate cases in which the server is aware that it has
encountered an error or is otherwise incapable of performing the request.

35 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 4.b HTTP Status Codes - explained

36 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 HTTP Status Codes – explained (contd…)

37 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 HTTP Status Codes – explained (contd…)

38 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 HTTP Status Codes – explained (contd…)

39 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 4.c Typical Load test metrics – server based

• Number of users and/or sessions

• Average session time

• Number of page views

• Average page views per session

• Peak period (e.g., 75% of traffic is from 11:00 AM-4:00 PM)

• Number of hits

• Average page size

• Most requested pages

• Average time spend on page

• New users vs. returning users

• Frequency of visits (e.g., 75% of users made one visit)

• Demographics

• Client information such as browser, browser version, Java script support, Java script enable/disable, and so on.

40 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 4.d Typical Load test metrics – user based

• Number of users
• Session length
• User activities and frequency of activities per session
• Think/Read/Data-input time
• Percentage by functional group
• Percentage by human speed
• Percentage by human patience (cancellation rates)
• Percentage by domain expertise (speed)
• Percentage by familiarity (speed)
• Percentage by demographics (arrival rates)

41 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 4.e Results Evaluation

 Often the first indication that something is wrong is the end user
response times start to climb. Knowing which pages are failing
will help you narrow down where the problem is.

• Whichever load test tool you use, it will need to produce

reports that will highlight the following:
• Page response time by load level
• Completed and abandoned session by load level
• Page views and page hits by load level
• HTTP and network errors by load level
• Concurrent user by minute
• Missing links report, if applicable
• Full detailed report which includes response time by page and
by transaction, lost sales opportunities, analysis and
recommendations

42 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 4.f Performance Monitoring Counters

• Monitoring and Tuning Hardware Performance

• Monitoring Memory
• Monitoring Processor Capacity
• Monitoring Multiprocessor Systems
• Monitoring Network Capacity and Bandwidth
• Monitoring and Optimizing the Hard Disk
• Monitoring and Tuning Web Application Performance
• ASP Counters
• WWW Service Counters
• Tuning Web Applications

43 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Performance Monitoring Counters (2)

 If a Web application runs slowly or does not perform properly, the

performance of your server can be affected. This topic describes how
to monitor and tune applications to improve your server's performance
• ASP Counters
• Active Server Pages\ Requests/Sec
• Active Server Pages\ Requests Executing
• Active Server Pages\ Request Wait Time
• Active Server Pages\ Request Execution Time
• Active Server Pages\ Requests Queued
• WWW Service Counters
• WWW service\ CGI Requests/sec
• WWW service\ ISAPI Extension Requests/Sec
• WWW service\ Get Requests/sec
• WWW service\ Post Requests/Sec

44 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 4.g Provisions for Performance Test Engineer

(1) Knowledge
 Project Management Plan (Performance Agreements, Service
Level Agreements)
 Infrastructure Topology (Server, details, versions of SW,
communication frequency)
 Database Schemas (and access rights!)
 Installation and Configuration Notes
 Requirements, Design documents and developer notes
 Sequence of User action (derived from use cases)
 Approval from Functional Testing

45 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Provisions ( Contd.. )

(2) Hardware
• Load Testing Servers
• Load Testing Clients (Hosts)
• Load Balancer (Optional)
• External Storage (Discs)

46 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Provisions ( Contd.. )

3) Data and Access rights

• Accounts and password for administrators
• Test data from the Production (a representative data at least!)
• Security certificates (if any)
• Server images (Ghost Backups etc)

47 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Provisions ( Contd.. )

(4) Software
• Load testing software
• Installation Package storage and Locations
• Access to CVS, VSS for the source code
• Special Programming software for the Load testing

48 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 4.h Tools for Performance Testing

49 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL

 Thank You.

50 © 2013 WIPRO LTD | WWW.WIPRO.COM | CONFIDENTIAL