Why performance testing?

 Do want to use an Movie Booking web-site to book a ticket

and wait for 15 minutes to know that there are no seats available?
Why performance testing?

How many of us like to click on a button and see the result –

“Error : Page not found”?
 Bad performance is bad for business
 Lack of proper performance testing results in loss of revenue,
loss of credibility, loss of customers
Why Applications become slow?

Multiple users using the application at the same point of time

Load on the application increases

Applications become slow

What is Performance Testing Check
Performance of
application

Application Under
Creating the test
load on the
application (Lets assume Gmail
application)
What are Vusers Can we have
Image real users
there were
1000 users

These
were real
users

Virtual Users
OR
Vusers
Types of users

 User Base (100000)  registered

 Application Users / Online users (100) (90 +10)
 Concurrent users (10)

Key Facts
 Number of concurrent users is not same as the number of
application users
Performance testing tools and protocols

8
 What is
Architecture?
What is architecture?
 An architecture is a

 combination of software and system components and connections

10
Importance of architecture

 Helps in
 Designing smaller and more manageable pieces of application
 Understanding dependencies between components
 Enables interaction
 Between different parties (stakeholders) who intend to develop the computer-
based system
 Between different systems and system components

11
What is a server

Software (Computer Program) or Hardware which

 Understands your request
 Process your request
 Send the response back
Application Server (APP Server)

An application server is a server program

in a computer in a distributed network that
provides the business logic
for an application program
 Web Server

An web server is full implementation of HTTP Protocol

A Browser is partial implementation of Http Protocol

 Application layers
Presentation layer
Accepts user inputs, displays data and any results computation on the user
interface

Application processing layer

Provides application specific functionality
e.g., in a banking system, banking functions such as open account, close
account, view account details etc.

Data management layer

It manages the system database

15
Client/Server - 2 Tier architecture

 Improves multi-user updating.

 These days again this model is picking up because of mobile apps

There are 2 types of Models

16
3-Tier Architecture
Presentation
Layer
Business logic tier between presentation logic and data tier
Business Logic
Layer

Data Access
Layer

Data

PC Database
(Thin Client) Server
Application
Server

17
Three tier architecture

 Advantages
 Separate processor might be present for each application layer

 This architecture is simpler to

 manage than fat-client model and also gives better performance
than a thin-client model

 Can be scaled easily

 more servers can be added as demands increase

18
N-Tier (Multi-Tier Architecture)
Business Logic
Layer

Presentation Layer Data Access Layer

Enterprise Data

Database
PC (Thin Client) Web Server Application Server
Server
 Specialized web servers were introduced which can generate presentation content
which is transferred to the browser on the client tier, which takes care of rendering
the user interfaces

19
What is performance testing?

 PT is testing an application for speed, stability and scalability in “Production

like Environment” under virtual user load to meet Non-Functional
requirements. (NFR’s)
 Speed: How fast the application responds to your request. In other words,
Response time should be less
 Stability: Performance should not degrade even after using the
Application for very long time.
 Scalability: Application’s ability to handle increasing user without
performance degradation
Purpose of Performance Testing

 To identify Performance Bottlenecks

 Functionality of the application should not be changed under real
world conditions

Key Notes:

Performance Tester are required to identify Performance Bottlenecks but not Defects
(Bugs)
Important Terms in PT

 PT: Performance testing

 NFR : Non-Functional testing.
 Response Time - Time taken by the system to respond to a specific
transaction request
 Think Time –Time taken for selecting new transaction after receiving
the response for previous transaction.
 Throughput - Transactions per second, Bytes per second
90 percentile response time

 When we execute a Performance Test, same transaction will be executed

multiple times.
 Since, the same transaction is executed multiple times, multiple response
times are captured
 Refer
Client willtobe
inserted spreadsheet
provided formin
with Max, 90% ,RTAvg,
calculations.
and 90% response times.
 LoadRunner automatically calculate the 90% response times
Performance Testing VS Performance
Engineering
Performance testing is a subset of performance engineering
Performance Testing Performance Engineering

Performance Testing usually deals PE not only deals with identifying the
with identifying issues in the issues but also eliminating the same.
Application Under Test
Types of Performance Testing

Load Testing

 This is a mandatory test which will be done prior to any test.

 This test usually simulate current user load on the AUT.
 Usually Peak load for the application is considered for this test.
 This test ensures that the AUT is stable and handle expected peak
load once the application is deployed.
Scenario Load Test – MERCURY SAMPLE PROJECT
Name
Scenario Load Test – Duration 1 hour.
Type
Scenario To simulate the peak Load and to monitor the performance of
Objective the MERCURY SAMPLE PROJECT online system

Steps The online load will be maintained at steady state for 1 hour
with only critical transactions
Entry All the Monitors are in place
Criteria Test Data is set-up
Shakedown completed successfully

Exit Criteria Response times meet the SLA

Test completion report is agreed upon by stakeholders
Stress Testing
 Stress Test is conducted by increasing the user load gradually until the application
breaks.
 Objective of this test is to obtain the breaking point or saturation point.
 Usually once the breaking point is attained there is a noticeable increase in errors
and also there is a big increase in response time.
 This test is performed to check if application can accept spikes.

Spike Testing
 Spike Testing is considered to be subset of Stress Testing.
 It is done by increasing the user load beyond anticipated load for short periods of
time.
Scenario Stress Test
Name
Scenario Type Stress Test – Duration N/A

Scenario To objective is verify that the application can handle the projected growth
Objective and to discover the breaking point of MERCURY SAMPLE PROJECT
Online

Steps 1. Ramp up to 150% of peak load volume and thereafter continuously

increase load until breaking point is found
Break point – When the error rate is more than 20% or response times
are really high

Entry Criteria All the Monitors are in place

Test Data is set-up
Peak Load test completed successfully

Exit Criteria Test completion report is agreed upon by stakeholders

Endurance Testing

 It is also called as Soak Test.

 This test is performed for long periods of time (8 hrs, 16 hrs, 1 day, 3
day) with expected user load.
 Purpose of this test is to identify performance bottlenecks like
Memory Leaks, connection leaks etc.
Scenario Soak Test – MERCURY SAMPLE PROJECT
Name

Scenario Type Endurance – Duration 8 hour.

Scenario To discover memory issues and bottlenecks that might occur under daily
Objective usage of the application

Steps Steady state Test is maintained for 8 hour with half the Peak Load.

Entry Criteria All the Monitors are in place

Test Data is set-up
Peak Load test completed successfully

Exit Criteria Test completion report is agreed upon by stakeholders

Scalability Testing

 Test is performed using the user load considering the growth of the
application under test down the years.
 This test is performed to check the capability to scale up or scale out
in terms of User Load.
Volume Testing

 It is a load test except that huge data populated in the database.

 The data populated in the DB is expected down the years.
 Purpose of this test is to check if there is any change in the response
time with increase DB volumes.
Performance Testing Lifecycle Stages
Non-functional Requirement Gathering and
Analysis

Test Requirement Test Strategy and

Planning

Test Plan Test Design and

Development

Iterations

Test Scripts Test Execution

Bottleneck
Analysis

Test Results Test Result

Analysis

Final Test Reports

Test Reports
Workload Load Modeling
Work load

 Includes the number of virtual users and the volume of transactions per user

 Estimation for number of users required for a given volume or vice-versa

 Collect information on
 various combination of business transactions

 For each business transaction

 information collected is probable number of users performing same transactions simultaneously
 during normal operation
 peak time operations.

 Refer to below work load modeling example

Little’s LAW

N = X * (Rt)

 N = No. of Concurrent Users

 X = Throughput (TPS)
 Rt = Response time (Secs)

N = X * (Rt + Zt) (For system with think time)

 Zt = Think Time
 Industry standard for Zt is 10 secs

Rt + Zt = Script Execution Time

Little’s LAW

 For Work load Modeling, Pacing is required to be calculated. So the formula

would be

N = X * (Rt + Zt + Pacing)

Rt + Zt = SET
NFR Gathering
Infrastructure Details

 As part of NFR gathering, PT team is supposed to capture the configuration details of

both production environment and performance testing environment
 Data collected
 Operating System of each server– OS version
 Hardware configuration of each server – Number of CPU’s, speed, memory etc..
 Details of server clusters.
 Support software on each server – Web server, application server, database

 Example
 Apache Tomcat Server 8
 1.6GHz processors with 32MB Cache, 16GB Memory, 100GB Hard Disks, HP-UX and
Java ES pre-installed
Transaction Details

 As part of NFR gathering, PT team is supposed to captures list of

performance critical transactions.

 What are Performance Critical Transactions:

 Transactions which are executed Frequently.
 Transactions which are critical for Business.
 Transactions that are suspected to have high resource requirements
Transaction Details (Contd..)

Example
For a banking application, following transactions are critical
 Account Summary Details
 Checking Transaction History
 Balance Transfer
 Login
 Logout

Following transactions are not critical

 Change Password
 Change Username
 Change Theme
 Order Checkbook
 Open a new account
Scalability Related data

 Few years down the line, there is chance that user load on the application
might increase because of the business growth.

 Because of this increased load, the performance of the application might get
affected.

 So, performance Testing team is supposed to capture the future user volume
growth.

Example
 There is a 100% increase in user load annually.
Workload Related data

 Application usage patterns and volumes for each performance Critical Transaction is
captured as part of this model.
 This data is identified by
 Interviewing Clients
 Analyzing existing logs
 Example
Data related to response times and other
metrics

 As part of NFR Collection, Performance Testing Team is suppose to capture

Response Time (RT) related data.

 After the Performance test execution, the test results are compared with the
SLA to determine if the application meets performance expectations
 Examples -
 When application is subjected to 1000 concurrent user load, “Login”
transaction should not take more then 2 seconds to complete
 When application is subjected to 1000 concurrent user load, CPU
utilization of the WebApp Server should never cross 60%.
DB Data

 Database volumes affect round times for operations that access the database

 To simulate realistic load on the application the DB tables should be loaded

with sufficient number of records

 The data retention model is used to capture the key DB tables and estimated
number of records in each of these tables

 All this data will help in simulating realistic database volumes during test
execution

45
LoadRunner
 Components of LR
• Virtual User Generator

• Controller

• Analyzer

• Load Generator
 Virtual User Generator (VuGen)
• VuGen is the main component of LR which is used to
create scripts to simulate user actions on the AUT.

• Scripts are created in C language of JavaScript

Language.

• The scripts are usually generated by recording the

events between client and server.

• VuGen component provides the Performance Tester

with the replay option.

• Note: Scripts are executed using VuGen for not

applying the load. It is done only for debugging
purpose.
 Controller
Controller is configured to

• Number of Vusers

• Number of scripts to be executed.

• Number of LG’s

• Design Ramp up and ramp down

• Define the group name

 Load Generator
• Load Generators are systems that will create Virtual
users.

• Depending on the hardware configuration of the LG,

number of Vusers generated by LG changes.

• Consider a case, where LG memory is 500 MB. Since

each Vusers is required to have 2.3 MB of memory, this
LG can support 220 Vusers (Approx)

• In the above case, we might require 5 LG’s to get a

Vusers load of 1000 users
 Load Generator
• Controller component of LR has the ability to control
multiple load generators

• Also remember that your Vusers scripts will be

downloaded to LG machines during the scenario
execution. So, results needs to be collated by
controller from all the LG’s after the execution.

• User needs to have the appropriate license to access

Load Generator and Controller
 Analyzer

• This component will provide the Test Results.

• On further analysis of this test results, one can identify

Performance Bottlenecks of the Application Under Test.

• Test report can also be prepared using this component.

LoadRunner Architecture
&
Installation
LR Architecture
Where to install LoadRunner components?
• In a real time set-up..

 Vugen is installed on the performance tester’s

machine / desktop.

 Controller & LG’s are installed on different

windows machines / servers.

 Note: Each LG requires a separate machine

 Analysis is installed on the performance tester’s

machine / desktop.

• For our session..

 All the components are installed on your desktop

Identify hardware and software needed for installation
 Installation of LR
• LR can be downloaded from following link

http://www8.hp.com/us/en/software-
solutions/loadrunner-load-testing/try-now.html

• Select Free trail

 Installation of LR

• Complete the sign up process

• Download starts
 Installation of LR
• Browse to DVD folder and Double click on set-up file to
start the installation
 Installation of LR
• Select the required installation option as “LoadRunner
Full Setup” and finish the installation.

 Note: Installation process might require latest

version of Java. So, when prompted, go ahead
and install

• Once the installation is complete, all the components

are installed on your desktop.
 Installation of LR

• For additional help, refer to the “HP LoadRunner

Installation Guide” in the DVD folder
 LR Basic Flow

Create Vuser Create Run Scenarios

Scripts using Scenarios using using
VuGen Controller controller
component Component component

Tune system Analyze

based on Results using
Analysis “Analysis”Com
ponents
Virtual User Generator
Functioning of Virtual User Generator

• VuGen creates scripts by recording the activity

between the client and server when a user performs
actions on the application.
Functioning of Virtual User Generator
Virtual User

• It is not the real users. It is a tool generated user. In

other words, human users are replaced by virtual
users.

Virtual User Script

• The actions performed by the human users are

recorded in the form of a script. The scripts, when
replayed emulate the real user performing the business
actions.
Functioning of Virtual User Generator

Each Virtual user script will always have 3 default

transactions.

• Vuser_init:

• Actions:

• Vuser_end:
Steps to Create a Script in VUGEN
 Steps to Create a Script

• Understand the AUT

• Record various transactions using VUGEN

• Enhance the script

• Play back to make sure there are no issues with the

script
Protocol Advisor
 Protocol Advisor

• Select Record > Protocol Advisor > Analyze Application.

• Try to walk through a variety of business processes to

make sure that your results are comprehensive. Click
Finish Analyzing to end the analysis and display the
results.

• As per the results, select the protocol and create a

new Vuser Script
71
Recording Options
 Recording Options
• General

 Recording

 Script

 Protocol

 Code Generation

• Correlations

 Configuration
Recording Options

 Automatically close any application process when VuGen stops recording

 When End Transaction is inserted during recording time, automatically insert
lr_think_time in the script
 Generate recorded event logs in event viewer of the system
 During recording, if user is pausing for more that 2 seconds, then auto generate
lr_think_time(<actual think time duration>).
 If actual pause during recording is less than 2 seconds, ignore think time
 Maximum number of lines of code that can be stored in one action.c file
74
Recording Options

• General – Protocol option shows all the protocols chosen for the recording of the
script

75
 Recording Options

• General – Recording has two options

• HTML Based Script
• Generates steps for each user action.
• The script size is small
• Preferable when we are interested in measuring entire page load time
• URL Based Script
• Records all requests and resources from the sever
• Script size is larger
• Preferred when we want to measure individual page components (resources, non resources)
load times 76
 Recording Options

• Network – Port Mapping is used to configure the port setting for capturing user
actions
• Three options available
• Socket level data
• WinInet level data
• Socket level and WinInet level data

77
Recording script using Vugen

Emulate the user actions

78
Recording script using Vugen

Script created
after
recording
ends

79
HTML Based Script Vs URL Based
Script
HTML Based Script Vs URL Based Script

HTML Based Script URL Based Script

The script size is small Script size is larger

Preferred when we want to

Preferable when we are
measure individual page
interested in measuring
components (resources, non
entire page load time
resources) load times
Transactions
 Transactions

• Transactions measures the system performance

resulting from one or more user actions

• Only means of measuring application response time

and transaction pass/fail count
Inserting transactions

34
Think Time
 Think Time

• The delay caused by the user between two subsequent

requests is called think time

• Inserted automatically by VuGen during recording

depending upon the Recording Options
 Think Time

• Inserted automatically by VuGen

during recording

• Think time settings can be

customized as needed using run-time
settings

87
Comments
 Comments

• A comment can be added to provide additional

information on the script

• For inserting comments use

 //

 /* ……..*/
Check Points
 Check Points
• Two types of Check Points are available with VuGen

 Text CP
 Image CP

• It verifies if a particular text or image is present on

the web page.

• By adding the Check Point, one can confirm if a

particular transaction has passed or failed.
Text checkpoints
 Text checkpoints

• Check Point is placed on a particular text available on

the Web Page.

• Web_reg_find() function is used for inserting the text

check point
 To add text check point for text, use web_reg_find() function
 This function can be added during recording, typed directly in the script or
can be added using steps toolbox.

94
Image Check Point
 Image Check Point

• Check Point is placed by using a ALT or SRC attribute of

a particular image available on the Web Page.

• web_image_check() function is used for inserting the

text check point
 To add verification check point for image, use web_image_check()
function
 This function can be typed directly in the script or can be added using
steps toolbox.
Rendezvous Point
 Rendezvous Point
• Rendezvous point is used to synchronize all the Vusers
to perform a particular transaction at the same point
of time.

• When a Vuser reaches the rendezvous point, the

Controller holds execution of the Vuser until all the
other participating Vusers reach the point.

• When all the participating vusers reach the rendezvous

point, these Vusers will be released all at once to
create a spike.
 Double click on
Steps Tool Bar and
give the rendezvous
name and click ok
100
Rendezvous
point should be
inserted before
start of a
transaction

101
Parameterization
 Parameterization

• It is a process by which a hard coded value is replaced

with a parameter in the script.

• This option helps the script to execute with multiple

values, thereby, simulating a real time scenario
 Parameterization

Steps Involved for Parametrization

• Create a Parameter

• Assign the values to the Parameter

• Replace the Hard-coded values in the script with these

Parameters
 Select the value of the field on which you want to apply parameterization and
right click and select “Replace with a parameter” as shown below.

 “Select or Create parameter” window will be opened.

105
 Give the parameter name, by default Parameter type and Original value will be
populated. You can change the Parameter type as required.
 Click on “Properties” button.
 “Parameter Properties” window will be opened.

 Click on “Create Table” and enter the data by adding a new column if needed.
106
 Close the Window
 You can create different tables for different parameters or you can add all
parameters in same table with different columns and you can make the other
parameters use the same table by selecting the same “File path”.

107
Correlation
 Correlation
• There are some dynamic values in the script which
changes from iteration to iteration.

• Since these values are dynamic in nature, their value

changes with each execution.

• So, there is a need for these dynamic values to be

captured from server response and pass it subsequently
to any part of script.

• Process of capturing these values using

web_Reg_Save_param_Ex function and pass it
wherever required is called correlation.
Automatic Correlation
 Automatic Correlation

• Scan the script for identifying the values to be

correlated

• Vugen will provide the recommendations.

• Click on the value to be correlated and click on

“Apply”
Manual Correlation
 Manual Correlation
• Identify the values to be correlated : Record the
same script twice and compare both the recordings to
identify the values

• Search for that value in the server response: Once,

the value is identified, check for that value in the
server response to identify left boundary and right
boundary

• Correlate: Correlate that value using

web_Reg_save_param_ex using LB and RB

• Replace: Replace the dynamic value present in the

script with the LR parameter of correlation function.
Defining Rules for Correlation
 Defining Rules for Correlation

• Rules can defined in the recording options for the

automatic correlation.

• This feature is useful, if your project contains lot of

scripts and there is necessity to correlate same values
multiple time.
Run Time Settings
 Run Time Settings
• Run logic

• Pacing

• Log

• Think Time

• Speed Emulation

• Browser Emulation

• Preferences
Run Logic
Run logic : Allows manipulation of the run logic by setting different iteration values

118
Pacing
Pacing – Provides different options regarding when the next iteration
starts

119
Log
Provides various logging options

120
Think Time
Provides different options of how the think times are replayed during
run-time.

121
Network
Provides Network speed simulation options

122
Browser
Simulates browser during runtime

123
Internet Protocol
Preferences: Set preference for verification check point and other advanced options

124
Internet Protocol
Advanced Options: Set advanced options like step download timeout period, Keep alive
connections by

125
Other Miscellaneous options
 Logs
• Replay

• Recording

• Generation
 Regenerating the Script

• Using this option, Script can be regenerated without

doing the recording again.
Controller
 Controller
• Execution of test scripts happens using the LR
component called Controller.

• Although it is possible to execute the scripts using

VuGen, Load cannot be applied on the AUT. Execution
using VuGen is done only for debugging purpose.

• Actual Load on the AUT can be applied using controller

as test can be executed using multiple scripts with
multiple Vusers.

• Test designed with multiple scripts using multiple

Vusers is called Test Scenario.
Internal Working of the Controller
Controller

Vusers Vusers

Server(s
)

LG1 LG2
Test Scenario Creation
 Test Scenario

• Test designed with multiple scripts using multiple

Vusers is called Test Scenario.

• Each Test Scenario can contain one or more groups.

• A scenario contains all the information about groups of

Vusers which are required to emulate human users
during test

• Steps involved for Test Scenario creation

 Create Test scenario

 Configure Run-time Settings

 Test Scenario Creation
• Selecting the type of Test Scenario

• Adding the test scripts to the test scenario

• Selecting the required number of LG’s

• Configure the schedule (Defining the Ramp-up, Ramp-

Down and Steady state)

• Selecting Scenario Schedule

• Setting the SLA (Optional Step)

Test Scenario Type
Selecting the Type of Test Scenario
• Manual Scenario

• Goal-Oriented Scenario
Adding Scripts to the Scenario
Adding Scripts to the Scenario

Select scripts to added

Click on Details to change the Group Name
Configure Load Generators
 Configure Load Generators
• Scenario  Load Generator

• Add new LG’s

 Add LG to the Group
• Add the LG to the group.
Defining the Scenario Schedule
 Scenario Schedule
• Schedule By Scenario

• Schedule By Group

• Basic Schedule

• Real world schedule

Configure the Schedule
 Configure the Schedule
Defining the Ramp-up, Ramp-Down and Steady state
Set-up the SLA’s
 Set-up the SLA’s
• The SLA in LoadRunner or Controller, gives an
opportunity to you to test your application against an
SLA.
Configuring Run Time Settings
 Run Time Settings
• Run logic

• Pacing

• Log

• Think Time

• Speed Emulation

• Browser Emulation
Test Execution
 Test Execution

• Set-up the Result Directory for storing the results.

• Execute the Test

 Test Execution
• Steps during Test Scenario execution

 Required number of Vusers can be generated to

create the load on AUT.

 Each Vuser can be controlled using options:

Initialize, Run, Stop.

 Controller can display status messages for all

Vusers under execution

 Collects and presents live data for all defined

metrics of server resources

• Steps after Test Scenario execution

 Collects and organizes performance data

Analyzer
 Analysis
• Analyzer

 Generates graphs

 Includes utilities for report generation (HTML

format and Word format)

 Helps in identification of bottlenecks

 Analysis of Test Results
• The Analysis session contains summary Reports and
basic graphs like Running Vusers, Throughput,
Transaction response time, Hits per sec..

• The Summary Report contains SLA, running Vusers etc…

• Summary report

 90 Percentile

 SLA

 Running Vusers

 Hits per second

 Throughput

 Average Transaction Response Time

 OS level(CPU & Memory utilization)

Graphs - Options
 Graphs - Options
• Graphs in the Analysis session can be

 Enhanced for its granularity.

 It can be drilled down

 Merged

 Overlay

 Tile

 Correlate
Test Report Generation
 Test Report Generation
• Once Analysis is done, a Test report can be generated
in any one of the formats.