Introduction

This document provides an overview into the Performance Testing that a testing service can help cover, linking
the aims of the engagement to how these can be met and covering some general pre-requisites and common
factors that need to be taken into account in any proposed testing exercise.

Performance Objectives
The performance objectives (or requirements) of a system refer to the ability of that system to continue to
perform tasks to accepted levels under different load volumes and patterns. Load, in this sense, typically refers
to the number of users on a system concurrently, or currently performing an action. In the case of a batch
process, where the volume of transactions are processed sequentially or in a queuing system etc., this is
considered out of scope for a standard performance testing engagement.

It is important for these objectives to be identified as part of a project’s delivery requirements, and it is these
requirements that determine the need for a performance test exercise in the first instance. Requirements
need to be realistic and relativity to a baseline should be considered.

Performance Test Types

Depending on the performance objectives, different types of performance testing may need to be performed.
The following is a brief overview of some of these.

Load testing. Load testing is the process of executing a concurrent user load and/or a system load onto a
system, at incremented concurrency ramp-up rates, to determine its stability, performance and integrity.
During load testing system components are monitored and correlated with transaction response times. At a
point where a system becomes unstable due to erratic or dramatically extended response times the system
will have reached its benchmark. At this point an analysis should take place to identify the bottlenecks and the
tuning required to resolve them. Typically the goals of load testing are:
• Benchmark a systems performance i.e. to determine its thresholds before instability occurs;
• Identify and rectify performance issues in iterative cycles of load testing and performance tuning.

Stress testing. This type of testing is to determine or validate an application’s behaviour when it is pushed
beyond normal or peak load conditions. The goal of stress testing is to reveal application bugs that surface only
under high load conditions. These bugs can include such things as synchronization issues, race conditions, and
memory leaks. Stress testing enables you to identify application weak points, and shows how the application
behaves under extreme load conditions.

Soak Testing. Soak Testing is a type of performance test that verifies a system's stability and performance
characteristics over an extended period of time. It is typical in this type of performance test to maintain a
certain level of user concurrency for an extended period of time. This type of test can identify issues relating to
memory allocation, log file handles, and database resource utilization. Typically, issues are identified in
shorter, targeted performance tests. Soak Testing provides a measure of a system's stability over an extended
period of time.

Spike Testing. Spike testing is a subset of stress testing. A spike test is a type of performance test focused on
determining or validating the performance characteristics of the product under test when subjected to
workload models and load volumes that repeatedly increase beyond anticipated production operations for
short periods of time.
Measurements / Data Collection
Depending on the performance testing objectives, the subsequent test types and on the individual project,
different data will need to be collected and analysed to give conclusions on perceived quality of the application
under test in terms of performance. The following is a description of some of the Performance test
measurements.

Average Page Load Time

The page load time is a direct indication of the performance levels from a user experience perspective. As an
initial step, the time taken for loading a specific web page is taken and used as a baseline. Further load times
of the same page are captured when the system is under load to give a comparison against the baseline. It may
be worth capturing an average of multiple readings for the baseline, and doing the same at each load level.

To capture the page load time itself, one can use a timer, a performance tool or other specialist websites to
capture this information. So long as the capture method remains consistent, a relative measurement can be
used to compare against the baseline accordingly.

There are many factors that influence the page load times, e.g. traffic, internet speed, page size; and from a
performance testing perspective, it is important to note that relative, and not actual times, is the basis for
deducing the capability of a system.

Server Page Response Time / Time to First Byte

Similar to the page load time, the web server page response time is a good indication of the performance
levels from a web client perspective. Also referred to as the Time to First Byte (TTFB), it is, as the name
suggests, a measure of the time taken for a server to respond and ‘provide’ the first byte of data to a given
page load request. The advantage of this measurement over the page load time is that it negates the external
factors that typically influence page load times, but of course does not give a ‘true’ user experience view.

Most performance tools will provide means of capturing this metric as standard, and as per page load time (as
well as any performance related metric) a baseline will need to be captured. Further measurements of the
TTFB under different load patterns will provide a relative measurement against the captured baseline.

CPU Utilisation
The CPU utilisation of a server will directly impact the perceived performance for an end-user, especially if
utilisation is ‘high’. ‘High’ is very subjective and it is normal to have an arbitrary threshold set to allow for
measurement. This arbitrary value may be a prescribed value from a business requirement, or a company-wide
policy. Either way, a baseline should be captured when no load is applied, and subsequent measurements
under load can be used to compare against this baseline, the prescribed business requirement and any
company-wide policy.

Depending on the system under test, there may be multiple servers involved. For example, a typical web
application will probably consist of at least one web, an application and a database server. Subsequently, CPU
Utilisation will need to be considered for each of the three servers and measured accordingly.

Memory (RAM) Utilisation

Like CPU utilisation, the level of RAM utilisation of a server will directly impact the performance for an end-
user. ‘High’ usage is again very subjective and similar to CPU utilisation, an arbitrary threshold needs to be set
to allow for measurement. This arbitrary value may be a prescribed value from a business requirement, or a
company-wide policy. Either way, a baseline should be captured when no load is applied, and subsequent
measurements under load can be used to compare against this baseline, the prescribed business requirement
and any company-wide policy.

Depending on the system under test, there may be multiple servers involved. For example, a typical web
application will probably consist of at least one web, an application and a database server. Subsequently, RAM
Utilisation will need to be considered for each of the three servers and measured accordingly.
Throughput
A commonly used metric for in Performance testing is Throughput, and depending on your take on this, it can
be the number of transactions or data usage over a given time for an application. The idea of this metric is to
check that throughput remains consistent with expectations. As an example, during a soak test where the load
remains constant, you would expect the throughput to remain pretty static as well, and a simple throughput vs
time graph will highlight this fact with a ‘flatline’. Alternatively, if you are consistently increasing load, you
would expect the throughput to increase in line with this; failure to do so may highlight an issue.

Baselining
Baselining has been mentioned a few times whilst discussing measurements, but there are different
approaches to doing this and how it is approached depends on the context of the objectives of the
performance testing exercise.

1. Like-for-like or As-per-live. This is typically what is suggested as the ideal solution whereby the volume
of transactions are as per Live, and if feasible will provide the basis for accurate reflection of the live
environment prior to any additional performance testing load being applied. If the environment is a
fully scaled environment, it will also provide the mechanism to find the stress point of the system.
2. No Load. This could be useful for getting the baseline for a web page load or server request response
when there are no other activities present, which in turn can be used to get relative measurements
once load is applied. This will allow for an indication of the impact of the component of the system
under test.
3. Ad-hoc. If you have an existing test environment shared across many development functions, any
baselines recorded will vary from performance test run to performance test run. Relative Loaded
measurements can be taking for the performance test exercise, but interpretations of the results will
be the key in deriving any value from the exercise itself.

Performance Test Environment

The diagram below is a very simplistic view of the components required for a performance test:
- The actual application under test, which is treated as a black box from the view of the controller.
- A Load Controller and Loading Mechanism
- Monitoring capabilities

[Cloud] Load
Generation

Controller

Monitor Server

Application Under Test

Application Under Test
The physical Test Environment architecture of the hardware and software of the system under test will help
determine the value of the test results that will be obtained. Typically, there is a high financial cost to be able
to maintain a separate environment that is comparable to Live – this cost can be from physical hardware to
software licences as well as the network infrastructure and maintenance overheads to provide support for all
of these.

The following is a list of some of the items for consideration:

- Physical Servers and their administration
o Web Server(s)
o Application Server(s)
o Database Server(s)
- Load Balancer(s)
- Additional Network Traffic / Bandwidth
- Port Security / Firewall maintenance
- Software Licence costs / maintenance
- Deployment costs

Therefore, there is often the need to acquire a reduced scale testing environment, or utilise existing, shared
test environments. The viability of using such resource limited environments depends on the objectives of the
exercise in question.

Load Test Controller

To allow for controlled load levels, a Load Test Controller and a Load generation tool (often provided as one
tool) will need to be utilised and this tool must be executing on a server of some sort. Load generation tools
create multiple threads (/users) and perform pre-defined actions (often repeatedly) for a period of time.
Generating and maintaining this load, whichever tool is used, [ironically] is a resource-intensive task, and as
such, typically would be installed on a dedicated server.

The Performance test exercise does not need to limit itself to one Load Test Controller, and multiple can be set
up on multiple servers depending on the loading patterns desired. As an extension to this, any number of users
can add additional load to the application under test, with the added benefit of obtaining a qualitative
measure of the perceived performance levels.

Further to this, many of the Performance Testing Tools on the market will provide a mechanism to Load from a
Cloud-based location. In this scenario, the controller is still local in location, be the ‘users’ will be replicated to
be based in whatever location that the specific Performance Tool creators provide, and in other words, provide
a ‘truer’ customer experience in terms of the Network Traffic taking into account the real World Wide Web in
its’ performance measures.

The choice of which of the models above is used depends [again] on the desired objectives of the exercise; the
most important thing is to understand your loading patterns, and any external load that may be applied to the
system under test and use the measurements captured (Section 4) to understand the impact (if any).

Monitoring Server
Monitoring is a key part of any Performance engagement and depending on the setup and tools used; this can
be a separate activity or one that is controlled by the Controller. Many Performance tools provide monitoring
capabilities inclusive of the software, and even if these meet the needs of the engagement(s) in question, it is
worth considering having this setup on a dedicated server to not impact the performance of the Load Test
Controller.

Examples of existing monitoring tools include DBAs use of SQL Sentry or Operations use of Microsoft SCOM
and ProMon, but any tool that allows for monitoring any system should be considered.
Third Party Integrations
It is important to understand the makeup of the system under test and wider interactions with any third party
systems. There will be situations when the third party system is within scope and other situations where it is
out of scope. With this in mind, considerations of the third party system must be taken into account when
evaluating results; alternatively, the third party system can be ‘stubbed out’ with realistic time frames
attributed to the stub.

Test Cases / User Journeys

The Test Cases, often referred to as User Journeys, are the different functional aspects of the system under
test that are of interest. It can be as simple as selecting a specific link on a webpage or the process of
submitting an application for a product; and, from a non-user interface perspective, it can even be individual
triggered service calls to a system. The main aim is to replicate individual actions that can be triggered against
the system under test.

A test scenario can consist of a single or a number of these test cases to build up a ‘live-like’ user journey, for
example, a user may navigate a website for a period of time, then log in, then perform some purchase before
logging out again. Each of these are on its own a user journey, but combined creates a user scenario or one
overall user journey. How these are combined, or whether they need to be depends on the context of the
testing.

It is worth noting that the anticipated purpose of the test cases / user journeys is to allow for the performance
testing of the system under test, and journeys where the different layers of the system are representatively
accessed is the key aim. Creating multiple [, similar] test cases accessing the same system and altering the
inputs whilst giving different outputs (ie. functional testing) is not the aim of performance testing.

Load Patterns
The Load pattern specifies for each of the User Journeys, how many users will be executing said journey, when
they are going to execute that journey and for how long. The loading pattern, will directly impact the
performance of the system under test and, planned carefully will allow for meaningful conclusions to be drawn
on the relative performance of the system.

Depending on the purpose of the performance test, a different loading pattern would be applied:
Base-lining. For this, a simple load pattern of zero load across the board would be applied. With no users on
the system, monitoring of the servers of the system under test will provide a reference point for subsequent
tests where there is load.

Background Noise. Where you are attempting to mimic anticipated Live traffic, the system under test will
need to be applied with load that is consistent with what the current (or anticipated) live system has or will
have. Any additional load of the performance scenarios under test will then have a realistic performance
impact on a ‘Live’ system.

Load testing. Depending on the aim of the Load Testing, a different load pattern would be applied. To analyse
the specific impact a specific function, a system may start with a (base-lining) zero load environment and
subject it to a simple/small load of the function(s) in question. To understand the impact on the overall Live
system of the functions in question, an expected load of said function may be applied to the Background Noise
(suggested above). Where the aim to highlight potential coding issues, concurrency will need to be considered,
and the load and scenarios triggered will have to take this into account.

Stress testing. As this type of testing is to monitor an application’s behaviour when it is pushed extreme level
of loads, the performance test environment being used is critical, and in an ideal world needs to be like-for-like
with Live. The loading pattern of background noise needs to as accurate as possible, and throughout the
testing, the load of background noise needs to be increased continually until acceptable levels of performance
are no longer being met. Without a realistic environment and/or specific requirements to allow for activity to
be reflective of a real situation, the conclusions drawn from stress testing will carry reliability risks.
Soak Testing. As the aim of Soak Testing is to verify a system's stability over an extended period of time, a load
pattern consistent with expected levels of activity should be applied. Where time permits, subjecting the
system to an anticipated future higher level (but not stress-like) can also be informative.

Test Deliverables

Initial Engagement
Analyse Agree Scope Produce Create Tests Document Results
Requirements Test Plan and Execute and Findings

The activities of a Test Engineer for Performance Testing are very similar to that for any other form of testing –
ie. refer to the Fundamental Test Process of Plan -> Prep -> Run -> Improve -> Close – with a strong emphasis
on the requirements gathering and scope. Test input should be provided from the initial engagement and
receiving the performance requirements all the way until the delivery of the product, using any expertise
knowledge on performance testing of the tester(such as those discussed in this document thus far).

With regards to physical documentation for a performance exercise, typically a performance test plan and a
test report will be created.

Performance Test Plan

As with a standard Test Plan, the aim of this document is to state out the planned test activities of the
performance testing engagement, paying particular attention to the scope of the exercise and how that can be
achieved, including the test scenarios and load patterns that will be applied. A section is dedicated to the
potential risks of the exercise in delivering the aims taking into account the proposed actions. Attached is an
example of a recent engagement which addresses these points.

Performance Test Report

Following the execution of Performance Tests, it is important to document the results in a manner that can be
understood by any interested parties. On top of listing the results and measurements, sections are dedicated
to both Conclusions and Recommendation providing a commentary based on the analysis of the results from a
Testing perspective.

These conclusions should be presented back to the team and stakeholders, with a decision to close or
revisiting the requirements [and a further performance test exercise] as likely outcomes.

Tools
There are plenty of performance testing tools that are available on the market, both open-source and licenced
options. Considerations should be given to the following:
- Cost of licensing
- Controller Ease of use
- Concurrency options
- Loading options
- Test Case Management
- Monitoring/Reporting capabilities

Any Google search will provide you with a list, here is the result of one I did earlier today:
http://www.softwaretestinghelp.com/performance-testing-tools-load-testing-tools/