Systems Integration - US India

Performance Testing Process

Authors Email ids

Prathyusha Kodukulla [email protected]
Madhuri Godi [email protected]
Rough Draft Created Date 10/01/2017

Final Draft Created Date 25/01/2017

CONTENTS
1. Introduction
2. Purpose of performance testing
3. Current Challenges
4. Methodology
5. Performance Tools
6. Summary
7. References
8. Contacts

2
1. Introduction:
Long wait times, delays, errors and service interruptions tests the patience of
website users. A quick and responsive website that can keep visitors happy and
engaged is the dire need of today’s online entrepreneurs who face stiff
competition from their counterparts. Since, every fraction of a second matters
when it comes to web performance, it becomes inevitable for businesses to
address the performance challenges of their website in the quickest possible
time.
An adept performance testing strategy that identifies the bottlenecks and
different areas of performance testing including calculation of application
performance, investigating best methods for performance upholds the
performance and scalability of a site and should thus be adopted.
Performance testing is an extremely important element before any software
product is launched in the market. A well-tested software product ensures
customers satisfaction, retention and loyalty, and eventually helps create strong
brand equity.
This whitepaper delves into volume and load testing, inspecting different
sections of a successful load and performance testing process, finding the
attributes of a load and performance testing tool that is trustable.
Apart from this, this paper addresses the challenges created as a result of
technological up-gradation and advent of new technologies that are abstract and
complex.

2. Purpose of Performance Testing:

Certain performance bottlenecks are identified in the testing of web applications
for most of the software development projects. Performance is a crucial success
factor. Project managers should plan in advance for performance in the initial
stages of design and development, as performance will be on the agenda during
the first production run. Most web applications have better performance as long
as they cater to few users. But what would happen if large number of users work
with the application simultaneously? At this point Load and Performance Testing
(L&P) becomes a major factor. By simulating a large number of web users, load
and network traffic can be generated on the system and can be driven to their
performance limit.
There are several reasons to perform L&P tests. Since, there is no 100% certainty
whether or not multi-user software makes all its functions and methods available

3
 to all its users, (and is therefore multi-user capable), it is possible to find high
percentage of faults with performance testing. These faults could not have been
identified by individual manual testing, independent of whether unit tests or
system tests were performed. Performance testing allows a realistic check of the
configuration of interfaces to other systems or subsystems and computers. In
addition, performance tests can also detect bottlenecks in the host systems.
Performance tests essentially concern the following topics:
1. Consumption of resources
2. Response time
3. Large load / number of users

People especially from operations are mainly interested in the system resource
consumption and are interested in knowing what to expect with the introduction
of an application to the production system.

3. Current Challenges:
4
Prior to the era when development frameworks were not introduced, application
development used to be done using machine-oriented programming languages. This
way programmers developed a more precise view of the target platform, on which
the application code was deployed. For this reason, they could identify certain
performance related risks more clearly than in the current time. New methodologies
and technologies promote the use of distributed applications, which offers high
flexibility for interaction between components and composition of services to
construct application systems.
The advanced world of Web and Mobile applications has changed the way
companies promote, market, and sell their products besides automating key
business processes, making it faster and easier to enter orders, processing
payments, or even tracking data.

The large volumes of information gathered by Web applications can help companies
define more precise marketing strategies, target specific customers, while offer
better and more personalized services to their clients. But more the companies rely
on Web applications for running their businesses and generating revenues, greater is
the risk of failure due to the complexity behind the Internet.
At the early age of online commerce, most companies didn’t invest any time in pre-
production performance testing. At the very best they performed manual testing,
expecting their infrastructure to support the user load once the application went
live.
It wasn’t indeed uncommon to see a website going down right/ crashed immediately
after sites were launched or after the execution of a major upgrade etc. However,
businesses can’t afford to endure even short term interruptions, as the new
generation of online-savvy consumers expect reliable services and timely response
at all times. If a site cannot accept an order or is taking too long to display a cart for
selecting a product, customers won’t hesitate going elsewhere.
Today’s web applications are also more complex than ever before. Scaling a
multifaceted, integrated infrastructure from end-to-end means managing the
performance and capacities of individual components within each tier while
measuring the overall transaction response time of the entire system.

4. Methodology:
5
 The methodology described in this paper has the following physical deliverables:
 Assessment report
 Test strategy/plan
 Test scripts
 Test scenarios
 Test results
 Results summary(s)
 Test report and presentation

Following a proper methodology guarantees a successful performance test

project. It is important that the same methodology is followed on all projects.
This ensures that the same deliverables are produced and quality can be
measured against this. Comparisons can be made between different projects by
evaluating the deliverables. Projects that were the most successful are identified
easily and it is also easy to identify what went wrong or was done wrong on the
projects that were less successful or where less value was added.
Value added can be determined by looking at system improvement as plotted on
graphs or given as percentages for areas where improvements were made. In
Figure 1 two average response time graphs are shown.

Figure 1 – First test results

6
 The Average Transaction Response Time graph displays the average time taken
to perform transactions during each second of the load test scenario run. The x-
axis represents the elapsed time from the beginning of the scenario run. The y-
axis represents the average response time (in seconds) of each transaction.

Figure 2 – Final Test Results

The first graph shows the response time for the first test and the second graph
for the final test of the performance test project and the improvement (value
added) is obvious. These graphs are available after testing so show it to the
people that matter.

The performance testing methodology consists of six phases. Each phase is

completed with a deliverable or deliverables. Table 1 shows each phase with the
expected deliverable(s).

Phase Deliverable
1 - Project assessment Assessment report
2 - Planning Test plan
3 - Scripting Test scripts
4 - Test execution Test scenarios, test results
5 - Results analysis Results summary
7
 6 - Reporting Performance test report and presentation

4.1 Project Assessment:

The most successful projects that I have completed all started with a formal
assessment. This phase determines whether the work can be done and if so,
how? The assessment is a process of gathering information. Requirements are
analyzed and the system and architecture studied to determine whether the
requirements can be met with what is available in the specific environment.
People should also be informed about your own requirements to perform the
testing, including the time required to get useful results.
Expectations should be managed for the duration of the project. People usually
assume that results can be delivered very easily and very quickly. If you are using
some automation tool, as is usually the case with performance testing, this is
even more applicable. The assessment process is where expectation
management starts.

4.2 Planning:
The information gathered during the project assessment is used for planning the
performance testing and to start the test plan. The performance test plan must
contain all the detail and acts as a check list and reference for test execution.
The test plan forms the backbone of the testing process and is a working
document that is updated as the project progresses. A fully completed test plan
guarantees that no details are left out for the test execution.

4.3 Scripting:
When you find problems during initial testing, the test tool and scripts are always
blamed first. It is essential that you are 100% sure that your scripts do not cause
any problems or errors on the system.
Understanding the tool and how it interacts with the system is just as important.
You need to be in a position where you can completely trust the tool and your
scripts. You need to be able to confirm this and at the same time gain the
respect of the necessary people.
Scripting is where the testing starts. It is extremely important that you familiarize
yourself 100% with the application. A good tester will get a feel for the system
right from the start. As you learn the system and processes to script, make notes
of response times and slow or very busy processes. All of these may be potential
8
bottlenecks in the system. When you start executing scripts, monitor the
processes you identified closely and you may identify the first problem before
any formal performance or load test was done.
Monitoring starts during the scripting phase as well. When running a script for
the first time the response times for measured transactions should be noted
already. Keeping track of these response times is very important as changes in
system behavior can be spotted by running one script only and save the time of
preparing and setting up full load tests. Always run individual scripts at least
once after every system change or implementation.
Keeping an eye on everything every time an individual script is run saves a lot of
time and trouble. It is necessary to run the scripts on every new release and
ensure they are working fine. If not, they should prepare fresh script relevant to
code changes using techniques like parameterization & handling of dynamic
values etc. There is nothing worse than starting a full-blown load test with
unique data for multiple users in place only to find that something is not right
and the whole data and scenario setup have to be repeated. Attention to detail
and understanding system behavior help to avoid these situations.

4.4 Test Execution:

There are different views and ways of talking about performance testing. Most
people talk about load testing or stress testing. Although there isn’t a definite
right or wrong I refer to performance testing as a whole and this includes the
testing as described in Table 3. The methodology was developed using this
terminology.

Performance Testing
Type of Test Description
Baseline test Establish performance baselines
Load test Emulate production load on the system
Stress test Load the system to breakpoint
Soak test Test the system over a long period of time
Volume test High data volumes / throughput. Database
growth

Baseline tests:
Baseline tests are often mentioned but also ignored. However, they hold far
more value than just establishing performance baselines and are one of the most
important steps in this methodology. With some effort and time taken to
examine details, up to 85% of performance problems can be identified and
9
solved during baseline test runs. Unfortunately there is often not enough time
for proper baseline testing, so it is important to include and plan for baseline
tests right from the beginning of a project.

Baseline tests are done with each script individually. Typically each script is run
with 1, 2, 5, 10 and 20 users. The maximum number of users will differ from
project to project and is also dependent on the type of transaction or business
process scripted. In some cases 5 or 10 users may be the maximum for a specific
script.
Full monitoring should be done during the baseline testing. All results must be
saved and analyzed. The advantage of this is that all measurements are specific
to the one process or transaction and problems can be identified without the
trouble of isolating the process causing the problem or error. Figure 6 shows the
response times of one script. The example shown is the result of a 20-user test
that caused very high CPU utilization, Figure 7. The high CPU utilization only
happened with the one script.

Figure 6 – Response time graph for one script with 3 measurements

10
 Figure 7 – CPU utilization very high for 20 users performing a single
process

The problem was identified during baseline testing and isolated to the one script.
No time was wasted for setting up and preparing for a full load test with multiple
scripts and then spending more time trying to isolate the cause of high CPU from
a typical load test response time graph as can be seen in Figure 8. The objective
is to iron out problems early during baseline testing and have an almost “clear
run” when the first full load test is done. This saves everyone the frustration of
trying to establish the cause of a problem or bottleneck with numerous users and
transactions all being monitored at the same time.

Figure 8 – Isolating problems between many processes can be impossible

11
Following the methodology with all baseline tests completed before the load test
runs begin, ensures that every test run produces meaningful results. This is a
very important aspect of the test process as early, meaningful results make
people see your efforts as well as the value you are adding right from the start.

Load tests:
When I started my performance testing career, I was trained how to plan, script
and execute load tests. Although scripts were run individually during the
scripting phase and for data preparation, the aim was to prepare for a load test.
One of the main drawbacks of this was that the first test results were only
available after a long period of preparation, often without meaningful or even
readable results. Although problems were identified by this method, it was very
difficult and time consuming to determine the cause of the problem. Add to this
the very “busy” graphs from load test results and it becomes clear why a more
productive and meaningful methodology should be followed.

These factors resulted in the development of a new methodology that delivered

quick and meaningful results. Load testing is now only a part of the performance
testing project and most problems are identified and solved before the first load
test run.

Capacity Test:
A capacity test is a test to determine how many users your application can
handle before either performance or stability becomes unacceptable. By
knowing the number of users your application can handle “successfully”, you will
have better visibility into events that might push your site beyond its limitations.
This is a way to avoid potential problems in the future.

Spike Test:
Spike testing is a type of load test. The object of this type of performance test is
to verify a system's stability during bursts of concurrent user and or system
activity to varying degrees of load over varying time periods.

Stress test:
A stress test is run to determine the breakpoint of the system. This should be
done once all problems that stem from the performance testing have been
resolved. The results of a stress test can be used for capacity planning and it

12
gives administrators a view of how the system breaks and recovers. Plan to
include at least one stress test towards the end of the project.

Soak test:
A soak test is a load test that runs over a long period of time. Memory leaks are
probably the most common problem to look for during a soak test, but
connection timeouts are often found and database growth can also be
monitored. Include all the relevant people when planning the soak test and ask
them what they want monitored for their specific area. Time and other logistical
issues such as test data are some of the main problems to overcome when
planning and executing a proper soak test. The test should run as long as
possible, or until a specific trend can be identified through some of the monitors.
If any serious defects are present, this will most probably determine the duration
of the soak test.

Volume test:
Volume testing refers to size and more specific database and file sizes. This is not
always a requirement, but can be important depending on the type of
application being tested. Database size can influence performance greatly and
this should be put forward as a risk if the performance testing is done against a
small database compared to what the size would be in production. With volume
testing load might not be required in the form of high user numbers and careful
planning of how to execute the test is needed.

4.5 Results Analysis:

Results analysis is perhaps the most challenging aspect of performance testing. It
starts with the design of scenarios and tests that will give you the right “picture”
when you look at the results at the end of a test run. Meaningful results are not
always achieved and I believe the results that you want from a test must be the
goal to work for when designing performance test scenarios.

Test results are the most important deliverable for the performance tester. This
is after all the most effective way of showing the success of the testing. At the
beginning of the paper I mentioned the first and the last test results from a
project. This is the goal to work for. The comparison between the first and the

13
last test run. How much improvement is shown as a result of the performance
testing?

There are two main rules to follow to ensure successful results analysis:
1. Save everything.
Name and save the results of every test run you do. Use a proper and sensible
naming convention to make referencing at a later stage easy.
2. Keep track of everything.
Make notes on why a test fails or why performance is poor or different than
before. Also keep notes of why results are good or better than before. What was
changed? Add the changes and the effect they have on system performance to
the results summary after each run. Save it all for later reference and to use in
the final test report.

Performance testing is an iterative process with many test runs. A short results
summary is the most effective way to communicate results between test runs
and most often the time between test runs is not enough to compile a full test
report. The summary documents are good physical deliverables that make your
effort more visible to the people making the investment.

4.6 Reporting:
The last methodology phase is to report back on the findings and progress of the
whole project. A full performance test report is delivered with a presentation to
communicate the content of the report to the relevant people.

The aim is to explain the content of the final report and answer questions
anyone might have about the testing and findings. I have learnt through
experience that a report on its own is not very effective and most people never
read it. Do both the report and presentation in a manner that non-technical
people can understand it as well.

The final report doesn’t refer to the results of one specific test and covers the
findings of the test process as a whole. Graphs are included mainly for
comparison with the emphasis on performance improvement throughout the
project. Detailed results are not included but a reference to the relevant results
summaries are given where specific issues are discussed.

5. Performance Tools:

14
LoadFocus is an All-In-One cloud testing platform for load testing and
performance testing, Website Speed Testing, Automated Website Testing for
Websites, Mobile Applications and APIs. Easy and cost-effective way to test your
Websites, Mobile/Web Applications, Web Services and APIs

The Grinder is a Java load testing framework that makes it easy to run a
distributed test using many load injector machines. It is freely available under a
BSD-style open-source license.

Gatling is an open-source load testing framework based on Scala, Akka and

Netty. Project’s aims include: High performance, Ready-to-present HTML
reports, Scenario recorder and developer-friendly DSL

Tsung is an open-source multi-protocol distributed load testing tool. The

purpose of Tsung is to simulate users in order to test the scalability and
performance of IP based client/server applications.

Apache JMeter™ is a graphical server performance testing tool, for both static
and dynamic resources (files or CGI, Servlets, Perl scripts). Apache JMeter™ is
open source, a 100% pure Java application designed to load test functional
behavior and measure performance.

OpenWebLoad is a tool for load testing web applications. It aims to be easy to

use and providing near real-time performance measurements of the application
under test.

JCrawler is an open source (under the CPL) Stress-Testing Tool for web-
applications. You can give JCrawler a set of starting URLs and it will begin
crawling from that point onwards, going through any URLs it can find on its way
and generating load on the web application.

WebLOAD is easy to use, supports most of our requests. It is easy to

create/record scripts for web applications especially dotnet applications. Easy
correlation and helps even a starter to handle dynamic values. Easy integration
with Dynatrace. Good analysis report. Great reporting.

LoadComplete is a desktop tool for load, stress, scalability testing of websites

and web applications. LoadComplete allows you to generate load from virtual
machines, on premise computers, or even the cloud.

15
LoadRunner – Load testing software that gives you an accurate picture of end-
to-end system performance to identify and resolve issues before applications go-
live.

IBM® Rational® Performance Tester is a performance testing solution that

validates the scalability of web and server applications. Rational Performance
Tester identifies the presence and cause of system performance bottlenecks and
reduces load testing complexity.

6. Summary:
Performance Testing is very crucial for any project. Without analyzing the
performance of any project, it cannot be guaranteed that it would give a good
user experience. The methodology described in this paper has been proven on
various projects with various architectures. It is by no means the only
methodology that works, but it does give you assurance of positive results.
Visibility of the value added and the guarantee of success are the main reasons
for developing and implementing this methodology.

7. References:
 http://blog.smartbear.com/software-quality/7-types-of-web-
performance-tests-and-how-they-fit-into-your-testing-cycle/
 http://www.testingperformance.org/
 https://www.tutorialspoint.com/software_testing_dictionary/
 http://istqbexamcertification.com/what-is-performance-testing-in-
software/
 http://softwaretestingfundamentals.com/performance-testing/
 https://loadfocus.com/blog/2016/07/28/best-performance-testing-tools-
and-load-testing-tools/

8. Contacts

Krishna Chirala, Phani

DC Specialist Senior, Deloitte Consulting LLP
[email protected]

Kandagadla Suresh
DC Manager, Deloitte Consulting LLP
[email protected]

16
Deloitte Consulting LLP

This white paper includes data and information that shall not be disclosed outside the government and shall not be duplicated, used, or
disclosed—in whole or in part—for any purpose other than consideration of this white paper. In no event shall any part of this white
paper be used in connection with the development of specifications or work statements with respect to any solicitation subject to full and
open competition requirements. This restriction does not limit the government's right to use information contained in these data if they
are obtained from another source without restriction.