Open Source Performance Testing

Using
Apache JMeter

A Success Story Of

A Leading Financial Company in the U.S

By
Naga, Sravanthi (119500)
Performance Testing CoE
 Open Source Performance Testing Using Apache JMeter

Table of Contents

Abstract ..................................................................................................................... 3
1.0 Introduction ...................................................................................................... 4
1.1 About the Organization ..................................................................................................... 4
2.0 Performance Testing ....................................................................................... 4
2.1 What is Performance Testing? ......................................................................................... 4
2.2 Need for Performance Testing ......................................................................................... 4
2.3 Costs associated with Performance testing ................................................................... 5
2.4 Open Source Solution ....................................................................................................... 5
3.0 Success Story .................................................................................................. 6
3.1 Application Architecture – An example ........................................................................... 6
3.2 Core Performance Testing Activities ............................................................................... 7
3.3 Implementation: Challenges faced .................................................................................. 7
3.4 Implementation: Open Source Benefits .......................................................................... 8
3.4.1 Implementation: Apache JMeter Features & Overview ................................................... 9
3.4.2 Implementation: Apache JMeter Limitations ..................................................................... 9
3.5 Implementation: Value Additions from Cognizant ....................................................... 10
3.5.1 Introduction to JMeter – GLog Reporting Framework .............................................. 10
3.5.2 Features .................................................................................................................... 10
3.5.3 Benefits ..................................................................................................................... 10
3.6 Implementation: Lessons learnt ..................................................................................... 11
3.7 Client specific benefits .................................................................................................... 12

White Paper Page 2 of 12

Open Source Performance Testing Using Apache JMeter

Abstract
In early 2007, Cognizant’s performance testing team had started its journey in the Open
Source arena. During this tenure of 1.5 years, we’ve gained considerable mastery in some
of the popular Open Source tools backed by a lot of success stories, new innovations,
continuous Research & Development of new tools & utilities, new frameworks being built in
this exciting Open Source galaxy.

What are the reasons for this growing affinity towards the Open Source tools? The reasons
being not just the Cost savings, but Open source solutions have much to offer performance
test engineers. It provides an alternative choice to commercial test tools. With proper
planning and evaluation, the Open source Performance team can become a cost effective
alternative solution for an organization’s performance testing needs. Having Performance
testers with the right skill sets and with programming knowledge can go a long way in
making the Open source Performance testing beneficial in the true sense.

This white paper explains the success story of a leading financial organization in United
States and how performance testing activities were successfully carried out by using the
Open Source tool Apache JMeter without compromising on quality. It also depicts the value
additions that the Cognizant’s performance testing team has provided to the customer.
Without the rigor described in this paper, performance testing could mean lots of invested
dollars on the tool and the support from the tool.

White Paper Page 3 of 12

Open Source Performance Testing Using Apache JMeter

1.0 Introduction
1.1 About the Organization
The company is a leading financial organization with a clearly defined mission: to help craft
a vision for ideal financial future - especially when it comes to retirement - and then help you
make it a reality. It is the next generation of one of the most respected financial services
companies. With over two million clients and one of the largest advisor networks in the
country, it is ushering in a dynamic new era of financial planning - one where we can help
you pursue your dreams with greater confidence, increased peace of mind and the benefits
of an enduring relationship with an advisor. Over 110 years in financial services, more than
10,000 personal financial advisors in more than 3,500 offices nationwide and a heritage of
corporate philanthropy.

2.0 Performance Testing

2.1 What is Performance Testing?
Performance testing of an application / system is basically the process of understanding
how the application and its operating environment behave at various user loads. In general,
it is performed by simulating virtual users to determine / validate the scalability, availability,
robustness, hardware & software resource utilization of the application thus ideally paving
the way for effective capacity planning of the system for future usage.

One of the main objectives of performance testing is to help maintain the system with low
latency, high throughput, and low utilization.

Various types of Performance Testing include:

 Load Testing
 Stress Testing
 Volume Testing
 Spike Testing
 Endurance Testing

2.2 Need for Performance Testing

Performance Testing sets the ‘best possible’ performance expectation under a given
configuration of infrastructure. It also highlights early in the testing process if changes need
to be made before application goes into production. Some of the reasons for which
Performance Testing has to be mandatorily carried out are:

White Paper Page 4 of 12

Open Source Performance Testing Using Apache JMeter

 Poor Response Time A critical webpage may take one minute to show up. The expected
response time is less than 10 seconds.

 Scalability When 100 users from new branch were introduced into the system, the
application was unable to handle the additional load and response times degraded from 5
seconds to 50 seconds.

 Batch Window Issues The batch upload that is supposed to happen in a 1 AM to 7 AM

window does not complete till 9:30 AM. This impacts the availability of data for online
applications.

 Availability The application server needs to be restarted owing to resource leaks within
the application. This causes a downtime of atleast 0.5 hours a day.

 Resource Capacity Planning An application consumes 90% CPU on a server that is

supposed to host another application after 3 months. There is no extra capacity to take this
additional load.

2.3 Costs associated with Performance testing

Performance testing usually has huge costs associated with it. The costs associated with
Performance testing can be categorized as

 The Investment made on the Performance testing tool.

 Time consumed in Preparation of test data.
 The Load Testing Infrastructure / environment setup should be as close as
possible to the production environments.
 In case of administrative organizations, acquiring the required permissions is a
tedious process, since performance tests typically require many administrative rights.

Bottom-line is while there are many costs associated with Performance testing, the
Investment on the tool is considered to be the most indispensable contributor to the total
cost. Most of the commercial tools require separate licenses to be procured based on the
user load which contributes to huge Performance testing effort cost itself.

2.4 Open Source Solution

Quest for better alternative The Investment on the tool has lead to search out for better
alternative solutions. This quest has ended in the Open Source Performance Testing tools
without sacrificing quality.

What is Open Source? ‘Open Source’ is defined as a software code that is available for
users to examine and change freely without violating any patents, copyright laws, or
licensing agreements. The idea with open source is that when everyone can work together
and build upon existing tools, the ultimate result is much better software. It is a way for
White Paper Page 5 of 12
Open Source Performance Testing Using Apache JMeter

many companies and individuals to collaborate and improve software that each person
could not do alone.

 Why should one go for open source?

 No cost – Absolutely FREE!!

 Evolving Software (Can be customized to users’ needs).
 High portability.
 Scalable and offers an effective solution for growing business demands.
 Huge open source forum technical support which stands in par with any other
commercial tool support. In fact it’s more advantageous as there is no cost or
logistics involved like other commercial tool technical support.

Wow!
In summary, open source solutions are not just about cost savings. It is more
customizable and user friendly, highly portable across various platforms and offers an
effective solution for small growing business demands. The leading financial organization
had exactly used this success formula to performance test its various kinds of Web & Web-
service based applications.

Curious to know how? Its time get to the success story now…

3.0 Success Story

3.1 Application Architecture – An example
Browser

System and Site

Authentication Authorization
Administrator

Site, Page, Custom Dynamic Portal Common

Portlet & Portlets Module Content
Navigation Management
System

Database

White Paper Page 6 of 12

Open Source Performance Testing Using Apache JMeter

 Multi-Tier Architecture.
 Portlet data is customized according to user queries and data is retrieved from the
database.
 The data is then fed into the Common Content management system after which it
is returned to the browser.
 So thus all the various Web & Web-service based applications which were
involved in this Content management system were successfully tested using the
Open Source Performance Testing Tool Apache JMeter.

3.2 Core Performance Testing Activities

The following are some of the key activities during the successful implementation of
Performance testing for the leading financial organization using JMeter:

 Understand and Analyze the Business Requirements.

 Define Performance Test Goals and create a Performance Test Plan.
 Create JMeter Scripts for identified critical business flows and execute various load
test scenarios.
 Prepare customized analysis summary reports for every test execution and an
overall Engagement Summary Report at the end of the engagement.
 Based on the performance bottlenecks idenitified, provide suggestions to the
application development team for tuning the application. Test executions are thus
run in iterative manner until the desired acceptance criteria are met.

3.3 Implementation: Challenges faced

Being the first venture into the world of Open source Performance testing, the following
were the challenges.

 Newer concepts and various ways of implementing different protocols had to be

explored from scratch. For instance Bean Shell Technology, Handling FTP requests
and dealing with various types of Java based requests.
 The reporting aspect of the tool was very limited. The data given by the tool was
in a raw form and a lot of manual effort had to be involved in order to derive
meaningful results and to generate elegant presentable reports.
 Lot of other open source JMeter utilities and plug-ins had to be be explored in order
to effectively replicate the scenarios.
 The associates had to be highly flexible – For instance JMeter cannot record the
AJAX specific requests. However it will be able to play them back. Hence those
missing requests, headers had to be captured from other HTTP profiling tools like
HTTP Watch and Packet Sniffing tools such as Wireshark and had to be placed in
JMeter. Hence one has to be highly flexible in terms of technical expertise and
usage of various tools and technologies in order to yield the maximum benefits out
of Open Source.

White Paper Page 7 of 12

Open Source Performance Testing Using Apache JMeter

3.4 Implementation: Open Source Benefits

 Huge Open Source Support - Since the tool has a huge open source support
community, learning new protocols were relatively easy and the queries posted almost
had solutions. This is beneficial as most commercial tools include separate cost for
providing technical support.

 No need of High-tech infrastructure - Doesn’t demand a state of the art

infrastructure for load testing.

 Distributed Testing - Supports multiple load-injectors managed by a single

controller.

 User-friendly GUI - Most of the popular Performance Testing open source tools
have a very good GUI thereby allowing faster operations and more precise timings.
Hence learning and adapting to the tool becomes easier over a period of time.

 No Licenses and Maintenance Issues - Most commercial tools require separate

licenses to be procured based on the user load specifications which contribute to huge
tool cost. In addition to that, there involves an extra burden of maintaining the
licenses. With the use of Open Source tools no such cost / efforts are involved to
explicitly maintain the tool.

White Paper Page 8 of 12

Open Source Performance Testing Using Apache JMeter

3.4.1 Implementation: Apache JMeter Features & Overview

Apache JMeter is a powerful desktop performance tool from the Apache Jakarta project,
written in Java, for load-testing web pages, web applications, and other static and dynamic
resources including databases, files, Servlets, Perl scripts, Java Objects, FTP Servers, and
more. The main component in JMeter is the ‘Java Swing-based Graphical User Interface’
which can be used both for Scripting and Execution.

 Excellent cost saving solution for small projects as it is an open source tool.
 Robust in handling complex test scenarios that demand ‘n’ number of virtual users.
 Complete portability and supports 100% all the Java based applications.
 Less scripting efforts as compared to other tools because of its user effective GUI.
 Is used to conduct Load, Stress, Volume & Endurance tests on the Web & Web-
service based applications.
 HTTP & HTTPS load testing can be done without any adding additional pluggable
samplers.
 Since it is Java based, the tool was highly compatible with most of the Java based
requests i.e. it can be used to directly test the Java requests, JDBC requests and JMS
publisher & Subscriber. Hence it is a very good cost effective solution for small
development activities as well.

3.4.2 Implementation: Apache JMeter Limitations

 Reporting Framework Limitations

 Does not provide enough statistics: Most of the open source tools (JMeter as
well) does not provide the client side statistics summary reports in a
presentable format as available with the other popular commercial tools such
as HP Load Runner and Borland Silk Performer.
 Huge Log Files: Added to that JMeter provides the log details for every user
and every iteration thus in turn causing the log files generated to be very huge.

White Paper Page 9 of 12

Open Source Performance Testing Using Apache JMeter

 Manual Effort: Analyzing such voluminous log files involves a lot of manual
effort in order to create highly sophisticated analysis reports.
 Time Consuming: Above all, creating the final customer presentable summary
report consumes huge amount of time.

 AJAX components cannot be interpreted by JMeter.

 Cannot collate the Running Vusers data correcly in a Distributed Load Testing
conducted using multiple Load Generators.
 Cannot monitor the Server related components.
 There is no feature which allows to pass all the requests concurrently at the same time
unlike in other commercial tools.

3.5 Implementation: Value Additions from Cognizant

3.5.1 Introduction to JMeter – GLog Reporting Framework

‘JMeter – GLog’ is a new age Java reporting tool which is essentially used to plot the
graphs and generate reports for the Open Source tool Apache JMeter. It presents a smart
solution to customize and generate elegant reports from the log files within fraction of
mouse clicks, thus bringing them in par with any other commercial Performance Testing
Tool.

3.5.2 Features
 The tool can spawn any number of records, transcending Excel’s 65K or 1048K
limitation.
 Capable of collating the log files from multiple load generators within fraction of mouse
clicks.
 The tool has a rich palette of features which allows the users to zoom, print, merge
and save the graphs.
 The observations can be included in the same screen as the report window, thus
easing the task for the end user.

3.5.3 Benefits
 The major benefit is one can drastically reduce the time consumed in generation of
elegant final summary reports.
 Automated generation of both PDF and MS-Word which includes elegant and
sophisticated reports with all possible client side statistics as any other popular
performance testing tool.
 No external database is required.
 No more installations - simply unzip the file and start using it.
 Above all one can save one’s productive time

White Paper Page 10 of 12

Open Source Performance Testing Using Apache JMeter

3.6 Implementation: Lessons learnt

JMeter was an excellent choice as this project's performance test tool. The following lessons
learned provide additional detail.

 Team Skill set: If the group in charge of implementing and executing the performance test
effort does not have a member with a solid programming background, the open source
initiative will surely fail. There must be a person on the team capable of addressing issues
related just not only to usage of the performance tool, but making changes and/or
modifications to the open source code. The team must have the appropriate skill set to
properly implement as well as use these tools. The first implication was that in order to
prepare proper performance test scripts the JMeter users should have sufficient Java
technical skills.

 Training: The Cognizant Performance Testing team had to independently learn on how to
use the tool and also offer training to the other team members on their learning. This clearly
states that there is the necessity to get the right skills and experience with performance
testing and use of performance test tools to make the project a success.

Limited Support: The open source community does an excellent job of providing online
support. However, depending on the timeline of one’s performance testing effort, the type of
support needed may not come as fast as one would like to be. As a result of the unavailability
of a dedicated support team from Apache, the Performance Testing team had to overcome
the problems faced during their testing activities.

Reporting Framework: JMeter generates a huge output file and a lot of manual effort is
needed to create sophisticated analysis report. Analyzing JMeter logs and creating graphs
is a tedious and time consuming process. To overcome the analyzing issue and time
constraint, a work around was carried out to extract details from voluminous data. Using this
method, average, maximum, minimum response time, number of passed and failed
transactions, throughput can be extracted with minimum effort.
White Paper Page 11 of 12
Open Source Performance Testing Using Apache JMeter

However now a Reporting framework is readily available which had made life easier 

3.7 Client specific benefits

 The entire Open Source framework provided major cost savings to the client without
compromising on the quality of the deliverables.
 JMeter was easy to install and of medium complexity to learn and hence the effort involved
in adapting to the tool was minimal.
 Being developed on Java, the tool was highly portable that made us explore different
options thus inturn helping the client in Performance testing of different technologies.
 The Application was tested with upto 2400 Active Users volumes thus helping the client
identify possible production performance issues beforehand.
 Since it is stage by stage performance test, the system bottlenecks were ironed out thus
reducing the chances of production outages.
 Cognizant’s Performance Testing Team was highly flexible and adaptive to the various
challenges pose by the Open Source tools. The team was able to overcome the reporting
limitation by developing a complete reporting framework thus making it as an entire Open
source packaged solution to the customer.

White Paper Page 12 of 12