Who does load tuning after load testing?

Tuning software is done by a developer who is responsible for improving the performance of the system.
Testers of performance can suggest or recommend to the development team for software tuning.
However, only developers do tuning.
In load testing, what is tuning software?

Software tuning is the process of fixing the bottlenecked software, identifying the level of software
bottlenecks by getting the database and code profiled. Modifying or fine tuning the software to fix the
bottlenecks is called tuning software.
In load testing, what is tuning hardware?

If during the load test, there is a specific bottleneck identified, the process of bottleneck elimination is
called tuning the hardware.
In load testing, what are concurrent hits of a user?

When many users run the load tests, it is called a concurrent user hit when users hit an application step
common without any difference in terms of milliseconds. The point of concurrency will be added as one of
the virtual user steps. Thus, all zip line users will wait at the point of concurrency if they reach this ahead
of time. Once the point of concurrency is hit by all users, they can begin to hit the requests.
What are the criteria for entering and exiting in testing performance?

You can begin testing application performance during the designs. After evaluating these tests, you can
collect and analyze the results for performance to be improved. The process of performance tuning will be
done throughout the life cycle of the application development. Performance tuning is done based on
factors like application release time and application stability user requirement, scalability under load,
reliability, performance tolerance and stress tolerance criteria. The end criteria in some projects are
defined based on the performance of the client, defined for each application section. When a product gets
to the level expected then this is considered as performance testing end criteria.
What is a protocol?

A protocol is a set of defined rules for information transmission between 2 or more multiple systems is
called a protocol. This is used for test script recording in most of the load and performance testing tools.
So as a load test /performance engineer, you need to be familiar with the used protocol for script

recording. You can get the related protocol information not from the developers but actually from the
architects.
What drawbacks do manual load tests have?

The manual load testing drawbacks are:

It is very expensive to do Manual Testing, as real users charge by the hour.

With manual load testing, load testing for longer durations like for 7 days wont be possible, as
users really work a maximum of eight hours daily.

You will not get accuracy for results correlation as there are delays between actions of users.

It is hard to do results collection as the results capture each other.

It is hard to do.

What is the life-cycle of testing?

Planning the Test

Developing the Test

Execution of the Test

Analysis of Results

What are the tools of testing performance?

There are popular commercial tools for testing which include:

Borland Silk Performer, which allows behavior prediction of environments of e-business to be

predicted before it is deployed, regardless of complexity and size.

IBMs Rational Performance Testing- allows the comparison of test metrics versus running tests.

RadViews WebLoad- allows comparison of test metrics versus running tests.

Compuwares QAload- is used for database and web based systems.

HPs LoadRunner- this is for other applications and web applications. It provides various
environment applications, database and platforms. Evaluating each tracking of bottlenecks and
components measurements is monitored by the number of servers. You might want to check out
this course entitled HP Quick Test Professional that helps you learn how to use HP Quick Test
Professional to automate a process of testing for web applications.

What are performance testing sub-genres?

Here is the list of Performance Testing sub-genres:

Soak testing is performed to understand the behavior of the application when applying load for
long time periods and what happens to the applications response time and stability.

Spike testing is the application change that happens when suddenly large numbers of users
decreases or increases.

Volume testing is application testing to determine how much data amounts it can handle effective
and efficiency.

Stress testing is done for evaluating the performance of a system by increasing user numbers to
more than the specified requirements limits. This is done to comprehend the levels an application
will crash.

Load testing is done to examine an applications performance for specified load expected by
increasing the number of users performing specific application tasks in a specific period of time on
the application.

Why does performance testing need to happen?

Performance testing is done for evaluating the performance of an application under some stress and load
condition. Generally, this is measured in terms of the user activitys response time. It is designed for
testing a whole systems performance at high stress and high load conditions. For example, if customers
like withdrawing funds from an ATM. If the response of the machine after card insertion takes more than
five minute the system fails to function. Performance Testing Types include capacity, stress and load.
Capacity is the measure of the capacity overall and what time a time of response becomes unacceptable.
Stress examines behavior of application under activity in peak bursts. Load is analogous to testing volume
and to determine how applications deal with data in large amounts.
How to identify a memory leak?

You need to run the load testing for longer durations when you are trying to identify a leak in memory.
When you are running test loads for longer durations, the memory of the server in question will gradually
grow if there is a leak in memory.
What is profiling?

Profiling is a process of pinpointing a bottleneck performance at minute levels. This is done by

performance teams for engineering which includes developers or performance testers. You can profile in
any application layer getting tested. If you need to do application profiling you might need to use tools for
performance profiling of application servers. When profiling an application server, you identify issues at
the level of code such as memory intensive APIs If the database is what you are profiling using the tools

for database profiling, you can identify a number of things such as a full table scan query, high cost
queries and the number of executed SQLs.
What are the automated testing phases of performance?

There are involved phases in testing automated performance and these are:

Design/Planning- this is the first phase where teams gather the performance testing requirements.
This can include team requirements, system requirements, technical requirements and business
requirements.

Build- this phase is made up of requirements that are automated and gathered during the phase of
planning/design.

Execution- this phase is actually accomplished in multiple phases. It consists of various testing
types like benchmarking testing and baseline testing.

Tuning and Analyzing- during testing of performance, you can capture all the related system
details like System Resources and Response Time for identifying the systems major bottlenecks.
After these are identified, you will then be able to improve the overall performance of the system.

What is throughput in performance testing?

In Performance Testing, throughput is the data amount sent by the server in response to the request of the
client in a given time period. It can also be the unit number of work you can handle per time unit. You
measure throughput in terms of per second requests, reports per year, hits per second or daily calls, etc.
In many cases, you calculate throughputs in bits per second. Generally you get higher performance with
higher throughput value.
How is JMeter spike testing performed?

To perform spike testing, you need to understand what change happens on applications when suddenly
large numbers of users decrease or increase. You can achieve JMeter spike testing using a timer that is
synchronized. You block the threads by getting the timer synchronized until a specific thread number is
getting a blocking. Next, one a large load is instantaneously created, release these.
What are baseline tests?

These are tests done on applications before coming to conclusions. It can be either the validation or
verification process which gives an idea of what the next stage needs to do. It is a testing technique that is
very important, and if properly done, eighty five per cent of problems with performance can be solved and
identified when proper tests of baseline are accomplished.
How are benchmark and baseline tests different?

The process of running tests in sets is to capture information performance is baseline testing. This info
can be used as a reference point when changes in the future are made to applications. On the other hand,
benchmark tests are the process of comparing the performance of your system against industry standards
giving by other organizations. You can, for example, run baseline tests on applications, analyze the
collected results and modify many indexes on the database of a SQL Server before running the identical
test once more using the previous same results to find out whether or not new results are the same, worse
or better. Here is a course entitled Software Testing which will help you learn all about testing software
from experts in the industry.
Q1. Why Performance Testing is performed?
Performance Testing is performed to evaluate application performance under some load and stress
condition. It is generally measured in terms of response time for the user activity. It is designed to test the
whole performance of the system at high load and stress condition.
Example: Customer like to withdraw money from an ATM counter, customer inserts debit or credit card and
wait for the response. If system takes more than 5 min. then according to requirements system functioning is
fail.
Type of Performance Testing:

Load: analogous to volume testing and determine how application deal with large amount of data.

Stress: examine application behavior under peak bursts of activity.

Capacity: measure overall capacity and determine at what time response time become
unacceptable.

Q2. What are tools of performance testing?

Following are some popular commercial testing tools are:

LoadRunner(HP): this for web and other application. It provides a variety of application
environments, platforms and database. Number of server monitors to evaluate the performance
measurement of each component and tracking of bottlenecks.

QAload(Compuware): used for load testing of web, database and char-based system.

WebLoad(RadView): it allows comparing of running test vs. test metrics.

Rational Performance Tester (IBM): used to identify presence and cause of system performance
bottlenecks.

Silk Performer (Borland): allow prediction of behavior of e-business environment before it is

deployed, regardless of size and complexity.

Q3. Explain the sub-genres of Performance testing.

Following are the sub-genres of Performance Testing:

Load Testing: it is conducted to examine the performance of application for a specific expected
load. Load can be increased by increasing the number of user performing a specific task on the
application in a specific time period.

Stress Testing: is conducted to evaluate a system performance by increasing the number of user
more than the limits of its specified requirements. It is performed to understand at which level
application crash.

Volume Testing: test an application in order to determine how much amount of data it can handle
efficiently and effectively.

Spike Testing: what changes happens on the application when suddenly large number of user
increased or decreased.

Soak Testing: is performed to understand the application behavior when we apply load for a long
period of time what happens on the stability and response time of application.

Q4.What is performance tuning?

To improve the system performance we follow a mechanism, known as Performance tuning. To improve the
systems performance there are two types of tuning performed:

Hardware tuning: Optimizing, adding or replacing the hardware components of the system and
changes in the infrastructure level to improve the systems performance is called hardware tuning.

Software tuning: Identifying the software level bottlenecks by profiling the code, database etc. Fine
tuning or modifying the software to fix the bottlenecks is called software tuning.

Q5. What is concurrent user hits in load testing?

When the multiple users, without any time difference, hits on a same event of the application under the load
test is called a concurrent user hit. The concurrency point is added so that multiple Virtual User can work on
a single event of the application. By adding concurrency point, the virtual users will wait for the other Virtual
users which are running the scripts, if they reach early. When all the users reached to the concurrency point,
only then they start hitting the requests.
Q6. What is the need for Performance testing?
Performance testing is needed to verify the below:

Response time of application for the intended number of users

Maximum load resisting capacity of application.

Capacity of application to handling the number of transactions.

Stability of application under expected and unexpected user load.

Ensuring that users have proper response time on production

Q7. What is the reason behind performing automated load testing?

Following drawbacks of manual Load Testing that leads to Automation load testing:

Difficult to measure the performance of the application accurately.

Difficult to do synchronization between the users.

Number of real time users are required to involve in Performance Testing

Difficult to analyze and identify the results & bottlenecks.

Increases the infrastructure cost

Q8. What are the exiting and entering criteria in the performance testing?
We can start the performance testing of application during the design. After the execution of the performance
testing, we collected the results and analyzed them to improve the performance. The performance tuning
processed will be performed throughout the application development life cycle. Performance tuning is
performed which is based on factors like release time of application and user requirements of application
stability, reliability and scalability under load, stress and performance tolerance criteria. In some projects the
end criteria is defined based on the client performance requirements defined for each section of the
application. When product reaches to the expected level then that can be considered as the end criteria for
performance testing.
Q9.How do you identify the performance bottlenecks situations?
Performance Bottlenecks can identify by monitoring the application against load and stress condition. To find
bottleneck situation in performance testing we use Load Runner because provides different types of
monitors like run-time monitor, web resource monitor, network delay monitor, firewall monitor, database
server monitor, ERP server resources monitor and Java performance monitor. These monitors can help to us
to determine the condition which causes increased response time of the application. The measurements of
performance of the application are based on response time, throughput, hits per sec, network delay graphs,
etc.
Q10. What activities are performed during performance testing of any application?
Following activities are performed during testing of application:
1. Create user scenarios
2. User Distribution
3. Scripting
4. Dry run of the application
5. Running load test and analyzing the result.,
Q11. How can we perform spike testing in JMeter?
Spike Testing is performed to understand what changes happens on the application when suddenly large
number of user increased or decreased. Sudden changes in the number of user by increasing or decreasing
at certain point of application and then monitoring the behavior. In JMeter spike testing can be achieved
using Synchronizing Timer. The threads are blocked by synchronizing the timer until a particular number of
threads have been blocked, and then release them at once thus creating large instantaneous load.
Q12. What is distributed load testing?
Distributed load testing: in this we test the application for a number of users accessing the application at a
same time. In distributed load testing test cases are execute to determine the application behavior. Now
application behavior is monitored, recorded and analyzed when multiple users concurrently use the system.
Distributed load testing is the process using which multiple systems can be used for simulating load of large
number of users. The reason for doing the distributed load testing is that to overcome the limitation single
system to generate large number of threads.
Q13. Explain the basic requirements of Performance test plan.
Any Software Performance Test Plan should have the minimum contents as mentioned below:

Performance Test Strategy and scope definitions.

Test process and methodologies.

Test tool details.

Test cases details including scripting and script maintenance mechanisms.

Resource allocations and responsibilities for Testers.

Risk management definitions.

Test Start /Stop criteria along with Pass/Fail criteria definitions.

Test environment setup requirements.

Virtual Users, Load, Volume Load Definitions for Different Performance Test Phases.

Results Analysis and Reporting format definitions

Q14. What is throughput in Performance Testing?

Throughput in Performance testing is the amount of data sent by the server in responds to the client request
in a given period of time or it is the number of units of work that can be handled per unit of time. The
throughput is measured in terms of requests per second, calls per day, hits per second, reports per year, etc.
In most of the cases, the throughput is calculated in bits per seconds. Higher the throughput value, higher
the performance of the application It is includes the client side statistics.
Q15. What are the automated Performance testing phases?
The phases involved in automated performance testing are:

Planning/Design: This is the primary phase where team will be gathering the requirements of the
performance testing. Requirements can be Business, Technical, System and Team requirements.

Build: This phase consists of automating the requirements collected during the design phase.

Execution: it is done in multiple phases. It consists of various types of testing like baseline,
benchmarking testing

Analyzing and tuning: During the performance testing we will be capturing all the details related to
the system like Response time and System Resources for identifying the major bottlenecks of the
system. After the bottlenecks are identified we have to tune the system to improve the overall
performance.

Q16. What is Performance Testing?

Performance Testing is performed to determine response time of the some components of the system
perform under a particular workload. It is generally measured in terms of response time for the user activity.
It is designed to test the overall performance of the system at high load and stress condition It identifies the
drawback of the architectural design which helps to tune the application. It includes the following:

Increasing number of users interacting with the system.

Determine the Response time.

Repeating the load consistently.

Monitoring the system components under controlled load.

Providing robust analysis and reporting engines.

Q17. What is baseline testing?

Baseline testing is a testing which is performed on the application before coming to any conclusion. It can be
either the verification or validation process which provides an idea of what the next stage has to do. It is very
important testing technique, if done properly, 85% of performance problems can be identified and solved
when proper baseline tests are done.
Q18. What is the testing lifecycle?
There is no standard testing life cycle, but it is consist of following phases:

Test Planning (Test Strategy, Test Plan, Test Bed Creation)

Test Development (Test Procedures, Test Scenarios, Test Cases)

Test Execution

Result Analysis (compare Expected to Actual results)

Defect Tracking

Reporting

Q19. What is the difference between baseline and benchmark testing?

The differences between baseline and benchmark testing are:

Baseline testing is the process of running a set of tests to capture performance information. This
information can be used as a point of reference when in future changes are made to the application
where as Benchmarking is the process of comparing your system performance against an industry
standard that is given by some other organization.

Example: We can run baseline test of an application, collect and analyze results, and then modify
several indexes on a SQL Server database and run the same test again, using the previous results
to determine whether or not the new results were better, worse, or about the same.

Client-Side Performance Counters

LoadUIWeb can track various web server metrics during the load test execution (see Performance Monitoring Basic Concepts). Below is a list of counters that the test engine measures during the test run on the computer
where LoadUIWeb is running. These counters do not require a preliminary configuration and therefore are
available to every load test.

Counter

Description

Virtual users

The number of virtual users that are currently involved in the test.

Passed requests

The number of successful requests that were sent to the tested server since the last
measurement of this parameter. Since the measurements are done every second, this
is actually the number of successful requests sent per second.

Time to first byte

The time (in seconds) between the first byte of the user request and the first byte of the
server response.

Time to last byte

The time (in seconds) between the first byte of the user request and the last byte of the
server response.

Response transfer time

The time (in seconds) between the first byte and the last byte of the response arriving.

Page load time

The time (in seconds) spent to download the complete page including all images,
scripts, CSS files, and so on.

Request transfer speed

The speed of data transfer (in megabytes per second) when the request was sent to
the server.

Response transfer speed

The speed of data transfer (in megabytes per second) when the server sent back the

Counter

Description
response.

Request throughput

The speed of data transfer (in kilobytes per second) when the server was receiving the
request.

Response throughput

The speed of data transfer (in kilobytes per second) when the server was processing
the request.

Errors

The number of errors that have occurred since the last call.

Warnings

The number of warnings that have occurred since the last call.

QOS Page Errors

The number of errors that occur when the page load time exceeds the specified
threshold. The measurements are made every second.

QOS Response Errors

The number of errors that occur when the response transfer time exceeds the specified
threshold. The measurements are made every second.

Failed Validations

The number of failed validations that have occurred by the current moment during the
test run.

What Are Performance Metrics Used in Load Testing?

There are many measurements that you can use when load testing. The following metrics are key
performance indicators for your web application or web site.
Response metrics show the performance measurement from a user perspective. Volume graphs show the
traffic generated by the load testing tool against the target web application.

Response Metrics Is My Site Weathering the Storm?

Average Response Time

Error Rate

Slowest (Peak) Response Time

Volume Measurements How Strong is the Storm?

Requests per Second

Throughput Kilobytes per Second

Concurrent Users
In LoadStorm, the performance metrics are plotted in one-minute increments. All of the load generating
servers feed data back to the LoadStorm reporting engine.
Calculations are applied to the raw data from every request and response, which results in objective metrics
that are useful to determine the effectiveness of your target web application to handle the load.

Example of a real time volume graph

Each virtual user takes actions that simulate real world behavior, and each time a page is hit, there musts be
several pieces of data tracked. This data is used by web developers, testers, QA managers, and
performance engineers to validate the systems ability to scale to the volume define in system requirements.

Average Response Time

When you measure every request and every response to those requests, you will have data for the round
trip of what is sent from a browser and how long it takes the target web application to deliver what was
needed.
For example, one request will be a web pagelets say the home page of the web site. The load testing
system will simulate the users browser in sending a request for the home.html resource. On the targets
side, the request is received by the web server, it makes further requests of the application to dynamically
build the page, and when the full HTML document is compiled, the web server returns that document along
with a response header.
The Average Response Time takes into consideration every round trip request/response cycle up until that
point in time of the load test and calculates the mathematical mean of all response times.
The resulting metric is a reflection of the speed of the web application being tested the BEST indicator of
how the target site is performing from the users perspective. The Average Response Time includes the
delivery of HTML, images, CSS, XML, Javascript files, and any other resource being used. Thus, the
average will be significantly affected by any slow components.
Response times can be measured as either:

Time to First Byte

Time to Last Byte

Some people like to know when the first byte of the response is received by the load generator (simulated
browser). This shows how long the request took to get there and how long the server took to start replying.
However, that is only part of the real equation. It seems to be much more valuable to know the entire cycle of
response that encompasses the duration of download for the resource. Meaning, why would I want to know
only part of the response time? What is most important is what the user experiences, and that includes the
delivery of the full payload from the server. A user wants to see the HTML page which requires receipt of

the full document. So the Time to Last Byte would be preferred as a Key Performance Indicator (KPI) over
Time to First Byte.

Peak Response Time

Similar to the previous metric, Peak Response Time is measuring the round trip of a request/response cycle.
However the peak will tell us what is the LONGEST cycle at this point in the test.
For example, if we are looking at a graph that is showing 5 minutes into the load test that the Peak
Response Time is 12 seconds, then we now know one of our requests took that long. The average may still
be sub-second because our other resources had speedy response.
The Peak Response Time shows us that at least one of our resources are potentially problematic. It can
reflect an anomaly in the application where a specific request was mishandled by the target system. Usually
though, there will be an expensive database query involved in fulfilling a certain request such as a page
that makes it take much longer, and this metric is great to expose those issues.
Typically images and stylesheets are not the slowest (although they can be when a mistake is made like
using a BMP file). In a web application, the process of dynamically building the HTML document from
application logic and database queries is usually the most time intensive part of the system. It is less
common, yet occurs more often with open source apps, to have very slow Javascript files because of their
enormous size. Large files can produce slow responses that will show up in Peak Response Time, so be
careful when using big images or calling big JS libraries. Many times, you really only need less than 20% of
the Javascript inside those libraries. Lazy coders wont take the trouble to clean out the other 80%, and that
will hurt their system performance.

Error Rate
It is to be expected that some errors may occur when processing requests, especially under load. Most of
the time you will see errors begin to be reported when the load has reached a point that exceeds the web
applications ability to deliver what is necessary.
The Error Rate is the mathematical calculation that produces a percentage of problem requests to all
requests. The percentage reflects how many responses are HTTP status codes indicating an error on the
server, as well as any request that never gets a response.
The web server will return an HTTP Status Code in the response header. Normal codes are usually 200
(OK) or something in the 3xx range indicating a redirect on the server. A common error code is 500, which
means the web server knows it has a problem with fulfilling that request. That of course doesnt tell you what
caused the problem, but at least you know that the server knows there is a definitive technical defect in the
functioning of the system somewhere.
It is much trickier to measure something you never receive, so an error code can be reported by the load
testing tool for a condition not indicated by the server. Specifically, the tool must wait for some period of time
before it quits listening for a response. The tool must determine when it will give up on a request and
declare a timeout condition. Timeouts will not a code received from a web server, so the tool must choose a
code such as a 408 to represent the timeout error.
Other errors can be hard to describe because they do not occur at the HTTP level. A good example is when
the web server refuses a connection at the TCP network layer. There is no way to receive an HTTP Status

Code for this, thus the load testing tool must choose some error code to use for reporting this condition back
to you in the load testing results. A code of 417 is what LoadStorm reports.
Error Rate is a significant metric because it measure performance failure in the application. It tells you how
many failed requests are occurring at a particular point in time of your load test. The value of this metric is
most evident when you can easily see the percentage of problems increase significantly as the higher load
produces more errors. In many load tests, this climb in Error Rate will be drastic. This rapid rise in errors tells
you where the target system is stressed beyond its ability to deliver adequate performance.
No one can define the tolerance for Error Rate in your web application. Some testers consider less than 1%
Error Rate successful if the test is delivering greater than 95% of the maximum expected traffic. However,
other testers consider any errors to be a big problem and work to eliminate them. It is not uncommon to have
a few errors in web applications especially when you are dealing with thousands of concurrent users.

Throughput
Throughput is the measurement of bandwidth consumed during the test. It shows how much data is flowing
back and forth from your servers.
Throughput is measured in units of Kilobytes Per Second.

Requests per Second

RPS is the measurement of how many requests are being sent to the target server. It includes requests for
HTML pages, CSS stylesheets, XML documents, JavaScript libraries, images and Flash/multimedia files.
RPS will be affected by how many resources are called from the sites pages. Some sites can have 50-100
images per page, and as long as these images are small in size (e.g. <25k), than the RPS will be higher
than long text pages with few images that are dynamically generated from database queries. The reason for
this is that images and other static resources are served by the web server or a Content Delivery Network,
and there is virtually no expensive processing that must take place before that resource is sent to the
browser (i.e. LoadStorm).

Concurrent Users
Concurrent users is the most common way to express the load being applied during a test. This metric is
measuring how many virtual users are active at any particular point in time. It does not equate to RPS
because one user can generate a high number of requests, and each vuser will not constantly be generating
requests.
A virtual user does what a real user does as specified by the scenarios and steps that you have created in
the load testing tool. If there are 1,000 vusers, then there are 1,000 scenarios running at that particular time.
Many of those 1,000 vusers may be spawning requests at the same time, but there are many vusers that are
not because of think time. Simply put, think time is the pause between vuser actions that simulates what
happens with a real user as he or she reads the page received before clicking again.

Other Thoughts on Load Testing Metrics

On SOA Testing blog, they list the most important load testing metrics in their context as:

Response time: Its the most important parameter to reflect the quality of a Web Service. Response
time is the total time it takes after the client sends a request till it gets a response. This includes the time the
message remains in transit on the network, which cant be measured exclusively by any load-testing tool. So
were restricted to testing Web Services deployed on a local machine. The result will be a graph measuring
the average response time against the number of virtual users.

Number of transactions passed/failed: This parameter simply shows the total number of transactions
passed or failed.

Throughput: Its measured in bytes and represents the amount of data that the virtual users receive
from the server at any given second. We can compare this graph to the response-time graph to see how the
throughput affects transaction performance.

Load size: The number of concurrent virtual users trying to access the Web Service at any particular
instance in an interval of time.

CPU utilization: The amount of CPU time used by the Web Service while processing the request.

Memory utilization: The amount of memory used by the Web Service while processing the request.

Wait Time (Average Latency): The time it takes from when a request is sent until the first byte is
received.