Comparison between LoadRunner and OpenSTA

Item Description LoadRunner OpenSTA

Protocols The communication Many supported. Protocols are HTTP 1.0 / 1.1 / HTTPS (SSL)
protocols that can be charged per item. Has a multi- only.
captured manipulated protocol recording feature.
and replayed by the
tool.

Playback functions Replaying of the script Extended logging supports Similar playback facilities, but
and script debugging view of parameter values and no integrated comparison
facilities. Server messages. Also view function. The debugging
and comparison with ‘recorded’ functions are in the controller,
version of web page view and including set break points and
client response messages. single stepping.
Debugging facilities in script
generator, step and
breakpoints.

Scripting language The medium used to Called TSL, it uses standard Called SCL, it uses a
represent the syntax for “C” & allows C proprietary, “BASIC” like
captured protocol function libraries to be added. language that has special
data and manipulate Has extensive customized automation scripting facilities.
the data for play- functions for the different Is limited in available functions,
back. protocols supported by the such as string manipulation and
tool. supports direct DOM
addressing.

Extensibility The ability to increase Additional TSL or “C” function SCL script modules can be
the functionality of libraries, limited to functional defined in â??Includeâ?? files.
the tool. capabilities of the tool. Open Source therefore new tool
functionality can be added using
C++.
Scripting Interface The interfaces Captures in several modes, Has low-level HTTP protocol
supplied by the tool high level context based and view and provides graphical
application for the low level HTTP view. Has both tree representation of the â??
purpose of script a graphical tree structure & a Document Object Modelâ??
editing. script view. Script view has (DOM) structure. Viewable
function sensitive entry. captured HTML rendering and
addressable server-header
table. Some language sensitive,
syntax color coding
functionality.

Correlation The task of Automated correlation Manual correlation using

substituting values in faculties. Including during graphical interactive â??
dynamic data to recording, after recording and DOMâ?? structure. Facility to
enable successful comparing recordings with automatically generate script
playback. playback results. Not available code to aid variable
for all modes of capture. substitution.
Cookie Management Detection, recording HTTP header cookies are HTTP header cookies are
and playback of HTTP managed automatically and managed automatically and can
cookies. Both tools can be manipulated manually if be manipulated manually if
need additional code required. required.
to manage JavaScript
generated cookies.

Parameterization Automatically Extensive facilities for data Extensive facilities for data
changing dynamic entry including wizard interface entry including wizard interface
data values for more to DB interrogation. No to automatically generate test
accurate emulation of standard function to lock data data. Standard functions for
real users. Often sources and maintain sequential, random and
essential for session uniqueness of concurrently pseudo-random data-file
management. accessed data across access. Has a standard common
distributed tests. locking facility for maintaining
uniqueness of parameters for
an individual load injector or
across all injectors on a
distributed test.

Controller Application that Facilities for real-time Facilities for real-time

manages and monitoring. Automatic scenario monitoring. Simple drag & drop
conducts a test. generation. Individual control multi-scenario test
of vusers, scripts and groups of configuration supporting
scripts. Scheduling, percentage modular scripting and addition
runs & iterations. of new scenarios/Vus at run-
time. No automatic scenario
generation. Allows test-wide
and user-specific HTTP
monitoring and debugging while
multi-user load in progress.

Monitoring Resource usage Many supported. On-line Supports Integrated real-time,

information is graphics during execution. graph view of Windows
captured during Supplied with Apache, NT/2000 Performance
execution. It can be Netscape & IIS, other monitors (Perfmon) & SNMP collectors.
shown during are charged per item. Results Various measurements of test
execution and used to used for reporting. New facility progress including VU-specific,
build performance to allow remote users to custom status and activity
reports. monitor real-time results via a information. Web-relay allows
browser based interface. Note: monitors to run on remote
To monitor through a firewall machines beyond firewall. On-
requires TCPIP access through line graphics during execution,
a specific port. Future versions & monitored results used for
of LoadRunner should use reporting.
HTTP messaging & avoid this
issue.
Distributed tests The ability to Supports multiple load- Supports multiple load-injectors
distribute the injectors managed by a single managed by a single controller.
generation of load controller. Uses TCP/IP if on the same
across multiple load- network or the Web-relay
injector machines. feature uses HTTP to control
load-injectors located within
remote DMZs.

IP Spoofing The ability to emulate Supports IP spoofing including No inbuilt features to enable IP
the behaviour of automated router updates for spoofing.
different IP addresses IP forwarding.
accessing a system.
Particularly useful
with Load balancing
systems.

WAN/LAN emulation The ability to emulate New feature added to version No inbuilt features to enable
the behaviour of 7.6. Allows the emulation of WAN/LAN emulation.
different network latency, packet loss, link faults
infrastructures during & dynamic routing effects over
a test. the LAN used in a test.
Requires a special license.
Caching The ability to emulate Can control browser cache No specific faculties, although
the caching of pages emulation during playback and can be emulated with simple
as performed by a control setting for each script code.
web browser. individual vuser.
User connection The ability to emulate Can emulate different network No inbuilt features to emulate
speed emulation the different network speeds during playback user-connection speed
speeds that can be emulation.
used by real users.
Distributed/Remote To generate large Can control multiple load Can control multiple load
load generation loads additional load generators and collect results. generators and collect results.
generator machines
are required. These
machines must be
centrally controlled.
Can control load generators Can control load generators
located at remote networks located at remote networks
(through a firewall) if required (through a firewall) if required
using HTTP ports. using HTTP ports.

Reporting & analysis The facilities to Sophisticated & large range of Simple charts and graphs
examine and charts and graphs with overlay sufficient for analysing key load
investigate the results facilities. -Automatic report related statistics and resource
of a test including generation into MS-Word. usage monitors. Resource
timers and monitored Analyser is a separate usage monitors supports graph
resources and display application that can be overlays. Can be exported to
the results in distributed to users. Microsoft Excel. No license
graphical format. restrictions on OpenSTA
distribution thus stats can be
viewed by any user with access.
-Free tools and excel macros
available through public user-
forums.
Scalability The ability of the tool Resource limitations are Main resource used is RAM.
to generate numbers number of threads and RAM. Tested for simple ASP pages,
of virtual users and Approx 1 vuser per ½ Mb reaching up to 3,000 users on
the corresponding RAM for NT/W2K. Windows 95, load generator of 1Gb RAM on a
resource usage. 98 & Unix are less efficient. single P4 processor & W2K.
Actual resource use Max. approx 1,500 VUs per PC. Unconfirmed report of a limit of
depends on the 1664 Vus per Win2K machine
number, size and for complex scripts. Suspected
complexity of the thread limitation. No licensing
scripts. limitations.

Initial Costs Purchase costs of From £16,000 for basic FREE� download from
software and licenses, package with no virtual users.
SourceForge via
excluding upgrades or Additional charge for each www.OpenSTA.org. Downloads
support. protocol and monitored available: Previous versions;
resource and for virtual users.
Automatic installer or current
source code (with simple build
instructions for MS C++ Visual
Studio 6).
Costs of Virtual users Most commercial tools Prices vary but for the purpose FREE. No licensing limitations.
charge on the basis of of expectation: additional
the number of virtual vusers cost from £10K for 25
user available. Extra VUs to £66K for 1000 VUs.
hardware is an Also temporary VU days at
additional cost. £3.50 each per day (min
1000). This is not a quoted
price.

Support & The support services By M.I., approx 1/5 of initial

Various independent resources.
consultancy available for the tool cost per annum. Includes etest associates charged from
and some costs. upgrades. MI & their partner
£50 per incident for remote
companies also provide tech support. Consultancy rates
consultancy (including etest
available on request.Numerous
associates). online resources including web
and e-mail forums. Upgrades
are free (approx every 3-6
months)
Training The training services MI has a range of courses from Specialist companies provide
available for the tool. around £400 per day per tailored training, prices vary.
person. Many partners also
provide training.
System requirements The operating MS windows 2000, NT4 MS windows 2000, NT4 (sp5+),
systems required to (sp6a),XP-Pro (also 95 & 98 XP-Pro.
host the tool. (Not the cannot run multi-threaded) -
OS of the system Load generators also support
under test). limited generator functionality
on: Unix: HP, Solaris, Linux

Hardware The hardware Min: Pentium 350 & 128M - Min : Pentium 200 & 80MB
requirements. requirements to host Load generators: Pentium RAM. Pref: Pentium 500MHz+ &
the tool. (Not the 1GHz & 1 MB per VU. 128MB+ RAM.
hardware of the
system under test).
Access to source code Availability of the Unavailable. Open Source GNU public license
source code of the - ‘C++’ Language.
tool itself.

A users opinion. Opinion of Internet Has a very user friendly user Easy to use interface and
performance interface and fantastic excellent scalability.
engineers who have monitoring & results analysis. Inbuilt results analysis is
experience with both Automatic correlation and slightly limited in
tools. improved script recording comparison to
facilities can aid productivity. LoadRunner. The captured
Very flexible scripting functions data is open and can also
& great help documents. be exported to Excel in
Complexity of options and two clicks. Drag & drop
layout of controller are scenario set-up and
drawbacks. The availability (at control is very intuitive
a cost) of numerous protocols and easy to interact with,
for recording is a significant simplifying creation of
strength. scenarios with modular
scripts. Manual correlation
can be a headache but is
eased slightly by using
third-party ‘Diff’ tools and
the inbuilt GUI DOM
addressing feature. A
relative shortage of
standard script language
functions but enough to
get most HTTP load
testing jobs done. If not,
with “Includes” support
and the availability of the
source code, it is very
extensible