An AUTOMATED TESTING INSTITUTE Publication - www.automatedtestinginstitute.

com

AutomAted ....... S t
oftware
SPECIAL ISSUE 2010

eSting

MAGAZINE

Performance Testing Virtualized System

excerPts from the mentorA/Ati WebinAr on virtuAlizAtion

The Winner Is...

leArn more About this yeArs Ati AutomAtion honors Winners

Moving Your TesT Lab

To The

Cloud
SPECIAL 2ND ANNUAL ATI AUTOMATION HONORS COVERAGE

Are You Contributing Content Yet?

The Automated Testing Institute relies heavily on the automated testing community in order to deliver up-to-date and relevant content. Thats why weve made it even easier for you to contribute content directly to the ATI Online Reference! Register and let your voice be heard today!

As a registered user you can submit content directly to the site, providing you with content control and the ability to network with like minded individuals.

Community Comments Box

>> Community Comments Box - this comments box, available on the home page of the site, provides an opportunity for users to post micro comments in real time. >> AnnounCements & Blog Posts - if you have interesting tool announcements, or you have a concept that youd like to blog about, submit a post directly to the Ati online reference today. At Ati, you have a community of individuals that would love to hear what you have to say. Your site profile will include a list of your submitted articles. >> AutomAtion events - Do you know about a cool automated testing meetup, webinar or conference? let the rest of us know about it by posting it on the Ati site. Add the date, time and venue so people will know where to go and when to be there.

Announcements & Blog Posts

Automation Events

learn more today at http//www.about.automatedtestinginstitute.com

AutomAted S t
oftware eSting
Special Edition 2010, Volume 2, Issue 4

Contents

Cloud
9

Virtualization CoVerage
Moving Your TesT Lab To The CLoud
Most organizations that employ in-house virtualization for testing are at least studying the potential benefits of the cloud and many are making the move, either in whole or in part. Read this article to learn what those reasons are and how you might go about moving your test lab to the cloud. By Randy Hayes

exCerTs FroM: six CriTiCaL suCCess FaCTors For PerForManCe TesTing virTuaLized sYsTeMs 38
Peformance testers must understand the fundamentals of virtualization, differences among virtual server technologies, virtual workload modeling, key performance bottlenecks of virtualized systems and adapting testing for virtualization. This article provide some excerpts from the webinar that provides all of this information fto you. By Dan Downing & Dion Johnson

2nd annual ati automation HonorS CoVerage

FinaLisTs and Winners announCed
Best Open Source UNIT Automated Test Tool 13 Best Open Source FUNCTIONAL Automated Test Tool 14 Best Open Source PERFORMANCE Automated Test Tool 15 Best Commercial FUNCTIONAL Automated Test Tool 18 Best Commercial PERFORMANCE Automated Test Tool 20 Best Automated Testing Book 21 Best Automated Testing Blog 21 Best Automated Testing Forum 21

12

and The aTi auToMaTion honors Winner is...

22

You know who the winners are, but do you really KNOW who the winners are? Learn more about them.

ColumnS & departmentS

ediToriaL
The best is Yet To Come Why the best is OK.
special edition 2010

The Automated Software Testing (AST) Magazine is an Automated Testing Institute (ATI) publication. For more information regarding the magazine visit http://www.astmagazine.automatedtestinginstitute.com

www.automatedtestinginstitute.com

Automated software testing magazine

Editorial

The besT is Yet To Come

by Dion Johnson
Thats right. The title of this article has the B word in it. Im not exactly sure when it happened, but at some point the word best became profanity. It became something to fear or shun. It became the favorite target for those that sought to set themselves apart, maybe even above the masses... the masses and their other destructive devices like testing standards, wikipedia, and ESPNs top ten plays. Are these things really destructive, though? They are definitely not perfect, and may occassionally present disagreeable information, but they have true value: the value of collective wisdom. Which brings me back to the word best. Best is not a word to be feared, but rather embraced, and that is exactly what the Automated Testing Institute has done by organizing the 2nd Annual ATI Automation Honors awards. ATI invited its community to get

intelligent enough to know that the tools and resources that have been named the best may not be the absolute best in every single circumstance that you can muster. Naming them the best doesnt end evaluation, but rather helps individual users and organizations begin and more

Then, the short list may be more closely evaluated to see if the best translates into the best for you. So, dont be afraid of the word best. Often if you shine a light on those that demonize the word, youll find that they actually do promote the best concept,

...at some point the word best became profanity

quickly conclude evaluations. Take, for example, the tool categories in which awards are presented. There are hundreds - maybe thousands - of tools that can fit into any given category. ATIs tool index alone has thousands of tools listed, and the index is far from exhaustive. So when people are looking for tools to meet their needs, where do they start? Generally organizations dont have time and resources to evaluate hundreds of tools. This is the benefit that something like the ATI Automation Honors provides. By highlighting tools that have been collectively named the best by a community of users that share needs that are similar to yours, you may now more readily refine your search and create a more effective short list. but without using the word best. Then if you probe deeper, youll find that the best that they promote is their own personal doctrine. Not only is this special issue of the AST magazine not afraid of the word best, it is dedicated to it. The 2nd Annual ATI Honors awards winners and finalists are highlighted throughout this issue which provides descriptions of as well as feedback from the awardees. In addition, this issue shines a light on the topic of virtualization. Mr. Randy Hayes delivers an engaging article on cloudbased virtualization entitled, Moving Your Test Lab To The Cloud. Also, excerpts are provided from the Mentora/ATI sponsored webinar entitled, Six Critical Success Factors for Performance Testing Virtualized Systems. special edition 2010

together and collectively identify a set of automated testing tools and resources that may be considered the best, with the belief that no one will mistake the word best to mean perfect. The IT discipline is filled with people that are 4

Automated software testing magazine

www.automatedtestinginstitute.com

For more information, visit http://www.networking.automatedtestinginstitute.com

special edition 2010 www.automatedtestinginstitute.com Automated software testing magazine 5

Authors and Events

Whos In This Issue?
tests for the Altair 8080 in 1979. Then in 1983 he created the first automated test tool for the PC called AutoTester for DOS. Currently Hayes is co-founder and CEO of Capacity Calibration, a software company that specializes in cloud-based Web load testing. Incorporated in 2008, Capacity Calibration develops and markets test automation solutions for medium and large companies. Its flagship product is CapCal, a Web performance testing tool intended to help reduce the effort of script creation and maintenance.

randy hayes wrote diagnostic

AutomAted S t
oftware eSting
Managing Editor Dion Johnson Contributing Editor Edward Torrie Director of Marketing and Events Christine Johnson

A PUBLICATION OF THE AUTOMATED TESTING INSTITUTE

12 years of experience and has led hundreds of performance projects on applications ranging from eCommerce to ERP for companies ranging from startups to global enterprises. He has authored numerous presentations, white papers and articles on performance testing, presented at STAR, HP Software Universe, Software Test and Performance, and Workshop on Performance and Reliability conferences. He is widely recognized among the software testing and performance community, and holds positions on the Advisory boards at STP and WOPR among others.

dan downing has over

CONTACT US AST Magazine [email protected] ATI Online Reference [email protected]

aTi honors Winners

Ati and Partner events

September 2011 Verify/ATI Conference
www.verifyati.com

are featured in this issue. Open Source, Commercial and other tools and resources get the recognition they deserve!
6 Automated software testing magazine

www.automatedtestinginstitute.com

special edition 2010

ATI Automation Honors

Celebrating Excellence in the Discipline of Software Test Automation See the Winners Now!
www.atihonors.automatedtestinginstitute.com

2nd Annual

special edition 2010

www.automatedtestinginstitute.com

Automated software testing magazine

Moving You
To The
by Hayes Randy

Clo
special edition 2010

Automated software testing magazine

www.automatedtestinginstitute.com

Moving to the Cloud

ur

oud
I
special edition 2010

TesT Lab

n a recent study, IBM reported that the average enterprise devotes up to 50 percent of its entire technology infrastructure to development and testing, but typically up to 90 percent of the test infrastructure remains idle. Whether or not this describes your company, the fact remains that test automation has increased both the size and number of test labs, where a test lab is a room or space dedicated to computers used exclusively for testing. With the advent of virtualization, computers in a test lab have been made far more productive along with the people using them just the ability to store different configurations for use in testing has been a great boon for testers and developers everywhere. Cloud computing, of course, is really just a specialized way of using virtualization, but it relieves you of the need to purchase, store and maintain your own hardware and provides virtually unlimited capacity for when you need it. Most organizations that use VMware or Virtual PC are at least studying the potential benefits of the cloud and many are making the move, either in whole or in part, for all the right reasons. What those reasons are and how to go about making the switch are the topics of this article.
www.automatedtestinginstitute.com Automated software testing magazine 9

Types of Cloud Computing

Its become as tricky to define the cloud as it was to define artificial intelligence in the 80s (if you can remember that far back). Microsofts Azure, for example, provides just about everything but the Windows platform itself as a cloud-based service, while Amazon and Rackspace provide both the Windows and the Linux platforms as a service. For the purpose of test automation you will most often need the Windows platform as a service, since most functional test tools run on Windows (with some exceptions, of course). So for the purposes of this

Test labs with 10 to 20 or even more machines in them are quite common, even though they are going the way of the dodo bird as of 2010. Because they consume space and power and are usually idle a great deal of the time, they are a prime candidate for migrating to the cloud. In fact there is probably no better example of why cloud computing makes sense than a test lab for these very reasons.

to load testing and even then it wasnt enough. But for functional testing you may want to avail yourself of the added capacity of the cloud for a number of reasons. You might want to run a number of tests in parallel and thus cut down on the time needed to complete a lengthy test run. For cross browser testing you may want to run tests in all the major browsers instead of just one. Practically anything you can do with virtualization in house you can do in the cloud, but without the limits inherent in a lab of any size.

Benef of the Cloud its for Testing

Aside from the space and power consumption, what makes the cloud better than using virtualization in house? Well,

Choosing the Right Cloud

Everyone knows that Amazon was the first to introduce cloud computing to

Aside from the space and power consumption, what makes the cloud better than using virtualization in house? Well, the most obvious example is load testing

article we will confine our discussion to clouds that offer the Windows platform as a service, and thus compatible with tools like HP Mercury QuickTest Pro, Micro Focus SilkTest, Smart Bear Test Complete, Worksoft Certify as so forth.

the most obvious example is load testing, in which many machines are needed to generate load. How many is a matter of the size of the load needed, but I have seen test labs with 100 computers stacked from floor to ceiling dedicated entirely

Test labs with 1O to 2O or even more machines in them are quite common, even though they are going the way of the dodo bird as of 2O1O.
10 Automated software testing magazine

the masses and remains by far the most popular and well known cloud. The Amazon Compute Cloud (EC2) supports a wide range of Windows and Linux platforms in all sizes, and takes just minutes to configure and deploy custom test machine images. Rackspace has also been an innovator in this space and has recently announced support for Windows as well as Linux platforms. Pricing ranges from 2 cents an hour to over $1 an hour based on the size of the machine in terms of memory and disk space as well as the speed and number of CPU cores. One of the benefits of the Amazon cloud is something called Virtual Private Cloud, which allows you to configure your test machine instances to run inside your firewall. Unless the application under test runs on the client machine itself you will often need to have your test machines run on the internal network, and VPC supports this. A workaround might special edition 2010

www.automatedtestinginstitute.com

Moving to the Cloud

be to install VPN software on your test machine images, but in any case it is a matter to consider. The location of data centers is also a factor, and Amazon has a presence in the USA, Europe and Asia. While this may not be as important for functional testing it is definitely a factor in load testing, where network latency is a key metric in test results. options you require and attach them to your instances on startup. The default is all ports closed, so even to use RDP and SSH you will have to create a security group with these ports open. When you spin up a cloud instance you are given a public IP address that anyone can access. This IP address lives for the duration of that instances lifetime so if you want to maintain that same address in the future Amazon provides elastic IP addresses which are static and allow you to associate them with any instance. If you need to make sure nobody outside your organization can access your test machines you should consider using VPC so that your instances run on a sub-net inside your private network. the difference between life and death fortunately most web apps are nowhere near as sensitive).

Making the Switch

In general, the procedure for setting up a test lab on the cloud is not much different than setting one up in a virtualized environment. Start with a plain vanilla system, install and configure whatever software you need, and save it. The magic, of course, is most visible when you need to set up and tear down your instances watching a dozen or more test machines come to life in a matter of minutes is fun to watch, and even more fun when they are churning out tests and test results. One limiting factor, however, is the availability of Windows platforms neither Amazon nor Rackspace supports anything but Windows Server editions, whereas VMware supports XP, Vista and Windows 7 as well. With Microsoft promising to provide Windows as a service on the Azure cloud, maybe they will lift this limitation. But in the meantime, many companies are employing a hybrid approach that uses a combination of virtual machines and cloud machines. The days of dedicating a physical computer to run a single test are gone have you noticed?

Some Drawbacks and Limitations

OK, so youve set up an account at the cloud provider of your choice and fired up an instance of a Windows or Linux machine. Now what? Well, like any vanilla virtual machine youve got to copy whatever files you need, including programs to install. For this you need to rely on FTP commands, which is cumbersome compared to copying files from one drive to another. You will need to use RDP for Windows or SSH for Linux to log in, then attach to yet another server to download your files. With Amazons Virtual Private Cloud things are easier since you can see the machine from inside your firewall, but still you may find yourself doing a lot of manual work and wish it could be automated for you. Finally, a remote connection is never as fast as a local one so screens may not appear as snappy as you are accustomed to. And yet for the number of advantages you get these limitations are worth it for most people.

Load Testing as Killer App for the Cloud

Hardly anything else comes close to load testing as a perfect reason to use the cloud. Depending on how big of a test you need to run you can easily require dozens if not hundreds of computers to generate the load. It simply is not practical for most companies to have this many computers on hand for testing purposes, especially if they are only needed from time to time. With cloud computing you can spin up as many as you need, use them for your tests, and tear them down again. There is a running debate about the accuracy of load tests that use the cloud for load generation (or any virtual machines for that matter) because of the slowing down of the system clock that can occur if the cloud providers resources become overly taxed. Its been our experience that this effect, while certainly a possibility, occurs rarely enough and is subtle enough to have no visible effect on the results. In any case, if you need 100 computers to run a load test what are your options other than using the cloud? Its one of those trade-offs that is easy to make for most companies (unless you are testing medical or transportation equipment where milliseconds can mean

Wait! Theres More! Want more of the cloud? Check out the Volume 2 Issue 1 (V2, I1) of the Automated Software Testing Magazine (March 2010 Issue). V2 I1 has articles on how automated testing is leading the cloud charge and how open source tools are adapting to the cloud. Find back issues at:
http://www.astmagazine. automatedtestinginstitute.com

What about Security

Security has always been a major concern, and even more so nowadays with so many cyber threats that exist. While the data centers themselves are physically secure, what you expose to the outside world is entirely up to you. With Amazon, for instance, the use of security groups allows you to set the firewall special edition 2010

www.automatedtestinginstitute.com

Automated software testing magazine

11

2nd Annual ATI ............ ............ Automation ............ Honors

Pioneers in
Honoring tHe best
tools and resources

in automation

The 8 award categories are: best open source, unit, functional and performance test tools; best commercial unit, functional and performance test tools; best automated testing book, best blog and best forum site. Subcategories exist in each of the main tool categories and are specific to technology, such as C++, Web and Java. Also, each tool category has a subcategory that is not technology specific, but instead honors the tool irrespective of technology for its feature list, ease of use, support and other pertinent factors. 12 Automated software testing magazine

......................

u ap so

beck

www.automatedtestinginstitute.com

Ju n it
Kent

e st et Te pl m o c derek

i
Ole Lensmar

langone
special edition 2010

Best Open Source Unit Test Tool

Sub-category

C++

Google C++ Testing Framework Producer: Google

cfix Producer: Johannes Passing

Check Producer: Chris Pickett

Winner
JUnit Producer: David Saff, Erich Gamma, Erik G. H. Meade, Kent Beck

Finalist
TestNG Producer: Cdric Beust

Finalist
UISpec4J Producer: Rgis Medina, Pascal Pratmarty

Sub-category

Java

Winner
NUnit Producer: Charlie Poole

Finalist
csUnit Producer: Jacob Jake W. An-

Finalist
xUnit.net Producer: Brad Wilson, Jim Newkirk

Sub-category

.NET

derson, Piers Lawson, Markus Renschler and Manfred Lange

Winner
JUnit Producer: David Saff, Erich Gamma, Erik G. H. Meade, Kent Beck

Finalist
NUnit Producer: Charlie Poole cfix Producer: Johannes Passing

Finalist
TestNG Producer: Cdric Beust

Overall

Sub-category

Winner

Runner-up

Finalist

Finalist
13

special edition 2010

www.automatedtestinginstitute.com

Automated software testing magazine

Best Open Source Fu

Sub-category
Selenium Producer: OpenQA Sahi Producer: Tyto Software WebTest Producer: Canoo Watir
Producer: Bret Pettichord

Web

Winner
T-Plan Robot Producer: T-Plan Ltd.

Runner-up

Finalist

Finalist
UISpec4J Producer: Rgis Medina, Pascal Pratmarty

Jemmy Producer: Alexandre (Shura) Iline

Sub-category

Java

Winner
SWTBot Producer: Charlie Poole

Runner-up

Finalist

White Producer: Vivek Singh

Finalist

Finalist
White Producer: Vivek Singh

Sub-category

Fitnesse Producer: Micah Martin, Robert Martin

T-Plan Robot Producer: T-Plan Ltd.

.NET

Winner

Finalist

Finalist

14 Automated software testing magazine

www.automatedtestinginstitute.com

special edition 2010

unctional Test Tool

Flash/Flex
Sub-category
Flex Monkey Producer: Gorilla Logic Flash Selenium Producer: Paulo Caroli, Sachin Sudheendra FunFx Producer: Aslak Hellesy, Stefan Magnus Landr, Peter Nicolai Motzfeldt

Winner
Selenium Producer: OpenQA Jsystem Producer: Ignis Software

Finalist
Fitnesse Producer: Micah Martin, Robert Martin

Finalist
White Producer: Vivek Singh

Overall

Sub-category

Winner

Runner-up

Finalist

Finalist

For us, this award proves that some do consider free technologies instead of just blindly paying big bucks

Shura Iline

Jemmy

special edition 2010

www.automatedtestinginstitute.com

Automated software testing magazine

15

Best Open Source Performance Test Tool

Sub-category
JMeter
Producer: Apache Software Foundation

Grinder Producer: Philip Aston

C++

fwptt Producer: Bogdan Damian

TestMaker
Producer: PushToTest

Winner
Software Foundation

Runner-up
SoapUI Producer: eviware

Finalist
Grinder Producer: Philip Aston

Finalist
Pylot
Producer: Corey Goldberg

Sub-category

JMeter Producer: Apache

Java

Winner
JMeter
Producer: Apache Software Foundation

Runner-up
SoapUI Producer: eviware

Finalist
Grinder Producer: Philip Aston

Finalist
TestMaker
Producer: PushToTest

Overall

Sub-category

Winner

Runner-up

Finalist

Finalist

This prize goes out to the community. Thank you. Thank you so much guys.
16 Automated software testing magazine www.automatedtestinginstitute.com

Niclas and Ole

SoapUI

special edition 2010

Are You Contributing Content Yet?

The Automated Testing Institute relies heavily on the automated testing community in order to deliver up-to-date and relevant content. Thats why weve made it even easier for you to contribute content directly to the ATI Online Reference! Register and let your voice be heard today!

As a registered user you can submit content directly to the site, providing you with content control and the ability to network with like minded individuals.

Community Comments Box

>> Community Comments Box - this comments box, available on the home page of the site, provides an opportunity for users to post micro comments in real time. >> AnnounCements & Blog Posts - if you have interesting tool announcements, or you have a concept that youd like to blog about, submit a post directly to the Ati online reference today. At Ati, you have a community of individuals that would love to hear what you have to say. Your site profile will include a list of your submitted articles. >> AutomAtion events - Do you know about a cool automated testing meetup, webinar or conference? let the rest of us know about it by posting it on the Ati site. Add the date, time and venue so people will know where to go and when to be there.

Announcements & Blog Posts

Automation Events

learn more today at http//www.about.automatedtestinginstitute.com

Best Commercial Fu
Sub-category
SilkTest Producer: Micro Focus Certify Producer: Worksoft Rational Functional Tester Producer: IBM

Web

Winner
TestComplete Producer: SmartBear Software

Finalist
QA Wizard Producer: Seapine Software Qf-Test Producer: QFS

Finalist
Squish Producer: Froglogic

Sub-category

Java

Winner
Ranorex Producer: Ranorex GmbH

Runner-up

Finalist

Finalist
SilkTest Producer: Micro Focus

Sub-category

QA Wizard Producer: Seapine Software

.NET

Winner
Ranorex Producer: Ranorex GmbH

Finalist
eggPlant Producer: TestPlant

Finalist
RIATest Producer: Cogitek Inc

Flash/Flex

Sub-category

Winner

Finalist

Finalist

18 Automated software testing magazine

www.automatedtestinginstitute.com

special edition 2010

unctional Test Tool

Web Services/Soa
Sub-category
SilkPerfomer for SOA Producer: Micro Focus Lisa Test Producer: iTKO SOATest Producer: Parasoft

Winner
TestComplete Producer: SmartBear Software

Finalist
Ranorex Producer: Ranorex GmbH

Finalist
eggPlant Producer: TestPlant

Overall

Sub-category

Winner

Runner-up

Finalist

QA Wizard Producer: Seapine Software

SilkTest Producer: Micro Focus

Finalist

Finalist

I would like to thank ATI for organizing this event, and would like to thank all of you who voted for us. Jen Herget
CEO, Ranorex GmbH
special edition 2010 www.automatedtestinginstitute.com Automated software testing magazine 19 19

Best Commercial Performance Test Tool

Web Services Web/HTTPS
Sub-category
LoadRunner Producer: Hewlett Packard AQTime Producer: SmartBear Software Producer: IBM
Rational Performance Tester

SilkPerformer
Producer: Micro Focus

Winner
LoadRunner Producer: Hewlett Packard

Runner-up

Finalist

Finalist
SOATest Producer: Parasoft

Sub-category

QA Load Producer: Micro Focus

Winner
LoadRunner Producer: Hewlett Packard

Finalist
AQTime Producer: SmartBear Software
Rational Performance Tester

Finalist
SilkPerformer Producer: Micro Focus

Overall

Sub-category

Producer: IBM

Winner

Runner-up

Finalist

Finalist

Wed like to thank the Automated Testing Institute...for this prestigious award.
President, SmartBear Software
20 Automated software testing magazine www.automatedtestinginstitute.com special edition 2010

Derek Langone

Best Automated testing Book, Blog & Forum

Best Forum
SQAForums Producer: BetaSoft Inc
Knowledge Networks

AdvancedQTP Forum Producer: SOLMAR

One Stop Testing

TekTips
Producer: Tecumseh Group

Producer: Vyom Technosoft Pvt. Ltd.

Winner

Runner-up
Autonomicon Producer: Nick Olivo

Finalist

Finalist
Corey Goldberg Blog Producer: Corey Goldberg

Automated Testing and QA Producer: Peter Kartashov

Best Blog

Winner
Finalist

Runner-up
Testlabs.com Producer: TestLabs

Finalist

I.M. Testy Producer: Testing Mentor

Finalist
Expert Shell Scripting (Experts Voice in Open Source)

Best Book

QTP Unplugged Producer: Tarun Lalwani

Software Performance and Scalability: A Quantitative Approach

Producer: Henry H. Liu

Producer: Ron Peters

Winner

Finalist
www.automatedtestinginstitute.com

Finalist
Automated software testing magazine 21

special edition 2010

And the AtI AUTOmATIOn HOnORS Winner is...

TI A uto A tion ma nors Ho

The AST Magazine Focuses on the ATI Automation Honors Award Winners
he ATI Automation Honors awards accept nominations from you, the automation community, and rely on your votes to determine who the best of the best really are. Awards are presented in several categories from the corporate level down to the practitioner level, resulting in a celebration of the best and most relevant new releases of tools, and the best print and online resources. Weve told you who the winners are, but do you really KNOW who the winners are? This article provides you with the opportunity to learn more about this elite group of tools and resources.
22 Automated software testing magazine www.automatedtestinginstitute.com special edition 2010

2nd AnnuAl ATI AuTomATIon Honors

We dont think there is (or will be) a single best framework - you have to pick the right tool for the particular task you are tackling.
GoogleTest FAQs

http://code.google.com/p/googletest/wiki/FAQ

Google C++ Testing Framework

oogle C++ Testing Framework, also known as Google Test, is based on the xUnit architecture, and helps write C++ tests on a variety of platforms, including Linux, Windows, and Mac. Its features include the ability to isolates tests by running each of them on different objects and grouping related tests into test cases that have the ability to share data and subroutines. Google Test is also used by the following notable projects: The Chromium projects The LLVM compiler Protocol Buffers (Googles data interchange format)

JUnit

Unit is an open source unit testing framework for the Java programming language. Created by Kent Beck and Erich Gamma, JUnit has been important in the creation of test-driven development. JUnit features include: Assertions for testing expected results, test fixtures for sharing code, common test data and test runners for running tests. JUnit has been ported to other languages including ATI Honors Winner NUnits C#.

Honors received
1st Place Winner: Best Open Source UNIT Automated Test Tool - Java 1st Place Winner: Best Open Source UNIT Automated Test Tool - Overall

Honors received
1 Place Winner: Best Open Source UNIT Automated Test Tool - C++
st

Last eLigibLe reLease

4.8.1

Last eLigibLe reLease

1.4.0

Producer/Project admin
David Saff, Erich Gamma, Erik G. H. Meade, Kent Beck

Producer/Project admin
Google

Fun Fact(s):
JUnit has also won the Best Java Performance Monitoring/Testing Tool award in the 2001 and 2002 JavaWorld Editors Choice Awards (ECA)

Fun Fact(s):
The GoogleTest FAQs page begins answering one of its questions by saying, First, lets say clearly that we dont want to get into the debate of which C++ testing framework is the best.. The automation community has, however, decided to settle this debate for them by naming them the best in the C++ category. While it is true that no single tool is the best for every single purpose or person, GoogleTest has earned its way onto anyones short list of tools to consider for unit testing C++ applications.

Web Presence/Project Home:

Web Presence/Project Home:

http://code.google.com/p/googletest/ special edition 2010

honor
Kent Beck (Emailed Statement)

http://sourceforge.net/projects/junit/files/

Thank you for the

Extreme Programming Creator and JUnit developer

www.automatedtestinginstitute.com

Automated software testing magazine

23

2nd AnnuAl ATI AuTomATIon Honors

Its very exciting that the Automated Testing Institute

this way. Thanks for noticing folks!

Charlie Poole (Acceptance Speech Excerpt)
NUnit Representative

chosen to honor nunit in

has
Sahi

NUnit
Unit is a unit-testing framework for all .Net languages that is based on the xUnit architecture. The current production release, version 2.5, is the sixth major release, is written entirely in C# and has been completely redesigned to take advantage of many .NET language features, for example custom attributes and other reflection related capabilities. Charlie Poole, NUnit Developer has identified the following top ten reasons to try NUnit 2.5 (http:// nunit.net/blogs/?p=66#comments): Data-Driven Tests, Theories, Inline Expected Exception Tests, Generic Support, Lambda Support, Out-of-Process Execution and Runtime Selection, PNUnit, Source Code Display, Timeout and Delayed Constraints, Threading Attributes.

ahi is a tool for automation of web applications. Developed in Java and JavaScript, it uses simple JavaScript to execute events on the browser. Features include in-browser controls, text based scripts, Ant support for playback of suites of tests, and multithreaded playback. It supports HTTP and HTTPS. Sahi runs as a proxy server and the browser needs to use the Sahi server as its proxy. Sahi then injects JavaScript so that it can access elements in the webpage.

Honors received
runner uP: Best Open Source FUNCTIONAL Automated Test Tool - Web

Last eLigibLe reLease

v3 (2009-09-22)

Honors received
1st Place Winner: Best Open Source UNIT Automated Test Tool - .NET runner uP: Best Open Source UNIT Automated Test Tool - Overall

Producer/Project admin
Tyto Software

Fun Fact(s):
According to Sahis website, products like ThoughtWorks Studios Twist and the commercial version of PushToTests TestMaker (an ATI nominee in the Open Source Functional Category) leverage Sahis technology to achieve easy automation across browsers.

Last eLigibLe reLease

2.5.3 (Version 2.5.3.9345)

Producer/Project admin
Charlie Poole

Fun Fact(s):
NUnit, like many other xUnit frameworks, was originally transliterated from JUnit. The 2.0 release was a total redesign that was created as though it was done completely in C# from the start.

Web Presence/Project Home:

http://sahi.co.in/w/

Web Presence/Project Home:

http://www.nunit.org

Your acknowledgement [makes] us even more focused

Narayan Raman (Acceptance Speech Excerpt)
Sahi Representative

24 Automated software testing magazine

www.automatedtestinginstitute.com

special edition 2010

2nd AnnuAl ATI AuTomATIon Honors

Had there been another testing tool that could test Javascript uI features in all browsers on all
platforms, believe me, I would have
Jason Huggins (Selenium Blog Interview Excerpt)

saved lots of time *not* writing my own tool.

Jemmy
emmy is a tool allowing you to create automated tests for Java GUI applications. Tests are written in Java, using Jemmy as a regular Java library.

Co-creator of Selenium http://googletesting.blogspot.com/2007/09/seleniums-inventor.html

Selenium

elenium is a suite of tools used to test web applications. The suite includes:

Selenium IDE Selenium RC Selenium Grid

Honors received
runner uP: Best Open Source FUNCTIONAL Automated Test Tool - Java

It provides a record/playback feature along with a test domain specific language (DSL) to write tests in a number of popular programming languages, including Java, Ruby, Groovy, Python, PHP, and Perl. Selenium deploys on Windows, Linux, and Macintosh platforms. Selenium was originally developed by Jason Huggins, who was later joined by other programmers and testers at ThoughtWorks.

Last eLigibLe reLease

Nov 20, 2009

Producer/Project admin
Shura Iline

Fun Fact(s):
JSystems solution for testing Swing application is composed of an agent that uses a Jemmy package to simulate user operations on the Swing UI components. So, once again one ATI Automation Honors finalist is linked to another. Great tools stick together! (http://www.jsystemtest.org/?q=node/81).

Honors received
1st Place Winner: Best Open Source FUNCTIONAL Automated Test Tool - Web 1st Place Winner: Best Open Source FUNCTIONAL Automated Test Tool - Overall

Last eLigibLe reLease

1.0.5

Web Presence/Project Home:

https://jemmy.dev.java.net/

Producer/Project admin
OpenQA

Fun Fact(s):
People often think of Selenium as one tool, but it is actually a suite of tools. There are three main tools in the suite, but in total, there are 8!

For us this proves some do consider free technologies

Shura Iline (Acceptance Speech Excerpt)
Jemmy Representative

Web Presence/Project Home:

http://seleniumhq.org/ special edition 2010

www.automatedtestinginstitute.com

Automated software testing magazine 25

2nd AnnuAl ATI AuTomATIon Honors

Enables customers,
and their programmers software to should

testers,
learn do, what to and

automatically
compare that to what it
Fitnesse One Minute Description Excerpt
http://www.fitnesse.org/FitNesse.UserGuide.OneMinuteDescription

actually does do.

Fitnesse
itNesse is a lightweight, open-source framework that enables customers, testers, and programmers on a software development team to collaborate on and share what the software under test should do, create acceptance tests for checking expected behavior against actual behavior, and automate these acceptance tests.

T-Plan Robot

-Plan Robot is a Java Functional automated testing tool that is developed on generic image based testing principles, and provides a human-like approach to software testing. Founded in 2005, the VNCRobot project had delivered more than 20 binary releases of free software for automated black box GUI testing over the RFB protocol until July 2009 when it was acquired by T-Plan Ltd and renamed to T-Plan Robot.

Honors received
1st Plance Winner: Best Open Source FUNCTIONAL Automated Test Tool - .NET

Honors received
1st Place Winner: Best Open Source FUNCTIONAL Automated Test Tool - Java

Last eLigibLe reLease

20100303

Last eLigibLe reLease

2.0.3

Producer/Project admin
Micah Martin, Robert Martin

Producer/Project admin
T-Plan Ltd.

Fun Fact(s):
Fitnesse is really a front end for developing and running FIT (Framework for Integrated Test) tests, and is a web server, wiki and automated testing tool all rolled into one.

Fun Fact(s):
The original version, VNCRobot 1.x, was released as freeware, but came into the world of open source in T-Plan Robot version 2.0 (http:// www.t-plan.com/robot/docs/faq.html#license). Although open source, the T-Plan Robot source code is not open for any public contributions. Instead, if you, Jane/Joe Automator, want to update the functionality, you have to do so via plug-ins. (http://www.t-plan.com/robot/docs/ faq.html#project)

Web Presence/Project Home:

http://sourceforge.net/projects/fitnesse/

Web Presence/Project Home:

http://www.t-plan.com/robot/

Thanks for this...

Charlie Wheeler (Emailed Statement)
T-Plan Robot Sales & Marketing Manager

special edition 2010

26 Automated software testing magazine

www.automatedtestinginstitute.com

2nd AnnuAl ATI AuTomATIon Honors

on behalf of the Jsystem team and community I want to say thank you for giving us this award.
Yoram Shamir (1st annual awardsAcceptance Speech Excerpt)
CEO Ignis Software

JSystem
System is a professional open-source tool for writing and managing automated system tests. Written in Java, JSystem is based on several open source java projects, including JUnit for writing tests, Ant used to execute scenarios, and Eclipse as the development environment. Jsystem is comprised of the following components: JSystem Services (Java API), JSystem Drivers, JSystem GUI Interface (JRunner), JSystem Agent and the JSystem Eclipse plug-in

FlexMonkey
lexMonkey is an Adobe AIR application used for testing Flex and AIR based applications. Providing the functionality to record, playback and verify Flex UI interactions, FlexMonkey also generates ActionScriptbased testing scripts that you can easily include within a continuous integration environment.

Honors received
runner uP: Best Open Source FUNCTIONAL Automated Test Tool Flash/Flex

Honors received
runner uP: Best Open Source FUNCTIONAL Automated Test Tool - Overall

Last eLigibLe reLease

4.0

Last eLigibLe reLease

5.4.0

Producer/Project admin
Gorilla Logic

Producer/Project admin
Ignis Software

Fun Fact(s):
Based on the Adobe automation API. Stuart Stern stumbled upon it and built functionality on top of it to create Flex Monkey.

Fun Fact(s):
People often think of Selenium as one tool, but it is actually a suite of tools. There are three main tools in the suite, but in total, there are 8!

Web Presence/Project Home:

http://www.gorillalogic.com/flexmonkey

Web Presence/Project Home:

http://www.jsystemtest.org/

[are going to] over as uI development platforms of Stuart Stern (TheFlexShow podcast excerpt) choice. Gorilla Logic President/CEO, Creator of Flex Monkey

like flex ultimately take

Things

special edition 2010

www.automatedtestinginstitute.com

http://www.theflexshow.com

Automated software testing magazine 27

2nd AnnuAl ATI AuTomATIon Honors

The

hands

Grinder puts control right into developers

and provides

Philip Aston (Java Performance Tuning interview excerpt)

visibility of issues.
JMeter

early

Creator of The Grinder http://www.javaperformancetuning.com/news/interview031.shtml

The Grinder
he Grinder is a Java load testing framework that makes it easy to run a distributed test using many load injector machines. Key features Generic Approach Load test anything that has a Java API. Flexible Scripting Tests are written in the powerful Jython scripting language. Distributed Framework Mature HTTP Support

Meter is designed to load test functional behavior and measure performance. JMeter may be used to test performance on static and dynamic resources (Servlets, Perl scripts, Java Objects, Queries, and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types.

Honors received
1st Place Winner: Best Open Source PERFORMANCE Automated Test Tool - Web 1st Place Winner: Best Open Source PERFORMANCE Automated Test Tool - Web
Services/SOA

1st Place Winner: Best Open Source PERFORMANCE Automated Test Tool - Overall

Honors received
runner uP: Best Open Source PERFORMANCE Automated Test Tool - Web

Last eLigibLe reLease

2.3.3

Producer/Project admin
Apache Software Foundation

Last eLigibLe reLease

3.3

Fun Fact(s):
Stefano Mazzocchi of the Apache Software Foundation was the original developer of JMeter. He wrote it primarily to test the performance of Apache JServ (a project that has since been replaced by the Apache Tomcat project).

Producer/Project admin
Philip Aston

Fun Fact(s):
The Grinder was originally developed for the book Professional Java 2 Enterprise Edition with BEA WebLogic Server by Paco Gmez and Peter Zadrozny. It soon found a life outside of the books pages and now finds itself being honored as a full fledge testing tool.

Web Presence/Project Home:

http://grinder.sourceforge.net/

Web Presence/Project Home:

http://jakarta.apache.org/jmeter/index.html

Jmeter can be used to simulate a heavy load on a server

JMeter Site Introduction
http://jakarta.apache.org/jmeter/index.html

28 Automated software testing magazine

www.automatedtestinginstitute.com

special edition 2010

2nd AnnuAl ATI AuTomATIon Honors

Wed like to thank you guys at the ATI Automation Awards. And we also have to thank our faithful users which are the main key factor to our being here.
Ole & Niclas (Acceptance Speech Excerpt)
Co-creators of soapUI

soapUI
oapUI is an open source Web Service Testing Tool for Service Oriented Architecture (SOA). Its functionality mainly covers web service inspection, invoking, development, simulation and mocking, functional testing, load and compliance testing. Productivity enhancement features can be found in the soapUI pro version. soapUI was initially released on Sourceforge during September 2005 and is distributed under the terms of the GNU Lesser General Public License with the application and source code also provided on the basis of the Open Source GNU Lesser General Public License.

SilkTest

ilkTest is a functional and regression automated testing tool that promotes collaboration among different stakeholders such as QA professionals, business analysts and developers, allowing various groups to participate in a continuous quality process. It allows scripts to be developed visually (no script), or with VB.NET, C# or Java environments.

Honors received
1st Place Winner: Best Commercial FUNCTIONAL Automated Test Tool - Web

Last eLigibLe reLease

V2009 R2

Producer/Project admin
Micro Focus

Honors received
runner uP: Best Open Source PERFORMANCE Automated Test Tool - Web Services/SOA runner uP: Best Open Source PERFORMANCE Automated Test Tool - Overall

Fun Fact(s):
At one point in time SilkTest was produced by Segue Software, but Segue was acquired by Borland in 2006. Borland was then acquired by Micro Focus International in 2009. So a lot of people were confused about where to go for the tool. Adding to the confusion is the fact that as of the date of this publication, you can find SilkTest listed on both the Microfocus site (http://www.microfocus.com/ products/silk/silktest/index.aspx) as well as a separate Borland site (http://www.borland.com/us/products/silk/silktest/index.html). In their defense, however, the Borland site has a line of subtext under the Borland name reading, A Micro Focus Company.

Last eLigibLe reLease

3.0

Producer/Project admin
eviware

Fun Fact(s):
With more than 2,000,000 downloads, it is the most used tool for SOA testing.

Web Presence/Project Home:

http://www.soapui.org/

Web Presence/Project Home: http://www.microfocus.com/products/silk/silktest/index.aspx

Fastest script creation of any functional test automation tool.

Micro Focus Site
http://www.microfocus.com/products/silk/silktest/index.aspx

special edition 2010

www.automatedtestinginstitute.com

Automated software testing magazine 29

2nd AnnuAl ATI AuTomATIon Honors

Wed like to thank the Automated Testing Institute

for selecting testing tool

TestComplete
Derek Langone (Acceptance Speech Excerpt)
SmartBear President

our automated

TestComplete
estComplete is an automated testing tool that lets you create, manage and run tests for any Windows, Web or Rich Client software. TestCompletes script-free keyword-driven testing, test visualizer and user-friendly interface makes learning automated testing available for a wider audience of testers.

QA Wizard

A Wizard Pro automates the functional and regression testing of Web, Windows, and Java applications, and load testing of Web applications. It uses an object-based record and playback engine that works with a repository in which objects may be stored. It also has features to aid in data-driven scripting and validation of expected results.

Honors received
runner uP: Best Commercial FUNCTIONAL Automated Test Tool - Java

Honors received
1st Place Winner: Best Commercial FUNCTIONAL Automated Test Tool - Java 1st Place Winner: Best Commercial FUNCTIONAL Automated Test Tool - Overall

Last eLigibLe reLease

2010.1

Producer/Project admin
Seapine Software

Last eLigibLe reLease

7.52

Fun Fact(s):
QAWizard is no stranger to awards. It has won several APEX awards that honor excellence in publications. They won for their users guides and product tours. In addition, the QA Wizard documentation also received a STC (Society For Technical Communication) award. The tool was also a finalist for the 15th Annual Jolt Awards.

Producer/Project admin
SmartBear Software

Fun Fact(s):
TestComplete has a connection with competitor SilkTest. According to the Smartbear site, Joe Krivickas, CEO of Smartbear, was the CEO of Segue Software. Seque was the company that owned the Silk product line, including SilkTest (now a MicroFocus product), which became used by tens of thousands of development teams. Joe may not have been the biggest fan of Borlands takeover of Silk however, since according to the site, he felt the Borland takeover of Segue didnt allow him to finish what he started. As he moved on with Smartbear, he took his Segue-collaborator, Ian McLeod, the previous head of R&D at Segue, with him. Ian is now an Executive VP at Smartbear. Small world.

Web Presence/Project Home:

f u n c t i o n a l
testing with load
Rick Riccetti (Newsletter Excerpt)
Seapine Software President and CEO http://www.seapine.com/pr.php?id=182

http://www.seapine.com/qawizard.html

...now easily combine

Web Presence/Project Home:

http://www.automatedqa.com/products/testcomplete/ 30 Automated software testing magazine

www.automatedtestinginstitute.com

special edition 2010

2nd AnnuAl ATI AuTomATIon Honors

ranorex is proud
to accept this award and to continue to provide a powerful testing tool to the industry
Jen Herget (Acceptance Speech Excerpt)
Ranorex CEO

Ranorex
anorex is a Windows UI test automation framework for testing many different application types including Winforms, WPF, Silverlight, Adobe Flash/ Flex, Web 2.0, Ajax, .NET and Java. The Ranorex Library is the powerful core of the Ranorex Automation Framework. Regardless how the Ranorex Library is used to implement tests, a Ranorex test can be implemented as a simple EXE file. In addition, ranorex also has key features such as capture/replay and an object repository browser.

SilkPerformer for SOA

icro Focus SilkPerformer offers a cost-effective, enterprise-class tool for software application performance and load testing, delivering automated software load, stress and performance testing in an open and sharable model.

Honors received
1st Place Winner: Best Commercial FUNCTIONAL Automated Test Tool - Web/Web Services

Last eLigibLe reLease

v2009 R2

Honors received
1st Place Winner: Best Commercial FUNCTIONAL Automated Test Tool - .Net 1st Place Winner: Best Commercial FUNCTIONAL Automated Test Tool - Flash/Flex runner uP: Best Commercial FUNCTIONAL Automated Test Tool - Overall

Producer/Project admin
Micro Focus

Fun Fact(s):
Through an interesting twist of fate, SilkPerformer and its competitor QALoad are now owned by the same company. The Silk product line was previously owned by Borland, while QALoad was a Compuware product. In 2009 when Micro Focus acquired Borland and the Quality Solutions portion of Compuware, SilkPerformer and QALoad came under the same umbrella. The only question that remains, is how long Micro Focus will continue to support two different tools that have the same purpose

Last eLigibLe reLease

2.2.2

Producer/Project admin
Ranorex GmbH

Fun Fact(s):
Ranorex is so dedicated to automation that they even went as far as to automate the ATI Honors voting process for its users! The created an automated script that their users could download that automatically selected Ranorex in the pertinent categories. The user still had to manually fill out the CAPTCHA (which is the whole point of CAPTCHA), but this extra effort really shows that Ranorex has automation coursing through its very veins!

or scalability
problems early
Micro Focus Site
http://www.microfocus.com/products/silk/silkperformer.aspx

Web Presence/Project Home: http://www.microfocus.com/products/silk/silkperformer.aspx

finding performance

Web Presence/Project Home:

http://www.ranorex.com/ special edition 2010

www.automatedtestinginstitute.com

Automated software testing magazine

31

2nd AnnuAl ATI AuTomATIon Honors

At smartBear we are dedicated to providing best in breed technology at an affordable price for the QA and development teams that are focused on software Quality
Derek Langone (Acceptance Speech Excerpt)
SmartBear President

AQTime
QTime is a perfor mance profiler and memory allocation debugger for 32-bit and 64-bit Windows and .NET-connected applications. AQtime can be integrated into Microsoft Visual Studio, Borland Developer Studio or CodeGear RAD Studio or it can run as a stand-alone application. With AQtime, you can not only find a bottleneck, but also determine what caused that bottleneck.

LoadRunner

ewlett-Packard(HP) Loadrunner (LR) is a performance and load testing product for examining system behavior and performance, while generating actual system load. LoadRunner can emulate hundreds or thousands of concurrent users to put the application through the rigors of reallife user loads, while collecting information from key infrastructure components.

Honors received
1st Place Winner: Best Commercial PERFORMANCE Automated Test Tool - Web/HTTPS 1st Place Winner: Best Commercial PERFORMANCE Automated Test Tool - Web Services/SOA 1st Place Winner: Best Commercial PERFORMANCE Automated Test Tool - Overall

Honors received
runner uP: Best Commercial PERFORMANCE Automated Test Tool - Web/HTTPS runner uP: Best Commercial PERFORMANCE Automated Test Tool - Overall

Last eLigibLe reLease

6.4

Last eLigibLe reLease

9.5

Producer/Project admin
Hewlett Packard

Producer/Project admin
SmartBear Software

Fun Fact(s):
HP LR holds 77% load testing marketshare worldwide!

Fun Fact(s):
Borland, according to a SmartBear case study is on the record as having used AQTime to tune Delphi 2006. This is particularly interesting given that Joe Krivickas, CEO of Smartbear, used to be CEO of Seque, the company that produces a competing line of automation products (the Silk products) and was bought by Borland. See TestComplete Fun Fact for more information.

Web Presence/Project Home:

Web Presence/Project Home:

http://www.automatedqa.com/products/aqtime/

https://h10078.www1.hp.com/cda/hpms/display/main/hpms_ content.jsp?zn=bto&cp=1-11-15-17^8_4000_100__

...prevent costly
Hewlett Packard Site

application performance problems

32 Automated software testing magazine

www.automatedtestinginstitute.com

special edition 2010

2nd AnnuAl ATI AuTomATIon Honors

I would like to thank all those that voted for QTP unplugged as the best book in [the] testing automation category.
Tarun Lalwani (Acceptance Speech Excerpt)
Author

QTP Unplugged
P QuickTest Professional is a functional test automation tool. It supports a Record and Playback framework out of the box, where we can record and capture our interactions with the application under test and then replay those actions later. With this book you will learn - Basic concepts of QTP - Working without Object repository using Descriptive Programming - Advanced concepts of QTP - Working with external tools Microsoft Word, Outlook, Excel Integrating QTP Scripts with Quality Center - Real life Automation problems and their solutions

Automated Testing and QA

utomated Testing and QA (At4QA) is a blog focused strictly on software testing topics, and more specifically automated software testing topics.

Honors received
1st Place Winner: Best Automated Testing Blog

Producer/Primary bLogger
Peter Kartashov

bLogger ProFiLe
* I sense this world as I am & * Quality is like health - nobody can buy it...

Honors received
1st Place Winner: Best Automated Testing Book

Fun Fact(s):
On August 31, Automated Testing and QA published a blog post that referenced 5 other blogs. 4 out of 5 of the blogs are associated with past and present ATI Automation Honors winners and finalist including: Google Testing Blog, SmartBear Blog, I.M. Testy Blog, Corey Goldberg Blog.

autHors
Tarun Lalwani

PubLisHer
KnowledgeInbox

Web Presence/Project Home:

http://at4qa.blogspot.com/

special edition 2010

Im [an] enthusiast of test automation and this is why Im running At4QA Peter Kartashov (Acceptance Speech Excerpt) blog.
www.automatedtestinginstitute.com

At4QA Primary Blogger

Automated software testing magazine

33

2nd AnnuAl ATI AuTomATIon Honors

Its wonderful to see that some organization has awards focused on software testing categories
AJ Alhait (1st Annual Acceptance Speech Excerpt)
SQAForums

SQAForums

Autonomicon
utonomicon is a blog that features thoughts on software automation and software demonstrations.

QAForums is an internet forum site that is created by BetaSoft, Inc.

BetaSoft Inc. is a privately held corporation based in San Jose, California. Founded in 1995, its main business function is providing software testing and quality assurance services throughout the United States and Canada.

Honors received
runner uP: Best Automated Testing Blog

Honors received
1st Place Winner: Best Automated Testing Forum

Producer/Primary bLogger
Nick Olivo

Producer/admin
Betasoft Inc.

bLogger ProFiLe
Im an SE, an automated tester and a tech writer rolled into one.

Web Presence/Project Home:

Web Presence/Project Home:

http://www.sqaforums.com/ubbthreads.php

http://autonomicon.blogspot.com/

AdvancedQTP Forum
O L M A R sponsors a leading resource hub for HPs QuickTest Professional - www. AdvancedQTP.com, which helps - for free thousands of professionals with its top class team of experts and materials. SOLMAR is a global provider of professional services that range from consultancy and integration, thru development and project management, to training, job placement and outsourcing.

Established in January 2008, SOLMAR is already a leading global provider of training and coaching in the field of testing automation.

Honors received
runner uP: Best Automated Testing Forum

Producer/admin
Solmar

Web Presence/Project Home:

http://www.advancedqtp.com/

34 Automated software testing magazine

www.automatedtestinginstitute.com

special edition 2010

http://www.googleautomation.com
special edition 2010 www.automatedtestinginstitute.com Automated software testing magazine 35

36 Automated software testing magazine

www.automatedtestinginstitute.com

special edition 2010

special edition 2010

www.automatedtestinginstitute.com

Automated software testing magazine

37

r e m c o x Fr e

for Performance Testin With

Dan
Downing

Six Critical Succe

t

&
Dion
Johnson

Webinar
On October 26 the Automated Testing Institute co-sponsored a webinar with Mentora. Below are excerpts from that webinar.

Note:

For the full webinar visit:

impact, challenges and requirements surrounding virtualization and found that the adoption of server virtualization is expanding rapidly. As part of their study, IT decision makers were polled and asked to reveal the percentage of servers they had virtualized in 2009 along with the percentage of servers that they planned on having virtualized in 2011. For 2009 the survey peak was already at around 21 to 50 percent, which is pretty high when you think about how pervasive huge racks of dedicated servers was just a few short years ago. But then looking forward to 2011 the survey reveals that the peak will be closer to 70% range.
special edition 2010

http://webinars.automatedtestinginstitute.com

For questions related to the webinar or performance testing of virtualized systems, feel free to contact Dan Downing at:
[email protected] STATe OF AdOPTION OF SeRveR vIRTUALIzATION
Dion Johnson The reason I, as representative of the Automated Testing Institute or ATI, wanted to participate in this webinar was because server virtualization is truly a great topic to cover. The very purpose of ATI is to provide a comprehensive resource of timely information relative and pertinent to testing and test automation, and there are few topics more timely and pertinent right now than virtualization. In 2009 the research firm, Forrester Consulting evaluated the 38 Automated software testing magazine

www.automatedtestinginstitute.com

ess Factors
[As] an IT consultant and as the managing editor of the Automated Software Testing Magazine, these findings are definitely in line with what Ive seen personally and in line with the feedback that Ive received from our readership. So there is clearly a strong tide headed towards virtualization that has already begun and promises to continue into the future. So as testers, specifically performance testers, we need to make sure were ahead of the game, because it promises to not only change how we test, but what we test. Because as virtualization moves more to production environments, we wont just be using virtualization in our development environments to test software applications, but instead well be doing more testing of the virtualized system itself. Plus, this move to virtualization also promises to introduce new or enhanced challenges that we may face

Webinar Recap

ng Virtualized Systems

ANATOMy OF A vIRTUAL SySTeM

Dan Downing So moving right on to the anatomy.

To learn about the top challenges feared most by IT decision makers, view the full webinar at:
http://webinars.automatedtestinginstitute.com
special edition 2010

This little diagram in the center of the slide [above], were going to walk through that here for a minute. The box labeled hardware in the middle, that black box, is known in VM speak as the host. And its basically the hardware that provides the platform for virtualization. Unlike just your typical server, it typically is a hardware that has multi all over it: multi-cpu, multi-core, multi-threaded. Typically a CPU board will present numerous sockets, multiple sockets of memory. Often times were looking at 64-bit systems because we can address expanded memory over the 4 gigabyte limit. And another thing that is multi in a good host is a multi network interface card. Typically one of the bottlenecks of virtualized systems can be bandwidth throughput, and so relying on a single or even a master/ slave NIC is not good enough for a good host. Sitting below the hardware box - that box labeled storage - is nothing more than a bunch of disk drives accessible to the host over [an] efficient network connection. - [This is a] so called storage area network, or network attached storage, one of those flavors of external storage with fast IO buses that can be hooked up to the host either through standard Ethernet cabling or sometimes through optimized fiber channel
Automated software testing magazine 39

www.automatedtestinginstitute.com

or faster optimized IO and that also contain protocols or embed (or use) protocols that off load some of the overhead that often the CPU has to do in order to do physical IO. Again, hinting at one of the other vulnerabilities we will

bring into play here, primarily from understanding where the potential vulnerabilities of a virtualized system can be. One of the things that weve learned from experience is to look at the vulnerabilities as being the places where there

So as testers, specifically performance testers, we need to make sure were ahead of the game, because it promises to not only change how we test, but what we test.
talk about shortly that we want to be sure to understand about virtual systems. Sitting above the hardware box or the host rectangle, there, in the red box, is another box labeled hypervisor. Hypervisor is a virtual term that refers to the piece of virtualized software and there [are] actually two types that this comes in: Type 1 is the one illustrated here or attempted to illustrate, where the hypervisor is the native OS running on the host. VMware is probably the leading example of the type 1 hypervisor. Theres actually a type 2 and that is where the hypervisor or the virtualization software itself run as a guest operating system on top of whatever the hardwares native OS is. An example of this is Windows 2008 server which comes embedded with a virtualization layer that is optional. And well talk about some of these underlying technologies towards the end. And finally the boxes across the top are the slices that the host will be sliced into; and each will have allocated to it some amount of CPU, some amount of memory and will share the peripheral that is the disk IO and the network IO and whatever other peripheral that may be involved, but those are the two main ones - based on how those VMs are configured using the management utility of the hypervisor. is potential limiting resource. Number 1 area is in disk IO. Perhaps you remember our diagram where I suggested the anatomy of a good virtualized system contains fast network attached storage with efficient protocols and thats because IO is one of the bottlenecks and we want to be able to maximize the IO throughput of the host because its going to be shared by all of the various virtual partitions that that host supports. So in this first bullet were looking at a piece of the previous output from VMware capacity planner and it shows a big red X on the left hand side and that happens

To learn why someone might choose one hypervisor type over the other, view the full webinar at:
http://webinars.automatedtestinginstitute.com

PeRFORMANCe TeSTING RULeS OF ThUMb WheN evALUATING CAPACITy PLANNING deCISIONS

Dan Downing Yes, there are some heuristics we can 40 Automated software testing magazine

to be the Oracle database server in this instance. And the number 142 thats highlighted on the right is under disk IO and that says were tracking 142 IOs per second. Well, that may be relatively large, or not, depending again on the capacity of that particular system and its IO bus, but typically thats going to be flagged by a good capacity planning software as not a good candidate and thats where the red X comes into play. So this really points to us as testers that, hey anytime were testing a virtualized system
special edition 2010

www.automatedtestinginstitute.com

disk IO, [potential] contention at the disk level is something to look for. Did high disk IO servers get virtualized? And if so, lets really focus to make sure that we didnt undermine scalability and performance. The number 1 vulnerability regarding a virtualized system comes sort of in two related flavors and that is redundancy and load balancing. If you think about it, most systems - most complex systems - have multiple web servers, multiple app servers, a database cluster, perhaps some file servers, mail servers, other helper systems and they all occur in more than one instance - theres three app servers, theres five web servers and so forth. And certainly I think it should be pretty obvious that the reason those dedicated systems have those capabilities is number 1 for load balancing you know the load coming into our front door is more than one web server can support or more than one app server can manage and secondly if one of those hardware boxes fails its got helpers to take up the load and hopefully not affect end user experience too dramatically.

C ontent C oday T
Community Comments Box

ontribute

To learn more performance testing implications and some specific performance testing techniques view the full webinar at:
http://webinars.automatedtestinginstitute.com

Announcements & Blog Posts

Automation Events

As a registered user you can submit content directly to the site, providing you with content control and the ability to network with like minded individuals.

learn more today at

http//www.about.automatedtestinginstitute.com

special edition 2010

www.automatedtestinginstitute.com

Automated software testing magazine

41

42 Automated software testing magazine

www.automatedtestinginstitute.com

special edition 2010

special edition 2010

www.automatedtestinginstitute.com

Automated software testing magazine

43

44 Automated software testing magazine

www.automatedtestinginstitute.com

special edition 2010