Utomated: Oftware Esting
Utomated: Oftware Esting
com
A
S
utomated .......
T
oftware esting MAGAZINE
December 2010 $8.95
TechnicaL
vsvs
Non- Technical
Skills in Test Automation
Automation
Supercharged
Mod-out Your
Automation To Make
It More Effective
Showing Up On
Management’s
Radar
6 Inevitable Questions
You Must Be Prepared
To Answer
www.verifyati.com
A
S
utomated
T
oftware esting
December 2010, Volume 2, Issue 5
Contents
Cover Story
Technical Vs Non-Technical Skills 16
There is a general perception that testers must be or become technical, i.e. programmers, if they are to become
involved in automation. This is not necessarily the case, however. Read this article to learn more about the role of
the testers and test automators in test automation. By Dorothy Graham
Features
Automation Supercharged 24
Just like many automobile enthusiasts “mod-out” their muscle cars, you have the option to mod-out your automation for a
higher degree of technical output. By J.L. Perlin
As a registered user you can submit content directly to the site, providing you with
content control and the ability to network with like minded individuals.
Community
Comments Box >> Community Comments Box - This comments box, available on the home page of
the site, provides an opportunity for users to post micro comments in real time.
>> Announcements & Blog Posts - If you have interesting tool announcements, or
you have a concept that you’d like to blog about, submit a post directly to the ATI
Announcements & Online Reference today. At ATI, you have a community of individuals that would love
Blog Posts to hear what you have to say. Your site profile will include a list of your submitted
articles.
>> Automation Events - Do you know about a cool automated testing meetup,
webinar or conference? Let the rest of us know about it by posting it on the ATI site.
Automation Add the date, time and venue so people will know where to go and when to be there.
Events
Learn more today at http//www.about.automatedtestinginstitute.com
Authors and Events
Despite the title, this article is more about the OUTs, 1. Create template structure
than the Ins; the Output to be more exact. Very often one 2. Connect to test management tool data store
of the serious limiting factors of test management tools is
the format in which data comes out. Test teams and their 3. Open template file
stakeholders get used to managing and seeing data, such as 4. Query header data
test case reports, in a specific way. For example, it’s not
5. Add header data to report
uncommon for testers to manage and maintain test
cases using Microsoft Word templates, but as the test 6. Query dynamic data
bed grows and this approach becomes increasingly 7. Insert rows in the dynamic data report table
unmanageable, project leads often work to introduce
a test management tool. This move towards a test 8. Add dynamic data to report
management tool is often met by resistance for a number of 9. Save and Close template
reasons, and one of those reasons is that the tool
won’t allow them to export the test cases Step 1. Create template structure
in the format they’re used to
This step is the only one
seeing. Sometimes this can
that is not programmatic.
be mitigated by easing
It calls for the creation of a
all pertinent parties into
formatted Word template file
the new way of viewing
(.dot or .dotx). This template
the data. Other times it’s not
file will be accessed and used
that simple. While it may be fine for the
by the custom exporter script
test designer to change the way they manage and
that generates a test case from the
view the data, there may be contractual obligations that
test management tool to MSWord.
force the project team to provide stakeholders (sometimes
The use of a template greatly reduces the amount of code that
government stakeholders) with test case reports in a specific
the custom exporter will require, because instead of building
format. So whether the issues are contractual or cultural, there
the Word report from scratch, the script will simply extract the
is often a need to print out custom Word reports. This can be
desired data from the tool and place it in the predefined template
accomplished in 9 Steps:
• QueryHeaderData
• QueryTestStepData
C
and insert rows (using the InsertRows as illustrated in lines 46 through 52.
method) in the table based on the data
obtained from the data store. Steps 9. Save and Close template ontribute
C
Steps 8. Add dynamic data to report The final step in the process is to
ontent
close and save the generated report file.
Adding the dynamic data to the report Line 55 uses the saveas method to save
is similar to adding the header data, the file with a desired file name. Lines
T
except for the fact that the table cells in 58 through 60 close Word and do some
oday
which data will be entered have been variable cleanup.
Automation Supercharged
(Continued from page 27)
tools also allow the test engineer to take a for multiple AUT servers to be used to
snapshot of the AUT state and attach it to advance other testing objectives.
Community
a defect report to help the developers in
the debugging process. The functionality Closing Comments Box
provided by the VM management and
test management tool integration can There are many factors to take into
greatly reduce the amount of time account when fine tuning your automation
required for building test environments, effort, including environment, available
and can provide some added value to resources, and the track time that you Announcements &
the development team by enabling the are looking to meet or beat. All of the Blog Posts
test engineer to send more detailed methods mentioned above may not apply
information about a failed test in much to your particular scenario, but you can
less time. take bits and pieces of this information
The use of virtual machines also to fine tune your effort to increase its
reduces testing’s footprint and costs, performance, reliability, agility, speed,
allowing funds previously allocated and horsepower! Automation
Events
http//www.about.automatedtestinginstitute.com
Ajar Source
It’s Free...For a Fee
Ah, open source software is great! people (or as you call it teams).
Let’s take a moment to list what we love Now your Pro version has had the
about it: ability to do teaming since at least
version 2.0. I can’t help but think
• It has a large community of that this being a critical function
of a CRM system is intentionally
developers supporting it
left out in order to steer people
• It offers users the ability to to the commercial version of you
customize it product. Am I wrong here?”
• It is free
Many feel duped by commercial open
Wait a minute. The above list needs to be source products, because they are of the
amended: opinion that the “open source” portion of
• It has large community of the software is really not useful without
developers supporting it the features offered in the commercial
• It offers users the ability to extension of the product.
customize it
• It is free Others say commercial open source
producers absolutely have the right to
That’s right, open source software is sell certain portions of their software
not always free. Just in case you were to the public while remaining under the
a lot of debate, regarding whether these umbrella that is open source. Those that
wondering, this is not some thinly veiled
products are really and truly open source. fall on this side of the fence often lean
reference to the costs associated with
open source documentation and expertise It is argued by many that they are not, on the Free Software Definition, which is
or the lack there-of. Open source software and that the companies are merely sometimes linked to open source and open
is literally not always free. Believe it or using the “open source” moniker as a source licenses. This definition reads that:
not, ‘Totally Free’ is not a prerequisite for marketing strategy for software that is
software to be considered open source, and in reality commercial. ZDnet has an “Free software” is a matter of
many companies are taking that loophole article entitled, “Commericial open liberty, not price. To understand
to the bank… literally. Take Redhat for source, a misnomer”, that is all about the concept, you should think of
By DorothyGraham
automation discipline
16 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2010
S
eemingly at odds, both these interconnected skillsets
T
Technical
ec
hnical
are necessary for the successful implementation of
n-
No
not have technical skills should not thereby be prohibited from writing and running automated tests.
TestAutomationSkills one of a ‘tester’, and that any manual Is test automation a technical
tester can pick up and learn how to use task?
Existing Perception an automated testing tool. Although the The answer to this question depends
skills [of a tester] … are still needed to on what you include as part of “test
The automation of test execution implement AST, a complement of skills automation”. If you view it as the direct
is a popular application of computer similar to the broad range of skill sets use of a test execution tool, i.e. writing,
technology to itself. There are a number needed to develop the software product editing and running scripts written in
of books about test automation. Many itself is needed.” (Dustin, E., Garrett, the tool’s scripting language, then it is
of them do not appear to mention skills T. and Gauf, B. 2009. Implementing a technical task, and programming (i.e.
needed (or it was not obvious if they Automated Software Testing. Addison scripting) skills are needed.
did). There is a general perception that Wesley/Pearson Education, Boston, MA.
testers must be or become technical, p 225) Another technical aspect of test
i.e. programmers, if they are to become automation is the design of the testware
involved in automation, although there are A paper by Mosaic mentions architecture – the structure and relationship
a few exceptions that mention a distinction three roles: “Manual Test Engineer”, of all of the items of testware that comprise
between testers and automators. “Automation Test Engineer” and “Lead the artifacts required for automated tests
Automator” ([author unknown] 2002. to successfully run. The design of the
Linda Hayes in her useful booklet
Staffing your test automation team. testware architecture is a critical aspect
on automation says: “… developing
Mosaic Inc, Chicago IL. http://www. for successful test automation, and the
test scripts is essentially a form of
mosaicinc.com/mosaicinc/successful_ skills needed for this include technical
programming; for this role, a more
test.htm). In this model, the design of expertise, as well as knowledge of how
technical background is needed.”
tests (i.e. the tester’s role) is done by both the tests are to be used. The person who
(Hayes, L. G. 1995. Automated Testing
test engineers; the automation work (i.e. designs the testware architecture may be
Handbook. Software Testing Institute,
test automator’s role) is done by the lead called a test automator, test architect, or
Dallas, TX.) She distinguishes between
lead automator.
“Test Developers” i.e. testers, and “Script automator and automation test engineer.
Developers”, which is part of the role of a The key distinction is who designs the
test automator. tests, which in my view is best done by Constructing automated tests
Dustin et al says: “When people think the tester, but collaborating with the is not entirely a technical
of AST [Automated Software Testing], test automator for tests that are to be process
they often think the skill set required is automated. The construction of the automation
K ey
didn’t want to become technical, so he
concentrated on manual testing, but the
others all became good test automators.
T erminology
There were two interesting side-effects of
the testers’ newly acquired skill set. First,
they had a lot more sympathy for the
developers, as they now understood first-
hand the frustrations of trying to get the
• Test automation: the computer-assisted running of soft- computer to do what you wanted it to do.
ware tests, i.e. the automation of test execution. Second, they found that the developers
treated them with a bit more respect, as
• Test automator: A person who builds and maintains the they now also had some development
testware associated with automated tests. skills. This led to a better relationship
between the developers and testers.
• Tester: A person who identifies test conditions, designs Another example where it worked
test cases and verifies test results. A tester may also very well to have all of the testers become
build and execute tests and compare test results. automators is described in a chapter by
Lisa Crispin in our forthcoming book
• Testware: The artifacts required to plan, design and ex- (Crispin, L. Zero to 100% Regression
ecute tests, such as documentation, scripts, inputs, ex- Test Automation in one year: an Agile
Approach to Automation 2010. In
pected outcomes, set-up and clean-up procedures, files, Graham, D. and Fewster, M. Experiences
databases, environments, and any additional software of Test Automation. [Publisher not yet
or utilities used in testing. determined]). An agile team of 9 to
12 people were all involved in doing
manual regression testing, so were highly
motivated to automate 20% of their work,
architecture, and the scripts and other determined, but this is a task that is done and everyone became involved in the
testware that will be used to run automated by a tester; the implementation of the test automation.
tests is a technical task, but automated is what is done by the automator.
testing is not just the structure of the
architecture and scripts. A separate team of test
TestersToAutomators? automators?
The whole purpose of test automation
is to make it possible to run tests with I have seen other organizations where
minimal human involvement in test Testers become automators? a separate team is set up to automate tests,
execution (and comparison). leaving the testers free to concentrate on
I have seen it work well to have a designing tests and running manual tests.
There is a need for testers to be able team of manual testers embarking on an As the automation team gets going, they
to use automated tests, both to write tests automation project, where all (or nearly automate tests nominated by the testers,
to be run automatically, and to run those all) of the testers effectively become freeing the testers from having to do
tests and view the results. The tests that programmers, i.e. test programmers, those tests manually. The automation
are to be automated could be technical
team provides
tests, such as
a service to the
those written
by developers The content of the test needs to be testers, designing
the testware
as part of
Te s t - D r i v e ndetermined, but this is a task that is done by architecture and
structure of the
Development or
unit or integration a tester; the implementation of the test tests, and assisting
where needed
testing,
system
but
and is what is done by the automator when problems are
encountered with
acceptance tests
the automated
can also be automated, and the testers who or scripters. At a former colleague’s tests. For example, if an automated test
write those tests are not always technical company, five out of the team of six fails, it could be because of a software
(i.e. software developers). testers went on the tool vendor’s training fault (in which case the tester would
The content of the test needs to be course and became familiar with the tool’s have found a bug), but it could fail for a
As a registered user you can submit content directly to the site, providing you with
content control and the ability to network with like minded individuals.
Community
Comments Box >> Community Comments Box - This comments box, available on the home page of
the site, provides an opportunity for users to post micro comments in real time.
>> Announcements & Blog Posts - If you have interesting tool announcements, or
you have a concept that you’d like to blog about, submit a post directly to the ATI
Announcements & Online Reference today. At ATI, you have a community of individuals that would love
Blog Posts to hear what you have to say. Your site profile will include a list of your submitted
articles.
>> Automation Events - Do you know about a cool automated testing meetup,
webinar or conference? Let the rest of us know about it by posting it on the ATI site.
Automation Add the date, time and venue so people will know where to go and when to be there.
Events
Learn more today at http//www.about.automatedtestinginstitute.com
December 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 19
technical reason such as a problem with to the software being tested should be automators to make it easy for the testers
the environment, a missing testware item confined to only one testware artifact (or a to kick off a set of tests, for example
(i.e. a bug in the automated testware), or minimum number if this is not practical). by providing options in a user-friend
a problem with the tool itself. The tester, This layer gives good maintainability interface to the automation.
not being technical, will need technical to the automated test regime. The testers also need to receive and
assistance to identify the source of the understand the results of the automated
problem. tests, and the way in which this information
Tester Layer
So we have the situation where test is communicated to them is also designed
automation does require technical skills, If all of the testers are technical, by the test automator.
but we have testers who do not have those such as developers who are doing Test-
This separation of the tester from the
skills – can this really work? Yes it can, Driven Design or unit testing, then this
automation is needed for the automation
but it needs two key separations or layers layer is not as critical. The Tester layer of
to grow within an organization and to give
of abstraction. abstraction is needed when system testers
long-lasting benefits and wide-spread
or user acceptance testers want to use test
acceptance.
automation, but do not want to become
AutomationSuccessNeeds technical.
LayersOfAbstration SummaryAndConclusion
In order to achieve this, the non-
Technical Layer technical testers must be able to both write
tests (that can then be run automatically) Test automation does need technical
Technical aspects are very important and also to run tests, i.e. to “kick off” a set skill – for those who are closest to the tool
for test automation. A good testware of automated tests. itself.
architecture will implement good software
If the testware architecture uses a The skills of the tester and the skills
of the test automator may be found in the
The skills of the tester and the skills same person, but it may work better to
have different people performing the two
of the test automator may be found in the roles.
The test automator’s role is critical in
same person, but it may work better establishing a modular and well-structured
testware architecture, separating the tool
to have different people performing the from the testware, and providing a tester-
friendly interface to the testware for non-
two roles. technical testers.
Non-technical people can be good
keyword-driven approach, the testers testers; testers should be able to use test
development practices for the testware,
can write tests using keywords that are automation without needing to have
separating the tool itself and the direct
related to the business knowledge or technical skills. Getting to this point,
scripting of the tool from the software or
domain knowledge that they are familiar however, does require good technical
scriptware that calls and uses the lower
with. Yes, they do have to follow the support.
level scripts. Modularity and reuse are
key factors in minimizing maintenance of correct syntax for the keywords, but tools
automated testware. If something changes enable this to be relatively easy to do, for
in the software, the testware will need to example by providing a drop-down list of Wait! There’s More!
reflect that change. With lower levels of valid keywords and checking the syntax
scripting (a recorded test or linear script of parameters entered to the keywords.
This article is adapted from a white
being the lowest), a small change to a The keywords are implemented (i.e. paper written for the Conference of the
screen can result in making “magnetic programmed) by test automators, using Association of Software Testing (CAST)
trash” of the automated tests (Kaner, C., the scripting language of the tool, or using 2010 held from August 2 - 4.
Falk, J. and Nguyen, H. Q. 1993. Testing any other programming language that
Computer Software. Van Nostrand they know and is appropriate. The testers Also, for more information on technical
Reinhold, NY.). are not involved in the implementation and non-technical test automation skills,
of the keywords, but they are able to use visit the Automation Body of Knowledge
If possible, the testware should be (ABOK) portion of the ATI Online
designed so that it can cope with changes them to write tests.
Reference Site:
in the software under test without needing The testers also need to be able to
any changes to the testware. If this is select a set of tests to be run automatically. http://www.abok.automatedtestinginstitute.com
not possible, the effects of any change This can be implemented by the test
ATI Automation
Honors
Celebrating Excellence in the
Discipline of Software Test
Automation
www.veri
22 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2010
nference 2011
ifyati.com
December 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 23
Automation
Superch
By J.L. Perlin
B
y implementing the
harged
techniques discussed in
this article you will be
able to reduce the amount of
time required to perform standard
automation tasks, expand the availability
of various testing resources, increase
environmental agility, and fine tune your
research and evaluation efforts. Some of the
methods that allow you to supercharge your
automation effort not only increase performance
for the testing team, but for other project teams, as well.
in the Wait statement. The automated test code, making it easier to manage and Increase Agility
then tries to execute the step following increase performance. When available it
the “Wait” statement, and fails, because may be good practice to make use of this
the AUT is not ready for this step to be A virtual machine is an efficient,
built-in functionality.
executed. Or conversely, the application isolated duplicate of a real machine that
Speaking of memory, undiagnosed supports the implementation of multiple
may have completed its action in much
less time than allocated in the “Wait” memory leaks are a common culprit complete configurations (including
statement, forcing the script to wait longer when it comes to test execution speed operating systems, AUT versions, etc.) on
than is necessary before moving on to degradation. Some automated testing one single box. For example, you can have
the next step. These unnecessary pauses tools (and in many cases the AUTs a virtual server available in your testing
can add up to a considerable amount of themselves) have undetected memory lab that has several different installations
wasted time when executing many tests. leaks. As each test opens, executes, and of the AUT, each being a different version
When possible, it is best to use alternative closes the effect of the memory leak(s) and/or configuration (operating system,
methods to have the test script sync with compounds – eventually slowing test database, etc.)
the AUT. One solution is to identify an execution to a crawl, or in many cases This allows your team to quickly
Showing Up n
Management’s
Radar
6 Inevitable Questions You Must Be Prepared
To Answer
28 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2010
“Often, test automation begins as just a
notion”
I t may be a simple notion by a testing team is soaring, but it is flying under the radar because
the initial notion was never really developed into a
resource that either got a hold of a commercial plan for how exactly test automation would help the
test tool that the company shelved, or perhaps project. The sheer activity of scripting is enough to
downloaded an open source automated test tool. provide a feeling of progress, and so it continues,
Maybe it is a notion by upper-level management that with no questions asked. This low pressure work
began a contract environment can
year with an ...the initial notion was never really developed be enjoyable,
ample budget and often useful
that they are all into a plan for how exactly test automation for a fledgling
too eager to burn automated
on massive tool
would help the project. testing initiative,
license purchases, but don’t be
service contracts and training. However acted upon and lulled into thinking this is how it will remain forever.
by whom ever it is acted upon, this notion is typically Eventually the project budget will begin to wear thin,
the same in all cases; it is a vague conception of test or the development and testing efforts will begin to
automation benefiting the testing efforts of the project. experience increased pressures, and alas management’s
So this notion is typically brought to bear through radar will pick up on the test automation effort. When
tool acquisition and proof-of-concept scripting. The this happens, if you’re not prepared, your automation
6 Inevitable Questions
1. What is the development level of effort?
proof-of-concept scripting, like an airplane on a effort will get shot out of the sky.
runway, results in the automation effort taking off into In order to remain airborne once you appear on
the friendly skies of the project with increased script management’s radar, there are 6 questions that you
development and maybe even the hiring of addition must be ready to provide answers to.
resources to support that effort. The automated testing
In fact, people like and want real Machines can do checking, but they can’t do
cases, they want to hear that you or testing, because they’re missing the ability
whoever else did the same to evaluate. Here, I don’t mean “evaluation”
for some of “Fortune 100” in the sense of performing a calculation
companies. Moreover they want to and setting a bit. I mean evaluation in the
know how, why and for how
sense of making a determination about what
much they had got that. Your
people value; what they might choose or
potential Clients want to
know risks, issues, lessons prefer. In the second way of interpreting the
learnt and benefits of your question, automating exploratory testing is
proposal from other guys who impossible—but using automation as part of
had got the experience. an exploratory process is entirely possible
www.veri
36 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2010
nference 2011
ifyati.com
December 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 37
Hot Topics in Automation
www.verifyati.com
40 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2010