TEST AUTOMATION

eGuide

COPYRIGHT 2016
 TEST AUTOMATION
eGuide
3
Between the mainstream adoption of agile practices and the growing DevOps movement, you may have noticed the word
Bringing the Value of
“continuous” popping up a lot: continuous integration, continuous testing, continuous delivery, continuous deployment.
Your Test Automation
Efforts Front and Center
In today’s continuous-everything, app-centric world, test automation isn’t a nice-to-have; it’s a must-have. To stay competitive in the
marketplace, organizations must automate as many tests as possible to release products as quickly as possible. To maintain an
5 edge in the job market, testers must grow their technical skills to fit into this rapid-release, automated environment.
Why Your Test
Automation Efforts
Should Tackle Data First

7 In this Test Automation eGuide

Selenium: The Open
Source, the Myth, the Bringing the Value of Your Test Automation Efforts Automation That Learns: Making Your Computer
Legend Front and Center Work for You
Once you’ve adopted test automation, you should determine whether It’s been suggested that because automation can only do checking,

10 it’s actually yielding the expected benefits—and you’ll want to keep

these benefits visible to stakeholders to reinforce the value. A metrics
automation cannot learn. But if you’re talking about the acquisition
of knowledge through experience and study, Jeremy Carey-Dressler
Automation That Learns: dashboard aligned with the organization goals and business objec- believes automation can, in fact, learn—with a tester adding some
Making Your Computer tives shows you’re on the right track. additional code to capture and analyze more available data.
Work for You
Why Your Test Automation Efforts Should Tackle The Top 5 Test Automation Mistakes
13 Data First Test automation is a valuable process, but it can be difficult to inte-
Automation projects often start by tackling the technical issues, but grate into your existing test strategy. Hundreds of people weighed in
The Top 5 Test
Linda Hayes says a specific data environment should be established about the biggest mistakes they’ve made when automating. Here are
Automation Mistakes
first. If you can’t control, define, and predict your data, you won’t five of the most popular answers—and advice on how we can stop
have the repeatability that makes test automation practical—but it repeating these mistakes.
14 makes sense for manual testing, too.
The Buzz on Test The Buzz on Test Automation
Automation Selenium: The Open Source, the Myth, the Legend Some of the industry’s best and brightest share their thoughts on
Many people wonder what it means that Selenium is open source, Test Automation and its role in agile, QA, and the shift to continuous
15 and further, what the community element of that paradigm brings to integration and delivery.
the table. This article addresses some of the common misconceptions
Additional Resources for about that situation, as well as details some of the benefits of the Additional Resources for Test Automation
Test Automation community behind a product like Selenium. Invaluable resources to keep you, your organization, and your practic-
es at the forefront of the Test Automation movement.

COPYRIGHT 2016 2
 TEST AUTOMATION
eGuide
3
Bringing the Value of Bringing the Value of Your Test
Automation Efforts Front and Center
Your Test Automation
Efforts Front and Center

5 By Michael Sowers
Why Your Test
Automation Efforts Sometimes, in order to get everyone on board with adopting test au-
Should Tackle Data First tomation, you have to build a business case to highlight the poten-
tial benefits. Once you’ve convinced the organization to make that
7 investment, you should determine whether it’s actually yielding the
predicted benefits—and you’ll want to keep these benefits visible to
Selenium: The Open
key stakeholders to reinforce the value.
Source, the Myth, the
Legend
How do we frame the quantitative and
10 qualitative benefits of test automation
Automation That Learns:
Making Your Computer
in a way that links to the organizational
Work for You objectives and business goals?
13 We tend to have many metrics in place to track the progress of
testing and measure the degree of readiness or risks in our software
The Top 5 Test
products. These metrics include test effectiveness, software quality,
Automation Mistakes
test status, resources, issues, and so forth, but what about metrics
for the test automation platform? How do we frame the quantitative
14 and qualitative benefits of test automation in a way that links to the It’s important to design the dashboard at the beginning of your
The Buzz on Test organizational objectives and business goals? This is an important automation project in order to ensure that the approaches and
Automation element of planning and implementing our automation architecture. tools you choose will yield the measures and metrics you need. Five
dimensions will usually cover the range of information that the team
15 One of the approaches that has worked well for organizations I’ve and key stakeholders require, but of course you can trim or expand
worked with is a test automation dashboard. A useful dashboard these to meet your needs:
Additional Resources for should reflect values that align with the organization and business
Test Automation goals, be agile and adaptable, and contain information that is action- Capabilities include the automation platform features, interface
able and meaningful, with just enough information that it’s still easy capabilities, integrations, types of systems under tests that are sup-
to use and maintain. ported, and scripting languages supported.

COPYRIGHT 2016 3
 TEST AUTOMATION
eGuide
3 Usage includes the number of users, teams, and projects; the fre- Challenges capture the constraints or issues with the tools or auto-
Bringing the Value of quency of use of a given automation feature; the rate of automation mation platform.
Your Test Automation adoption; and the frequency of use of test suites.
Efforts Front and Center Investment encompasses the costs (in time or money), such as
Benefits are metrics such as rate of unit test, story test, and non- defects in the automation platform, licenses, platform maintenance,
5 functional automation; ease of learning and use of the automation training and retraining, operation (including hosting, servers, etc.),
platform; scripting or execution efficiency; degree of automation the time to build, maintain, and execute automated tests, and the
Why Your Test suite code coverage; speed and efficiency of the test automation time to investigate defects in the automation test infrastructure or
Automation Efforts infrastructure; hours saved versus manual testing; number of addi- test suites.
Should Tackle Data First tional test cycles achieved per sprint; percent of increased require-

A useful dashboard should

ments, functional or nonfunctional testing, or structural coverage
7 achieved; and how much earlier defects are found.
Selenium: The Open
Source, the Myth, the
reflect values that align
Legend
with the organization
10 and business goals, be
Automation That Learns:
Making Your Computer agile and adaptable,
Work for You
and contain information
13 that is actionable and
meaningful.
The Top 5 Test
Automation Mistakes

14 The dimensions and associated metrics that are applicable can be

The Buzz on Test displayed in a graphical dashboard format, with simple or more
Automation complex measures associated with each “dial.” I find trend mea-
sures more useful than absolutes.
15
The critical element of any dashboard is ensuring that it’s aligned
Additional Resources for with the organization’s goals and business objectives, and having
Test Automation just enough measures in place to guide the continual refinement of
the automation platform. {end}

COPYRIGHT 2016 4
 TEST AUTOMATION
eGuide
3
Bringing the Value of Why Your Test Automation Efforts
Should Tackle Data First
Your Test Automation
Efforts Front and Center

5 By Linda Hayes
Why Your Test
Automation Efforts
Should Tackle Data First
There is no question that automation poses technical challenges;
you have to get your tool and application to play nice together
If you can’t control, define, and
before automation is even possible. But the most difficult obstacle to predict the state of the data, you
7 success is more mundane: It’s the data. If you can’t control, define,
and predict the state of the data, you won’t have the repeatability
won’t have the repeatability that
Selenium: The Open
Source, the Myth, the that makes automation practical. makes automation practical.
Legend
It starts with the data in the test environment. For a hospital, this gers, or make payments. Defining this data is an integral part of

10 includes which rooms are available, what doctors are on staff, and
which medications are in inventory. For an airline, this means know-
developing your test cases. For a manual test, you can adapt on the
fly by looking for available rooms or experimenting with different
Automation That Learns: ing what routes are flown, which flights are scheduled, and how flights, but an automated test must know the values in advance.
Making Your Computer many seats are available for each trip. You have to control the state Trying to adapt during automated execution is theoretically possi-
Work for You of the data when you start execution because your test must know ble, but usually impractical; it leads to complex logic that introduces
what to expect. ambiguity and instability.
13
Next comes the data you provide during the test. As you execute, Finally, you must be able to predict the result. This is harder than
The Top 5 Test
you will create or transform the data: admit patients, book passen- it sounds. You can’t get away with saying “verify that the value is
Automation Mistakes

14
The Buzz on Test
Automation

15
Additional Resources for
Test Automation

COPYRIGHT 2016 5
 TEST AUTOMATION
eGuide
3 correct” like you may be able to in
Bringing the Value of a manual test; you have to specify
Your Test Automation precisely what the correct response
Efforts Front and Center is. Manual testers have so much trib-
al knowledge that they are used to
5 winging it, and reducing experience
to exact values takes extra thought
Why Your Test and effort.
Automation Efforts
Should Tackle Data First Too often, automation projects
start by tackling the technical is-
7 sues first, and only once execution
becomes possible do they start to
Selenium: The Open
trip over the issues with the data.
Source, the Myth, the
In my experience, the data issues
Legend
should come first because they take

10 longer to resolve and may involve

other departments and resources.
Automation That Learns: Subject matter experts, database
Making Your Computer administrators, network engineers,
Work for You and hardware support teams may have to be orchestrated before controlled, predictable data environment, then not only will manual
a robust and repeatable solution can emerge. Waiting until you are testing be far more efficient, but the transition to automation will be
13 starting the test cycle is too late. significantly less painful.

The Top 5 Test

To those who argue that the technical issues have to be resolved be- In the end, if you have resolved the data challenges but find the
Automation Mistakes
fore the data issues matter, I say that having a comprehensive data technical issues are insurmountable, you have still gained produc-
strategy enables both manual and automated testing. While manual tivity. But if you resolve the technical issues first and then cannot
14 testers can and do adapt on the fly, the truth is that this takes time. address the data issues, you have lost time and effort. It may not be
The Buzz on Test The majority of most manual test effort is spent searching for or cre- as fun or as satisfying as slaying technical dragons, but addressing
Automation ating the conditions required for the test. If everyone can rely on a the data leads to ultimate success. {end}

15
Additional Resources for
In my experience, the data issues should come first
Test Automation because they take longer to resolve and may involve
other departments and resources.
COPYRIGHT 2016 6
 TEST AUTOMATION
eGuide
3
Bringing the Value of Selenium: The Open Source, the
Myth, the Legend
Your Test Automation
Efforts Front and Center

5 By Brian Van Stone

Why Your Test
Automation Efforts When it comes time to select a tool for test automation, most orga- I’m not quite sure on the origin of the myth that open source soft-
Should Tackle Data First nizations will create a list of candidates and whittle it down based on ware is not worthy of the enterprise, but the IT community is certain-
various measures: cost, features, alignment to existing standards, ly coming around on this one.
7 suitability for the software development lifecycle, and more. With the
wide-scale adoption that Selenium has received in organizations of Open source software is slow to release updates and fixes
Selenium: The Open
varying sizes and across many industries, the tool has found its way While I won’t vehemently shout “False!” at anyone who says this
Source, the Myth, the
onto most company’s radars. to me, I will debate the idea quite heavily. The hard part of talking
Legend
about this misconception is that it sounds feasible. A community of
I’m not quite sure on the origin of the myth
10 that open source software is not worthy of the
developers who have other full-time jobs and aren’t getting paid to
develop the software? Of course people assume that they’ll release
Automation That Learns: enterprise, but the IT community is certainly slower and be less responsive!
Making Your Computer coming around on this one.
Work for You
But from one place to another, this can raise some very interest-
13 ing questions. Many wonder what it means to them that Selenium
is open source, and further, what the community element of that
The Top 5 Test
paradigm brings to the table. Here, I address some of the common
Automation Mistakes
misconceptions of this situation, as well as talk about some of the
real benefits of the community behind a product like Selenium.
14
The Buzz on Test The Myths
Automation Open source software is not enterprise-worthy
Enterprise-quality open source software has existed for a very long
15 time. I’m a big fan of the Apache Software Foundation (ASF), and
you don’t need to look any further to find dozens of examples of
Additional Resources for high-quality open source software that have become competitive in
Test Automation the industry—and, in a handful of cases, industry standard. ASF was
releasing tools like Tomcat and Ant as early as 2000.

COPYRIGHT 2016 7
 TEST AUTOMATION
eGuide
3 Luckily, this is not the reality. While the scenario described here is Open source software compromises security
Bringing the Value of entirely possible, the same can happen with proprietary software. This one I don’t have a problem shouting about. I’ll grant that in
Your Test Automation Also, the developers behind Selenium and similarly successful open organizations with lots of proprietary information it makes sense to
Efforts Front and Center source projects have these challenges much higher on their list of have blanket policies about protection of source code. That being
priorities than you might guess at first. Couple this with the fact that said, I have never heard an argument approaching legitimate claims
5 you can see the source, find the bug, fix it, build it, use it, and share it that supported this point.
with the community, and now you’re firing on all cylinders. If a bug is
Why Your Test problematic enough to frustrate you, it’s frustrating plenty of others. Security is about a lot more than software. It’s about arsenals of
Automation Efforts Call enough attention to it and the community will respond in force. software, and systems designed to protect that software and your in-
Should Tackle Data First formation. I’m not sure what angle people want to take on this, or if

A large community of driven

this rumor is simply an artifact of a fear of the unknown, but let’s be
7 quite clear: An implementation of Selenium, or some other reputable
Selenium: The Open people will generate good open source software, does not inherently compromise the security
of your organization or its information.
Source, the Myth, the
Legend ideas. This isn’t a maybe or a
likelihood; it happens. The Reality
10 With these open source myths out of the way, let’s turn the conver-
sation toward some of the real benefits of the community behind Se-
Automation That Learns: lenium. While some can already be inferred from the content above,
Making Your Computer there are a few notable ones that deserve some special attention.
Work for You
A strong community drives rapid feedback
13 The developers of Selenium are users of the product, and vice versa.
This is incredibly powerful. When I’ve been in a position in the past
The Top 5 Test
to have sales engineers demo a product to me that seems useful to
Automation Mistakes
IT organizations, I’ve often asked, “Do you use the tool in house?”
The question seems pretty innocent, but it has some strong implica-
14 tions. Do you trust the software enough to use it? Do you believe in
The Buzz on Test the power of the tool enough to want to leverage it?
Automation
History has shown us that people build better things, whatever they
15 may be, when they are strongly invested in the success of their prod-
uct and when they believe in it. Open source software does this for
Additional Resources for us, seemingly for free as a byproduct of the process.
Test Automation
More concretely, this means that the community behind an open
source product is strongly incentivized to provide rapid and con-

COPYRIGHT 2016 8
 TEST AUTOMATION
eGuide
3 tinuous feedback in order to drive long-term improvement in the
Bringing the Value of solution that is delivered. It’s actually that simple.
Your Test Automation
Efforts Front and Center A thousand heads are better than one
There are a lot of smart people in this world. Why not leverage that? A
5 large community of driven people will generate good ideas. This isn’t
a maybe or a likelihood; it happens. Selenium and other open source
Why Your Test projects are what they are today because of their communities.
Automation Efforts
Should Tackle Data First Thanks to the ingenuity, devotion, and real-world experience of
thousands of users, Selenium is constantly progressing toward being
7 a better and more comprehensive browser automation solution. And
this isn’t just a result of the vision of a handful of people, but rather
Selenium: The Open
a reactive and evolutionary process grooming Selenium into the
Source, the Myth, the
solution that real people, doing real work, want it to be.
Legend

10 Many users, many solutions

While a community of active users can be crucial in growing a
Selenium. My experience was extremely positive, the topics of the
talks were interesting and ran the full spectrum between business
Automation That Learns: product into the solution the market needs, this community is also a and technology, and the organizers really do a wonderful job of
Making Your Computer massive 24/7 support network for IT professionals using the product. making all of the attendees feel like they’re participating in some-
Work for You I personally think this is the most obvious and direct benefit of the thing great. If you want to experience the power of the Selenium
open source community. This is so powerful that it happens across community firsthand, there is no better way.
13 all of human interaction on the web. Just think about the Stack
Exchange network, MSDN, or even something like Pinterest. People At the end of the day, open source software can experience all the
The Top 5 Test
with similar goals band together on the web every day to collectively same problems we’re used to with proprietary software, and it isn’t
Automation Mistakes
solve challenges or share knowledge. the solution for everything—but this isn’t because community-driv-
en software is bad. Software development is a creative process, and
14 While it is clear that this trend is not unique to open source commu- under any paradigm it requires iterative improvement. The nature
The Buzz on Test nities, open source software development more strongly encourages of software creation necessitates support and feedback and is aided
Automation this type of community interaction than other methods. It’s a classic by the agility to implement this feedback in a reasonable amount
example of not having to reinvent the wheel when someone else has of time. A strong community of invested users supports these core
15 already solved your problem and shared the solution, and now there tenets of software development in compelling ways.
is more content to draw from than ever before.
Additional Resources for If you’re still skeptical, attend a Selenium conference or try your
Test Automation Open Source: It Takes a Village hand at contributing code or bug reports. I’m sure you’ll find this
I had the pleasure of attending the Selenium Conference, and I group to be very receptive, and with any luck, the community will
would strongly recommend this conference to anyone interested in grow one user stronger.

COPYRIGHT 2016 9
 TEST AUTOMATION
eGuide
3
Bringing the Value of Automation That Learns: Making
Your Computer Work for You
Your Test Automation
Efforts Front and Center

5 By Jeremy Carey-Dressler
Why Your Test
Automation Efforts I have heard it suggested that because automation can only do
Should Tackle Data First checking, automation cannot learn. By “learning,” I will use the
dictionary definition, “the acquisition of knowledge or skills through
7 experience, study, or by being taught.” If I claim that the act of pro-
gramming was teaching a machine, and that was how automation
Selenium: The Open
learns, it would not be a particularly meaningful claim. Instead, what
Source, the Myth, the
I am talking about is the acquisition of knowledge through experi-
Legend
ence and study. I believe automation can, in fact, learn, with a tester

10 adding some additional code around the automation.

Automation That Learns: Recording Data

Making Your Computer One high-level example of a computer learning is simply using a
Work for You database. A computer can clearly “learn” through the experiences
it has, then record that information into a database. For example, a
13 database can log how long a test runs by capturing those run times.
Each test execution will be a new row, and over time the computer
The Top 5 Test
will “know” about how long a test will take. The automation can
Automation Mistakes
then detect if a test is broken by checking a test’s execution time
compared to the average. For that matter, the test could also notice
14 a trend of a test taking longer and longer and perhaps notify the
The Buzz on Test team that the system is getting slower. Obviously these are not In a real-world example, at my current company we have a load
Automation evaluations that replace human intelligence, but they are heuristics a test that runs on a nightly basis. We run light load tests to see if the
human might care about. system has slowed down in any significant ways. Currently a human
15 must review the results, but we plan on adding a formula to decide

Additional Resources for

I believe automation can, in fact, learn, if the change in performance is too drastic and should be reviewed
by a human. The formula would take into account previous runs
Test Automation with a tester adding some additional and decide if the trend is statistically significant or not. Further into
code around the automation. the future we want to tie this to builds so that we know which code
changes are likely the cause of any slowdowns.

COPYRIGHT 2016 10
 TEST AUTOMATION
eGuide
3
Bringing the Value of
Complex search, such as what Google does, is an area that is difficult
to test because of the lack of an oracle. One way to handle this prob-
Given enough searches, you
Your Test Automation
Efforts Front and Center
lem is to take search results between runs and see how different the
results are—this way you can judge if a new version of search is an
demonstrate a trend showing
improvement or not. if the search feature, on
5 average, provided better
One way to do the validation is by looking at the frequency rate of
Why Your Test
Automation Efforts
words you used to search. So, say you search for “Dog” and “Dog”
appears twenty-five times in the top ten items that came back from
results to a user.
Should Tackle Data First your search. You can store this data, then on the next version of
search try “Dog” again. With this, you can create a trend line to individual test that passes or fails, you can consider the entire suite
7 demonstrate whether the search is improving. Maybe the next time of tests as one large test. That is to say, given enough searches, you
you search for “Dog” you find the word twenty-four times. While this demonstrate a trend showing if the search feature, on average, pro-
Selenium: The Open
test might appear like a failure, it doesn’t mean the search engine vided better results to a user.
Source, the Myth, the
is worse overall. Rather than thinking of each search term as an
Legend
In a previous job, we had a system that would take a screenshot of

10 any failed tests. We would manually look at these for patterns before
doing any serious triage because it often was the fastest way. As
Automation That Learns: the number of tests increased, we noticed many fails had similar
Making Your Computer failing screenshots, but the screenshots often were not exactly the
Work for You same. There would be five tests that hit HTTP server errors, but there
would be nothing the automation could find in common because the
13 tests were executed in different browsers.

The Top 5 Test

One of my brilliant coworkers came up with the idea of using the
Automation Mistakes
average luminosity (brightness) of the screenshot and sorting based
on that. Luminosity worked because browsers generally render
14 brightness the same way, while font often renders differently. Using
The Buzz on Test the average luminosity, which is just a number, as a sorting method,
Automation all similar images would be grouped together and it would become
apparent that those tests failed for the same reason, regardless of
15 browser. While we didn’t do this, the next step would have been to
compare the resulting luminosity value to previous test-run image
Additional Resources for luminosity. It could, with some additional safety check, then mark
Test Automation the test as “failed like previous run” and thus save having a tester
need to reexamine the results.

COPYRIGHT 2016 11
 TEST AUTOMATION
eGuide
3 Learn at Runtime
Bringing the Value of There are other ways to leverage knowledge around testing. I
Your Test Automation cowrote a system that automatically did first-level triage on all the
Efforts Front and Center automated tests. We noticed there were patterns we found in our
triage that occurred over and over again. We had an e-commerce
5 website akin to Amazon that had a variety of interesting code paths.
You might use a credit card, or you might use a gift card. You might
Why Your Test order in English, or you might order in Portuguese. You might use
Automation Efforts the search page, the product details page, and cart, but not check
Should Tackle Data First out. When we hand-triaged these tests, we would notice a pattern,
like “All these tests are failing at search . . . better go test that.”
7 It would take thirty minutes to notice the pattern and find the bug.
These were things the automation basically knew about each test,
Selenium: The Open
and it also knew what the ultimate test result was.
Source, the Myth, the
Legend

10 We noticed there were patterns

Automation That Learns:
we found in our triage that
Making Your Computer occurred over and over again.
Work for You
So we developed a system where the automation would consider all
13 the inputs and all the outputs and determine the most likely cause.
Inputs would include things like the name of the test and any pa-
The Top 5 Test
rameters in our parameterized tests, like payment method. Outputs
Automation Mistakes
would include pages visited, the exception type and message, and
page the test failed on. We then would look at all the fails and decide
14 what was common and assign that as the most likely cause. This did intelligence. There are many other ways you can have your tools
The Buzz on Test not mean a tester didn’t have to manually review the results; it just “learn” everything from production traffic to what tests are flaky.
Automation took a lot less time. How much of that you should automate depends on your environ-
ment and needs.
15 Summing It All Up
There is lots of data that automation throws away or ignores, Finally, when you find yourself not getting enough value from auto-
Additional Resources for preventing it from learning. If you capture this sort of data when it mation, or you’re spending too much time on analysis work a com-
Test Automation makes sense, you can analyze it, and perhaps even have the comput- puter could do, it might be best to see if you can get your computer
er check for things that are obviously wrong. Be careful, though— to learn on its own. {end}
“smart” automation cannot completely replace the need for human

COPYRIGHT 2016 12
 TEST AUTOMATION
eGuide
3
Bringing the Value of
Your Test Automation
The Top 5 Test Automation Mistakes
By Melissa Tondi
Efforts Front and Center

5 I’ve heard from hundreds of people about the biggest mistakes Not Knowing What to Automate
they’ve made in their pursuit of test automation success. Here are The goal should be to automate the right tests. In our selection pro-
Why Your Test five of the most popular answers—and recommendations on how cess, we determine the build verification tests and smoke tests first
Automation Efforts we can stop repeating these mistakes. and automatically make them candidates to automate. These are by
Should Tackle Data First far the most executed tests throughout a cycle for us, so this aligns
Automating Everything well with our strategy. However, we also have clear criteria to ensure
7 Not all tests can or should be automated, so I’m surprised when there is some thought process around determining whether the ROI
I hear that teams are still being measured on the number of tests makes automation worth it.
Selenium: The Open
automated versus the overall value the automation is bringing to the
Source, the Myth, the
Legend team. A better way to approach this is to have a selection process to Thinking Automation Takes Too Much Time
quickly determine when something should be considered for auto- Don’t get caught in the trap of thinking it takes less time to run a test

10 mation. When you are held to a meaningless metric, your creative

license is essentially removed, and you may not be performing the
manually than to automate it. Sometimes that’s the case, but be real-
istic with the number of times you run the test. Even though it takes
Automation That Learns: most valuable tests. only a few minutes to run it manually once, if you run it many times
Making Your Computer within a sprint, performing the test manually may exceed the time it
Work for You Having an Unclear Strategy takes to automate it. The key is to make sure the tests are meaningful
If you don’t have a clearly defined automation strategy, it will be to the goals of the project team and their results are provided quickly.
13 harder to measure success across teams. Our strategy focuses on
automating our build verification tests first, smoke tests second, Treating Automation as a Separate Role
The Top 5 Test
then a “dealer’s choice” of what makes the most sense for the spe- I see a lot of organizations use their automation engineers in a
Automation Mistakes
cific solution. The key is that the first two are standard across teams, matrixed role where they only automate the tests a manual tester
and more measurable and meaningful. instructs them to. On the other hand, I’ve also seen automation
14 engineers look down on traditional testing activities and refuse to
The Buzz on Test do anything other than automate tests. Somehow, we decided that
Automation A good tester will utilize all tools automation is so specialized that it’s more valuable than other kinds

and technologies available of testing. These engineers should still be intimately involved in the
15 product they are testing.
to ensure their solutions are
Additional Resources for
Test Automation comprehensive and efficient. A good tester will utilize all tools and technologies available to
ensure their solutions are comprehensive and efficient. Test automa-
tion is a valuable process, and if you avoid these common mistakes,
you’ll be able to take full advantage of its effectiveness.

COPYRIGHT 2016 13
 TEST AUTOMATION
eGuide
3
Bringing the Value of
Your Test Automation
The Buzz on Test Automation
Efforts Front and Center
INSIGHT FROM AROUND THE INDUSTRY

5 On Automation ROI: On Automating the Right Tests: By insisting that real

“What we found from customers who have “A lot of time is wasted on overly simplistic
Why Your Test gone the test automation route is, while they tests automated just for the sake of automat- testing activities cannot
Automation Efforts
Should Tackle Data First
have been able to speed up the amount of ing a test. This is especially true of automat- be automated at all,
testing they’re doing, decreasing the amount ed tests that attempt to emulate customer
of time it takes to do that testing, they’ve interactions, or GUI automation. Well-de- testers are left out of
7 actually been taking that cost savings and signed automated tests with specific purpose the continuous testing
re-investing it into the application.” can be very helpful in a project, and can also
Selenium: The Open
» Josh Galde free up a tester’s time to test and find ways
conversation.
Source, the Myth, the
to improve their product.”
Legend
On Agile Test Automation: » Bj Rollison
On Automation and Enterprise IT:
10 “It’s very difficult to imagine a project where
there is no test automation being done, On the Human Side of Automation: The design of test automation tools may
Automation That Learns: especially if they’re practicing agile because “If we divorce the notion of ‘automation’ from already have reached a boundary where
Making Your Computer agile embraces test automation right from the notions of ‘test cases’ and ‘test scripts,’ it can no longer cost-efficiently keep pace
Work for You the get go.” we can begin to think of automation as a with change in enterprise IT systems, and
» Kalyan Konda judicious use of technology to help humans where alternative solutions can no longer be
13 do their jobs; in our case, those humans are ignored.
On Domain Expertise: testers. This broadens our idea of automation » Martin Ivison
The Top 5 Test
“I want people to have thoughts and ideas to include non-test-case-based tools that will
Automation Mistakes
on how to test. I want my automation people assist humans in performing their testing.” On Failing the Right Way:
to be able to write their own test cases. I » Paul Grizzaffi Using automation as an example, we may be
14 need them to understand the domain to able to execute quicker once the initial tests
The Buzz on Test make sure what they’re automating is the On the Testers’ Role in Automation: have been written, but that means we fail
Automation right thing.” “By insisting that real testing activities can- faster, not that we find the right failures in
» Mary Thorn not be automated at all, testers are left out the code quicker.
15 of the continuous testing conversation. That » Leanne Howard
introduces the risk that important improve-
Additional Resources for It’s very difficult to imagine ments in test automation will be shaped
Test Automation a project where there is no more by software engineers than by the
test automation being done. testing community.”
» Stefan Friese

COPYRIGHT 2016 14
 TEST AUTOMATION
eGuide
3
Bringing the Value of
Additional Resources
Your Test Automation MORE INFORMATION FOR SOFTWARE PROFESSIONALS
Efforts Front and Center

5 StickyMinds is home to thousands of software testing resources,

CLICK
including articles, Better Software magazine articles, conference
HERE
Why Your Test presentations, and interviews with industry notables.
Automation Efforts
Should Tackle Data First
N A R R O W Y O U R S E A R C H T O A S P E C I F I C T Y P E O F C O N T E N T:
7
StickyMinds Articles Better Software Magazine TechWell Conference Interviews
Selenium: The Open Articles Presentations
StickyMinds articles cover a Each year, TechWell interviews
Source, the Myth, the wide range of software testing Better Software magazine is a Couldn’t make it to a STAR software dozens of software professionals
Legend topics including test automation, digital quarterly filled with expert testing conference? TechWell including well-known thought
test management, test design analysis, how-to articles, and real- conference presentations are leaders, seasoned practitioners,
10 techniques, agile testing, test
process improvement, test tools,
world case studies covering all
aspects of software development.
available to StickyMinds members
soon after a conference ends.
and respected conference
speakers. Click here to read,
Automation That Learns: and much more. Click here to Click here to join StickyMinds and Click here to join StickyMinds and listen to, and watch interviews
Making Your Computer read test automation articles on access test automation articles access test automation conference with test automation experts.
StickyMinds. from Better Software magazine. presentations.
Work for You

13
The Top 5 Test Conferences Providing Innovative Ideas & Solutions for Test/QA Professionals
Automation Mistakes

14 • Keynotes
The Buzz on Test • Tutorials
Automation • Training Classes
• Networking Sessions
15 • Conference sessions
• The Testing Expo
Additional Resources for • The Testing & Quality
Test Automation Leadership Summit

COPYRIGHT 2016 15