The Evolution

of Test
Automation
Results from the 2018
Tricentis and TechWell
Survey

BROUGHT TO YOU BY:

The Evolution of Test Automation

Table of
Contents

Introduction
3
Current Automation Levels and Future Outlook
4
Which Tests Are Being Automated, and by Whom?
6
Benefits and Challenges
8
Test Automation Was Born out of Necessity
10
Test Automation Tools and Frameworks
12
Conclusion
14
2
The Evolution of Test Automation

Introduction

ver the past few years, agile and DevOps Tricentis and TechWell Corporation partnered
have moved from buzzwords to common on a survey to measure how widespread test
practice. So much so, in fact, that if you haven’t automation is today and gather data on how
already implemented agile or DevOps, you are much those numbers are expected to change
lagging behind. With the increasing popularity over the next few years. The survey also looks
of these collaborative, fast-moving, adaptable at how test automation is being integrated into
principles, the software development and testing teams; the benefits and challenges organizations
lifecycle has become lightning-fast. Because this face; the relationship between agile, DevOps,
lifecycle is so fast, test automation is mandatory to and test automation adoption; and the tools and
maintain the quality of your software. If you aren’t frameworks organizations are utilizing for their
automating testing, you’re probably losing sleep automation efforts.
worrying that something catastrophic is going to
happen and you won’t even know about it. The data collected from 173 software testers,
developers, QA decision-makers, and business
Organizations of every size now recognize the users was analyzed with an eye toward how
need to speed up their processes if they want to enterprise-level companies compare to smaller
stay competitive in the marketplace. While still the companies when it comes to implementing test
best choice for some scenarios, manual testing automation. The results suggest differences in the
processes are no longer sufficient on their own. rate of adoption as well as the challenges each
Test automation is now a must-have rather than a company faces.
nice-to-have.

3
The Evolution of Test Automation

Current Automation Levels and

Future Outlook
Survey results indicate widespread test automation adoption still has a way to go, with more than three-
quarters of respondents automating less than 50% of their testing. (Fig. 1)

What percentage of your total testing efforts are currently automated?

4
More than 50% 24%

41% to 50% 10%

31% to 40% 6%

21% to 30% 12%

11% to 20% 17%

1% to 10% 23%

0% 7%
0 5 10 15 20 25

Fig. 1 Percentage of Respondents

However, the results predict growth in automation over the next year. Almost half of respondents (44%)
anticipate 50% or more of their testing efforts will be automated in the next twelve months. Automating
50% of their testing is a massive undertaking and investment for each of these organizations. It requires
that a team or teams of automation engineers or business testers learn new tools to implement automation
successfully. At large organizations, millions of dollars will be invested to hire new employees so they won’t
fall behind in time to market and development over this year. Of the organizations that do not currently
automate any testing, 10% plan to start automating within the next year. (Fig. 3)

In the next 12 months, what percentage of your total testing effort do you expect to be automated?

More than 50% 44%

41% to 50% 12%

31% to 40% 8%

21% to 30% 15%

11% to 20% 10%

1% to 10% 8%

0% 3%

0 10 20 30 40 50

Fig. 2 Percentage of Respondents

4
The Evolution of Test Automation

If your organization doesn't currently automate testing, is it considering implementing test automation in the future?

We currently automate testing 57%

Yes, within the next year 10%

Yes, within the next two years 2%

We have no plans to implement test automation 2%

Not applicable 27%

Other 2%

0 10 20 30 40 50 60

Percentage of Respondents
Fig. 3

Our survey results show that a larger company size doesn’t equate to a higher rate of automation. Almost
a third (29%) of mid-size companies currently automate 50% or more of their testing, while only 17% of
enterprises are automating at that level. However, more than half of enterprise organizations (57%) report
that they expect to have reached the level of 50% or higher automation within the next year. This is evidence
that process and skill change is by far the largest hurdle in adopting test automation. Test automation tools
have significantly improved; the hard part is changing the people and the processes.

How Do You Measure Success?

Top 6 Key Performance Indicators
50%

43% 43%

25%

23%
17% 14% 14%
0%

Automated Number Requirement Defect Cost per Defect

test coverage of builds coverage density automated turnaround
passed/failed test time

5
 The Evolution of Test Automation

Which Tests Are Being

Automated, and by Whom?
Regression tests are by far the most frequently automated test type (86%), followed by repeated execution
(46%), load (29%), performance (29%), and cross-browser (29%) testing. (Fig. 4) Most testing (63%) is still
being done solely by QA, but we are seeing a rise in collaborative testing teams that include both QA and
developers (27%). In a small percentage of organizations (4%), developers are responsible for most of the
testing. (Fig. 5)

Which types of testing are you currently automating?

(Respondents could select more than one answer)

4
We don’t automate testing 0%
Regression testing 86%
Load testing 29%
Repeated execution 46%
Performance testing 29%
Database testing 17%
Cross-browser testing 29%
Not applicable 6%
Other 17%
0 20 40 60 80 100

Percentage of Respondents
Fig. 4

Who performs most of the testing at your organization?

QA 63%

5
Developers 4%
QA and developers share testing responsibilities 27%
Not applicable 0%
Other 6%

0 10 20 30 40 50 60 70 80

Percentage of Respondents
Fig. 5

“Some of us would like to implement automated testing, but

we have no budget or support from upper management.”
—Systems analyst/QA tester

6
 The Evolution of Test Automation

Our survey looked at whether there was a difference in the kinds of testing QA teams and development
teams are doing. We found that QA teams do most of the UI testing (79%) in many organizations, while dev
teams perform the bulk of the unit testing (73%). (Fig. 6) Development teams also frequently contribute to
automated testing efforts at the API level. (Fig. 7)

Which types of automated testing are your QA and Development teams primarily performing?
(Respondents could select more than one answer)

80
79% QA
70 73%

60 Development team

6
50
Percentage of 49%
Respondents 40

30
28%
20
20%
Fig. 6 19%
10 15%
12% 12%
8%
0
Unit testing API testing UI testing Not applicable Other

Beyond unit level testing, where do developers tend to contribute to automated testing the most?

At the API level 38%

At the UI level 14%

Not applicable 39%

Other 9%

0 5 10 15 20 25 30 35 40

Fig. 7 Percentage of Respondents

Test Automation: Love It or Hate It?

Survey respondents weigh in on their test automation experiences

“Test automation improves your ability “We are not at the point yet where our “We haven’t automated enough to see
to determine stability of code build and testing cycles have been significantly significant improvements as of yet, but
readiness for deeper testing.” reduced—obviously we are automating have high expectations as we roll out
—Automation consultant the wrong things, or not automating more widespread automation.”
very efficiently.” —QA team lead —QA analyst

7
The Evolution of Test Automation

We were also curious how test automation strategies differ among organizations. More than half of
respondents report that their organization’s test automation strategy is decentralized, in that several
smaller QA teams work on different projects, while 35% have a centralized automation strategy where the
individual teams report to a single manager. (Fig. 8) We also found that 61% of mid-size companies follow a
decentralized automation strategy, compared with 47% of enterprise-level companies.

Centralized (i.e., individual teams

11% report up to a single director
or manager)
35%
Would you describe your Decentralized (i.e., several smaller
test automation strategy as QA teams work on different projects
centralized or decentralized? throughout the organization)
54%

Fig. 8 Not applicable

Benefits and Challenges

As with any major undertaking, test automation comes with its share of benefits and challenges. Faster
testing cycles is the most cited benefit (63%), followed by improved test coverage (54%) and catching bugs
earlier (41%). (Fig. 9)

What benefits have you experienced as a result of adopting test automation?

(Respondents could select more than one answer)

We haven't seen any benefit 8%

Faster testing cycles/increased efficiency of testing 63%
Caught bugs earlier 41%
Improved test coverage 54%
Improved QA team's morale/motivation 36%
Faster time to market 28%
Improved information security 6%
Not applicable 4%
Other 9%

0 10 20 30 40 50 60 70 80

Percentage of Respondents
Fig. 9

8
 The Evolution of Test Automation

The major challenges reported by survey respondents include finding staff with the appropriate skill
set (55%), shifting organizational mindset to embrace test automation (43%), and creating data sets for
automated tests (43%). (Fig. 10)

What challenges have you experienced with test automation?

(Respondents could select more than one answer)

We haven't had any challenges 3%

Difficulty keeping track of automation efforts across multiple tools 27%

Difficulty finding staff with appropriate skill set 55%

Difficulty shifting team/organizational mindset to embrace test automation 43%

10 Difficulty finding or selecting the right tools for your needs

Difficulty creating test data for your automated tests

40%

43%

Not applicable/we don't automate testing 5%

Other 21%

0 10 20 30 40 50 60
Percentage of Respondents

Fig. 10

66%
of respondents say
Random functional automation is the
Finding: most valuable for reducing
manual testing efforts

9
 The Evolution of Test Automation

Test Automation Was Born

out of Necessity
Mainstream acceptance of agile methodologies has significantly changed the role of the tester. [1] The
iterative nature of agile, and the growing popularity of DevOps and its reliance on continuous integration
and delivery, necessitate a shift from primarily manual testing to at least some level of automated testing.
Our survey found that 72% of organizations are automating testing, 76% have adopted agile methodologies,
and 55% are using DevOps practices. (Fig. 11)

Which of the following practices has your organization adopted?

(Respondents could select more than one answer)

1
Automated software testing 72%
Agile methodologies 76%
DevOps 55%
Continuous integration 56%
None/not applicable 6%

0 10 20 30 40 50 60 70 80

Fig. 11 Percentage of Respondents

The level of DevOps adoption varies widely, with 23% of respondents currently using DevOps on 1% to
10% of projects and 20% using DevOps on at least half of their projects. (Fig. 12) Our survey found that
small companies have the highest and lowest rates of DevOps adoption, with 27% employing DevOps on
more than half of their projects and 21% not using DevOps at all. At the other end, 13% of enterprise-level
companies implement DevOps on more than half their projects, and only 10% haven’t started using DevOps.

What percentage of projects in your organization have adopted DevOps?

More than 50% 24%

41% to 50% 9%

31% to 40% 7%

21% to 30% 7%

11% to 20% 10%

1% to 10% 23%

0% 14%

Not applicable 10%

0 5 10 15 20 25

Fig. 12 Percentage of Respondents

10
 The Evolution of Test Automation

We found that DevOps is not the only new trend among our respondents. Our survey found that 34% of
respondents use behavior-driven development, 32% use test-driven development, and 32% use acceptance
test-driven development. (Fig. 13)

Which of these methods do you use in conjunction with test automation?

(Respondents could select more than one answer)

Test-driven development 32%

Behavior-driven development 34%

Acceptance test-driven development 32%

None/Not applicable 32%

10 15 20 25 30 35
0 5
Percentage of Respondents
Fig. 13

Does Maturity Matter?

How test automation benefits compare
at early-stage versus advanced-stage adoption
100%

88%
71% 68%
50%

46%
38%
20% 18% 0%
0%

Faster testing cycles Improved test Catch bugs earlier No benefits

coverage

1% to 10% automation 50%+ automation

11
The Evolution of Test Automation

Test Automation Tools

and Frameworks
All these tests aren’t going to automate themselves, so we wanted to find out what tools and frameworks
are the most popular. A large portion of respondents are using a mixture of open source and commercial
tools (47%), while 36% rely solely on open source tools. (Fig. 14)

5%
4%

Open source

Does your organization use 36% Commercial

open-source, commercial,
or a mixture of test Both open source and commercial
automation tools?
47% Custom-built
Fig. 14 8%
Not applicable

A small percentage of respondents (8%) exclusively use commercial tools. We asked those who aren’t using
open source tools why they aren’t using them. The answers included “open source tools don’t meet current
needs,” “it’s too hard to migrate the test data,” and “team members lack the appropriate skills.” (Fig. 15)

If you aren't currently using open source test automation tools, what’s preventing you from doing so?

We currently use open source test automation tools 52%

There is no open source technology currently available to meet my testing needs 6%

Significant organizational investment in commercial automation tools 3%

Our team lacks the skills needed to use open source tools 6%

It would be too difficult to migrate testing data 1%

Not applicable 25%

Other 7%

0 10 20 30 40 50 60

Percentage of Respondents
Fig. 15

12
The Evolution of Test Automation

Almost half of small companies (46%) report using open source tools exclusively. Enterprise-level companies
tend to vary their tool sources, with 20% using open source only, 50% using a mix of open source and
commercial tools, and 10% having their tools custom-built.

When it comes to which specific automation tools and frameworks are in demand, we discovered clear
favorites. Among automation tools, Selenium (73%), Cucumber (33%), and Appium (24%) lead the pack.
(Fig. 16) The popularity of Cucumber is not unexpected given that, as noted above, 34% of respondents have
adopted behavior-driven development. Among automation frameworks, JUnit (36%), TestNG (22%), and
NUnit (19%) are seeing the highest usage. (Fig. 17)

WHICH AUTOMATION TOOLS ARE YOU WHICH AUTOMATION FRAMEWORKS ARE

CURRENTLY USING? YOU CURRENTLY USING?
(Respondents could selec t more than one answer) (Respondents could selec t more than one answer)

Appium. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24% Galen Framework. . . . . . . . . . . . . . . . . . . . . . . . 2%

Cucumber . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33% Gauge. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1%
eggPlant. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3% JUnit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36%
HP UFT (Micro Focus). . . . . . . . . . . . . . . . . . . . 17% Jasmin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3%
IBM Rational Functional Tester. . . . . . . . . . . . . 1% Mocha. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6%
Katalon Studio. . . . . . . . . . . . . . . . . . . . . . . . . . . 4% NUnit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19%
Ranorex. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6% Robot Framework. . . . . . . . . . . . . . . . . . . . . . . 12%
Sahi. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2% Redwood HQ. . . . . . . . . . . . . . . . . . . . . . . . . . . . 1%
Selenium . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73% Serenity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3%
Silk. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2% Spock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2%
Squish. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1% TestNG. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22%
Telerik. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3% Not applicable. . . . . . . . . . . . . . . . . . . . . . . . . . 23%
TestComplete . . . . . . . . . . . . . . . . . . . . . . . . . . . 8% Other . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26%
Tricentis Tosca . . . . . . . . . . . . . . . . . . . . . . . . . 16%
Watir. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5% Fig. 17
WorkSoft . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2%
Not applicable. . . . . . . . . . . . . . . . . . . . . . . . . . . 6%
Other . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31%

Fig. 16

“We don’t use automation to ‘reduce’ manual testing;

we use it to augment it and allow our human testers to
focus on new functionality and high-value testing work.”
—Automation consultant

13
The Evolution of Test Automation

Conclusion

est automation is a necessity in today’s fast- In order to fully benefit from test automation,
paced development environments. Whether there will need to be a major cultural reset in
you work for a tech company or a highly regulated many organizations. Managers and test teams will
government entity, if you aren’t automating your need to communicate openly about the realities of
testing now, you soon will be. As the survey results automation to dispel long-held misconconceptions.
show, over the next twelve months, test automation Faster test cycles, improved test coverage, and
efforts are going to continue to gain traction. higher-quality software do not come without a cost.

As test automation rapidly scales up at enterprises As agile and DevOps become commonplace
of all sizes, we are seeing a growth in collaborative practices in many organizations, development
testing teams, as well as an increasing number cycle times are only going to get faster. A push
of developers doing testing and testers learning to implement test automation now will save your
technical automation skills. [1] organization frustration (and money) later.

References
[1] TechWell Corporation, State of the Software Testing Profession Survey Report

14
The Evolution of Test Automation

With the industry’s #1 Continuous Testing platform, Tricentis is

recognized for reinventing software testing for DevOps. Through agile
test management and advanced test automation optimized to support
150+ technologies, we provide automated insight into the business risks
of your software releases—transforming testing from a roadblock to a
catalyst for innovation. The result is accelerated software delivery speed,
improved cost efficiency, and reduced business risk.

Tricentis is the only vendor to achieve “leader” status in all three top
analyst reports (i.e., the “Triple Crown.”) This honor is based on our
technical leadership, innovation, and a Global 2000 customer base of
1600+ companies, including global enterprises such as Allianz, ANZ Bank,
Cisco, Dolby, Experian, First Data, HSBC, Merck, Office Depot, Samsung,
Swiss Re, Starbucks, Telstra, UBS, Vodafone, Whole Foods, and WorldPay.
Customers rely on Tricentis to achieve and sustain test automation rates
of over 90 percent—increasing risk coverage while accelerating testing to
keep pace with Agile and DevOps.

Tricentis has a global presence in Austria, Australia, Belgium, Denmark,

Germany, India, Netherlands, Singapore, Switzerland, Poland, United
States and the UK. To learn more, visit https://www.tricentis.com or follow
us on LinkedIn, Twitter, and Facebook.

For more information, visit www.tricentis.com.

15
2570 W El Camino Real, Suite 540, Mountain View, CA 94040
+1-650-383-8329 | tricentis.com