The Role of Testing in A DevOps Transformation
The Role of Testing in A DevOps Transformation
R E A L I T Y :
Table of Contents
E X ECU TIV E SU M MA RY
Purpose and Methods. . . . . . . . . . . . . . . . . . . . . . . . 3
Demographics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Summary of Key Findings. . . . . . . . . . . . . . . . . . . . . 3
2
2019 Tricentis Survey Report
EXECUTIVE SUMMARY
3
2019 Tricentis Survey Report
We asked respondents how far along they were with DevOps transformation and found that progress
towards transformation was positively correlated with the level of importance placed on testing. In
organizations where DevOps transformation is complete, 87% of respondents said testing is critical to
successful DevOps transformation, compared to 49% of testers in organizations that haven’t undergone
DevOps transformation and it isn’t a priority.
4
2019 Tricentis Survey Report
We also noted that testers were significantly more likely to feel that their businesses prioritized testing
in organizations that have completed DevOps transformation. Out of these respondents, 80% said the
business prioritizes testing, compared to 26% in organizations where DevOps isn’t a priority.
The finding that organizations who have successfully completed DevOps transformation are 4x more likely
to prioritize testing indicates that testers are a crucial contributor to the DevOps transformation process.
Yes No
80
80%
70
60
57% 56%
50
Percentage of 46%
Respondents 40 42% 41%
30
26%
20
10 13%
0
DevOps transformation DevOps transformation DevOps is a priority, DevOps is not a priority
is complete is in progress but we have not started
In striking contrast, of those organizations that haven’t prioritized DevOps, a majority (63%) have automated
less than 10% of testing effort and a mere 3% have reached 50% automation or more.
Still, not all DevOps organizations are successfully scaling test automation. In organizations where
DevOps transformation is complete, a significant number (27%) report automating just 10% or less of
their testing efforts—the same average rate reported by organizations where DevOps transformation is
in progress.
This finding is corroborated by the findings of the 2019 Gitlab Global Developer Survey [1], which found that
despite the increased delivery speed that DevOps delivers, most DevOps teams encounter the majority of
delays during the testing stage. Half of those surveyed called out testing as the biggest source of delay in the
development process, suggesting that scaling test automation is a challenge even for mature DevOps teams.
5
2019 Tricentis Survey Report
This is similar to the finding in the 2017-2018 World Quality report [2] that indicates average test automation
levels are around 20%, as well as the current year’s report [3] that found rates are between 14% and 18% for
specific test automation activities like functional, API, and performance testing.
While organizations are not significantly increasing their rates of test automation, most are at least testing
it out, so to speak. A 2018 TechWell and Tricentis survey report, The Evolution of Test Automation [4], found
that 72% of organizations in the Americas were doing at least some level of automated testing. A little over a
year later, our survey showed that 80% of respondents are automating some portion of their testing effort,
suggesting a moderate year-over-year increase.
80 85%
80%
76%
72%
60
Percentage of
4
55% 54% 56% 56%
Respondents
40
34%
20
21%
0
Automated software Agile DevOps Continuous Behavior-driven
testing methodologies integration development
6
2019 Tricentis Survey Report
The 2018 survey also found that 24% of respondents were automating more than 50% of testing effort while
44% said they expected to reach that level of automation in the next 12 months. Our 2019 survey shows
only 14% of respondents in the Americas (and 19% on average across all regions) actually automate at least
50% of testing effort. This indicates that the rate of test automation within organizations isn’t growing as
rapidly as many testers anticipated, suggesting that expectations for scaling test automation do not always
match the (often difficult) reality.
0% 8%
1% to 10% 28%
11% to 20% 16%
5
21% to 30% 18%
31% to 40% 9%
41% to 50% 6%
More than 50% 14%
0 5 10 15 20 25 30
Percentage of Respondents
In this year’s survey, 30% of respondents in the Americas (and 35% on average across all regions) expect
that their organizations will automate more than 50% of testing in the next 12 months — a hopeful, but
slightly more modest, expectation.
0% 3%
1% to 10% 13%
11% to 20% 11%
21% to 30% 16%
31% to 40% 15%
41% to 50% 12%
More than 50% 30%
0 5 10 15 20 25 30
Percentage of Respondents
Together, these findings suggest that testers and testing leaders understand that an advanced level of test
automation is critical for transformation, but that achieving it is more difficult than anticipated.
7
2019 Tricentis Survey Report
To get a better idea of why organizations struggle to scale test automation to the desired level, we asked
testers where they’re spending the most time today, what their biggest pain points are, and where they have
to make tradeoffs. We also looked at how these responses differ according to whether their organization’s
DevOps transformation was complete, in progress, scheduled to kick off, or not a priority at all.
How testers spend their time evolves significantly as DevOps maturity increases
Across all DevOps maturity levels, survey respondents report spending most of their time, on average,
performing these five tasks:
• Implementing tests
• Designing tests
• Planning tests
• Testing strategy
• Documenting issues
When we broke down the results by DevOps maturity, we found that in organizations where DevOps
transformation is complete or in progress, testing strategy and updating/maintaining automated tests
rank in the top 5 places testers spend their time. The only place testing strategy is first on the list is in
organizations where DevOps transformation is complete. It is not listed in the top five in organizations
where DevOps isn’t a priority.
In organizations where DevOps transformation hasn’t started, implementation is rated number one and
documentation (documenting issues found and documenting manual tests) replaces updating/maintaining
automated tests in the top 5.
DevOps Initiative Not Started DevOps Initiative In Progress DevOps Initiative Complete
8
2019 Tricentis Survey Report
26%
Test maintenance
25% Other
We wanted to know how testers’ pains change with level of automation and DevOps transformation, i.e., the
“rewards of test automation maturity.” We found some big differences in the most common pain points at
differing levels of both DevOps maturity and test automation.
Survey results indicate that almost every organization, at every level of DevOps maturity, struggles with test
maintenance. For those organizations that haven’t started DevOps transformation or that are planning to
scale test automation, it’s critical to get this under control, because it’s only going to become more difficult
as the transformation progresses. Test maintenance is the first problem teams adopting test automation
are likely to encounter.
We also found that the higher the DevOps maturity, the more pain test maintenance causes, and the
lower the DevOps maturity, the more pain test orchestration, or simply running test cases, causes. We can
extrapolate that because these teams are likely just getting started with test automation and automating
a narrower range of scenarios, they are not yet struggling with maintenance, but are still ironing out the
processes associated with orchestration.
9
2019 Tricentis Survey Report
When we broke down pain points by level of automation, we observed similar trends. Nearly a third (29%) of
organizations with lower levels of automation (0% to 10% of test effort) say test orchestration is a pain point
but only 9% of those with higher levels of automation (40%+ of test effort) struggle with test orchestration.
Organizations that are further along in their automation journey tend to cite test maintenance, test data
management, and test environment management as the most significant pain points.
Level of Automation
More than
0% 1% to 10% 11% to 20% 21% to 30% 31% to 40% 41% to 50%
50%
Test reporting 21% 8% 7% 3% 0% 0% 4%
Test maintenance 14% 17% 21% 32% 43% 18% 36%
Test data
14% 13% 25% 6% 29% 18% 32%
management
Test environment
14% 27% 25% 32% 14% 45% 20%
management
Test
14% 15% 11% 13% 0% 9% 0%
orchestration
In organizations with lower levels of automation (0% to 10% of test effort), test reporting is a common
struggle. In this category, 29% of respondents say test reporting is a pain point while only 4% of those with
higher-level automation (50%+ of test effort) agree. This suggests that as organizations kick off or scale a
test automation initiative, they struggle to adjust their reporting to the new processes. Reporting can be
especially difficult to iron out in organizations using multiple open source test automation tools, which
requires them to analyze results across multiple tools.
10
2019 Tricentis Survey Report
We found that more than half of testers (55%) are forced to make tradeoffs because of lack of time while
16% report tradeoffs occur because they don’t have the right tools and capabilities.
Insufficient budget
9%
55% Insufficient tester skill levels
9%
Insufficient tool availability and capabilities
Lack of time has implications for testers beyond making tradeoffs. Almost a third of respondents (32%) cite
not having enough time to test as the biggest problem with their organization’s testing process across the
board. Unclear stakeholder expectations (21%) also made the list of testing process problems testers have
to deal with.
0 5 10 15 20 25 30 35
Percentage of Respondents
11
2019 Tricentis Survey Report
With all of the emerging lifecycle models and testing practices over the past several years, we were curious
whether testing practices or lifecycle model impacts job satisfaction.
The survey didn’t reveal a relationship between job satisfaction and testing practices, such as AI-augmented
testing, BDD, DevOps, etc., but the results do show that a majority of testers are generally more satisfied than
dissatisfied with their jobs.
We did, however, see a correlation between the level of job satisfaction and lifecycle model. Those testers
working in a primarily waterfall environment reported lower levels of job satisfaction than testers working in
primarily agile environments. This could be due to several factors, including the increased collaboration and
communication on agile teams, the opportunity to learn new technical skills, and being able to find and fix
issues as they occur rather than waiting until production to discover an error or bug.
12
2019 Tricentis Survey Report
Over the past 15 years we’ve seen agile adoption explode within organizations of all sizes and industries.
Before DevOps came along, software testers were concerned that agile was going to put them all out of a job.
We wanted to know what effect agile really has had on testers, so we asked respondents to rate how agile
impacted various areas of testing in their organization. Some of the more significant findings show that as a
result of agile implementation within the organizations surveyed:
• Tester value increased
• Testing scope became broader
• Testing became more thorough
• Job satisfaction increased
Industry notables have been prematurely announcing the death of software testing for years [5], and yet
here we are. As time goes on and development methodologies wax and wane in popularity, there is only one
thing we can predict with certainty: change is inevitable.
While the widespread adoption of agile and DevOps prompted their share of hand wringing, the tester role
shows no sign that it will become obsolete. Though it will, without a doubt, be different.
13
2019 Tricentis Survey Report
Survey respondents identified several potential threats to their jobs, but expectation of programming/
scripting skills (33%) and testing being deprioritized (47%) are by far the biggest perceived threats. When we
look at the perceived threats by salary range, we found that half of respondents who earn less than $50K per
year are most threatened by testing being deprioritized while only 25% of those earning more than $150K
per year feel the same.
Given that testing being deprioritized and lacking programming skills are the top two threats to testing
jobs according to survey respondents, it makes a lot of sense that SDET or test automation roles (39%) and
DevOps roles (22%) are seen as the next great opportunities for testers. Preparing early for the future of
testing is an excellent risk mitigation strategy and a smart career move.
0 5 10 15 20 25 30 35 40
Percentage of Respondents
14
2019 Tricentis Survey Report
In organizations where DevOps transformation is complete, 66% of managers are very satisfied or satisfied
with their team’s work compared with 23% of managers in organizations where DevOps is not a priority.
Organizations that don’t automate testing report only 14% of managers are very satisfied or satisfied with
their team’s work. In those organizations with 50%+ automation, 64% of managers are very satisfied or
satisfied with the work their teams do.
How manager satisfaction changes with DevOps maturity and level of automation
Very
DevOps maturity Very satisfied Satisfied Neutral Dissatisfied
dissatisfied
DevOps transformation is
13% 53% 0% 7% 7%
complete
DevOps transformation is in
19% 33% 14% 7% 0%
progress
Very
Level of test automation Very satisfied Satisfied Neutral Dissatisfied
dissatisfied
15
2019 Tricentis Survey Report
We know that manager satisfaction with their team’s work increases with DevOps maturity and level of
automation, but how do these factors affect the testers themselves?
Testers in organizations that don’t prioritize DevOps report higher job satisfaction and lower job
dissatisfaction than testers in organizations that have completed DevOps transformation. Job satisfaction is
lowest among organizations that are currently undergoing, but haven’t completed, DevOps transformation.
How testers’ job satisfaction is affected by DevOps maturity and level of automation
Very
DevOps maturity Very satisfied Satisfied Neutral Dissatisfied
dissatisfied
DevOps transformation is
13% 33% 7% 13% 7%
complete
DevOps transformation is in
16% 28% 21% 11% 6%
progress
DevOps is a priority, but we
13% 38% 3% 19% 0%
have not started
DevOps is not a priority 9% 46% 20% 3% 6%
Very
Level of test automation Very satisfied Satisfied Neutral Dissatisfied
dissatisfied
0% 0% 29% 36% 14% 0%
1% to 10% 8% 40% 13% 6% 15%
11% to 20% 21% 25% 18% 18% 0%
21% to 30% 13% 39% 23% 10% 0%
31% to 40% 21% 36% 7% 7% 0%
41% to 50% 9% 36% 9% 18% 9%
16
2019 Tricentis Survey Report
Conclusion
We took a look at where testers are spending their time and whether DevOps maturity affected this. We
discovered that regardless of where an organization is in its DevOps transformation, testers spend the
most time implementing, designing, and planning tests; working on testing strategy; and documenting
issues. However, we did find that in organizations with greater DevOps maturity testers spend more time
on strategy and maintaining automated tests than their peers in organizations that haven’t implemented
DevOps at all. These testers spend the bulk of their time implementing tests and documenting issues and
manual tests.
Again, we find consensus across every level of DevOps maturity when it comes to the pain of test
maintenance, though testers in organizations with high DevOps maturity rate maintenance as their top
pain point. Test orchestration was more painful for those in low DevOps and low test automation adoption
companies.
Testers across the board feel threatened by the prospect of testing being deprioritized and by having to
learn more technical/programming skills. While it’s unlikely that the testing role will become obsolete, there
is no doubt that it is evolving, and testers will need to take steps now to keep up with the changing face of
testing. Likewise, technology leaders who are planning or in the midst of a DevOps transformation must
take steps to ensure their testing teams are enabled to transform with the rest of the business.
17
2019 Tricentis Survey Report
The survey found that manager satisfaction improves with increased levels of test automation and
DevOps maturity, and that in organizations that have completed DevOps transformation, managers are
three times more likely to be satisfied with their team’s work. Findings also show a four-fold increase in
manager satisfaction between organizations with 0% test automation (14%) and those with more than 50%
automation (64%). These findings suggest that when testers are set up for success and prioritized alongside
other key functions, they are better equipped to meet expectations, and their organizations are more likely
to succeed with their DevOps initiatives.
REFERENCES
[1] 2019 Global Developer Survey: DevSecOps. Gitlab.
18
2019 Tricentis Survey Report
Appendix 1:
Regional Comparison
of Key Survey Results
Testers from around the world contributed to the 2019 survey Expectations vs. Reality: The Role of Testing
in a DevOps Transformation. While the results show many global similarities in how the role of testing is
evolving to meet the demands of DevOps and increased automation, some regional differences emerged,
which are highlighted here.
The regions are broken down into Europe/Middle East/Africa (EMEA), the Americas, and Asia Pacific (APAC).
19
2019 Tricentis Survey Report
APAC 77%
AMERICAS 72%
EMEA 47%
45%
71%
50% 49% 26% 26%
24%
APAC 13% 9%
AMERICAS EMEA
20
2019 Tricentis Survey Report
21