2022 Software Testing & Quality Report

Annual Survey Report

3 Introduction

4-8 Current Trends in Development

9-22 Current Trends and Challenges in Testing

23-27 What Business Goals Does Testing Help Achieve?

28-29 Closing Thoughts

2022 Software Testing & Quality Report - 2

Introduction
Welcome to the 2022 Software Testing & Quality Report!

We developed this report in the hopes that it sheds light on the current state of the QA and software testing community. We aimed to learn more
about your current goals, challenges, and key focus areas like new approaches to development, trending automation techniques, frameworks, and
more.

So, what’s changed in the world of QA and software testing in the last year? (Beyond the need to set up a home office, that is.)

The last year has served as a catalyst for cultural transformation worldwide, and software testing is no exception. But how did it affect software
testing and QA specifically? What trends in testing and software development emerged? What were some of your primary objectives and business
goals? What key challenges did you face? These questions are what we set out to answer.

To answer these questions, we invited TestRail users and the broader testing community to answer an in-depth survey about their perspectives on
key QA and testing trends. This is the fourth annual survey we have run and we have now gathered enough data and insights to launch this inaugural
Software Testing & Quality Report. By looking at the trends, patterns, and sentiments revealed by your responses, we want to paint a picture of
where QA and the state of testing is headed and how you fit into the picture.

Over the last four years, we have received more than 15,000 responses to our annual surveys. To give you a sense of who is participating in these
surveys, here’s a breakdown of some of the key demographic highlights from this year’s questionnaire:

Most respondents were testers or QA engineers, followed by test leads and test automation engineers/SDETs

Respondents came from 30+ industries, with the largest segment of respondents currently employed by
companies that produce computer software

Respondents represented almost 100 countries around the world

As you read this report, be on the lookout for trends around agile, DevOps, and other development methodologies, shifts in the focus on different
testing methodologies, and surprising data about the adoption of test automation. You can also keep an eye out for questions about the business
impact and ROI of testing, top challenges faced by QA teams these days, and people’s primary objectives around quality right now.

Finally, on behalf of everyone here at TestRail, our genuine thanks go out to those who took the time to answer the survey and made this report
possible. As we march into 2022 and beyond, we hope some of the findings from this report help you and your team reflect on your current software
development and QA processes and support your continued efforts to improve the quality practice within your organization!

2022 Software Testing & Quality Report

Report--33
Current Trends in Development
Current Trends in Development

Development methodologies and techniques

When asked about which development methodologies or 81%
techniques you currently use, the vast majority of respondents 82%
use agile (81%). Meanwhile, scrum remains strong at 64%,
while test-driven development (TDD) and behavior-driven 64%
development (BDD) saw slight decreases from 2020.
65%
45%
The agile methodology has transformed the process
42%
of developing software and appears to be the primary 18%
development methodology used in the industry market. 20%
More organizations are undertaking digital transformations, 15%
including incorporating software and technology as core 16%
elements of their unique selling proposition, as they look to 15%
enhance the way they manage data and deliver digital services 15% 2021
to external customers as well as internal users. The agile
methodology is also closely tied to DevOps, which respondents
11% 2020
12%
identified with three percentage points more than in our 2019
Annual Survey. DevOps is slowly becoming more of an influence 5%
in the software testing market and, especially when combined 4%
with agile, promises to promote the overall goals of agile
culture in delivering high-quality software applications.
2%
1%

0% 25% 50% 75% 100%

*2021 data not available

2022 Software Testing & Quality Report - 5

 Current Trends in Development

Development methodologies and techniques

Over 80% of respondents felt they were satisfied or very
2021 satisfied with the development methodologies they already
have in place and only 6% felt they were not satisfied or
80% very not satisfied. Satisfaction with current development
methodologies depends on the applied development method(s)
used. With over 80% of respondents using agile or some form
of agile methodology, it seems likely that processes that involve
62% quick iterations and more collaboration are closely related to
60% satisfaction.

For those who were not satisfied with their development

40%
methodologies, common themes revolved around poor
communication and lack of automation.

“Each team is doing their own thing independently,

19% there is poor communication, and a lack of
20%
14% coordination between teams.”

5% “Testing needs to be closely knit with development as TDD helps

1% you stay on top in terms of quality of the product. Right now, we
0% feel that there is a big gap between developers and testers.”

“We are not fully developed and are too reliant on

manual testing.”

2022 Software Testing & Quality Report - 6

Current Trends in Development

Adopting methodologies and techniques

When asked if your team is considering adopting any development
methodologies or techniques in the next year, almost half of the
respondents didn’t know if their teams were considering changes 28%
in development methodologies. Those who were considering
changes in development methodologies were mainly considering
adopting agile, scrum, and DevOps—in that order. TDD and BDD 23%
are in the conversation too at 14% and 11%, respectively.
19%
Today, the changes in modern development require testing to
adapt. Global trends like privacy regulations, cybersecurity attacks, 2021
increased leadership attention to digital infrastructure, and even 14%
Covid-19 also impact testing methods. As modern development
practices quickly evolve, testing is required to keep up. By adapting 11%
testing practices, tools, and methods, teams can continue to ensure
quality and on-time deliveries.
4%
For those considering adopting new development
methodologies or techniques in the next year, it’s important 8%
to step back and ask questions to understand why you are
considering a change and what you are trying to fix. Are there
problems with the methods you’re using today? What are they?
4%
Who is affected by them? Or, is there the expectation that—
even if no large problem exists—an agile method would allow 44%
you to improve your current methods? Can you define what
improvements you’re anticipating or what you’re hoping to 0% 10% 20% 30% 40% 50%
achieve with a different approach?

2022 Software Testing & Quality Report - 7

 Current Trends in Development

Frequency of release cycles

59% of respondents ship new releases “I feel the need – the need for speed.” Pete “Maverick” Mitchell.
2021 at least every two weeks
These words from Top Gun (1986) rang true for fighter pilots
and race car drivers back then but they also stand true for
30%
28% the dynamic software development world of today, where it
is becoming necessary to deliver code faster, safer, and with
better quality.

20% 19%
18% When asked how frequent your organization’s release cycles
are, about 60% of respondents said they are releasing new
code as often as every two weeks. Over 75% said they are
10% releasing at least once a month. If quality has emerged as a
10% 8% 8% differentiator for businesses to expand their customer base,
enabling faster release cycles arguably comes next.
6%
4%
0%

77% of respondents ship new

releases at least once a month

2022 Software Testing & Quality Report - 8

 Current Trends and
Challenges in Testing
Current Trends in Testing
- 2021 - 2020
Testing types, techniques,
and methodologies
92% 42%

When asked about what testing types, techniques, and 93% 63% 41%
methodologies your team currently uses, 92% of respondents
said they do some kind of regression testing, 88% reported
doing functional testing, and 71% employ unit testing. 88% 49% 28%
90% 62% 35%
Notably, when comparing the data from responses in 2020 to
2021, there’s a six-point increase from last year of respondents
who do load or performance testing.
38% 53% 23%
This increase in load and performance testing shows us the 54% 20%
value put into various kinds of non-functional testing. Without
good performance testing, you don’t know how your system
will deal with expected—or unexpected—demands. This kind of
testing should be done from the start to determine viability and 47% 17%
to build reliable performance into the design and structure of
63% 53% 19%
the system.

71% 54% 9%
66% 49% 9%

32% 39% 1%
*2020 and/or 2021 data not available 46% 2%

2022 Software Testing & Quality Report - 10

 Current Trends in Testing

2021 Automated testing types

When it comes to automated testing types, regression testing
appears to be the most commonly automated type of testing,
with close to 70% of respondents taking advantage of it. ( As a
side note, 87% of respondents said they still do some manual
100%
regression testing too).

The conclusion is simple: regression testing is an indispensable

75% part of any software development project. It can save a lot
69% of time fixing the defects at earlier development stages and
in turn, cut budget needs for a project. We’re living in an era
where high-quality software is no longer optional. Preventing
49% defects is especially important because users don’t always
50% notice when an application improves, but they definitely notice
38% when it breaks. Testers need to use an approach to regression
32% testing that will identify defects along all paths that users could
28% take, even the ones that aren’t easy to anticipate. Whether
25% your approach to regression testing is automated or manual, it
should provide useful feedback quickly and give you the most
confidence that relevant scenarios are covered.

0%

2022 Software Testing & Quality Report - 11

Current Trends in Testing
2021
Adopting testing types
40%
Test automation continues to be a dominant software testing
33%
trend. It has become the key enabler for many modern software
development and deployment practices and is especially 29%
critical for companies that decide to implement agile and 28%
DevOps methodologies.
25%
24%
This trend is exemplified by the strong interest among
respondents in automating more regression testing (40% of 24%
respondents) and adopting other types of automation testing 24%
such as automated end-to-end, web UI, and mobile testing over
23%
the next year. Moreover, one-third of respondents said they’re
considering adopting load and performance testing in the 20%
coming year. 19%
17%
Regression testing continues to be recognized as one of the most
important testing methods to use to maintain the level of quality of 16%
your product. This importance—combined with the need to deliver 15%
quality at speed, the rise of agile development methodologies,
12%
and increases in the accuracy and efficiency of automated testing
frameworks—has meant that many teams have shifted or are 11%
planning to shift from manual testing to automation. Fortunately, Other 8%
due to its repetitive and predictable nature, regression testing is
an ideal candidate for automation. By automating regression tests, 7%
it is possible to achieve several desired outcomes that enhance 0% 10% 20% 30% 40%
team agility and product quality, like giving the team more time to
focus on higher-value exploratory testing and introducing other
types of testing.

2022 Software Testing & Quality Report - 12

 Current Trends in Testing

Testing techniques
2021 Compared to the 81% of respondents who were satisfied or
very satisfied with their development methodologies and
60% 74% of respondents were satisfied or very techniques, 74% had the same assessment of the testing
methodologies they have in place (see: Current Trends in
satisfied with their current testing techniques Development: Adopting Development Methodologies and
Techniques - Page 7). This 7% drop-off might be explained
59% by common themes revolving around wanting to increase
automation. Teams are experiencing an increasing pressure to
do more with less and for many teams, increasing automation is
40% necessary to keep pace with agile development.

20%
17% 15%
“Manual testing is fine but we need to be extensive
on the automation side as well.”
8%
1%
0% “There is not enough time spent on automation
because of the lack of training and skill sets.”

2022 Software Testing & Quality Report - 13

Current Trends in Testing

Current testing process 2021

When asked about satisfaction with current testing processes,

73% of respondents said they are satisfied or very satisfied,
which is in line with the satisfaction respondents reported 60% 73% of respondents were satisfied or very
about their testing techniques. Compared to the 81% of
respondents who felt they were satisfied or very satisfied with satisfied with their testing process
their development processes, there is an eight-point drop-off.
58%

40%

Here are some common insights from the

respondents regarding their satisfaction levels
with their current testing process.
20%
18% 15%
“Our business goals are unclear and there is too much
responsibility on testers to derive requirements.”
8%
1%
“We are working out of a google spreadsheet so there
are no details in scenarios and a lot of confusion 0%
around who is doing what.”

2022 Software Testing & Quality Report - 14

Current Trends in Testing

Testing responsibilities 2021

When asked to describe quality testing responsibilities in your

organization, almost half (46%) of respondents said that developers 52%
on their team are responsible for running their own test code.
In comparison, nearly a third of developers are responsible for
manually testing their own code (28%). This fact isn’t necessarily 49%
surprising considering the growth in the number of respondents
who reported their team using unit testing.
46%

More than half (52%) of respondents said that their manual

testers are their primary testers. Primary test roles are now at 42%
about two manual testers to every one automation or SDET.

30%

28%

24%

14%

10%

0% 20% 40% 60%

2022 Software Testing & Quality Report - 15

 Current Trends in Testing

Roles
100% 90% of respondents said testers define tests in their
90% organizations, which isn’t surprising. Interestingly, 82% of
respondents also report non-tester roles like developers,
2021 product managers, and business analysts helping to define
tests in their organizations, including 38% of respondents who
75% said developers help define tests.

50%
38%
27%
25%
17%

0%

2022 Software Testing & Quality Report - 16

 Current Trends in Testing

Test Automation
Most respondents said they use automation to test web
2021 applications (76%) and APIs (63%). Meanwhile, 43% of
respondents are automating their mobile application testing,
80% 29% of respondents are doing the same with desktop
76% applications, and 25% of respondents said they use automation
to test microservices.
63%
60%
By automating web application tests, testers can save time
and effort on monotonous tasks because automated tests can
be run continuously or scheduled at intervals. By offloading
43% these time-consuming tasks from testers, they can focus on
40% exploratory testing or other high-value tests that require a
human perspective.
29%
25% Automation should not be seen as just a plug-and-play system
20% that requires no human intervention. Effective automation
requires testers to have a thorough knowledge of the software
9% 9% under test, as well as an “automation first” mindset.
3%
0%

2022 Software Testing & Quality Report - 17

Current Trends in Testing
2021
Automated tests
For those who are automating tests, when asked how many
automated tests your organization runs each day, 57% of
respondents said that they run more than 100 automated tests 42%
each day in their organizations.

As your application under test becomes more complicated, 33%

it becomes even more imperative to start automating some
testing. But most times, your strategy around test automation
is more important than the pure number of tests you’ve
automated. The larger your automated test suite gets, the more 17%
time it will take to maintain with each new release. Focusing
on automating tests of critical-path workflows or high priority
scenarios can help you bring a more risk-based approach to
prioritizing which tests to automate. 5%

1%

1%

0% 10% 20% 30% 40% 50%

2022 Software Testing & Quality Report - 18

Current Trends in Testing
What percentage of your test are currently automated currently?
Percentage of automated tests
The rise of software development methodologies that require a
continuous and streamlined workflow, such as agile and DevOps,
have pushed teams to automate as much as possible. That 2021 2020 2019 2018
said, even though we are seeing a slight increase from last year, 50%
we’re consistently seeing about 40% percent of all tests run by
respondents’ teams being automated.

These responses tell us that even though automated testing is 40% 43%
trending back upwards, this does not mean that manual testing 42%
will disappear in the near future. In fact, each type of testing 38%
serves a different function in the QA process, and together they
complement each other. 30% 35%

20%

10%

0%

2022 Software Testing & Quality Report - 19

 Current Trends in Testing

Automated tests
For the past four years, respondents have expected to
2021 2020 2019 2018 automate over half of their tests. Compared to the percentage
of tests that respondents have actually automated, this data
indicated that they succeeded in automating approximately
80% 20% fewer tests than expected.

Moving from manual to automated testing without impeding

current development and test processes requires more than
60% 63% just hiring an automation engineer. Frameworks have to be
61% built, tooling has to be implemented, employees have to be
trained, and more. These things take time and effort. Doing all
56% of that without compromising the quality of the software being
54% released is a significant undertaking – but it’s doable.
40%

20%

0%

2022 Software Testing & Quality Report - 20

 2021
-2021 2020
-2020 2019
-2019 2018
-2018 *2019 and/or 2018 data not available

50%
45%

40%
40%
35%

30%
25% 26%
11%
21%
21% 21% 21%
21%
20% 20%
20% 19% 19%
18% 18%
17% 17% 17%
16% 14% 14%
12% 12% 11% 11%
10% 9% 9%
7% 7%
6% 6%

0%

Current Trends in Testing When asked which test automation frameworks teams used, Selenium appeared as the most popular choice even after a five-
point drop from 2020. Cypress is becoming increasingly more popular with a nine-point increase from 2020, while Appium and
Automated tools and frameworks Cucumber saw the same steady growth they’ve had since 2018. Other notable leaders include JUnit, TestNG, and Pytest, while
the number of respondents who answered “none” or “not used” continued to drop yearly.

2022 Software Testing & Quality Report - 21

Current Trends in Testing

Key challenges
Similar to 2020, developing and increasing automation is still the 45%
biggest challenge reported by the testing community, while having 48%
enough time to complete QA tasks is second—which may be a 36%
31%
contributing factor as to why people want to automate more. Users also
reported experiencing challenges with end-to-end testing, managing 35%
30%
testing data and environments, and driving testing earlier in the
33%
process. 31%
32%
Numbers speak volumes: the global automation testing
market size reached $12.6 billion in 2019, and this number is 28%
28%
expected to grow to $28.8 billion by 2024. The market is big
and enterprises are getting on the automation rails, but many 20%
of them are not able to fully enjoy the benefits of it and even
19%
fail without properly tackling the top challenges in automated 15%
testing. Whether it’s struggling to find the right framework
18%
or tool, lack of communication and collaboration among the
team, not implementing a proper test automation strategy, 17%
or even setting unrealistic expectations, automation is a 18%
challenge for many teams. 17%
18%
16%

11%
12% 2021 2020
When asked what key initiatives or challenges your teams are 11%
planning to focus on over the next 1-3 years, there were three
10%
common responses: Developing more automated tests, increasing 11%
automated testing, and taking advantage of CI/CD.
0% 10% 20% 30% 40% 50%

“We want to focus more on adapting the latest industry-standard technologies at

all levels of the CI/CD pipeline plus development and testing. We also want to have *2020 data not available
all of the regression testing automated.”

2022 Software Testing & Quality Report - 22

What Business Goals Does
Testing Help Achieve?
Business Goals

Metrics and KPIs

58%
When asked which metrics or KPIs your team uses to measure
the impact of your testing, almost two-thirds of respondents 52%
measure test pass/fail rate, closely followed by metrics like the 6%
number of defects reported in production and the total number 41%
of tests executed. 50%
37%
59%
This year’s responses suggest that there has been a shift to 31%
put more of a focus on metrics that are aligned with actual
value centers and key results around quality—like the number 28%
of defects detected in production—instead of just measuring 43%
testing activity. 28%

27%
When looking at trend comparisons from 2020, measuring 26%
the number of defects reported and development lead time 26%
58%
was way up. On the other hand, measuring the percentage of
automated versus manual tests, test execution numbers, and 26%
bugs found were down.
26%
33%
26%
2021 2020
26%

21%
10%
17%
34%
0% 20% 40% 60%

*2020 data not available

2022 Software Testing & Quality Report - 24

 Business Goals
The ROI for testing
When asked how return on investment (ROI) is calculated for
testing in your organization, 93% of respondents told us they don’t
explicitly measure the ROI of their testing efforts. However, 67%
of respondents do want to calculate the ROI for their testing and
are thinking about how to calculate ROI in the future (although
31% based on the open text responses to this question, we found that
there appears to be a lot of inconsistency in the definition and
understanding of ROI around testing in the first place).

There is so much pressure in software development projects to

36% reduce costs and reduce development timeframes, and testing
is a common target for cuts. Oftentimes, the benefits of testing
aren’t understood, go unnoticed, or don’t have revenue dollars
associated with them. For that reason, some consider the idea of
calculating ROI for software testing to be a myth. Some questions to
26% consider are: “What can we, as a testing community, do not only to
demonstrate improvements in the effectiveness of the QA process
but also to quantify the actual ROI of QA? Is it worth trying to quantify
ROI? Would quantifying the ROI of testing help make QA more visible
to the rest of the organization?”
7% 2021

Some of the responses we got regarding what ROI means described

0% 10% 20% 30% 40% calculating ROI by the ratio of manual versus automated tests, test
effort versus bugs in production, and investment in cost versus
revenue generated.

“ROI is the number of bugs found during pre-production versus “ROI is measured by calculated daily revenue
the number of bugs found in production. If we are seeing less generated versus the cost of the project.”
than 20% production bugs, that is successful.”

2022 Software Testing & Quality Report - 25

Business Goals

Primary objectives of QA
2021
QA is an activity that aims to evaluate the quality of a
software product and improve it by identifying defects
and potential risks present in the software before it gets 9.10
shipped to end-users. Clearly defining the objectives
of your testing not only helps communicate the value 8.38
and impact of QA to the rest of the organization but also
ensures effective testing. 8.26

7.86
When asked about the primary objectives for your
team’s software development and QA teams, the top 7.38
three responses you selected are tied directly to quality,
namely, increasing test coverage, reducing bugs in 6.71
production, and automating more tests. The following
two most commonly selected objectives address speed 6.65
to deployment, while the following four priorities revolve
around driving efficiency in testing. 6.38

6.28

4.23

3.90

3.55

0.0 2.00 4.00 6.00 8.00 10.00

2022 Software Testing & Quality Report - 26

 Business Goals
- 2021
2021 Successful habits of a QA team
When asked to rate how important various activities are to
8.22 your success, respondents reported that “being more efficient”
7.78 was the top activity driving their success. Other top priorities
like “making sure the correct tests are being run”, “tracking
7.72 test coverage”, “improving collaboration with the development
7.66 team”, and “automating more of your team’s test cases” may be
areas where respondents are looking to “be more efficient” to
7.64 achieve overall success.

7.60
These responses show us that people are looking at efficiency
7.50
and better collaboration to help them achieve their goals of
7.44 increased test coverage and achieving better quality overall.
7.40
7.28
7.06
6.98
6.36
6.28
5.94

0.0 2.00 4.00 6.00 8.00 10.00

2022 Software Testing & Quality Report - 27

Closing Thoughts
So now that you have read through this report, let’s revisit our initial question:

What’s going on in the world of testing and quality assurance?

With 81% of respondents using agile, it is clear that agile is here to stay. However, this does not mean it will remain static. Since their inception
in 2001, agile practices have continued to evolve. It was first about shifting from the waterfall model of software development to focusing on
iterative development using scrum, XP, and other methodologies, thereby removing many bottlenecks organizations had with more waterfall-style
development delivery. Agile is now about optimizing operations within the development process, using different practices and tools, and making the
entire process more efficient. It is about starting the testing process right from the beginning of the development lifecycle to catch defects early (i.e.
“shift left”) and distilling insights from the data collected from each stage of the development process.

The agile methodology quickly gained widespread adoption and became the industry standard. Yet, in a few years, a critical oversight arose: in its
initial iterations, agile left out the processes and requirements of the operations team that deployed and managed software products. This, along
with evolutions in technology and new distributed approaches to development, led to DevOps, an approach that aligns development and operations
teams around the core tenants of shipping high quality software early and often. As we found in this year’s report, the adoption of DevOps continues
its steady increase, with almost half of respondents now reporting that their teams employ at least some form of DevOps practices. DevOps has been
a trend for the last several years, but it is here to stay and will continue to be a big focus for organizations moving into the rest of 2022 and 2023.

DevOps enables teams to build, test, and release software faster and more reliably both by incorporating agile principles and by streamlining and
accelerating certain processes with increased automation, including test automation. In 2022, test automation is at the forefront of users’ minds
and larger organizations are executing more automated tests on a daily basis. This trend is exemplified by the strong interest among respondents in
adopting more automated regression testing (40% of respondents) and other automated testing methods such as automated web UI testing (29% of
respondents) and automated end-to-end testing (28% of respondents) over the next year (none of which are necessarily exclusive of each other).

Along with increasing automation, most respondents identified increasing test coverage and reducing bugs in production as primary business goals,
while efficiency, coverage tracking, and better collaboration with developers were called out as key drivers of success. We know that high-quality
products are not the result of chance or random luck. Setting goals, communicating well, and tracking KPIs are all part of the concerted effort to
create a culture of excellence in QA and software testing.

“Quality assurance” is often assumed to be the step that happens between writing code and getting it deployed to production. But software quality
doesn’t happen at just one point in time; it encompasses more than just your bug count. QA is a mindset and a culture that your entire team, company,
and industry should be involved in.

We look forward to reporting how these trends continue to develop next year. Thanks again to everyone on the team that collaborated on this report!

2022 Software Testing & Quality Report- 28

TestRail is a test management platform built to help testing, QA, and software
development teams ship rock-solid products faster. TestRail makes it easy
to create organized, repeatable, and streamlined processes so you can scale
test coverage, plan test executions more strategically, and spend more time
actually testing.

TestRail can be used in waterfall or agile projects and is flexible enough to be

used in any type of QA process.

Manage test cases, plans, and runs

Increase test coverage
Get real time insights into your QA progress
Integrate with issue trackers and test automation
Explore Enterprise for advanced features and security

Integrate with all of the tools in your stack to build quality into every stage of
your software development lifecycle. More than 10,000 organizations like NASA,
Apple, Microsoft, Activision Blizzard, and Amazon trust TestRail to power their
QA and test management processes.

Visit our website for more information or get started today!

Image: Track all of your test activities and quality metrics in a single platform
TryTry
Try TestRail
aafree
free 14-day for free
TestRail
14-day TestRail trial
trial to improve collaboration, evaluate risk, and release higher-quality software.

2022 Software Testing & Quality Report- 29