The State of

Quality Report

2022
 TABLE OF Executive Summary
Quality at Speed: Quality Assurance Techniques and Challenges
CONTENTS Common testing techniques

The most widely used QA techniques being the most effective ones

QA teams and challenges in high software quality

Level of satisfaction in software quality, QA practices and tools

Test Automation
The continued decline in open-source frameworks and tools' popularity

Race to adopt test automation

Persistent obstacles in applying automation tools

Factors to ensure positive ROI

Artificial Intelligence in Test Automation

Looking Ahead to 2022
Case Study Analysis
Appendix
Methodology

Demographics
 In today’s rapidly changing world, organizations have to look both inward and outward
for improvements in their software quality (QA) practices. They need to collect and analyze
information and data generated from their practices to determine their weaknesses and
strengths in delivering software quality at speed. At the same time, they have to keep updated
on the current status and trends of their respective industries.

Understanding how data analytics can assist software teams, since 2018 we have worked with
our partners to conduct annual studies to discover and report insights into the current state of
software quality. These annual reports also offer recommendations on what areas and how
software teams can improve their QA practices and processes.

From this year's report, we learned that a vast majority of software professionals consider test
automation the most effective QA technique. However, with only half of them applying test
automation to their projects, teams are likely to experience difficulties when implementing
this technique.

CotyCTORosenblath

at Katalon, Inc.
Frequent requirement changes and the lack of time are the main challenges to adopting test
automation and ensuring quality. Still, we see more testing activities using automation.
Besides regression testing, teams today use automation tools to analyze test results, generate
test data, perform performance tests, and other time-consuming activities.

The survey also shows that, despite the great potential of artificial intelligence, AI for test
automation is still in its initial stages. Teams are more concerned with employing QA practices
and tools effectively rather than improving the application of AI to their projects.

”
At Katalon, our mission is to help teams deliver better software faster. Our test automation
platforms allow easy automation of necessary but time-consuming activities, such as test
generation and execution. We also incorporate advanced AI capabilities in our products, so
that teams can improve the effectiveness of test automation.

Executive
Summary
In today’s rapidly changing world, organizations have to look both inward
and outward for improvements in their software quality (QA) practices.
They need to collect and analyze information and data generated from
their practices to determine their weaknesses and strengths in delivering
software quality at speed. At the same time, they must look for the
current status and trends of their respective industries.

Based on a survey of 3,000+ respondents, this report captures a

snapshot of the current state of software quality, presents insights, and
predicts trends in QA techniques, practices, and tools. As our study
surveyed software engineers, QA engineers, analysts, and managers, the
findings reflect the opinions and voices of people who are directly
involved in ensuring the quality of their software.

 Findings
1. Although it is the most popular QA technique, 2. Monitoring and testing in production
automated system testing is not as widely used as emerged as an answer for balancing
people would expect.
between quality and speed.
Software teams today employ a combination of QA techniques To achieve quality at speed, teams need to find the right
during software development. Apart from manual testing, balance between ensuring the quality of software and
which is performed by virtually all software teams, automated reducing the time needed to perform QA activities. They
system testing is the most widely used technique among all 11 cannot sacrifice valuable time to make error-free software.

techniques surveyed.

This survey found that monitoring and testing in

Still, just over a half (51%) of respondents applied automation production is the third most popular QA technique which
for system and integration testing, which is rather low given the was applied by 38% of respondents teams.

fact that it has become crucial for Agile and DevOps practices.
This finding suggests that software teams embrace the
possibility of errors in production and use the technique as
a way to strike a balance between quality and speed.

51% applied automation for

system and integration testing

38% used monitoring and testing
in production, making it the
third popular QA technique.

Executive Summary The State of Quality Report 2022 | 5

 3. Automated testing and code review/inspection 4. Frequent requirement change and the lack
are the two most effective QA techniques. of time to ensure quality are the most cited
challenges in delivering high-quality software.
Teams tend to use a QA technique when they find that it is
effective. This survey showed that the two most widely used Like our surveys in recent years, this survey found the same
QA techniques are also the most effective ones. They are two most common challenges in achieving high-quality
test automation (for integration, system, and acceptance software: frequent requirement changes reported by 46% of
testing) and code review/inspection, which were respondents and the lack of time to ensure quality by 39%.

recognized by 65% and 54% of respondents, respectively.

Frequent changes in requirements make the application of

65%
automation difficult as teams have to spend time revising
their automation tests to reflect the changes. With a lack of
Test automation time, teams tend to perform ad hoc manual testing instead

54%
of investing resources in more effective QA techniques such
as test automation.

Code review/inspection

Executive Summary The State of Quality Report 2022 | 6

 5. Professionals are satisfied with their software 6. The popularity of free open-source
quality upon delivery, but they are not equally automation tools continues to decline as
satisfied with their QA practices and tools. commercial tools are more powerful.
Tools play a key role in the success of test automation
When asked to offer their levels of satisfaction with the practices. Since 2018, our surveys have tracked the popularity
quality upon delivery, almost 8 out of 10 respondents of common tools for test automation.

rated their software quality as good or better.

One trend is clear: Selenium is the most widely used tool, but
its popularity has declined sharply over the last four years,
8/10 going from 86% in 2018 to 54% in 2020, and 37% this year.

Whereas, only 5 out of 10 respondents were satisfied

or very satisfied with their QA practices and tools. 86% 54% 37%

5/10 2018 2020 2021

This survey also shows that tools and frameworks such as
Katalon and Perfecto have become more powerful.
When it comes to applying test automation, many Commercial tools such as these eliminate the need to build
respondents were unhappy with their automation tools and automation frameworks based on Selenium and Appium for
practices, so they had to resort to manual testing instead. test automation.

Executive Summary The S tate of Quality R eport 2 0 22 | 7

 7. Test automation is increasingly 8. The most common challenges in applying
applied to more testing activities.
automation tools continue to persist.
Another trend found in this survey is that teams applied This year’s survey found that the most common
automation to more testing activities. While a majority of challenges with automation tools remain the same
respondents applied automation to regression testing as those reported in 2020.
(53%), they also applied automation to analyzing test
results (38%), generating test data (36%), and They include:
performing performance tests (34%)

The lack of skills and

It is possible that software teams have more mature and 37% experience in tools
powerful automation tools to perform more testing
activities automatically.

53% Regression testing 36% Frequently changing

requirements

38% Analyzing test results

These challenges are not necessarily essential as
automation practices and tools have the potential to
36% Generating test data eliminate the need for advanced skills required for
automation and to cope with changing requirements.

34% Performance testing

Executive Summary The S tate of Q uality Report 2022 | 8

 9. Most teams have achieved a high ROI 10. AI for test automation is still in its infancy.
with test automation.
Artificial intelligence (AI) offers great potential for test
Return on investment (ROI) is an important metric to automation. It can reduce or even eliminate the need for
measure the benefits of test automation. This survey asked human involvement in time-consuming activities such as
software professionals to rate their ROI in terms of the cost/ test design and generation, data preparation, test
time-saving of their investment in test automation.

execution, verification, and test maintenance.

Still, our survey found that less than 50% of respondents

use some AI capabilities for their test automation. They
mainly used AI for generating test cases and scripts, test
The good news is...
data generation, defect detection, and test prioritization
and selection. This finding suggests that organizations,
most respondents have seen a positive ROI
especially automation tool vendors, continue to improve
with the majority of respondents (63%) having
the adoption of AI for test automation.

experienced 20% or more cost/time-saving in

their test automation investment while only 6%
reported a loss and 9% reported no saving.

< 50%
Higher ROI is achieved when organizations of respondents use
provide sufficient training, choose the right some AI capabilities for
tools and tasks to automate, and demonstrate their test automation.
persistence in applying tools.

Executive Summary The State of Quality Report 2022 | 9

Quality at Speed:

Quality Assurance
Techniques and Challenges

There are many QA techniques that a software team can apply during the
development life cycle. They include, for example, manual testing, automated testing,
code review, code analysis, test-driven development, and pair programming. As these
techniques are different in terms of the application context and effectiveness, using
the right techniques is crucial for the success of achieving quality at speed.

This section reports the findings on the popularity of QA techniques, their

effectiveness, and challenges in achieving quality through using these techniques.

 Common testing techniques

Automated system testing is not as widely
used as people would expect.
51% Automated testing
To ensure software quality, teams can apply many QA techniques
ranging from code review and pair programming to automated
testing. In this survey, we asked participants which QA techniques 41% Code review
their teams applied during development.

The most commonly used techniques are automated testing (51%),

38% Monitoring and testing
in production
including integration, system, and acceptance, and excluding
automated unit testing, code review (41%), monitoring and testing
in production (38%), and automated unit testing (34%).

34% Automated unit testing

When it comes to ensuring the quality of the application, code review and
automation testing play a major role. Because doing effective code review itself can
ensure whether both functional and non-functional requirements are effectively met.
In addition to implementing automation for smoke, sanity, and regression,
organizations can save a lot of time in terms of quality build delivery.

-- Vishnu VS, QA Manager, Xilligence. -

Quality at Speed: Quality Assurance Techniques and Challenges The State of Quality Report 2022 | 11
 Which QA techniques does your team apply during development?

Automated testing

(integration, system, and acceptance) 51%

Code review/inspection

41%
(Artifact Review)

Monitoring and testing in production 38%

Automated unit testing 34%

Test-driven development 32%

Behavior-driven development 30%

Coding guidelines 28%

Static software analysis (including

28%
automated code analysis)

Dynamic software analysis 21%

Pair programming 21%

Model-driven development
20%
(including Model-based testing)

We omitted manual testing from the answers as it is presumably a QA technique performed by all software teams. If including automated unit

testing, 63% of respondents reported applying test automation in their projects. Besides these techniques, other widely used QA techniques

include test-driven development (32%), coding guidelines (30%), behavior-driven development (28%), and static software analysis (28%).

Q u a l i t y a t S p e e d : Q u a l i t y A s s u r a n c e Te c h n i q u e s a n d C h a l l e n g e s T h e S t a te o f Quality R e p o r t 2022 | 12
 Surprisingly, monitoring and testing in production is
the third most popular QA technique used by 38%
of respondents. It is an emerging technique that
involves monitoring and testing software in real
environments with real users and real data.

As a crucial part of DevOps and continuous delivery

practices, it has become a popular QA technique as Because of these inevitable gaps between the

more teams adopt these practices. To speed up testing and production environments, teams have

time to market, teams test their software builds to perform monitoring and testing in production.

early and frequently in production-like

environments.

The shift to this QA technique also serves as a way

to balance between quality and time to market:

Still, these environments cannot fully replicate the teams speed up the release of software builds and

true production environment in which users, data, continue testing them in production rather than

and usage scenarios are real. trying to produce perfect and error-free builds.

Quality at Speed: Quality Assurance Techniques and Challenges The State of Quality Report 2022 | 13
 The need to deliver quality software at speed has created
Automated testing

several trends in the software development landscape.

is likely used together with:

One is the increase in shift-left testing adoption through

employing various QA techniques early in the software

development lifecycle.
Automated unit testing
1
Our correlation analysis revealed that there are strong

positive relationships between QA techniques, which

Code review
means that teams tend to apply a number of QA 2
techniques together.

3 Coding guidelines

Monitoring and testing

4 in production

Q u a l i t y a t S p e e d : Q u a l i t y A s s u r a n c e Te c h n i q u e s a n d C h a l l e n g e s The St a te of Q ua l i t y Re p or t 2 02 2 | 14
 Two-thirds of respondents who performed
automated unit testing also applied automation Recommendations:
in integration, system, and acceptance testing.

This finding suggests teams embrace monitoring and

About 61% of those using code review also applied
testing in production as an approach to achieve
automated testing. Our survey also showed that
monitoring and testing in production is used in quality at speed. Regardless of how much they test
conjunction with most QA techniques, which is pre-release, there are inevitably bugs in production.
reasonable since there is no other alternative QA
techniques used in production. Thus, instead of wasting valuable time to make
bug-free software and sacrificing time to market,
teams need to prioritize monitoring and testing in
production. It is an important complement to

61%
of those using code pre-production testing. At the same time, teams
review also applied
can build mechanisms to recover software from
automated testing.
failure, making the practice of discovery and
recovery from failure an important part of DevOps.

Quality at Speed: Quality Assurance Techniques and Challenges The State of Quality Report 2022 | 15
 The most widely used QA techniques

being the most effective ones

Automated testing is the most effective QA

technique, followed by code review/inspection.

Software teams use a QA technique because of its

effectiveness. It is true as our survey revealed that the most

popular technique, automation for integration, system, and

acceptance testing, is also the most effective, reported by 65%

Noticeably, users of a technique tend to view it more

of respondents whose teams used this technique.

favorably than those who do not use it. For example, only 17%

of all respondents vouched for pair programming, while 41%

said the most popular technique

65% is also the most effective one. of users of the technique identified it as the most effective.

Code review, monitoring and testing in production, and

These results seem to suggest that it is worth

automated unit testing are other most effective techniques,

evaluating less popular techniques such as

which were applied by more than 50% of respondents. As we

dynamic software analysis, pair programming,

can see in this result, respondents generally viewed that the

and model-driven development if they are
most widely used QA techniques are also the most effective.

suitable for your project’s requirements and

environments. They could potentially turn out

applied code review, testing in

50% production and automated unit testing. to be effective for your projects.

Q u a l i t y a t S p e e d : Q u a l i t y A s s u r a n c e Te c h n i q u e s a n d C h a l l e n g e s The St a te of Q ua l i t y Re p or t 2 02 2 | 16
 Most Effective Techniques

Automated testing

65%
(integration, system, and acceptance

Code review/inspection

54%
(Artifact Review)

Automated unit testing 52%

Monitoring and testing in production 50%

Behavior-driven development 49%

Test-driven development 49%

Dynamic software analysis 46%

Static software analysis (including

43%
automated code analysis)

Coding guidelines 43%

Pair programming 41%

Model-driven development (including

38%
Model-based testing)

Q u a l i t y a t S p e e d : Q u a l i t y A s s u r a n c e Te c h n i q u e s a n d C h a l l e n g e s T h e S t a te o f Q u a l i t y R e p o r t 2 0 2 2 | 1 7
 QA teams and challenges in high software quality

Frequent requirement changes and the

lack of time to ensure quality are the

Common challenges you face when
most cited challenges in delivering
delivering high software quality
high-quality software.

Frequent changes in requirements 46%

The need to deliver quality at speed creates a shift in

QA approaches in which software teams have to

Lack of time to ensure quality 39%

employ faster release cycles, and test early and often

Applying test automation 36%

in production-like environments.

Lack of experienced and skilled

35%
human resources

However, frequent requirement changes and the lack

Short release life cycles 34%

of time directly go against implementing this shift.

Lack of resources 31%

Lack of mature processes 30%

Unclear/unrealistic goals and targets 29%

Lack of mature tools/technology/software 28%

Quality at Speed : Q u a l i t y A s s u r a n c e Te c h n i q u e s a n d Challenges The St a te of Q ua l i t y Re p or t 2 02 2 | 18

 These are the two most common challenges reported
by respondents in our survey with 46% citing frequent
requirement changes and 39% citing the lack of time
as obstacles for delivering high-quality software.

46% Frequent
requirement
39%
changes Lack of time

Frequent changes in requirements force teams to revise their When the team lacks time, they are unable to invest
tests for regression testing, which reduces the effectiveness of resources upfront for test automation, and instead resort
regression testing that is usually applied during automation. to manual testing. With the lack of automation, it
They also have to perform more manual testing instead, which becomes more challenging to pursue short and rapid
slows down the process of testing and delivery overall.

release cycles, leading to a vicious cycle of inefficiency.

Quality at Speed: Quality Assurance Techniques and Challenges The State of Quality Report 2022 | 19
 Level of satisfaction in software quality,

QA practices and tools

Although people are satisfied with their software

quality upon delivery, they are not equally

satisfied with their QA practices and tools.

The level of satisfaction related to software quality upon Despite challenges in delivering high-quality software, most of the

delivery is probably one of the most important measures of surveyed individuals are satisfied with their software quality. An

success for software teams. Our survey asked respondents overwhelming 78% of respondents rated their software quality

to offer their perceived levels of satisfaction with software upon delivery as good or better while just 7% and 14% of them

quality upon delivery, QA practices, and tools. reported poor and fair software quality upon delivery, respectively.

37%
32%

What is the level of your

14%
7% 9%
software upon delivery?

Poor Fair Good Very good Excellent

Q u a l i t y a t S p e e d : Q u a l i t y A s s u r a n c e Te c h n i q u e s a n d C h a l l e n g e s The St a te of Q ua l i t y Re p or t 2 02 2 | 20
 However, when asked whether they were satisfied with Roughly 20% and 22% of them were not satisfied with the
the QA practices and tools used for their projects, their QA practices and tools used for their projects, respectively.
satisfaction levels are not equally high compared to Many participants in our follow-up interviews noted that
those of software quality. About half of the they were not very happy with automation tools and had to
respondents said they were either satisfied or very rely on manual testing instead.

satisfied with their QA practices (53%) and tools (50%).

How would you rate your satisfaction

with your team's QA practices? 45%

27%

14%
6% 8%

Not at all
Not satisfied Neither satisfied
Satisfied Very satisfied
satisfied nor dissatisfied

Quality at Speed: Quality Assurance Techniques and Challenges The State of Quality Report 2022 | 21
 These results showed that while people are happy ...but they look forward to improving their practices
with their software quality upon delivery, they expect and using better tools in the long run. “We are satisfied
more from their QA practices and tools. At present, with the QA practices and tools used in our projects,”
they can overcome limitations in QA practices and said one interviewee, “but in a highly competitive
tools to deliver high-quality software...
market, tools with AI technology integrated seem to
be the right approach for the future.”

How would you rate your experience

with tools used for your project? 43%

28%

15%

7% 7%

Not at all
Not satisfied Neither satisfied
Satisfied Very satisfied
satisfied nor dissatisfied

Quality at Speed: Quality Assurance Techniques and Challenges The State of Quality Report 2022 | 22
Test Automation
Test automation is reported in this study as the most widely used and
effective QA technique. It is essential for achieving quality at speed in
today’s software development with Agile and DevOps methodologies.

This section describes the findings related to the automation tool

landscape, common challenges in applying test automation, and the
return-on-investment for test automation.
 The key benefit of automated testing (across your application layers)
is feedback. Ultimately, we have to mitigate against the introduction of
bad code into our software codebases.

When software is developed under a test-driven development paradigm,

and the tests are written with feedback and information sharing as the
goal, the inevitable bugs that our automated testing pipelines discover are
rapidly resolved. The reason why we want tests to run quickly, and to fail
quickly, is to get feedback.

Ron Powell

CircleCI

Manager, Marketing Insights and Strategy

Te s t A u to m a t i o n The State of Quality Report 2022 | 24

 The continued decline in popularity
of free open source automation tools

A successful application of test automation has to start Selenium continues to be the most popular automation tool,
with choosing the right set of tools. There is a wide which is used by 37% of respondents. Yet, its popularity has
variety of test automation tools that software teams can declined sharply over the last four years, going from 86% in
choose from, ranging from open source to commercial 2018 to 54% in 2020 in our previous surveys.

tools, and something in the middle.

Selenium 86% 54% 37%

2018 2020 2021

Te s t A u to m a t i o n The State of Quality Report 2022 | 25

 One reason is that automation tools and frameworks This trend of declining reliance on building custom

based on Selenium such as Katalon and Perfecto have frameworks using open source solutions for test automation

become more powerful to eliminate the need of building is good news for the software community as they would

custom frameworks using Selenium.

spend less time and expertise on building and integrating

various open-source solutions and more time focusing on

This survey also found Postman as a popular alternative what really matters – automated testing.

to Selenium for API testing and monitoring.

37%
What automation tools/
32%

frameworks is your team using?

21% 21%
18% 18%
16%
15%

12% 12% 12%

11%
10%
9% 9%
8%

Selenium Postman Katalon

TestNg Cucumber Apache
Appium SoapUI Cypress Google-
Browser-
Robot
Cross-
IBM
UFT Others

Studio JMeter Test Stack Framework Browser Rational..

Te s t A u t o m a t i o n The St a te of Q ua l i t y Re p or t 2 02 2 | 26
 The race to adopt test automation

Test automation is increasingly

applied to more testing activities Which of the following testing activities
It is a fact that when teams apply test automation,
does your team apply testautomation for?
regression testing is the activity they start with. This
Performing regression tests 53%
survey confirms this strategy with 53% of respondents
applying automation for regression testing.

Generating test cases and test scripts 42%

However, it is increasingly used for other activities Performing functional tests 41%
including analyzing test results (38%), generating test
data (36%), and performing performance tests (34%).

Performing UI tests 41%

Analyzing test results 38%
Generating test data 36%
Performing performance tests 34%
Performing unit tests 32%
Others 1%

Test Automation The State of Quality Report 2022 | 27

 This finding suggests that as automation tools become more
mature and powerful, they offer greater capabilities for
performing more testing activities that have been traditionally
done using manual testing. In other words, more testing activities
and types that have been so far tested manually will be performed
automatically using tools in the future.

The application of AI technologies to automate more testing

activities and to offer intelligent decision support for test
automation is likely a trend in the advancement of tools.

Test Automation The State of Quality Report 2022 | 28

 Persistent obstacles in

applying automation tools

The two most common challenges in applying test Indeed, the top five challenges remain the same as

automation are the lack of skills and experience in tools our previous survey results except that this year's

(37%) and frequently changing requirements (36%). This study shows an increase in the level of the challenge

finding is the same as our survey reported in 2020. QA related to script maintenance (32% and ranked 4th)

practices and tools have changed much over the years, but compared with the finding in 2020 where the

these challenges continue to persist.

challenge was ranked relatively low (5th).

Surprisingly, this survey shows that the tool offerings

The most common challenges in applying

is ranked the lowest although it was ranked high in

automation tools continue to persist.

our previous survey. This finding indicates that test

automation tools have become more mature and

powerful in satisfying testing needs.

37%
36%

Lack of skills and Frequently changing

experience in tools requirements

Te s t A u t o m a t i o n The St a te of Q ua l i t y Re p or t 2 02 2 | 29
 The most common challenges that your team
has when using test automation tools?

Lack of skills and experience in test

37%
automation tools

Requirements change too often 36%

Difficult to perform different types of testing

34%
(usability, performance, security, etc.)

Fragile test scripts and high costs of script maintenance 32%

Difficult to integrate different tools/frameworks 31%

Difficult to organize and manage test artifacts

29%
(test cases, scripts, data, reports, etc.)

The adoption of automation tools does not match the current

25%
process and testing practices

Lack of tutorials and support from the provider/community 24%

Lack of reporting and analytics capabilities for test results 22%

The tool offerings don’t match the team’s testing needs 22%

Others 2%

Te s t A u t o m a t i o n The S t a t e o f Q u a l i t y R e p o r t 2022 | 30
 Factors to ensure positive ROI

Most people have achieved a high ROI

for test automation.

Despite challenges, the good news is that the return on What is your return on investment (ROI)
investment for test automation is relatively high. Most of test automation for your project?
respondents have seen a positive ROI with the majority of
respondents (63%) reporting to have seen 20% or more of the 38%

cost/time-saving in their test automation investment. Only

6% of them experienced a loss, and 9% have seen no saving.

The level of ROI depends on a number of factors such as the 21% 21%

effectiveness of training, choosing the right automation tasks,

test maintainability, reusability, and persistence in applying 9%
the tool.

6%
5%

Organizations should provide sufficient training to their teams

Negative None (no cost/ Low (less than Average (20% High (20-50% Very high
on practices and tools when applying automation

(making a loss) time saving) 20% cost or cost or time cost or time (50%+ cost or
time saving) saving) saving) time saving)

Automation is known to be challenging and time-consuming

in the beginning, which causes many teams to give up. One of
our follow-up interviews pointed out that many companies
would acquire an automation tool and use it for initial days,
but eventually return to their regular practices.

Te s t A u to m a t i o n T he S tate of Q uality Report 2022 | 31

Artificial Intelligence
in Test Automation
The application of artificial intelligence to test automation has become an
exciting trend in recent years. The software industry has seen a sharp increase in
tools that offer AI capabilities for test automation activities ranging from smart
test creation and data generation to intelligent test maintenance and execution.

These advancements in the application of AI to software testing have the great

potential to overcome many challenges in automation.
 The advancements in AI open up opportunities to automate

many testing activities that have often been done manually.

With AI-enabled tools, time-consuming activities such as

test design and generation, data preparation, verification,

and test maintenance can be more efficient and effective.

Example

AI-enabled functionality can learn to

automatically revise test cases and scripts

whenever the AUT is changed, reducing the effort

spent for maintaining tests and enabling teams to

speed up the regression testing process.

A r t i fi c i a l I n t e l l i g e n c e i n Te s t A u t o m a t i o n The St a te of Q ua l i t y Re p or t 2 02 2 | 33
 AI for test automation
Scripts
is still in its infancy. 35%
generation

Despite people's high expectations, AI for automation is still in

its early stage of application. Our survey showed that less than

50% of respondents reported using some AI capabilities for

their test automation. This result is surprisingly low given that AI

has received a lot of attention from the test automation Test data
community in recent years.

32%
generation

Among those who use AI capabilities with their automation

tools, about a third use it for automated test cases and scripts

generation (35%), test data generation (32%), automated defect

detection (30%), and test prioritization and selection (30%).

Automated
defect detection
30%
Generating test cases, scripts and data is a crucial process for

test automation. Tools with AI capabilities can improve this

process by automatically creating test artifacts faster and better

than human testers. Also, the AI-enabled test prioritization and

selection capabilities allow testers to reduce the number of

tests needed and optimize the process of test execution.

Test prioritization
and selection 30%

A r t i fi c i a l I n t e l l i g e n c e i n Te s t A u t o m a t i o n T h e S t a te o f Q u a l i t y R e p o r t 2 0 2 2 | 3 4
 There are several reasons
for the low adoption of AI.

1 One reason is that AI capabilities for automation are

2 Another reason is the lack of tools that support AI for

still limited and underdeveloped. They are presently used for many interleaved activities of the testing life cycle. Today

simple and easy-to-automate activities such as generating many tools integrate AI to address just a few specific and

simple test cases from requirements using natural language narrow problems, e.g., handling test flakiness using auto-

processing (NLP) and comparing screens using computer healing and reducing script coding via exploring elements on

vision. Complex testing activities such as automated test data the user interface. This lack of comprehensive AI-enabled

creation, assertions generation, and test evaluation require tools requires teams to adopt various tools if they want to use

more mature AI technologies to handle.

AI technologies.

3 The complexity of the test automation process and

the challenge in introducing and integrating AI technologies

to automation tools are also the reasons. AI models require a

lot of data to train and validate, but software organizations are

not ready to share their data.

A r t i fi c i a l I n t e l l i g e n c e i n Te s t A u t o m a t i o n T h e S t a te o f Q u a l i t y R e p o r t 2 0 2 2 | 3 5
 Trends in AI for test automation
Although receiving a low level of adoption today, AI has a lot to offer
for test automation to support a wide range of testing activities.

Our follow-up interview participants suggested that integrating AI

capabilities in test automation tools could take the automation
testing process to a different league altogether. An automation tool
with AI capabilities can significantly reduce the effort for time-
consuming processes such as scripting, test design, test data
preparation, script maintenance, and failure analysis.

We expect more advanced AI technologies to be developed and

applied for test automation. More comprehensive automation tools
will leverage AI technologies to support a wide range of testing
activities, resulting in more autonomous testing life cycles.

Artificial Intelligence in Test Automation The State of Quality Report 2022 | 36

 W e will likely see further advancement and integration of AI
technologies in the following areas of testing in the future:

Test Planning and Management 2. Test Generation and Maintenance 3. Test Selection and Optimization

Intelligent software analysis

Automated test case, script, and data generation
Test prioritization and selection

Test effort estimation

Automated test migration
Test suite optimization

Intelligent test decision making

Test reuse analysis
Test impact analysis

Test efficacy and robustness analysis

Automated test repair
Test coverage optimization

4. Test Execution 5. Test Evaluation & Report 6. Test Monitoring

Applying computer vision to

Automated defect detection
Usage pattern analysis

automation (visual testing)

Intelligent test analytics
Performance analysis
Application exploration Defect prediction

Defect classification and triage

Artificial Intelligence in Test Automation T h e State o f Q ualit y Report 2 0 22 | 37

 Do you use the following AI capabilities
in your test automation tools?

35%
32%
30% 30%
24% 24% 23%

Automated test Automated test Automated defect Test prioritization Intelligent Automated defect Intelligent test
cases and script data generation detection and selection software analysis correction reporting
generation

Artificial Intelligence in Test Automation The State of Quality Report 2022 | 38

Looking Ahead
to 2022
 Our survey asked respondents about which QA practices Improvements in adopting automation should be a

they planned to improve and what quality goals they would high-priority action for many teams to address the

pursue in the coming year, finding that the increase in challenges in automation-related practices and tools

adoption of test automation continues to be a trend as that remain. Teams need to cope with the lack of skills

more than 60% of respondents said they are likely/very and resources for automation and find ways to

likely to improve their adoption of automation (automated overcome the problems caused by frequently

unit testing, automated test case/data generation, and test changing requirements.

execution) in the coming year.

Looking Ahead to 2022 The St a te of Q ua l i t y Re p or t 2 02 2 | 40

 The prolonged COVID-19 pandemic forces software These issues emphasize the necessity of
professionals to continue working from home, affecting their improving QA practices, processes, and tools
well-being and mental health. Mental health issues such as employed by teams.
stress, fatigue, exhaustion, and loss of motivation can
severely influence the effectiveness of QA activities.

What are your primary software

quality goals in 2022?
67%
32% 59%
55% 50%
39% 38%

5%

Improve QA Increase Increase test Improve code Reduce Improve the Others
processes and automated coverage quality technical application of AI
practices tests debt approaches

Looking Ahead to 2022 The State of Quality Report 2022 | 41

Case Study Analysis
Technology Solution Provider Reduces Time to Market,

Execution, Maintenance Time, and Stress With Katalon

 Summary

A leading technology solutions provider tapped Katalon to replace its legacy

test automation system. In doing so, the firm was able to significantly reduce

both maintenance and execution time, allowing it to better serve its

pharmaceutical company partners with powerful research that enables them

to get life-saving drugs to the market.

“We selected Katalon because it gives us the ROI we wanted and it’s very

stable. Now if our test script fails, we know it’s the AUT’s fault, not the

platform’s,” said the QA manager. “Instead of thinking about which test to

execute based on a code change, we can selectively execute those that are

relevant to the impact analysis result.”

Case Study Analysis The St a te of Q ua l i t y Re p or t 2 02 2 | 43

 The Challenge

The client, a leading technology solutions company for The company needed to streamline and simplify its

clinical trial innovation and evidence, aims to empower its software testing automation so that its engineers could

partners to transform lives by parlaying deep scientific spend less time maintaining scripts and more time

expertise at a global scale into the broadest possible analyzing them to identify problems and more rapidly

endpoint technology platform.

iron out product defects.

The company is committed to innovating the future of

10%
clinical trials through technology and having a positive
of their time

writing them.
impact on global healthcare.

90%
As the client began to rapidly scale its test automation of their time

maintaining scripts

operations, it found that its developers were spending

about 90% of their time maintaining scripts and only

10% of time writing them.

Consequently, the company could scale faster and better

serve its clients and partners, who are mainly large

pharmaceutical companies.

Case Study Analysis The St a te of Q ua l i t y Re p or t 2 02 2 | 44

 The software tested would be used to analyze data

from large clinical studies. And since the outcome

could potentially contribute to life-saving drugs,

engineers needed to be certain their testing

automation was as efficient and effective as possible.

“We have a system where we deliver studies to the customers, and every

study is different,” said the manager. “Before we release those studies

into production to users, we need to test them, and it becomes

impossible with the increasing volume of studies.

Scaling horizontally by delivering more resources to deliver more studies

was not a viable solution. So we started looking at automation so that we

didn’ t have to test every study again and again.”

Case Study Analysis The St a te of Q ua l i t y Re p or t 2 02 2 | 45

 The Solution

After conducting a thorough ROI analysis of several test automation

tools on the market, the client chose Katalon-specifically, Studio

Enterprise and Runtime Engine-for its testing automation solution,

and began using Katalon in 2018 to test web (user interface “Sometimes they worked,

functionality) and native mobile applications (both iOS and

sometimes they didn’ t,” said the manager.
Android), as well as APIs.

2018
“And the manual testers in my team

found it very challenging to start

Katalon
scripting right away, while still having

to constantly fix the old tests and have

The company had done manual testing for the last 11 years before

less effort to develop newer scripts.”

switching to test automation. After a number of years employing

Selenium-based frameworks, namely Robot Framework,

Protractor and Selenium itself, the client found that it was too

programming-centric and required too much maintenance.

Case Study Analysis The St a te of Q ua l i t y Re p or t 2 02 2 | 46

 Our work is in a heavily regulated industry so

we need a lot of documentation around whatever

we do. We need to document our test steps with

evidence because we are subject to 70 to 80 audits

a year. So documentation was a key part of it.

The QA manager and his team eventually built a

framework around the mentioned open-source test

automation framework to help the engineers automate

“We integrated the testing into our pipeline and we

it and to give the auditors confidence in the designs.

never had a pipeline that was clean-never. We were

“But what we found over time was that the always executing a test on a local machine and

maintenance became a nightmare for us,” he said. saying, ‘here it works, there it doesn’ t, and we don’ t

know why,’ so we wanted to move on and wanted to

do something that was more stable.”

Case Study Analysis The St a te of Q ua l i t y Re p or t 2 02 2 | 47

 Seeking more stability and faster releases, the company From validation, staging to production environments,

began using Katalon for web and mobile application there was a set of different profiles for their automated

testing. “Everybody was immediately happy with it,” the test scripts to run against.

manager said.

Additionally, the support for Docker images in Katalon

Studio has also made everything much easier and

Using GitLab as a CI/CD system, the client has 50 test

quicker with lower memory consumption of their

team’s machines.
suites running in tandem with about 150 scripts in total

and executes them using Kubernetes with one dynamic

port for each suite.

Test engineers have integrated their tests into pipelines

and execute the tests automatically with every code build.

Case Study Analysis The St a te of Q ua l i t y Re p or t 2 02 2 | 48

 The Results

More Efficient Use of Internal Resources Significantly Reduced Execution

And Maintenance Time

After a six-month-long process of rewriting their

legacy scripts to migrate to Katalon Studio, their Before, running through hundreds of test scripts, one after

team immediately saw the results they wanted, another, took the firm weeks. As a solution, the firm implemented

especially in optimizing internal resources.

some new key test designing concepts: limiting the execution time

to only 15 minutes per script and breaking those that exceed the

set time into multiple scripts to run parallely.

“It was definitely worth the effort. With the

“Our execution time did actually come down from several hours to

same team, I was able to handle more

minutes and executed all our test suites within 15 minutes – and

business,” the manager said. “Not that we

that’s for hundreds of scripts. Also, our maintenance time has gone

let people go - we took on more work with down by at least 60%. Katalon’s AI capabilities such as Self-healing

and Smart Wait help reduce maintenance time and test flakiness.”,
the same team and level of productivity.”

said the QA manager

15 mins

- 60% maintenance time

Case Study Analysis The St a te of Q ua l i t y Re p or t 2 02 2 | 49

 Transformation Of DevOps Infrastructure Productive Failure Troubleshooting

“It was very easy to plug test automation into our pipeline, at “Katalon has introduced a lot of features for debugging
any stage, with Katalon,” an engineer said.

that are really helpful. If our test fails at a particular point in

time we can start our execution from the same point again.

“Every test suite becomes a test automation stage for us.

Also, the biggest challenge we had with every other tool We don’t have to go back 50 steps and re-execute. We
was maintaining compatibility between the browser version can put breakpoints in between and do better debugging.”

and the driver version.

With Dockerization, we don’t have this problem and

Katalon makes Dockerization very easy. We can use any
version of Chrome we want.

And also, one test suite issue doesn’t have an impact on

another one. So if a node goes down, no problem, the test
will still get executed.”

Case Study Analysis The State of Quality Report 2022 | 50

 Enhanced Product Quality Security Compliance
“We’ve reduced our production defects from a large number to Operating in a highly regulated environment, not
lower single-digit numbers for each product,” the manager said.

having to work with additional third-party software was

key for the client’s data to not be at risk. Test reports, in
“A key part of our mission is delivering high-quality products to particular, were generated automatically and locally
our customers. Our work is also measured on what percentage stored in Katalon Studio.
of our requirements are covered through automation tests.

That is a key metric that I need to report to executive leadership

every month. And Katalon has helped me with that.”

Improved Time To Market and Reduced Stress Levels

“Switching to Katalon has allowed us to not miss a single

committed timeline in the last three years,” the manager said.
“And that’s something that has never happened before. We
used to say, ‘Well, we can’t release it now because we haven’t
completed testing.’ That case doesn’t happen anymore.”

Overall, the fear of unfinished tests and delivery dates nearing

no longer were piling on the pressure for their QA engineers.

Case Study Analysis The State of Quality Report 2022 | 51

Thank you for sticking with us till the end and becoming a part of “The State

of Quality Report 2022.” We hope the report's insights have been helpful to

both you and your customers to thrive better.

At Katalon, we always keep a close eye on every slightest change in the test

automation space to see where we can help steer the industry forward. And of

course, this mission is forever incomplete without your comments. Yes, yours.

So why not LEND US YOUR THOUGHTS and win a $30 E-gift card?

May we ask just 4 simple questions to improve our report?

Yes, sure
Appendix
 Methodology

This study was carried out in two phases. In the first Although we conducted the study by following a
phase, we conducted an online survey of 3,000+ thorough process of designing the questionnaire,
respondents with direct involvement in quality practices choosing audiences, and analyzing the results, we do not
of software development projects.

claim it to be fully scientific. All online surveys reflect the

opinions of audiences in online communities rather than
The survey consisted of 28 questions concerning the overall population of software development and
software quality techniques, test automation, and the testing communities.

adoption of AI. Our survey was distributed via various

channels of software development and testing
communities around the world.

In the second phase, we performed follow-up interviews

with experts and experienced professionals to gather
their insights and opinions about the results we obtained
in the first phase. The results obtained from the first
phase and the prepared questions were shared with the
interviewees during the interview.

Appendix The State of Quality Report 2022 | 5 4

 Demographics
Of over 3,000 respondents who answered our survey,
Professional role
QA engineers and managers make up the largest
group with a total of 44%, including 18% engineers for
manual testing, 15% engineers for automation testing, 38%
and the remaining 15% managers.

The second largest group consists of IT engineers,

analysts, and consultants who were involved in various
QA and development activities in their projects. The 18%
15%
participants in this survey also include software 10% 11%
engineers (10%) and a small group of senior managers
such as directors, vice presidents, and CTO (3%). 5%
3%

Automation Manual QA Software IT engineer/ QA Lead/ Director/ VP/ Others

QA engineer engineer engineer Analyst/ Project CTO
Consultant manager

Appendix The State of Quality Report 2022 | 55

 Team size

As this survey is focused on team experience, it

included only the respondents who worked with 13% 22%
at least one teammate. As the results show, the
majority of respondents worked in teams of 6 - 10 11%
members (32%).

Twenty-two percent of respondents belonged to

teams of 2 - 5 members, the same as those 22%
belonging to teams of 11 - 20 members. The
32%
remaining (24%) of respondents worked in teams
of 21 members or more.
2-5 members 6-10 members
11-20 members 21-30 members
More than 30 members

Appendix The State of Quality Report 2022 | 56

 What are your company’s industries?

Half of the participants in this survey worked in Software and Information Technology 50%

the software and information technology sector.

Banking, Financial Services and Insurance 26%
The data shows that the majority of them
developed software applications for other Healthcare and Life Sciences 20%

sectors. A quarter of participants come from the Telecom, Media, and Entertainment 19%
Banking, Financial Services, and Insurance sector.

Consumer Products, Retail, and Distribution 17%

The survey also consists of participants from other Government and Public Sector 15%
industries: Healthcare and Life Sciences (20%);
Manufacturing 14%
Telecom, Media, and Entertainment (19%);
Consumer Products, Retail, and Distribution Automotive 13%
(17%); and Government and Public Sector (15%).
Energy and Utilities 13%

Transportation & Logistics 11%

Others 4%

Appendix T he S tate o f Q ua l ity R eport 2022 | 57

Acknowledgements
Katalon Team would like to thank...

the 3,000+ respondents for their time and contribution and our media partner, Xilligence, Testbits, Nihilent,

by taking part in the research study this year,

CircleCI, Trigent, BMP Technologies, Syone, QAlified

who has helped us distribute the survey to a wider audience.

the testing communities who have graciously helped our

The identity of the participants in the research study
survey reached and informed more people,

and their responses shall remain confidential.

Shape your automation testing future with us!

Katalon is the leading provider of software test automation solutions. The company offers a flexible

platform for web, API, mobile, and desktop testing that fits teams and projects of any size, for any purpose

from test creation, execution, reporting to seamless integration with popular CI/CD systems.

Widely used by a global network of 100K enterprises and 850K users across 160+ countries, Katalon is

high-ranking on trustworthy review sites such as Gartner, Capterra, and IT Central Station. Katalon

solutions include Katalon Studio, Katalon TestOps, Katalon Recorder, Katalon TestCloud and Katalium.

For more information, visit www.katalon.com.

Our products:

General & Legal inquiries: [email protected]

License & Purchase inquiries: [email protected]

Partnership inquiries: [email protected] Follow us: