2024

State of
Continuous
Testing
Explore testing
trends from test
automation experts

© Perforce Software, Inc. All trademarks and registered

trademarks are the property of their respective owners. (0220KS24)
 Welcome to the 2024 State of
Continuous Testing Report
In the inaugural edition of the State of Continuous Testing Report,
our Perfecto and BlazeMeter experts teamed up to take a deep dive
into industry trends—including testing trends, automation rates, the
use of AI and ML, and concerns and priorities for 2024.

Perforce surveyed hundreds of leading digital enterprises across a wide

range of industries to answer these questions.

Here’s what we found, and where continuous testing is going this year.

KEY TAKEAWAYS
• Teams are embracing hybrid cloud testing.
To do so, a versatile platform is needed that can
effectively support hybrid environments.

• AI-augmented testing is a huge trend.

Teams are looking to incorporate AI into their testing
process but only some have currently implemented it.

• Lack of automation resources challenges teams the most.

Teams continue to prioritize automation but lack the resources
to fully implement it.

• Accessibility testing is among the top investment priorities.

Alongside performance automation testing, accessibility testing
continues to grow in importance.

2 | 2024 State of Continuous Testing © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0220KS24)
 Mobile Testing Remains Highly Important In addition, Perforce is proud to be the first in the industry
to offer UX load testing that simulates real-world conditions
Over half of respondents (54%) selected 4 or 5 when asked about the importance of native for your mobile devices. You can test both your mobile user
and mobile web apps to their testing strategy, indicating that testing remains a high priority. experience and your backend under load in the cloud and
scale up to two million virtual users. The solution simulates
ON A SCALE OF 1 (LOW) TO 5 (HIGH), PLEASE RATE HOW IMPORTANT realistic mobile traffic patterns. You can even generate
MOBILE NATIVE AND MOBILE WEB APPS ARE IN YOUR OVERALL detailed server and end-user experience reports, which
TESTING STRATEGY? helps teams measure and understand how a mobile app
is impacted under a load.

1 2 3 4 5 Related Reading: Achieving Continuous Testing

With Perforce

The release of Scriptless Mobile also helps teams

automate the most complex native mobile use cases
through simplified, rapid test creation with as many test
18% 10% 18% 20% 34% combinations as you need to boost your testing efficacy.

Related Reading: Automated Mobile Testing With Perfecto

Scriptless Mobile

As testing teams look to expand the reach of mobile testing, more use cases must be added
to their mobile testing suites. In addition, more types of testing, such as User Experience
(UX) testing, are becoming increasingly important.

Perforce’s continuous testing suite offers a comprehensive solution for mobile testing Over half of respondents
challenges, particularly in areas such as UX load testing. The user-centric interface makes (54%) selected 4 or 5 when
advanced testing accessible even to non-technical team members, enhancing usability
asked about the importance
across complex mobile use cases, including two-factor authentication (2FA), geolocation,
of native and mobile web apps
voice recognition, network virtualization, and more.
to their testing strategy.

3 | 2024 State of Continuous Testing © Perforce Software, Inc. All trademarks and registered
3 | 2024 State of Test Automation trademarks are the property of their respective owners. (0220KS24)
 Interest in AI Testing Surges, For many, it feels like AI is the future of testing. AI enables faster and
smarter test creation by generating scripts instantly, so teams no longer
But Implementation is Lagging rely on manual or hybrid coding. AI can also assist by generating reliable
test data—which is often a pain point for testers.
Artificial Intelligence (AI) and Machine Learning (ML) have been surging
in popularity, particularly over the past year. While organizations are hoping Test Data Pro optimizes test data generation, addressing common AI
to adopt AI and ML to help with the testing process, implementation is still implementation challenges by providing reliable and usable test data,
falling short. vastly expanded test coverage, increased and reliable accuracy, speed
of testing, and bolstered app resilience. This AI-driven data helps solve

HOW DO YOU FORSEE THE ROLE OF AI ENHANCING the problem teams often face of acquiring usable and reliable test data.

YOUR TESTING PROCESS IN THE NEAR FUTURE? Test Data Pro assists with security by preventing customers from using real
data instead of synthetic data.

Some examples of the type of data Test Data Pro can generate:

7% • Addresses
11%
6%
• Date and time
Interested but haven’t started

Unsure • Credit card numbers

28% Don’t see a significance for AI
• Bank account details
Already implementing AI techniques
• Seed lists
Actively researching & planning to adopt
48%
Related Reading: BlazeMeter Test Data Pro: AI-Driven Test Data
Generation & Beyond

Perforce is continually innovating to remain in step with developments

in technology and new releases to ensure a seamless testing experience
While 48% of respondents indicated they are interested in AI but have not
for customers that stays ahead of market trends.
yet started any initiatives, only 11% are already implementing AI techniques.
Related Viewing: The Future of Testing: A Conversation About the Use
The interest is strong, however, with 40% saying they are either already
of AI and ML
implementing or actively researching and planning to adopt AI—indicating
that AI will continue to be a predominate force in testing in 2024.

4 | 2024 State of Continuous Testing © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0220KS24)
 Hybrid Test Environments Teams Embracing New
Are the Most Popular Testing Trends
Hybrid test environments are the most popular testing environment (31%) The Internet of Things (IoT) is a trend that promises to continue gaining
followed closely by cloud-based solutions (29%). These two environments traction in 2024. Testing using virtual reality, blockchain application testing,
continue to beat out on-premise hardware, further validating the fact that and service mesh architecture testing are also growing in popularity.
hybrid and cloud-based testing is the way of the future.
ARE YOU CURRENTLY IMPLEMENTING OR CONSIDERING
WHICH TEST ENVIRONMENT D
 O YOU PRIMARILY USE? ANY OF THE FOLLOWING NEW TESTING APPROACHES?
CHECK ALL THAT APPLY.

31%

29%
On-premise hardware

Cloud-based solutions Testing using VR or AR environments 47%

19%
18%
Other
IoT device testing 50%
Virtual environments
Blockchain application testing 33%
Hybrid
Service mesh architecture testing 44%

3%

IoT is a booming market and will need to be integrated into the continuous
Hybrid and cloud testing are growing in popularity due to their versatility, testing process. Continuous testing, or shift-left testing, helps ensure IoT
flexibility, and generally lower costs. Teams are increasingly looking for devices remain functional. As IoT applications need to adapt frequently and
testing platforms that can support both hybrid and cloud environments. rapidly to new requests, organizations will need to adopt technologies and
Perforce supports this growing popularity of hybrid and cloud testing, processes to incorporate IoT device testing.
along with all forms of testing, including hybrid, cloud, and on-premise
Our solution enhances IoT testing through its Jenkins plugin, a popular
device testing. Regardless of your team’s unique testing needs, Perforce’s
CI/CD tool, offering streamlined integration and automation capabilities.
far-reaching support allows for comprehensive testing at scale in one
intuitive and easy-to-navigate location. Related Reading: IoT Continuous Testing With Jenkins & BlazeMeter

Related Reading: The Drawbacks of a DIY Mobile Device Lab Setup

5 | 2024 State of Continuous Testing © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0220KS24)
 Lack of Automation Resources Test environments, and acquiring usable and reliable
test data, can be a frustrating bottleneck to the testing
& Environment Concerns process. Traditional Service Virtualization applications

Challenges Teams Most can be time consuming and complex (not to mention
costly) to set up, and mocks and stubs do not always
In 2024, the most frustrating challenge facing teams is environmental concerns, including account for real-world scenarios.
test data, service virtualization, virtual devices, and more. Additionally, 47% of respondents
Mock Services directly address environmental concerns
indicated that lacking resources for creating test automation was a prominent pain point.
by creating on-demand Virtual Services in the CI/CD
pipeline, eliminating dependencies, and simplifying
WHAT DO YOU FIND THE MOST C
 HALLENGING ABOUT TESTING? test processes. Additionally, teams can benefit greatly
CHECK ALL THAT APPLY. from Mock Services solutions that are automated,
deployable, and maintainable, as well as enable teams

1% to solve their automation and environment issues in one

Environment concerns intuitive platform.
33%
47% Lacking resources for making test automation
Related Reading: Above & Beyond: BlazeMeter
Getting right coverage for test scenarios
Mock Services
38% Test instability/false negatives/flakiness
51%
29% Testing not prioritized within organization

Deciphering why tests fail

39% Setting up & maintaining lab platform

42% None of the above

3%
Other
47% of respondents indicated
that lacking resources for
creating test automation was
a prominent pain point.

6 | 2024 State of Continuous Testing © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0220KS24)
 Test Failure Analysis Increasingly Perforce’s Testing Analytics and Reporting help teams
find and resolve defects and failures in testing faster.
Time-Consuming With unified test reporting, you can quickly see all your
test automation results in one location, identify issues
Test failure analysis continues to take up time and resources in 2024. Combined with manual
faster with ML-powered detection and classification,
testing, these areas of the software testing lifecycle are particularly tedious for testing teams.
and identify bugs earlier in the test cycle with root cause
analysis. With advanced analytics and robust data,
WHAT IS THE MOST TIME-CONSUMING A
 CTIVITY FOR YOU WITHIN
Perforce is able to help teams identify and resolve the 3%
A TEST CYCLE?
of failures resulting from actual defects more efficiently.

Additional features of Testing Analytics and

54% Reporting include:

Test failure analysis
47%
Test environment setup • Heatmaps
41% Manual testing
39% • Noise Filtering
Advanced script creation
34% • Visual Validation
28% Script maintenance

Integrating different tools • CI Dashboard

Other
• Rich Artifacts
1%
None of the above
3%
• Root Cause Analysis

• And more
There are four main reasons why tests fail: Related Reading: Test Automation Strategy for Beginners
1. Environment issues

2. Test issues

3. Issues with test data Perforce is able to help teams

4. Actual defects identify and resolve the 3%
Based on the data gathered by Perforce, barely 3% of failures are a result of actual defects. of failures resulting from actual
Yet defects are the most important failures to analyze when looking through testing reports. defects more efficiently.
Organizations that do not have coordinated reporting methods ultimately spend more time
and resources than necessary on backlogs.

7 | 2024 State of Continuous Testing © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0220KS24)
 Improving Test Coverage Inside you will find:

• Expanded lists of top 32 devices/OSs by country.

is a Continuous Testing Priority • Practical tips for applying the data to your test strategy.
Improving test automation coverage is one of the key aspects of a
• Planning calendars with upcoming mobile & web release dates.
testing strategy that teams are focusing on in 2024. Making strategic
decisions about test coverage based on users, their location, device Related Reading: Test Coverage Guide
trends, and a team’s comfort with risk have come to the forefront of
priorities for organizations. Teams Investing More in
WHICH IS YOUR TOP PRIORITY IN THE NEXT
Accessibility Testing
6 TO 12 MONTHS? SELECT ALL THAT APPLY. Accessibility testing is a major trend in 2024 for organizations looking to
increase their user base and comply with regulatory requirements.

3% Integrating testing in CI/CD pipeline WHICH TRENDS IMPACTING TESTING ARE YOU LOOKING
22% TO INVEST IN THIS YEAR? SELECT ALL THAT APPLY.
57% Improving test automation coverage

57% Transitioning testing to cloud

Other
51%
Reduce time on regression testing
46%
Executing tests faster 48%
Progressive apps (PWAs)
35%
42% Reducing escaped defects
Test impact analysis & reporting
42%
Moving from manual to automated
53% Utilizing AI & ML in testing
Reducing cost of execution
21% Application security
53%
4% Low code/no code testing

Progressive apps (PWAs)

Perforce significantly contributes to improving test automation coverage by 51% Other
39%
offering expanded testing capabilities and insights into device and user trends.
The Test Coverage Guide provides detailed information on the most popular
devices by region, as well as insights into the latest device trends, market shifts,
and new releases, helping you to ensure full test coverage for your team.

8 | 2024 State of Continuous Testing © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0220KS24)
 Accessibility testing ensures that applications are usable to as many people as possible, including individuals with visual, hearing, physical,
or cognitive impairments.

The Web Content Accessibility Guidelines (WCAG), Section 508, and the Americans With Disabilities Act (ADA) help ensure that software
is as accessible as possible to all users. Prioritizing A11Y accessibility not only helps teams meet legal requirements—and avoid potential fines
of up to $150,000—but it also improves overall user experience for all users, not just those with disabilities.

Perforce makes it easier than ever to incorporate accessibility testing into your testing strategy with Apple VoiceOver and Android TalkBack.
This solution enables testers to test on devices in the cloud, gaining an advantage over competitors. Furthermore, teams can eliminate
dependency on having physical devices in hand and increase automation, scalability, and speed.

Related Reading: A11Y Accessibility Testing Trends

Teams Continuing to Focus on Non-Functional Testing

2024 will see organizations seeking to expand automation testing with more of their non-functional testing capabilities by focusing on API,
performance, and E2E testing.

WHICH TEST AUTOMATION TYPES ARE Y

 OUR MAIN FOCUS AREA IN 2024? SELECT ALL THAT APPLY.

6% 39% 62% 33% 63% 66%

Perforce makes it easier

than ever to incorporate
accessibility testing.
Other

Unit E2E Accessibility Performance API

Testing Testing Testing Testing Testing

9 | 2024 State of Continuous Testing © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0220KS24)
 Non-functional testing remains highly important for teams hoping to shift WHICH FRAMEWORKS DO YOU PLAN TO USE IN 2024?
their testing left. Incorporating non-functional testing such as performance SELECT ALL THAT APPLY.
testing and load testing is critical to an organization’s success.

Load Testing allows teams to run massively scalable, open source-based

Cypress 31%
performance tests against applications, including web and mobile apps
and APIs. In addition, API Testing and Monitoring lets teams know how Selenium 57%

their APIs are behaving so they can address issues quickly and efficiently. Flutter 11%
Service Virtualization (Mock Services) and Test Data further enable teams
Low Code/No Code 43%
to incorporate non-functional testing capabilities into their testing suite.
Playwright 29%
Related Reading: Performance Testing vs. Load Testing vs. Stress Testing Other 16%

Teams Embracing Low-Code/

No-Code/Scriptless Solutions Traditionally, making the transition from manual mobile testing to automated
mobile testing has not always been easy—until now. Scriptless Mobile
in 2024 helps teams automate the most complex native mobile use cases through
simplified, rapid test creation with as many test combinations as you need
Low-code and no-code/scriptless solutions are seeing an increase in
to boost your testing efficacy. In addition, Scriptless Mobile helps eliminate
popularity. Organizations are investing in tools that can be used by their
dependency on coding and frees up time to focus on business goals and
entire team regardless of coding skillset. As indicated in section 6 of
their application.
this report, manual testing was found by 54% of respondents to be the
most time-consuming portion of the testing process. No-code/scriptless Related Reading: Bell Mobility Transforms Application Testing With
testing significantly speeds up the testing process, which is why 43% of Perfecto Scriptless Mobile
respondents plan to utilize scriptless testing in 2024.
Scriptless testing offers the same powerful capabilities as more traditional
frameworks while also making automation accessible to teams of any skill
level. Selenium (57%) remains the most popular testing framework in 2024.
Another aspect of the data to note is that the next two most popular open-
source frameworks of respondents are Cypress (31%) and Playwright (29%).

Related Reading: Cypress vs. Selenium: What’s the Right Cross-Browser

Testing Solution for You?

10 | 2024 State of Continuous Testing © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0220KS24)
 A Year In Review and PRIORITIZING ACCESSIBILITY TESTING

What’s to Come in 2024 With regulatory requirements in place and the risk of fines—and with user
experience remaining a necessity for successful applications—teams are
This past year saw the growth of many new testing trends. As we look continuing to prioritize and invest in accessibility testing. Not only does
beyond 2024, Perforce’s continuous testing suite will continue to lead accessibility testing reduce the risk of legal repercussions, but it also
in innovation, offering cutting-edge solutions tailored to evolving testing increases the user base when more individuals can access your applications.
needs and trends.
Incorporating accessibility testing into your testing strategy with new
technologies such as Apple VoiceOver and Android TalkBack enables teams
EMBRACING ADVANCES IN TESTING TECHNOLOGY to increase automation, scalability, and speed—all while doing the right thing
As new technologies like AI and ML continue to take hold of the testing for their users.
landscape, teams will continue to look for ways to implement them into
BlazeMeter University offers a wide range of testing courses, from beginner
their testing strategies. Teams are already recognizing the importance
to advanced framework lessons—including Perfecto courses—so you can
of embracing AI and ML into their testing pipeline but have not yet made
continuously improve your testing and advance your skills. For testing teams
the leap to implementation. Going forward, it will be important for teams
and software developers, the continuous advancement of knowledge is a
to adopt platforms that are in step with new technologies and that are
crucial step in staying up to speed with current testing trends.
finding innovative ways to stay ahead of the curve.
We encourage you to take advantage of our free resources, including our
courses, signing up for a free trial of our platforms, or discovering how to
LACK OF TEST AUTOMATION RESOURCES &
boost productivity while saving on cost and reducing risk with the
ENVIRONMENT CONCERNS STILL MAJOR CHALLENGE
ROI Calculator.
A lack of automation resources, test environments, and acquiring reliable
test data continue to be roadblocks to testing teams in 2024. Organizations
YOUR PLATFORM FOR COMPLETE CONTINUOUS TESTING
will need to find solutions that address all these concerns while eliminating
dependencies. Mock Services, which are automated, deployable, and With Perforce, achieving application quality has never been easier. Perforce’s

maintainable, will help teams solve their automation and environmental comprehensive suite—Perfecto and BlazeMeter—provides a single source

issues while providing reliable and usable test data that allows for the testing for delivering sharper market focus, faster releases, greater speed to market,

of real-world scenarios. reduced costs, and improved quality. Maximize continuous testing and shift
quality left by requesting a custom demo or start testing today at perfecto.io
and blazemeter.com.

11 | 2024 State of Continuous Testing © Perforce Software, Inc. All trademarks and registered
trademarks are the property of their respective owners. (0220KS24)