Continuous

API Testing
and Monitoring:
Best Practices &
Buy-in Guide
Evaluate continuous API testing and
monitoring tools to significantly improve
API quality, security, and reliability

v2.1
Contents
Four Key API Testing Strategies 4
1. Use Dynamic DDT to Check Entire User Flows 4
2. Ensure API Tests Are Created with Domain Expertise 5
3. Invest in an API Testing Tool with a Modern Architecture 5
4. Adopt a New API Metric: E2E Functional Uptime 5

Four Important API Testing Solutions 6

Solution 1: Solve QA/Testing Bottlenecks with Transparent API Testing 6
Solution 2: Monitor APIs With or Without a CI/CD Pipeline 7
Solution 3: Reuse API Tests as Functional Uptime Monitors in Production 8
Solution 4: Create a Single Pane of Performance Analytics 9

API Testing Automation: Evaluation Criteria 11

Features Evaluation Checklist 11
Conclusion 12

About API Fortress 13

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 2
Shift-left API testing automation has been increasingly embraced by developers and QA teams
as essential to successful agile software development or a CI/CD pipeline. In theory, automated
testing early in the lifecycle should reduce costly late-lifecycle QA/testing bottlenecks. Yet
bottlenecks continue to plague developers and testers, and the number of undetected API bugs
that go live and reach end-users or hackers remains alarmingly high.

$2.8 trillion was spent in the U.S. due to poor quality software (CISQ)

One of the critical issues behind a failure of testing effectiveness at agile organizations involves
the complexity of managing the scope and scale of API testing, that is, in striking the right balance
between time-to-market and quality-to-market. Companies need to do more testing in less time,
but a major roadblock stands in the way - the diverse teams that share in API success tend to
work in silos.

Development, QA/Testing, and DevOps teams may struggle to find consensus on

which API metrics should be tracked to set KPIs

A core reason for silos in an agile organization stems from the wide gap in priorities and
accountability among the teams that own APIs. While developers may be most concerned
about delivering a shippable product on time, QA/testing teams may be more concerned about
validating whether the product satisfies the user story. Then DevOps or DevSecOps leaders
have uptime and reliability concerns that extend beyond the delivery sprint. Banking, financial
services, healthcare, and other types of enterprises that deal with advanced API security and
compliance issues run into more complications that divide siloed teams even further.

With the right API testing and monitoring tools, agile organizations can bring insight-driven
visibility and collaboration to distributed teams. This is needed to set companies on the right
path to transform API testing reliability and performance.

This white paper offers a breakdown of recommended API testing strategies along with insights
about using the right API metrics to set up more viable KPIs based on our years of experience at API
Fortress in working with several of the world’s most innovative retail, banking, financial services,
insurance, healthcare, telecom, education, entertainment, and aeronautics/aviation companies.
Additionally, this white paper includes a breakdown of best practices for implementing good API
testing and monitoring tools.

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 3
Four Key API Testing Strategies
Prior to agile development, CI/CD pipelines
and microservices, UI and Unit testing
dominated the testing pyramid. And then, the
only essential role of API testing was to verify Test Automation
the contract and check for a ping round trip.
Now, API tests must effectively act as end-
to-end tests that validate entire user stories. UI Tests

This new set of strategies (along with new

metrics) are needed to determine API testing

E2
ET
es
success. The following “Big Four” strategies

tin
g
have been derived from what we have seen in
next-generation approaches to API testing for
modern systems with contemporary toolchains
and CICD flows. We hope these strategies
help leaders of development, QA and testing,
enterprise architecture, DevOps, and product
teams to better quantify the efficiency and
efficacy of their current or proof-of-concept
API testing tools.

Strategy 1: Use Dynamic DDT to Check Entire User Flows

Modern API testing tools must be able to conduct multi-step integration tests across APIs to
check on how well API endpoints work together (the “API flow”). In the real world, APIs use
data that is not fixed or prebuilt. It follows that API testing tools should create API testing
paths that are unpredictable. The best way to do this is to leverage Data-driven Testing (DDT)
from active databases or highly variable (fake) testing data. The goal is to create tests that
involve multiple steps, including calling an API that can trigger a sequence of unique API calls
in an array. In some cases, testing systems include components that convert databases into
APIs to be used as these dynamic APIs. Additionally, modern API testing tools should automate
OAuth 2.0 flows to avoid disruption to DDT while also validating SSO and multi-factor
authorizations.

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 4
Strategy 2: Ensure API Tests Are Created with Domain Expertise

While developers can certainly write tests that verify the technical capabilities of the APIs
they build, it can be ineffective and/or inefficient for developers to create API tests that also
validate the user story. Instead, API tests from developers that simply prove whether an API is
working when used correctly can form the basis for more stringent and comprehensive tests
written by professional QAs with high knowledge of the problem and solution domains. These
holistic tests may include end-to-end integration tests that can check entire flows involving
arrays of APIs that may have been built or changed by many different teams. Ultimately, the
goal is to leverage one unified test with consistent domain knowledge for proactive and real-
time insight about API health throughout constant changes to code and databases.

Strategy 3: Invest in an API Testing Tool with a Modern Architecture

Modern API testing tools, in particular, tools that excel at testing APIs in agile development and
CI/CD pipelines, must offer a robust set of APIs for plug-and-play integrations with existing
and modern DevOps tools. Today, many companies benefit from sending all test results to a
best-in-breed analytics platform such as Elastic or Splunk, while delegating notifications to
best-in-breed solutions such as PagerDuty or Slack.

Additionally, with modern architectures, API testing tools can evolve along a far more
innovative roadmap. If a company is evaluating a number of API testing tools that are divided
between unified testing suites that piecemeal together multiple apps/services versus a best-
in-breed unified testing platform, the best-in-breed platform is almost always the better
choice. Most API owners should avoid vendor lock-in, and future-proof their increasingly agile
toolchains for cloud maturity and distributed services.

Strategy 4: Adopt a New API Metric: E2E Functional Uptime

On its own, API uptime (an HTTP 200 OK) is an outmoded metric for continuous API quality.
Modern API testing tools must go beyond uptime, and simplify validation of the user story by
checking the business logic and service layers for problems across functionality, reliability,
performance, and compliance as well as potential security vulnerabilities. A holistic verification
of these layers allows a good API testing tool to significantly improve on the accuracy and
consistency of uptime reporting, particularly, in reducing false-positives and ensuring adequate
testing coverage. Testing Centers of Excellence or smaller QA/Testing teams sometimes refer
to this new API metric as “Functional Uptime” or “End-to-End (E2E) Uptime.”

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 5
Four Important API Testing Solutions
Before breaking down the best practices of a good API testing tool, this section provides a
framework for how the best practices were selected. The focus of the assessment is geared
toward solving four key API testing solutions that are critical for any organization moving from
a monolithic stack to distributed services and cloud maturity.

Solution 1:
Solve QA/Testing Bottlenecks with Transparent API Testing

63%

32%

22% 23%

Plan Build Test/QA Release / Deploy

Most of the costly and time-consuming bottlenecks that hold up software development happen
in the QA/testing stage. Insufficient or poor collaboration between technical and product
teams (or line of business owners) with poor or no visibility into end-to-end API testing is
often at fault. True end-to-end API testing tools tackle this problem by making it easy for
stakeholders of all technical and coding backgrounds to work in parallel. Ultimately, the goal is
to minimize the risk of falling short of validating the user story without delaying go-to-market.

Another vital aspect of API testing transparency is to ensure that all teams are clear on the
scope of testing coverage. Insufficient attention to this detail may result in high numbers
of false-positives, which allow malfunctioning APIs to exist in production environments for
prolonged periods of time.

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 6
Solution 2: Monitor APIs With or Without a CI/CD Pipeline

Continuous API Testing and Monitoring in a CI Flow

Developers & QA CI/CD Pipeline Centralized API Testing and Debugging

Repo

Command Line Interface

Test Execution Results
IDE
APIF_Local

Successful shift-left API testing automation frees development teams to confidently run a
Continuous Integration (CI) flow. Most API testing tools simplify the CI flow by seamlessly
integrating with popular CI/CD platforms such as Jenkins, BitBucket, Azure DevOps, Bamboo,
TravisCI, GitLab CI, CircleCI and more. The latest API testing tools go further by additionally
offering a built-in scheduler. This empowers teams to deploy continuous API tests and
monitors without requiring a CI/CD pipeline. Critical advantages of this approach include:

• Faster Builds, Less Overhead:

Continuous API testing (monitoring) via a scheduler achieves the same automation
goals of a CI/CD pipeline with added capabilities to ensure that development/staging
environments are working for teams 24/7; this risk reduction can return many hours
back to teams

• Enhanced Testing Consistency and Accuracy:

Free from dependence on the CI/CD pipeline, continuous API tests/monitors remain
accurate and consistent throughout changes to the code repository and databases

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 7
If API testing automation is an implied part of any CI flow, then it follows that end-to-end API
monitoring should be an implied part of any CI flow that must avoid disastrous QA/testing
bottlenecks. With proper API monitoring feeding the right (end-to-end) API testing data points
to analytics tools, QA/testing or DevOps leaders can properly curate analytics for more useful
KPIs to track API performance.

True continuous testing does not exist without end-to-end API

monitoring throughout the API lifecycle

Solution 3: Reuse API Tests as Functional Uptime

Monitors in Production

Reusing Functional API Tests as Monitors Improves Quality, Security, and Reliability

Knowledge Live APIs

Build

API MONITOR

PRE-PRODUCTION PRODUCTION
• CONTINUOUS
• DATA-DRIVEN
• FUNCTIONAL
• END-TO-END
Test

API Mocks

Many of the same benefits of shifting API testing left also apply when shifting API testing right
(into production environments). Good API testing tools help to significantly reduce the risk of
any unforeseen bugs or crashes by making it easy to reuse shift-left tests built with sufficient
domain knowledge and coverage as shift-right tests, aka, API monitors. Automated tests in

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 8
production are exposed to the knowns and unknowns that APIs will encounter when live.
Production environments typically contain complex systems and data sets that staging does
not. Ultimately, by extending proper API testing across the entire API lifecycle, companies can
solve QA/testing bottlenecks and ship products faster with greater confidence.

In the best cases, a good API testing tool makes it easy to build continuous API tests or
API monitors from a holistic combination of data-driven, functional, load, and end-to-end
(integration) tests. With this level of modular reusability, both QA/testing and live monitoring/
SRE teams can test more and test better in less time. Further, good API testing tools should
be able to schedule these functional/E2E monitors in development, pre-production, and
production environments. In this way, technical teams can verify an API’s technical capabilities
in harmony with product/testing teams validating the user story.

Solution 4: Create a Single Pane of Performance Analytics

Most companies already have proven solutions in place for best-in-breed UI and infrastructure
monitoring. A good API testing tool makes it easy to extend existing best-in-breed investments
with best-in-breed API testing by allowing functional tests to be reused as end-to-end load

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 9
tests that can be scheduled continuously as API performance monitors. Automated API load
tests generated in this manner offer built-in domain knowledge along with coverage that
stretches across entire user flows to help teams quickly connect the dots about how and why
functional and integration problems may have an impact on API performance. Now, performance
bugs that may have escaped detection altogether or required costly QA/testing in a fully
virtualized environment to find can be detected with a continuous end-to-end API load test.

End-to-end load testing helps companies raise the bar on SLA guarantees
for internal and public APIs, accelerating an API-led future

With a good API testing tool, monitoring and SRE teams can instantly upgrade their analytics
dashboards with powerful new metrics about API performance under real world conditions.

The Build vs. Buy Decision

An alternative to purchasing an API testing and monitoring platform is to build
the suite in-house. Traditional API testing platforms may have pushed companies
toward DIY solutions due to high costs, a lack of software budget, unique
requirements, lack of configurability, and/or problems integrating with an existing
toolchain. Building a testing suite requires the same planning and development
required of any new product or application . For some companies, they have no
choice, but there are important items to consider when choosing to build in-house
including:

• Dependent on a few staff members or service providers who know how to fix,
customize, and update the suite. This leads to a problem if they choose to leave
the organization.
• Usually there is a lack of UI and reporting to help understand results.
• Documentation to train new users on using the suite rarely exists.
• No formal support procedure for teammates.
• User management and auditing is rarely implemented, leading to a lack of
collaboration.
• Building from scratch keeps you always behind the curve in terms of new
standards and requirements.

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 10
API Testing Automation: Evaluation Criteria
This final section of the white paper offers two grids that capture the key points of this paper,
and may help in the pursuit of the right API testing and monitoring tool.

API Testing/Monitoring: Features Evaluation Checklist

API Testing & Monitoring: Features Evaluation Checklist

Y or N
Category Feature
Score (1-5)

Cloud
Installation & Setup
On-Premises

Single Setup Integrations (Slack, JIRA, etc...)

Team Setup & User Roles

Flexibility in Methods of Use (GUI, command-line, plugins, etc...)

Test Generation
Test Creation
Test Editing

Assertion Library

Easy to Learn / Understand

Ability to Create End-to-End Tests

Ease of Creating End-to-End Tests

Reuse Code Snippets Between Tests

Universal Variables

Support for Custom Coding

Test Creation - Scripting
Scripting Libraries

Data Source Driven Tests

Test Data Management
Supported Data Formats

Source Control Management Integration with VCS (eg: Github)

Run Tests Against Any Environment

Test Execution
Works With Your CI/CD Platform (eg: Jenkins, Bamboo, TFS, etc...)

APIs & Webhooks for Execution Flexibility

Built in Scheduling

W W W. AP IF O RT RE S S .C O M

Download PDF Download Spreadsheet

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 11
Conclusion
The API economy has greatly accelerated sweeping changes across many industries, including
an even greater dependence on RESTful apps and web services. Like many organizations, your
enterprise may be rushing with greater urgency toward digital transformation. Or you may be
finalizing strategies to bolster your leading position with cloud and API technologies. Wherever
you are in your journey in the API economy, a key takeaway from this white paper is to make
sure that you know what makes modern API testing so different from traditional API testing as
well as UI, unit, browser, and infrastructure testing/monitoring. By knowing why modern API
testing must effectively become end-to-end testing, you can establish the right API metrics and
KPIs for API testing success.

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 12
About Us
API Fortress is the only API testing platform built from the ground up for collaborative API
testing in modern architectures. Our customers integrate us easily into their SDLC toolchains
for shift-left testing and functional uptime monitoring of internal APIs and external/public
APIs. The cloud-native API Fortress platform can be deployed via an on-premises container or
as a SaaS with low total cost of ownership. Many API Fortress customers build or improve upon
single pane of analytics across APIs, UI, and infrastructure via DataDog, Sauce Labs, MuleSoft,
Apigee, Kong, Jenkins, Splunk, BitBucket, Azure DevOps, TIBCO Mashery, WSO2, Oracle
Cloud, Axway, PagerDuty, xMatters, and more.

Want to Learn More?

Visit www.APIFortress.com to chat live, sign up for a Free Trial, or

Schedule a Demo with a sales or solutions engineer.

Start a Free Trial Schedule a Demo

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 13