Guide to End-to-End UI Testing

How Get Started with End-to-End Testing

Guide to End-to-End UI Testing | 1
Contents
4 | What is End-to-End Testing?

7 | Benefits of End-to-End Testing

9 | Types of End-to End Testing

11 | Common Pitfalls

13 | Getting Started

16 | Beyond E2E Testing

17 | The Bottom Line

Introduction
Software has become a critical part of almost
every business. With rising competition,
Agile development teams use continuous
delivery to quickly release new features into
production. Manually testing each change is
impossible in these situations. So most Agile
teams use test automation to ensure that
everything is working with each release.

End-to-end tests are the ultimate form

of test automation. They run through a
complete scenario from a user's perspective
to ensure it's not broken, taking into account
various microservices and APIs that wouldn't
otherwise appear in unit or integration tests.
As a result, teams can more confidently
deploy updates to users in real-time.

Let's take a look at how end-to-end testing

fits into test automation and how you can get
started on the right foot.

End-to-end tests provide businesses with a

lot of confidence when deploying new code.
Here’s how to get started on the right foot.

Guide to End-to-End UI Testing | 3

What is End-to-end, or E2E, testing is a software
development methodology designed to
While most tests take a bottom-up approach
beginning with functions, E2E tests take a top-

End-to-End Testing? ensure that a product works from an end-user

perspective. While they're similar to integration
down approach starting with a user workflow.
You define the experience you want users to
and systems tests, E2E tests focus on the user have and what those workflows look like before
experience rather than a user story. The goal is building E2E tests to verify that they work
to provide the highest level of confidence that correctly and don't break with new updates.
the software works before deployment.
The Test Pyramid
For example, an E2E test for an email
application might consist of typing the URL into To put E2E tests into context, it helps to understand
the address bar, signing in with the correct the test pyramid. While overly simplistic, the test
credentials, opening an unread email, clicking pyramid is a visual metaphor that tells you how
the reply button, writing an email, clicking the to think about different test layers. That is, you
send button, checking the sent folder, and should write tests with varying levels of granularity
then signing out of the application. The process and create fewer tests as you go from testing
covers an entire critical workflow. functions to testing workflows (e.g., get high-level).

Test Pyramid - Source: Automation Panda

Modern test pyramids have many layers: Who's Responsible? Different Approaches
| Unit Tests focus on individual Developers are typically responsible for E2E UI tests typically use Selenium and
methods and functions within classes, writing unit and integration tests covering user the WebDriver protocol to interact with an
components, or modules. stories they pull into development. But who's application through headless browsers. For
responsible for E2E tests that span several example, the test might spin up a Chromium
| Integration Tests verify that different
user stories? Unfortunately, the answer browser that clicks on an add-to-cart button
modules or services work well together
isn't always obvious, and the right decision and goes through the checkout process until
(e.g., database interactions).
depends on your team and the technology reaching the confirmation page to ensure

| End-to-End Tests replicate a user's choices you make to handle testing. everything works.

behavior within an application to test

Quality assurance teams are usually the While headless browsers are a good starting
the complete system.
best fit, but many Agile teams don't have point, it doesn't always account for mobile

| Acceptance Tests are formal tests centralized quality assurance teams. Instead, devices very well. CrossBrowserTesting is a
they rely on cross-functional DevOps teams cloud-based platform that makes it easy to
to ensure that an application satisfies
that step in and handle tests. However, execute tests across thousands of real devices
specific business requirements.
DevOps may not have the skills required to and browser configurations. If you have a
| Exploratory Tests are informal tests designed write complex test scripts, nor the time to native app, BitBar provides mobile app testing
to explore edge cases and try to identify bugs. write them if they're already busy. via frameworks like Appium.

Unit tests are at the bottom of the pyramid. TestComplete makes it easy for anyone E2E API tests typically use libraries that
They should constitute the bulk of an to create E2E tests without programming provide a DSL for creating HTTP requests
application's test suite since they run quickly knowledge. For instance, a DevOps team from an API and evaluating the responses.
and cover a narrow piece of functionality. member can use record-and-replay tools Often, these tests are helpful for single-page
At the top of the pyramid, slow, brittle, and to build tests without writing code, whereas applications that rely on APIs and have fickle
highly integrated E2E tests should be reserved developers can use keyword tests to speed up user interfaces. Unlike integration tests, E2E
for the most critical parts of the application creation. The platform even integrates with API tests go through an entire workflow rather
since they're a lot harder to maintain. most CI processes out of the box. than an isolated feature or story.

Guide to End-to-End UI Testing | 5

Common Challenges

E2E tests are notoriously flaky and may suddenly

fail for unpredictable reasons. For example,
browser quirks, animations, or timing issues can
trigger a false positive. These characteristics
translate to high testing costs since developers
must update tests every time the UI experiences
a minor change – a time-consuming process.

In addition, E2E tests that span multiple

microservices can become even more challenging
for testers. For example, you might need to spin
up several microservices on a local machine or
build server, which requires many resources. Or
you may need to use third-party APIs to complete
tests, which can be brittle or costly on their own.

Finally, E2E tests tend to be a lot slower than

conventional unit and integration tests. Therefore,
it might not be feasible to run these tests with each
deployment if you have many E2E tests. A better option
is usually cutting down on E2E tests to focus on the
most critical user workflows (e.g., a checkout process).

E2E tests help you confidently deploy applications

knowing that they won't break critical user experiences.
That said, they're more brittle and time-consuming
than unit tests and should run sparingly.

Guide to End-to-End UI Testing | 6

Benefits of End-to-end testing may seem like a lot of extra
work. Even worse, it might seem repetitive
#2 Expanded Test Coverage

End-to-End Testing if unit and integration tests already cover Most unit and integration tests focus on a small
piece of code or a single type of browser. In
the same workflows. In reality, E2E tests
provide many significant benefits that make reality, users come from all different kinds of

them worthwhile to add to your test suite, environments. For example, many teams forget to
especially when you factor in the cost of a run UI tests on mobile devices even though they
substantial defect reaching production. may account for a large portion of active daily users
– and a poor UX could translate to costly churn.
#1 Increased Confidence
E2E tests incorporate these different environments
Years ago, deployments were a stressful event, to ensure that your application works everywhere.
with months of work packed into each release.
While unit and integration tests prove that user
These days, continuous deployment could mean
stories pass, E2E tests focus on the actual user and
several releases each day. While releases are
their experience with the application. The expanded
more frequent, it's still stressful to kick off a
focus can help uncover a lot of potential bugs that
deployment process. Fortunately, tests can help
you may have otherwise missed.
reduce stress by proving that certain portions of
the code work properly. #3 Discover New Bugs
The most significant benefit of E2E testing is that it Most testing efforts seek to identify bugs before
provides you with confidence when deploying new they reach production. But, of course, many bugs
code. While unit tests may be passing, there's no silently move into production, where they can
guarantee that the connections between various grow into more significant problems over time. For
microservices or third-party APIs work. E2E tests example, an overlooked change to a third-party API
verify these vital user experiences work before can lead to missing database values that you may
deployment, providing peace of mind. not notice for some time.

Guide to End-to-End UI Testing | 7

E2E tests excel at catching these difficult-to- tests that take time away from feature development. night fixing a bug that brought down production
spot bugs since they test the microservices and While adding E2E tests may seem like a step in on a Friday release. Instead, they can confidently
APIs that tie everything together. Even in simple the wrong direction, they can cut down on time- deploy software knowing that key user workflows
applications, you can ensure that the removal consuming tasks like fixing bugs. Failing tests make it will be up and running no matter what.
of seemingly innocuous libraries doesn't impact much easier for developers to track down bugs and
the UI in unexpected ways. As a result, you can E2E tests may seem like a lot of extra work,
fix them before they become even more costly.
guarantee users a better user experience. but there are numerous advantages to
In addition to catching bugs, E2E tests avoid time- building them, and in the end, they can
#4 Conserve Test Resources consuming rollbacks if there's a significant problem help you discover potentially devastating
Many organizations are hesitant to add even more with a deployment. Teams don't have to spend all bugs more quickly.

Bugs become more expensive over time - Source: TestBytes

Types of End-to-End There are many different ways to classify and
break down end-to-end tests. While some
Code vs. Codeless

Testing of these breakdowns may sound nuanced, Code-based tools enable developers to write
test scripts that are similar to unit tests. There's
it's essential to understand the differences
to ensure that you have the proper tests. usually a DSL that allows them to find elements

Understanding these tests can also help you on a web page, click buttons or links, and add

decide what parts of an application you should input to form fields. The benefit of a code-based

test to achieve your goals. approach is that it might make more intuitive
sense to developers, and tests can be easily
Horizontal vs. Vertical Testing stored and executed.

Horizontal E2E testing involves testing Code-less tools are more accessible to non-
across different parts of the application. For developers and enable faster test creation. For
example, in an e-commerce application, you example, using tools like TestComplete, you can
might test the search capabilities, shipping record a browser session to capture different
calculator, and checkout process. You're steps. You can then edit these steps and create
moving across the user's journey to complete pass/fail tests to validate outputs. Finally, you
a specific workflow and ensuring that the can easily integrate these tests into your CI
workflow doesn't fail in production. processes with built-in integrations.

Vertical E2E testing involves testing different Browser-based (BB) vs.

layers within each of these horizontal steps.
Computer-based (CB)
For example, when a user searches for a
product on the e-commerce application, it Browser-based tests focus on user interfaces,
might trigger different API calls to a backend such as browsers. As mentioned earlier,
database. Thus, you're moving down the list you can use Selenium and other tools to
of steps required to execute a specific piece simulate a browser session across different
of functionality across various services. devices. However, since they have to spin up

Guide to End-to-End UI Testing | 9

a browser instance, these tests take the longest successfully adds a product to a user's shopping
to execute, while UI changes often cause them cart. The goal is to verify that the code works,
to break unexpectedly. so it's only run once on either a real browser or
using a series of API calls to mimic the browser.
Computer-based tests focus on HTTP calls to
a backend server. Since HTTP calls generate Performance tests ensure that workflows operate
responses in milliseconds, these tests are much under a realistic load. For instance, if there's an
faster than UI-based tests. However, testing API N+1 query in the code, it may pass a functional
endpoints doesn't necessarily provide as much test, but it would quickly fail in production under
certainty as testing UI components since an API load. LoadNinja and other load testing solutions
might work even though a button doesn't work simulate a realistic number of devices and help
on a web interface. you measure response times and ensure there
aren't any bugs.
Functional vs. Performance
There are many different ways to slice E2E
Functional tests ensure that a test passes from tests, but they are all focused on the user's
a technical standpoint. For example, a functional experience rather than testing specific
test might determine if an "Add to Cart" button technical components of a web application.

Guide to End-to-End UI Testing | 10

Common Pitfalls Unit and integration tests are straightforward. You
take the requirements from a user story, translate
Tools like CrossBrowserTesting make it easy to
run E2E tests across hundreds of different web
them into a test, and then write the code to pass and mobile devices and browsers. That way, you
the test. End-to-end tests are more nuanced in can ensure that the tests pass regardless of device,
that you have to determine the most critical user browser, or operating system. BitBar provides the
experiences, design the tests with the user in mind, same capabilities for native mobile applications or
and take the time to maintain them. tests built with native testing frameworks.

#1 Failing to Consider #2 Only Focusing on Functionality

Environments
Most unit and integration tests focus on

It's easy to fall into the trap of treating E2E tests functionality. For example, they test whether

like unit or integration tests. In particular, many a specific input results in the desired output.
developers run them in the same environment as However, since E2E tests focus on the
their other tests. The problem is that users may user experience, it's important to consider
be using entirely different devices, browsers, and functionality, performance, and ease of
operating systems, resulting in poor test coverage use. You should take a holistic view of the
and potential bugs reaching production. workflow from a user's perspective.

For example, many web applications are built with For example, load tests can help ensure that the
responsive layouts. However, if you're running E2E application performs well under certain realistic
tests on a desktop browser, you aren't verifying that load conditions. These tests can help identify
mobile formats are necessarily functioning correctly. bottlenecks that could severely degrade the user
As a result, you might miss a deployment that experience before they ever reach production.
breaks a mobile navigation menu or other elements Unfortunately, it's almost impossible for unit
that might be invisible to desktop-only tests. and integration tests alone to catch them.

Guide to End-to-End UI Testing | 11

LoadNinja makes it easy to create load tests cycle ensures that there's a system check
using the same record-and-replay capabilities in place before starting the test run.
as TestComplete. In addition, you can
At the same time, many Agile organizations
quickly load test APIs through an easy-to-use
are shifting their tests to the left. While
interface without writing and maintaining a
you cannot test a UI before it's built, you
large test codebase. Best of all, the load tests
can write E2E tests for APIs and other
are run in the cloud across tens of thousands
elements in advance. By running these
of real devices for the most realistic data.
tests earlier in the process, you can identify
defects at earlier stages rather than fixing
#3 Executing E2E Tests Before
them immediately before a release.
Functional Tests
There are several mistakes that
Automated tests should roughly follow the
test pyramid. Fast unit tests should run
development teams make with E2E
regularly (e.g., with each merge), whereas testing. In particular, don't forget to
slow E2E tests should run less frequently test across multiple environments, test
(e.g., each deployment). In addition to performance along with functionality,
avoiding long test times, running E2E and be sure to execute E2E tests after
tests toward the end of a development functional tests have already passed.

Guide to End-to-End UI Testing | 12

Getting Started Many Agile teams use test-driven development, or
TDD, to write unit and integration tests. However,
| CrossBrowserTesting is a test automation
tool that helps you run Selenium, Appium, or
while TDD is a well-defined process, end-to-end other tests across actual desktop and mobile
tests are much more ambiguous. As a result, browsers in the cloud. You can even use the tool
many companies manually test new features for manual exploratory testing and debugging.
rather than writing E2E tests, but unfortunately,
| BitBar makes it easy to automate native
that can result in bugs reaching production users.
mobile app testing across real devices in

Identify the Team & Resources the cloud. As with CrossBrowserTesting,

you can also use the tool for manual
E2E testing requires buy-in from the team exploratory testing and debugging.
members responsible for building and
maintaining them. In addition, they should have | LoadNinja makes it easy to build and run

access to the resources and software that can UI or API load tests in a fraction of the

help them do their job more efficiently. For time it takes with scripts. In addition to its

example, record-and-replay tools can create UI record-and-replay tool, the platform runs

tests much more quickly than script-based tests. tests across hundreds or thousands of real
browsers for the most realistic results.
SmartBear offers several solutions that
can simplify and streamline E2E testing for Find the Vital User Experiences
businesses large and small, including:
The next step in the process is identifying the most
| TestComplete is an automated UI testing critical user experiences. Gather stakeholders,
platform featuring AI-powered object developers, and testers to discuss the most
recognition and script or scriptless flexibility. important scenarios that you want to test before
You can use it to test every desktop, web, each release. These might be the most popular
and mobile application, along with enterprise workflows within an application or the most
applications like SAP or Salesforce. valuable workflows from a business standpoint.

Guide to End-to-End UI Testing | 13

Google Analytics is an excellent starting point for After cutting down on unnecessary tests, you
identifying these critical workflows. Click on the can further reduce the time it takes to run CI
User Flow button to view your application's most processes by running machines in parallel.
popular user flows in the Audience section. The TravisCI, CircleCI, and other tools make it easy
green boxes represent pages (with URLs), the gray to split tests across separate environments or
lines represent visitors flowing to different pages, machines to run in parallel. Cloud-based CI tools
and the red lines are drop-offs that exit the page. make it even easier by instantly provisioning new
machines on demand.
In addition, Google Analytics enables you to
track goal values, which can be helpful when Another common way to speed up CI processes
identifying the most valuable workflows. For is only running tests on updated parts of the
example, you might notice that people adding application. For example, you might set services
items to a wish list tend to convert well, even or integration points behind a flag that runs only
though they only account for a small fraction if required or by using test segregation tools so
of visitors. As a result, you may want to create that repositories with no cross-dependencies are
an E2E test for wishlist functionality. ignored, and the test suites are run excessively.

Integrate into CI and CD Processes Follow Up Using Relevant Metrics

Continuous integration and deployment It's always a good idea to define metrics that
are an essential part of the test automation you can use to measure the success of your
process. While automating unit tests is pretty testing efforts. After all, testing takes time
straightforward, the slow-running nature away from feature development, so you want
of E2E tests can make them a challenge. to ensure that it's providing real value to your
For example, you may need to consider organization. Fortunately, there are a handful
how many E2E tests you want to include of standard metrics that you can use to track
and how often you want to run them. performance easily.
The four key metrics to watch include: In the end, successful E2E testing should
result in fewer defects reaching production,
| Test Preparation shows the number of test
faster bug fixes in production, and a higher
cases you've prepped against the number that
level of overall test coverage. These features
you've planned.
give teams much more confidence when
| Test Progress indicates the number of tests deploying new code since they can worry
you've completed against the number that less about severe bugs reaching production
you've planned.
users and causing costly issues.
| Defect Status shows the number of defects
that were opened and closed during a specific Start by identifying the team members
timeframe, as well as the severity and priority. responsible for E2E testing and providing
E2E tests should enable you to fix defects faster them with the tools they need to create and
and notice a positive trend in these numbers. maintain them efficiently. Next, determine

| Test Availability tracks the amount of time the scenarios that you want to test based
you think you'll need to run tests in a specific on popularity and business value. And
environment against how much time is finally, integrate these tests into your CI/CD
required and how many environments. processes to ensure they run.

Guide to End-to-End UI Testing | 15

Beyond E2E Testing End-to-end tests are just one part of the test
pyramid, and it's essential to understand how
BDD tests are typically developed using
tools like Cucumber and create a kind of
they work with other components. While unit living documentation of features. While
tests live in their own world, acceptance tests are E2E tests ensure that critical parts of the
crucial where E2E tests may find fundamental application work, acceptance tests prove
overlaps. These tests both verify that an that individual features are working. In other
application works from a high level – with a few words, E2E tests focus on the user, whereas
key differences. acceptance tests focus on user stories.

Behavior-Driven Development Integrating E2E Tests with BDD

Behavior-driven development, or BDD, is a There is typically overlap between E2E and

process that involves collaboration between acceptance tests since they both involve
developers, testers, and stakeholders. The goal testing the UI layer. However, in some
is to create concrete examples that formalize cases, companies may run BDD tests as
a shared understanding of how an application E2E tests in their CI/CD processes. The right
should behave. These concrete examples become decision depends on your development
code that forms the basis for acceptance tests. team and if the BDD test cases cover the
same critical user experiences as E2E tests.
For example, a BDD acceptance test might
look like this:

Given there's a signed-in user,

And there's a product page,
When the user navigates to the
product details page,
And clicks the "Add to Cart" button,
Then the product should appear in their cart.
 Cucumber and TestComplete Simplify BDD - Source: SmartBear

TestComplete makes it easy to quickly convert BDD- CI/CD, meaning that it's automatically kept up-to- and APIs work together to produce an optimal
style feature files into automated tests with native date and verified over time. user experience. When incorporated with
support for Gherkin's Given-When-Then scenarios. CI/CD processes, it can become a valuable
Acceptance tests are similar to E2E tests in that
As a result, Cucumber can help define business goals part of your test automation suite.
they ensure features are working correctly.
and requirements and create feature files while
In some cases, the lines are blurred, and SmartBear provides various E2E testing tools
TestComplete builds step definitions and updates
companies use BDD tests as their E2E tests. that you can use to quickly get up and running
test scripts from the step definitions in real-time.
and reduce maintenance time, including
The key benefit of BDD is that you have access to The Bottom Line TestComplete, CrossBrowserTesting, BitBar,
living documentation. That is, you can quickly verify End-to-end tests provide the highest level of and LoadNinja. If you need help getting up
that features are working with human-readable confidence before deploying an application and running with these tools, we're happy
documentation of how they work. The documentation to production users. Testing from a user to walk you through the process and
also serves as a test case that can be integrated with perspective verifies that different microservices ensure you're following best practices.

Guide to End-to-End UI Testing | 17

 Try it out for yourself
You don’t just need to read about it. SmartBear UI Test solutions enable
end-to-end testing, ensuring you achieve the quality consumers demand.

Start right away with a free trial today.

Test every desktop, web, Cloud-based Easily Automate

mobile, and packaged app Mobile app testing solution Performance Testing

Try TestComplete for Free Try BitBar for Free Try LoadNinja for Free
Guide to End-to-End UI Testing | 19