The Modern Agile Test

Automation Playbook
How To Build a Rock-Solid Test Automation
Framework FAST that will Turbo-Charge Your
Team And Your Career

The Serenity Dojo

https://www.serenity-dojo.com
 01

DO YOU STRUGGLE WITH TEST AUTOMATION?

Do you have trouble keeping your test automation up to pace with new features
being built? Do you feel you don't have enough time to automate all the scenarios
you need to each sprint?

Are you pressured by management to deliver "100% automation", when you

struggle to keep up even with the backlog of regression tests you need to write?

Do you find your tests can be brittle and unreliable? Is it hard to know exactly what
went wrong when a test fails?

Do you find developers don't want to help out with automation tasks, and don't
respect the value of the automation work you do?

If so, you are not alone

According to our recent industry-wide survey, these are
all common problems among automation testers. We’ve
all been there. It’s frustrating.

The truth is, the classic ways that folk do and teach test
automation simply are not effective in modern, agile
projects.

It's not enough just to know the tooling.

Sure, it's a start. But you also need to know when and
how to use test automation, and what agile practices
can make it work effectively.

Because the goal isn't just to write a bunch of automated

tests. It's to make your team deliver faster.

The role of testers is changing

forever, and if you want to keep
pace, you need to be prepared

But why don't traditional test automation approaches work for

agile projects? Read on to find out...
 02

Traditional Test Automation Methods

Do Not Work For Agile Projects
Automated testing tools have been around for ages. Almost three decades in fact.

The thing is, traditional testing strategies haven't really evolved much over this time.

They still focus on automating your manual test cases, after the work is finished.

Now if you’re new to test automation, this approach can feel intuitive. It kinda feels like home
- it's the way you do things with manual testing.. There are even a host of vendor tools that
promise to let you magically automate your test cases without writing a single line of code.

Problem is, it doesn't work. Simply automating manual test cases is actually the worst
thing you can do.

Why? Because it'll slow you down

Writing automated scripts for your manual test cases traps
you in a vicious cycle of late test automation, which piles
on the pressure and makes it impossible to finish
automation in time for the end of each sprint.

Simply put, if you only start automating after the features

are done and tested, there's no way you'll have time to
complete all the automation work you need to do within a
sprint.

But that's not the only issue.

Worse, it slows your team down!

Sure, it might catch the occasional regression defect.

But when you automate your manual test cases after the
work is done, it's way too late in the game to help the
developers on the team. No one will say thanks.

If the devs missed something or screwed up, they need to

know about it when they are working on it, not
afterwards. In other words, during the sprint.
 03

Plus, it's an expensive way to find defects

Do you like wasting your time? I know I don't!

But the fact is, automating your manual test cases is a thankless, expensive chore.

When you automate manual test cases, the natural approach is to automate via the UI. It's
also the main approach all the vendor products promote.

But this is actually the slowest and most inefficient way to automate.

Sure, you need some UI tests, but having your whole test suite built on top of them is a recipe
for disaster.

Worse still, the number of defects you find when you automate your manual test cases is
actually much smaller than what you would find with with exploratory testing.

So while it has benefits for automated regression testing, we will see there are must more
cost-effective ways to build up a comprehensive set of regression tests and to actually find
defects in time.

But there's another issue...

You end up testing the wrong things!

Most test cases, and most automated test scripts, focus on testing what an application does.
You click on this button, then you enter this value, then you click on this other button, and so
on.

But by just concentrating on what the application does (or how the user interacts with the
application), it's easy to lose focus on something much more important: what the application
should do, and what the business needs it to do for them.

You spend so much time worrying about how to click this button or select that dropdown
value, that you forgot to check whether the application actually allows the users to do their job.

Which is the most important thing we can be checking! As the old saying goes, you can't see
the forest for the trees.

So what can we do? Read on to discover a much better approach...

 04

Agile Test Automation In Three Simple Steps

Step #1) Automate requirements, not test cases

"Cut out the middle-man"

Test cases are at best second-hand information - they are interpretations of the original
requirements documents or acceptance criteria. So automated test cases are, at best, third-
hand information.

Each step in the process takes time and effort (those manual test cases won't write
themselves) and introduces the risk of missing some important detail.

A much better approach is to cut out the middleman and write requirements that can be
automated. This involves talking to business and other team members, and agreeing on
acceptance criteria and a few key examples of expected behaviour before work starts, and
turning these examples into business-readable, automated tests.

The result is a set of executable specifications that you can use to clearly demonstrate to
business folk that the application does what it is supposed to do in their own language.

This simple strategy helped one tester I worked with decrease the time it takes to
write new automated tests by a whooping 80-90%!

This doesn’t mean executable specifications are your only tests. But they are the starting
point and the goal posts. Other tests, at different levels and written by developers or testers,
expand from there until you have precisely the level of confidence you need.
 05

The "Executable Specifications" mindset

This is what I call the "executable specification"

mindset.

Because the best agile test automation engineers

don’t write test scripts; they turn business
requirements into executable specifications.
And this changes everything.

Executable specifications, whether you do them

before work starts on a feature, or retrospectively
as documentation, are the key to getting more
efficient automation, higher quality results and
higher quality reporting.

SHOULD I WRITE TEST SCRIPTS OR EXECUTABLE SPECIFICATIONS?

TEST SCRIPTS EXECUTABLE SPECIFICATIONS

Are seen as the tester’s problem Encourage collaboration across the team

Can start before the feature even starts,

Are only written after a feature is finished
and you can even get developers involved!

Report on what the application is supposed

Report only what test cases you ran
to do

What does an executable specification look like?

Good question! Here's an example of what I mean...
 06

Step #2) Demonstrate the outcomes, not the application

When you automate manual test cases, it feels very natural to test everything through the
user interface. And, to be fair, when all you have is a UI automation tool, everything feels like
a UI test.

But when you work with executable specifications, everything changes.

Your goal is to demonstrate a business outcome, not specific interactions with the UI.

This means you have the freedom to automate some scenarios as UI tests, others as backend
or API tests, and yet others as a mix of both. Just pick the simplest, fastest way to prove that
the application allows your users to achieve a specific business goal.

This is the exact same technique that another team I know used to build a test suite
of over 1500 automated business scenarios that run in under 10 minutes!
 07

Step #3) Model behaviour all the way down

Writing (or recording) a new test script for each test case is unsustainable. You'll burn out, and
quickly!

If you are going to do agile test automation well, within the sprint, and still leave yourself
enough time for exploratory testing, you need a test suite that grows. And do to that, you’re
gonna need a plan.

The secret to scalable test suites is how they are organised. They use flexible components
that model business behaviour, because this is where you get the most valuable reuse.

Each layer breaks down and describes the behaviour of the next layer down, right down to
where you actually interact with the system. This way you maximise ease of maintenance,
reuse and stability.

This is the key to writing tests fast, too. One team of testers I know used this
approach to automate almost 800 rock solid business scenarios in under a month.
 08

Step #4) Go Beyond Test Automation

Too often, testers end up slaves to test

automation.

Have you felt the pressure to automate more

and more, while you struggle to automate
tests for the current sprint, not to mention
the growing backlog of regression tests that
need automating? Have you felt the pressure
to simply "churn out" automation scripts to
meet your KPIs?

The truth is, automation written after a feature is completed is the least powerful form of
automation. Sure, it might catch some regression issues, but it does little to make the team
work faster, or to make the team deliver more value.

With a well-designed executable specifications framework, on the other hand, you can do
much better. You can:
Collaborate more closely and more precisely with analysts and business folk, so that
misinterpreted requirements become a thing of the past
Encourage developers to participate as well, because it makes them go faster too! 

TEST AUTOMATION THAT SERVES THE WHOLE TEAM

In a nutshell, high performing teams don't build test automation frameworks. They build test
collaboration frameworks: frameworks that do not simply check what your application does,
after the fact but allow the team to work together to specify exactly what the application
should do, before work starts.

This is what flips the tables and puts test automation at the service of the whole team.
This way you can leverage your automated tests to TRULY benefit your whole team,
accelerate your delivery, impress your manager and turbo-charge your testing career.
 09

Don't be a slave to automation

Test Automation is essential in an agile project. As a tester you need to learn it, and you
need to learn how to do it well.

But test automation shouldn't mean an endless struggle with looming deadlines and falling
automated tests that nobody believes anymore.

Automation should free testers up to do deeper, smarter exploratory testing. It should

provide automated evidence on the state of the application, in terms that the business
understands and trusts.

And the approach that I've outlined in this document is a great way to deliver automation
that truly makes life easier for you as a tester, and also accelerates the development
process as a whole.

And that's great for your team and your career!

Build a test automation framework that lasts.

A framework you can be proud of.
 10

WHAT'S NEXT?
It’s our hope that this playbook will help you get a better idea of what really works when it
comes to building a scalable, sustainable agile test automation framework.

We'd love to continue the conversation! If you want to learn more about modern agile test
automation, come join our LinkedIn Group on the following URL:

https://bit.ly/agile-test-automation-secrets

If you're ready to talk about more tailored help in your test automation career, we can do
that too. At Serenity Dojo we have helped hundreds of testers from all background
become high performing agile test automation engineers, with a solid understanding
both of the automation tools, and of the best way to use the tools in modern projects.

I can't make test automation easy. It's not. But I can give you the skills
and mindset to make it easy for you.

So if you’d like to speak with us on how you too can learn agile test automation, we’d love
to talk to you. There is never any pressure on our clarity call sessions. Our objective is to
figure out what's stopping you becoming an awesome agile test automation engineer, and
then help you figure out what the best plan of action is.

You can book a call with us on the URL below:

https://www.serenity-dojo.com/apply

Feel free to send us an email (if you need anything) or if you just want to say ‘thanks for
the guide’. ([email protected]).

Looking forward to talking soon!

John Ferguson Smart

Founder
Serenity Dojo

Serenity Dojo,
Fleet Chambers, 8-9 Westmoreland St
DUBLIN, Ireland
 11

HOW CAN WE HELP YOU MASTER TEST AUTOMATION?

If you are serious about learning agile test automation, chances are we can help.

At the Serenity Dojo, we've been helping testers from all backgrounds become top-notch
agile test automation engineers for many many years. It's not for everyone (which is why
there is a selection process). But it does get great results.

See, the Serenity Dojo is more than a training course. It's a Community of Learning.

We threw away the book on conventional, dated, online training strategies and designed a
completely new approach from the ground up. We combine decades of industry
experience in agile and test automation with the latest advances in Learning Sciences to
get real results for our students.

Our goal with the Serenity Dojo Expansion Programme, is to give you the skill and
confidence you need to nail agile testing and test automation in virtually any domain. If
you'd like to know more, click on the button below to book a call:

https://www.serenity-dojo.com/apply
Or if you are on LinkedIn, reach out to John (mention this playbook)

https://www.linkedin.com/in/john-ferguson-smart/