Test
Automation
Fundamentals
Mikó Szilárd
2016
EPAM
University Material 2
Blue-chip clients rely on EPAM
University Material 3
SCHEDULE
9.12 Intro Monday 12:15
9.19 Unit testing 1
9.26 Unit testing 2
4 homeworks
10.03 Continuous integration 1
10.10 Continuous integration 2
10.17 Selenium WebDriver 1
10.24 Selenium WebDriver 2
10.31
11.07 Selenium WebDriver 3
11.14 Selenium WebDriver 4
11.21 Mobile Testing 1
11.28 Mobile Testing 2
12.05 Mobile Testing 3
University Material 4
Agenda
1 Introduction
2 Approaches
3 Tools and frameworks
4 Test Execution
University Material 5
Introduction
1 History
2 Motives
3 Advantages
4 Best Practices
5 Expectations and Reality
6 Two disciplines
University Material 6
Feedback Control Mechanism
University Material 7
40s – Concept of Automation
University Material 8
Future of
Automation
University Material 9
Motives
300
Million
250
Source lines of code
history
200
150
100
50
0
1980 1990 2000 2010 2020
Car Operating system
University Material 10
Motives
RISK
University Material 11
Motives
University Material 12
Repeatable &
schedulable
Fast test
execution
Reliable
Increased
functional
coverage
Automatic
reporting
University Material 13
Best Practices
Automation is recommended for
• Static and Repetitive testing
• Data driven testing
• High complexity testing
• Load and Performance testing
• Smoke testing
• Regression testing
University Material 14
Best Practices
Automation is NOT recommended for
• One-time, ad-hoc or exploratory testing
• Usability or look-and-feel testing
• Testing a feature under development
• Testing 3rd party services
• Cases where time and cost investment is too high
University Material 15
Expectations and Reality
• Automate at once • Automate gradually
• Cut testing budget • Increase test coverage
• Reduce testing time • Reinvest time savings
• Automate everything • Keep coverage and quality
• Every team can use automation tools • Skills and training needed for success
University Material 16
Two disciplines
• Test and automation are two different disciplines
Test Automation
Application expertise Development expertise
What to test How to automate
Test cases Test code
University Material 17
Approaches
1 Linear scripting – Record and Playback
2 Structured scripting - Library Architecture
3 Data Driven Testing
4 Keyword/Table Driven Testing
5 Hybrid Testing
6 Test Driven Development
7 Acceptance Test Driven Development
8 Behavior Driven Development
University Material 18
Linear scripting – Record and Playback
• Manual recording of each test step • Playing recorded test in the future
• Two types of verification • Differences reported as errors
– Insert verification checkpoints
– Automatic record of all responses
• Simplest approach
University Material 19
Linear scripting – Record and Playback
Advantages
• Little training and setup time
– Technical expertise not mandatory
• Develop new tests on the fly
• Able to capture steps for
– Later diagnosis
– Bug reproduction
• Could help understanding test scripting
University Material 20
Linear scripting – Record and Playback
Disadvantages
• Manual capture is unavoidable
• Hard or impossible to maintain
• Test data is hard coded
• Higher chance of redundancy
• Higher chance of false positives
University Material 21
Linear scripting – Record and Playback
Improvements
• For windowed applications
– Use fixed screen format
– Verify only a selected area
• Ensuring the same input data
– Use a database backup
– Use stubbed data
• Use variables to store dynamic data
University Material 22
Structured scripting - Library Architecture
• Create an abstraction layer for testing • Use control structures (conditionals, loops)
• Each module or function has its own test • Create a common library from reusable parts
scripts
• Build test cases using test functions
University Material 23
Structured scripting - Library Architecture
Advantages
• Scalable
• High level of code reuse
• Easy to adapt functionality changes
• Lower maintenance cost
University Material 24
Structured scripting - Library Architecture
Disadvantages
• Test data is hard coded
• Technical expertise required
• Longer planning time
• Longer test script preparation
• Deeper knowledge of the application
University Material 25
Data Driven Testing
• Test data shouldn’t be hard coded
• Divide test data and test code
• Storage options
– Database, XML, Excel or CSV file
• Read inputs and expected outputs as key-value pairs
• Run your tests against multiple sets of data
University Material 26
Data Driven Testing
Advantages
• Early test data creation is possible
• Script changes doesn’t affect test data
• One script can run with multiple sets of data
• Manipulating test data could mean
– Adding, removing or modifying test cases
• Lower maintenance costs
University Material 27
Data Driven Testing
Disadvantages
• Technical expertise required
• Longer planning time
• Longer test script preparation
• Test data should be maintained carefully
University Material 28
Keyword/Table Driven Testing
• Store test data and actions
separately from code
• Create keywords for actions
• Create keyword parameters from test data
• Create test cases with a series of
keywords
University Material 29
Keyword/Table Driven Testing
Main process stages
Test preparation
Defining test basis
Test design
Analyzing test basis Test case design Test data design
Manual execution
Automating test execution
Implementing keyword actions
Automated execution
University Material 30
Keyword/Table Driven Testing
Advantages
• Early test design creation is possible
• High level of code reuse
• Independence from test tools
• Readable test cases for stakeholders
• Testing work can be distributed
– Test design needs application expertise
– Implementation needs automation expertise
University Material 31
Keyword/Table Driven Testing
Disadvantages
• Very high initial investment
• High automation expertise required
– Creating and maintaining the framework
• Not feasible for smaller projects
University Material 32
Hybrid Testing
• Combination of other approaches
• Over time it’s a natural evolution step
– Structured data driven testing
– Structured keyword driven testing
• Aims to have all their advantages
• Has most of their disadvantages
– Initial investment and needed expertise are high
University Material 33
“If it's not worth testing, why are you wasting
your time working on it?” - Scott Ambler
University Material 34
Test Driven Development
• Introduced a new evolutionary approach
– Test first development
– Refactoring
• Break down requirements to a test case
• Improve on software only that much to pass the new test
• Refactor design flaws and rerun all tests to catch regression
• When all test passed you can start it over
University Material 35
Test Driven Development
University Material 36
Test Driven Development
• Focuses on developers
• Developers doing TDD need
– Strong understanding of requirements
– Knowledge at least in unit testing
• Good unit test characteristics are
– Runs fast
– Runs in isolation
– Easy to read and understand
– Uses real data
– Represents one step towards your overall goal
University Material 37
Acceptance Test Driven Development
• Extension of test driven development
• Acceptance tests drive the development process
– Creating a layer between requirements and developer tests
– Helping to communicate and understand customer needs
• Through them all stakeholders could be involved
– Customers, Business, Testers, Developers
University Material 38
Instead of „What should I test?”,
you might hear
„What’s the intended behavior?”.
- Neel Lakshminarayan
University Material 39
Behavior Driven Development
• Improvement of acceptance test driven development
• Business goals drive the development process
• Enables a common language between stakeholders
• Through the use of a simple domain-specific language
– Domain specific phrases in English sentences
University Material 40
Behavior Driven Development
• Acceptance tests should be written as a user story
– As a [role]
– I want [feature]
– so that [benefit]
• Acceptance criteria should be written as scenarios
– Given [initial context]
– When [event occurs]
– Then [ensure some outcomes]
University Material 41
Behavior Driven Development
User story example
Title: Customer withdraws cash
As a customer,
I want to withdraw cash from an ATM,
so that I don’t have to wait in line at the bank.
Scenario example
Scenario: Account is in credit
Given the account is in credit
And the card is valid
And the dispenser contains cash
When the customer requests cash
Then ensure the account is debited
And ensure the card is returned
And ensure cash is dispensed
University Material 42
Tools and Frameworks
• Test automation tools can help in automating such task as
– Product installation
– Test data creation
– API interaction
– GUI interaction
– Test execution
– Problem detection
– Defect logging
• They serve as a driving agent for the automation process
University Material 43
• An automation framework is not a • Frameworks put automation
tool which performs a specific task approaches into practice
• It is an infrastructure that provides • Building a framework is a
a unified solution strategic decision
University Material 44
Return on Investment
A profitability measure that evaluates the performance of a
business by dividing net profit by investment.
𝑁𝑒𝑡 𝑝𝑟𝑜𝑓𝑖𝑡 𝐺𝑟𝑜𝑠𝑠 𝑝𝑟𝑜𝑓𝑖𝑡 − 𝐼𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡
𝑅𝑂𝐼 = =
𝐼𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡 𝐼𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡
• Good metric to measure return of money per period
• Could be used as an indicator to compare different alternatives
• However it is risky since a pre calculation is based on assumptions
University Material 45
Test Execution
1 Test Log
2 Error Log
3 Analyzing Results
4 Reporting
5 Metrics
University Material 46
Analyzing Results
• Classify false positives
– Test environment
– Application changes
– Test error
• Avoid reporting the same failure
• Keep on eye on false negatives
– Test execution finishes surprisingly fast
– Verified message appears in a different place
University Material 47
Metrics
University Material 48
Questions
University Material 49
