Test Automation Framework Designs

Test Automation Framework Designs Designing standardized frameworks instead of simple scripting By Martin Lienhard

The Great Divide There has been a divide separating QA and Dev due in large part by 3 rd party tool vendors and their proprietary test tools Open source tools have helped significantly to bridge that divide, so that we can work together to create better software Most QAs have the bigger picture of what they want to test, and how to do it, but have trouble building robust automation frameworks, because they lack the technical skills. Most Developers focus on testing code in isolation, but lack the testing experience from a system-wide or end-to-end perspective, so they have trouble building a test automation framework that suits the needs of QA.

1 st Phase of Test Progression Most QA start out creating test scripts by using record & playback tools Lack conditional logic and looping Contain static data, UI locators, URLs, etc. Cannot be executed iteratively Cannot be reusable modules Should not be chained together as dependencies Lack configuration and global properties Cannot abstract away from the test tool or test runner

Record & Playback Test Scripts

2 nd Phase of Test Progression QA usually improves their tests by adding data-driven techniques Create variables and arrays to store data Create parameterized functions or methods to store data Add looping to recorded scripts

3 rd Phase of Test Progression QA usually improves their tests by creating a business function library Modular, reusable test libraries of functions Create file readers that pull in parameters from text files, CSV files, etc. Abstract out the data from the tests by externalizing data Create parameterized or iterative tests that are data-driven Abstract out the UI locators from the tests by externalizing UI maps Create global properties and configuration files A vast array of disparate data pools and UI maps are created, which usually become unmanageable after a couple of years

4 th Phase of Test Progression QA usually improves their tests by creating a keyword-driven library: Create ODBC functions that read data from spreadsheets and databases Create spreadsheets of keywords that represent business domain language Low-level keywords describe test operations High-level keywords describe business language Intermediate-level keywords map business functions to test operations Tests become readable by business users and non-technical QA

… continued Create spreadsheets of data that drive the tests Query data from the application database as input and verification points for tests A hierarchy of abstracted keyword spreadsheets are created Now non-technical BAs and QAs are creating automated tests A vast array of disparate keyword spreadsheets are created, which usually become unmanageable after a couple of years Nearly all 3 rd party vendor test tools live in this area today Most of these tools use legacy scripting languages, which are not compatible with today’s technology

5 th Phase of Test Progression QAs become more technical and improve their tests by using object oriented testing techniques: Objects are designed and created to further modularize and reuse code with very low maintenance Test code is stored in objects The page object design is used Component object designs are used Test data are stored in objects Configuration and properties are stored in objects Calls are made to the application and developer written APIs to make tests “smarter”

… continued Continuous integration environments are utilized to run tests QA moves into white box testing to drive quality deeper into the application UI tests are ran “headless” to make them execute faster QA expands the test framework to include multiple test tools and test runners Databases, web services, and APIs are created to reuse code and data across multiple test tools Test tools and 3 rd party utilities can be upgraded to newer versions with enhanced functionality with minor impact to the test framework

… continued Developers can now run the tests, and contribute to the overall test suites More 3 rd party vendors are rewriting their test tools to include object oriented languages to accommodate these technical QAs Due to the cost of the tools and their proprietary nature, it is difficult to place these tools into a continuous integration environment to share with developers

What is a test automation framework? A test automation framework is an application that allows you to write a series of tests without worrying about the constraints or limitations of the underlying test tools

Benefits of building a Test Framework Tool agnostic Abstracts away low level commands Utilize many test tools Test tools such as Selenium, WebDriver, etc. Test runner such as unit test frameworks Other common utilities Perform multiple levels and types of testing Functional, regression, load, performance, unit, integration, etc.

… continued Generic test API Common codebase and database Exception and error handling Modularized, reusable, and maintainable code and data Standardized test idioms Test configuration Configurable test suites Global test properties

… continued Automatic test versioning No need to get a specific version from the source control repository Always get the latest version Eliminates deployment to multiple machines for testing different versions of an application Deploy to a single server machine

… continued Multi-threading Run tests in parallel Runs on test machines in the grid or cloud Test across multiple environments and application versions simultaneously Continuous integration Run in a continuous integration environment Share tests and collaborate with other teams

Test Suite A collection of tests to run Have test parameters i.e.: application, environment, version, etc. Have test groups i.e.: regression, build, etc.

Test Configuration Any configurable test item that is separate from the test suite i.e.: web site URLs, database URLs, usernames, passwords, lab machine configurations, etc. Separated from the rest of the framework for information security and regulatory compliance

Test Properties Are thread specific Have global scope and can be accessed by any class method

Test Data Repository Abstracts away changes made only to test data Data can be stored in databases and files such as XML, JSON, XLS, TXT, CSV, etc. Data can be retrieved from data sources such as databases, web services, APIs, EDI, etc. Metadata can be used to describe additional information about the test data such as versions, application names, required fields, invalid formats, etc. Share data across multiple screens, applications, and tools

Parameterized Test example JUnit4 TestNG

Test Parameters Uses a hash map of key-value pairs Very useful for both functional and load testing Hash maps eliminate the need for Creating fixed method parameters that change frequently Changing data access objects that change frequently Hash maps can be easily populated from any data source Retrieve the test data and process the format for input to various test tools Handles version differences between test data

User Interface (UI) Mapping Abstracts away changes made only to UI object locators UI object identifiers can be stored in databases and files such as XML, JSON, XLS, TXT, CSV, etc. Metadata can be used to describe additional information about the UI objects such as versions, application names, form fields, etc. Share UI identifiers across multiple screens, applications, and tools

Test Controls Retrieve the UI locators and process the format for input to various test tools Handles version differences between UI locators Develop custom controls to mimic UI objects Self test verification

Test Control Interface example

Test Components A class object that represents a piece of a form, page, or screen Groups of common objects that can be found across more than one page Standardized component verification tests

Test Forms A class object that represents a piece a form Groups of components and controls that mimic the UI form Standardized form verification tests

Test Page or Screen A class object that represents a web page or screen – commonly referred to as “page objects” Groups of forms, components, and controls that mimic the UI page or screen Self test verification

Instances when NOT to return a new Page Object for each Page Submitting a page can return one of many different pages, or versions of pages, which would be represented by many different classes Page A could return B, C, or D Page A could return B v1.0, B v2.0, or Cv1.0, C v2.0, or D v1.0, D v2.0, etc. This means that you would need to do the following: Create multiple methods with similar names that return different page objects Page A.submitB() returns B Page A.submitC() returns C

… continued Return an page object of an inherited type, and then cast the type of the page in the script Page A.submit() returns page X Pages B and C inherit from page X Script checks the type of X and casts to B or C This can get very ugly in a script… Submitting a page may initiate an AJAX call, which can rewrite the same page Page A could display an AJAX (modal) dialog Page A could display some other AJAX widget Page A could be rewritten to look like page B

… continued If the page flow is not what is being tested, then an unexpected page or AJAX call could fail and kill the test Test case: We are trying to test the functionality of page Z The primary page flow is as follows: A to B, B to C, C to D, and D to Z The alternative page flows could also correctly occur, but are unpredictable base on many unknown variables: A to H, H to L, L to M, and M to Z A to R, R to S, and S to Z A to X, X to Y, and Y to Z

… continued What if page A unexpectedly returned page H, R, or X instead of page B? If page H, R, or X verified itself and asserted a failure, then how would the test ever reach page Z?

JavaScript & HTML DOM There are many instances when we need the test framework to mimic a human reading the content of a page and then making a decision on what action to perform next Verifying the list of items on a search results page Selecting items to purchase from a list Selecting elements to edit, update, or delete Scenarios like these usually occur as a result of some previous action, and display content generated dynamically to the user The test framework may have to query an application database or web service to provide the input to trigger the dynamic page content

… continued The test framework will then need to interrogate the page to determine if the expected elements, attributes, or content are present, and then make a decision on whether to perform an action on some element JavaScript can be used to inspect the HTML DOM to find these elements and their attributes and content JavaScript expressions can be evaluated by the test tool, and then return some result that the test framework can use to make a decision Checking if an element is present or getting a value will not work on a dynamically generated element

JS DOM example HTML JavaScript

Test Application Container A class object that represents the application, and contains the major page and component objects by version It provides the test services available to the test scripts Is thread specific

Container example Interface Class

Parallel Environment Tracks with Different Build Versions Parallel development tracks competing to deploy to production Run automated build verification tests Run automated functional and regression tests Run automated user acceptance tests Run automated tests for production patches and fixes Run automated sanity tests against production post-deployment Run automated tests across versions for production rollback strategy

Parallel Environment Tracks with Different Build Versions

Test Scripts Test scripts can be written using standard unit test frameworks Classes and methods are written using domain language Script details and complexity are abstracted to the framework layers Scripts are shorter and easier to read

Test Reports and Logs Reports can be generated from The unit test framework External reporting tools Custom report utilities can be written Test execution activity can be written to standard logging utilities Test failures, errors, and exceptions can be written to the reports and logs

Q&A Thank you for your time! Martin Lienhard [email_address]

Test Automation Framework Designs

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Test Automation Framework Designs (20)

More from Sauce Labs (20)

Recently uploaded (20)

Test Automation Framework Designs