Developing a Feedback-Driven Automated Testing Tool for Web Applications

Scott McMaster and Xun Yuan (Google) http://www.webtestingexplorer.org/

What Does a Modern Web Application Look Like?

Multiple application tiers -- client and server. Frequently updated.
Especially the user interface.

Browser side:
Hundreds/thousands of HTML elements. Dynamic -- always changing at runtime, frequently asynchronously.

The Current State of Web Application Testing

Heavy reliance on unit testing of individual classes/modules.
The majority of automated testing occurs here.

Ad-hoc and manual system/integration testing. Inefficient automated testing through the user interface:
Hard to maintain due to rapid changes. Focused on observing the browser to the exclusion of other application tiers.

The Future State: WebTestingExplorer

Supports Automated Exploratory Testing.
Automated: Easy to replay (and regenerate) interesting test cases. Exploratory: Systematically discovers new application states. Testing: Includes test case generation, oracle generation, real-time bug detection, and test case replay.

WebTestingExplorer Advantages
Efficiently test through the user interface. Test like a real user. Don't miss integration issues and things that are hard to observe.
Behind the scenes in the browser. Or on other application tiers.

Design: Testability
Optimize for the case where testers have detailed technical knowledge of the application-under-test.
And can improve testability (such as by adding unique id's).

Testable applications allow us to generate more robust tests.

Design: Code-Based Configuration and Customization

Testers use Java code.
Not a domain-specific language. Not declarative. Not highly reliant on XPath expressions.

Integrates with JUnit, WebDriver, Selenium, etc.

Design: Use Existing Libraries

Writing robust infrastructure to drive modern browsers and web applications is hard. --> WebTestingExplorer is built on WebDriver.
Inherits its low-level robustness. Easy for testers to extend.

Design: Performance, Stability, and Effectiveness

Many regions of a fully rendered web application are not interesting for testing.
"Like" buttons, display ads, common banners, etc.

-->Easy to operate on subsets of the UI.

Improves performance (less to look at). Improves stability (can skip unstable/changing regions). Improves effectiveness (can focus on the most interesting user scenarios).

WebTestingExplorer Workflow
Two Complementary Phases: 1. Exploratory
Incrementally searches for new application states. Writes out new test cases and oracles. Can search for bugs.

2. Replay
Executes tests generated by the Exploratory phase. For regression testing. A/B, comparison of actual and expected states.

Terminology
Element: An HTML element in the browser.
Actionable Element: An element that the application's end-user can interact with. Stateful Element: An element that the tester wants to examine for state changes. State: An abstraction of the current state of the web application, as defined by the tester. Action: Performed on an element to cause something to happen in the web application. Action Sequence: An ordered list of actions. Test Case: An action sequence (deemed interesting on the basis of state change) that is saved for future replay.

Workflow: Exploratory
Iteratively and systematically moving through the web application via the user interface.
Monitors for different types of bugs. Generates increasingly-long and sophisticated test cases (and oracles) for later replay.

Driven by feedback derived from state.

State: Definition
Def: An abstraction of the current state of the web application, customizable by the tester. WebTestingExplorer has a rich and extensible notion of state. Can theoretically include a combination of:
Web browser state (HTML, DOM, cookies, ...) Web server state (session data, variables, ...) Database state (persistent application data)

User-defined (but some state implementations are included and ready-touse!)

State: Browser-Based Examples

Stateful HTML Elements
All Count Visible Selected subset (e.g. only <input>'s)

Stateful Elements' Properties

All (HTML+CSS) Selected subset (e.g. font-weight, color)

Exploration and State

Changes in observed state between actions move the exploration forward into longer action sequences. High-level algorithm:
a. b. c. d. e. f. Pick an action sequence. Take a state snapshot. Do something. Take another state snapshot. Compare a) and c). Different? Then: Generate a new test case. Enqueue new action sequences to execute. g. Repeat.

Exploration Algorithm
Execute IAS AS: action sequence AI: actionable element IAS: initial action sequence SS: start state FS: final state

Extend IAS

DEQUE

No

FS == SS Yes Write Test Case

Execute AS Collect States Create Oracles

Extend AS

Exploration Algorithm
Execute IAS <IAS1, AI1>,<IAS1, AI2> ,....,<IAS1, AIn> <IAS2, AI1>,<IAS2, AI2> ,....,<IAS2, AIn> ... <IASm, AI1>,<IASm, AI2> ,....,<IASm, AIn> Extend IAS

DEQUE

No

FS == SS Yes Write Test Case

Execute AS Collect States Create Oracles

Extend AS

Oracles: Framework
WebTestingExplorer includes an oracle framework and some pre-defined oracles. Can be used to detect bugs during Exploratory and Replay phases. Oracles for regression testing can be generated during Exploratory phase, used during Replay phase.
May leverage initial and final state.

Oracles: Examples
HttpResponseOracle: Checks for allowed and disallowed HTTP response codes. JSErrorCollectorOracle: Looks for Javascript runtime errors. FinalStateCheckOracle: Uses initial and final state for regression testing. Also:
The framework allows testers to plug in custom oracle implementations.

Workflow: Replay Example (Regression Testing)

Exploration creates a test case.
Test case includes saved final state for the configured state checkers.

Replayer is configured with the FinalStateCheckOracle. Replayer replays the test case. At the end, a state snapshot is taken and compared to the final state saved in the test case. If they are different, a failure is recorded.

Configuration
WebTestingExplorer is highly configurable for both phases (Exploratory and Replay). Can limit the state space and restrict the action sequences to be explored. Examples:
Skip exploring common header and footer elements on the page. Exclude ads from the Actionable and Stateful elements. In an industrial web application, we have observed that we can generate interesting test cases by checking just two Stateful elements.

Configuration Points
Selectors for Actionable and Stateful web elements. Pre-packaged and custom Oracles. Action sequence filters to eliminate redundant or uninteresting exploration paths. Example: MaxRepeatedActionSequenceFilter eliminates action sequences that repeat the same action more than a set number of times. Action generators to create custom actions on specific elements. Action sequence prioritizer for test case prioritization during exploration. Custom wait conditions to increase stability while executing an action sequence.

Example: Custom Action Generator

Only explore links that execute javascript:

new TagActionGeneratorConfig("a") { @Override public Set<Action> generateActions(WebElementWithIdentifier elementWithId) { Set<Action> actions = Sets.newHashSet(); String href = elementWithId.getElement().getAttribute("href"); if (href != null && href.startsWith("javascript:")) { actions.add(new ClickAction(elementWithId.getIdentifier())); } return actions; } }

Related Work
Browser automation frameworks (Selenium, WebDriver).
Allow testers to automate test cases of their own design.

Model-based GUI test case generation (GUITAR).

Exploration model very focused on desktop applications (menus and modal dialogs).

Web site crawler (Crawljax).

Diffs the browser DOM tree based on a similarity metric. Defines invariants as Xpath expressions.

Future Work
Parallel exploration. Code coverage integration. Evaluate the effectiveness of different state abstractions. Integrated state from multiple tiers of the web application. Industrial applications and studies.

Questions?

Related Work: Crawljax