AWS Device Farm

Developer Guide
API Version 2015-06-23

AWS Device Farm Developer Guide

AWS Device Farm: Developer Guide

Copyright 2016 Amazon Web Services, Inc. and/or its affiliates. All rights reserved.
Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner
that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not
owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by
Amazon.

AWS Device Farm Developer Guide

Table of Contents
What Is AWS Device Farm? ............................................................................................................ 1
Automated App Testing ........................................................................................................... 1
Supported Test Types and Built-in Tests ............................................................................. 1
Remote Access Interaction ..................................................................................................... 2
Terminology .......................................................................................................................... 2
Setting Up ............................................................................................................................ 3
Setting Up .................................................................................................................................... 4
Step 1: Sign Up for AWS ......................................................................................................... 4
Step 2: Create or Use an IAM User in Your AWS Account ............................................................. 4
Step 3: Give the IAM User Permission to Access Device Farm ....................................................... 5
Next Step ............................................................................................................................. 6
Getting Started ............................................................................................................................. 7
Prerequisites ........................................................................................................................ 7
Step 1: Sign in to the Console .................................................................................................. 8
Step 2: Create a Project .......................................................................................................... 8
Step 3: Create and Start a Run ................................................................................................ 8
Step 4: View the Run's Results ................................................................................................ 9
Next Steps ........................................................................................................................... 9
Concepts ................................................................................................................................... 10
Devices .............................................................................................................................. 10
Supported Devices ...................................................................................................... 10
Device Pools ............................................................................................................... 10
Device Branding .......................................................................................................... 11
Device Slots ............................................................................................................... 11
Pre-Installed Device Apps ............................................................................................. 11
Device Capabilities ...................................................................................................... 11
Test Types .......................................................................................................................... 11
Android Test Types ....................................................................................................... 11
iOS Test Types ............................................................................................................ 12
Web Application Test Types ........................................................................................... 12
Runs ................................................................................................................................. 12
Run Configuration ........................................................................................................ 13
Run Files Retention ..................................................................................................... 13
Parallel Runs .............................................................................................................. 13
Instrumenting Apps ...................................................................................................... 13
Re-Signing Apps in Runs .............................................................................................. 13
Obfuscated Apps in Runs .............................................................................................. 13
Ads in Runs ................................................................................................................ 13
Media in Runs ............................................................................................................. 13
Common Tasks for Runs ............................................................................................... 14
Reports .............................................................................................................................. 14
Report Retention ......................................................................................................... 14
Report Components ..................................................................................................... 14
Performance Samples in Reports ................................................................................... 14
Logs in Reports ........................................................................................................... 14
Common Tasks for Reports ........................................................................................... 14
Sessions ............................................................................................................................ 14
Supported Devices for Remote Access ............................................................................ 15
Session Files Retention ................................................................................................ 15
Instrumenting Apps ...................................................................................................... 15
Re-Signing Apps in Sessions ......................................................................................... 15
Obfuscated Apps in Sessions ........................................................................................ 15
Purchase Device Slots .................................................................................................................. 16
Purchase a Device Slot with the Device Farm Console ............................................................... 16
Purchase a Device Slot with the AWS CLI ................................................................................ 17
API Version 2015-06-23
iii

AWS Device Farm Developer Guide

Purchase a Device Slot with the Device Farm API .....................................................................

Working with Projects ...................................................................................................................
Create a Project ..................................................................................................................
Prerequisites ..............................................................................................................
Create a Project with the Device Farm Console .................................................................
Create a Project with the AWS CLI .................................................................................
Create a Project with the Device Farm API .......................................................................
View the Projects List ...........................................................................................................
Prerequisites ..............................................................................................................
View the Projects List with the Device Farm Console ..........................................................
View the Projects List with the AWS CLI ..........................................................................
View the Projects List with the Device Farm API ................................................................
Working with Runs .......................................................................................................................
Create a Run ......................................................................................................................
Prerequisites ..............................................................................................................
Create a Run with the Device Farm Console .....................................................................
Create a Run with the AWS CLI .....................................................................................
Create a Run with the Device Farm API ...........................................................................
Next Steps .................................................................................................................
Stop a Run .........................................................................................................................
Prerequisites ..............................................................................................................
Stop a Run with the Device Farm Console .......................................................................
Stop a Run with the AWS CLI ........................................................................................
Stop a Run with the Device Farm API ..............................................................................
View a Runs List ..................................................................................................................
Prerequisites ..............................................................................................................
View a Runs List with the Device Farm Console ................................................................
View a Runs List with the AWS CLI .................................................................................
View a Runs List with the Device Farm API ......................................................................
Create a Device Pool ............................................................................................................
Prerequisites ..............................................................................................................
Create a Device Pool with the Device Farm Console ..........................................................
Create a Device Pool with the AWS CLI ...........................................................................
Create a Device Pool with the Device Farm API ................................................................
Analyze a Report .................................................................................................................
Prerequisites ..............................................................................................................
Console Icons .............................................................................................................
Open a Report with the Device Farm Console ...................................................................
Analyze a Report's Summary Page with the Device Farm Console .......................................
Analyze a Report's Unique Problems with the Device Farm Console .....................................
Analyze a Report by Device with the Device Farm Console .................................................
Analyze a Report by Suite with the Device Farm Console ...................................................
Analyze a Report by Test with the Device Farm Console .....................................................
Analyze Performance Data for a Problem, Device, Suite, or Test in a Report with the Device
Farm Console .............................................................................................................
Analyze Log Information for a Problem, Device, Suite, or Test in a Report with the Device Farm
Console .....................................................................................................................
Working with Test Types ................................................................................................................
Built-in Test Types ................................................................................................................
Custom Test Types ...............................................................................................................
Custom Android Test Types ...........................................................................................
Custom iOS Test Types .................................................................................................
Custom Web Application Test Types ................................................................................
Android Tests ......................................................................................................................
Built-in Test Types for Android ........................................................................................
Custom Test Types for Android .......................................................................................
Appium Java JUnit .......................................................................................................
Appium Java TestNG ....................................................................................................
API Version 2015-06-23
iv

19
20
20
20
20
21
21
21
21
22
22
22
23
23
23
24
25
25
25
25
26
26
27
28
28
29
29
29
29
29
30
30
31
31
31
31
31
31
32
32
33
33
34
35
35
38
38
38
38
38
39
39
39
39
39
44

AWS Device Farm Developer Guide

Appium Python ............................................................................................................ 48

Calabash ................................................................................................................... 51
Instrumentation ........................................................................................................... 53
UI Automator .............................................................................................................. 54
iOS Tests ........................................................................................................................... 56
Built-in Test Types for iOS .............................................................................................. 56
Custom Test Types ....................................................................................................... 56
Appium Java JUnit ....................................................................................................... 56
Appium Java TestNG .................................................................................................... 60
Appium Python ............................................................................................................ 65
Calabash ................................................................................................................... 68
UI Automation ............................................................................................................. 70
XCTest ....................................................................................................................... 71
XCTest UI ................................................................................................................... 72
Web App Tests .................................................................................................................... 74
Rules for Metered and Unmetered Devices ...................................................................... 74
Appium Java JUnit ....................................................................................................... 74
Appium Java TestNG .................................................................................................... 76
Appium Python ............................................................................................................ 78
Built-in Tests ....................................................................................................................... 82
Built-in Test Types ........................................................................................................ 82
Built-in: Explorer (Android) ............................................................................................. 82
Built-in: Fuzz (Android and iOS) ..................................................................................... 83
Working with Remote Access ......................................................................................................... 84
Create a Session ................................................................................................................. 84
Prerequisites .............................................................................................................. 85
Create a Session with the Device Farm Console ............................................................... 85
Next Steps ................................................................................................................. 86
Use a Session ..................................................................................................................... 87
Prerequisites .............................................................................................................. 87
Use a Session in the Device Farm Console ...................................................................... 87
Next Steps ................................................................................................................. 88
Get Session Results ............................................................................................................. 88
Prerequisites .............................................................................................................. 88
Viewing Session Details ................................................................................................ 88
Downloading Session Video or Logs ............................................................................... 89
CloudTrail Integration .................................................................................................................... 90
Device Farm Information in CloudTrail ..................................................................................... 90
Understanding Device Farm Log File Entries ............................................................................ 91
AWS CLI Reference ..................................................................................................................... 93
API Reference ............................................................................................................................. 94
Access Permissions Reference ...................................................................................................... 95
Create and Attach a Policy to an IAM User ............................................................................... 96
Action Syntax for Performing Actions in Device Farm ................................................................. 97
Limits ........................................................................................................................................ 98
Tools and Plugins ........................................................................................................................ 99
Jenkins CI Plugin ................................................................................................................. 99
Step 1: Install the Plugin .............................................................................................. 101
Step 2: Create an IAM User ......................................................................................... 102
Step 3: First-time configuration instructions ..................................................................... 103
Step 4: Use the Plugin ................................................................................................ 103
Dependencies ........................................................................................................... 104
Device Farm Gradle Plugin .................................................................................................. 104
Building the Device Farm Gradle Plugin ......................................................................... 104
Setting up the Device Farm Gradle Plugin ...................................................................... 105
Generating an IAM user .............................................................................................. 106
Configuring Test Types ................................................................................................ 107
Dependencies ........................................................................................................... 109
API Version 2015-06-23
v

AWS Device Farm Developer Guide

Document History ...................................................................................................................... 110

AWS Glossary ........................................................................................................................... 112

API Version 2015-06-23

vi

AWS Device Farm Developer Guide

Automated App Testing

What Is AWS Device Farm?

Device Farm is an app testing service that enables you to test and interact with your Android, iOS, and
Web apps on real, physical phones and tablets that are hosted by Amazon Web Services (AWS). There
are two main ways to use Device Farm:
Automated testing of apps using a variety of available testing frameworks
Remote access of devices onto which you can load, run, and interact with apps in real time

Automated App Testing

Device Farm allows you to upload your own tests or use built-in, script-free compatibility tests. Because
testing is automatically performed in parallel, tests on multiple devices begin in minutes.
A test report containing high-level results, low-level logs, pixel-to-pixel screenshots, and performance
data is updated as tests are completed.
Device Farm supports testing of native and hybrid Android, iOS, and Fire OS apps, including those created
with PhoneGap, Titanium, Xamarin, Unity, and other frameworks. It supports remote access of Android
apps for interactive testing.

Supported Test Types and Built-in Tests

Device Farm currently provides support for the following test types:
For Android:

Appium Java JUnit (p. 39)

Appium Java TestNG (p. 44)
Appium Python (p. 48)
Calabash (p. 51)

Instrumentation (p. 53) (JUnit, Espresso, Robotium, or any instrumentation-based tests)

UI Automator (p. 54)
Explorer (p. 82)
For iOS:

API Version 2015-06-23

1

AWS Device Farm Developer Guide

Remote Access Interaction

Appium Java JUnit (p. 56)

Appium Java TestNG (p. 60)
Appium Python (p. 65)
Calabash (p. 68)
UI Automation (p. 70)
XCTest (p. 71) (including KIF)
XCTest UI (p. 72)
For Web Apps:
Appium Java JUnit (p. 74)
Appium Java TestNG (p. 76)
Appium Python (p. 78)
If you do not have your own tests, you can use a built-in fuzz test. For more information, see Built-in:
Fuzz (Android and iOS) (p. 83).

Remote Access Interaction

Remote access allows you to swipe, gesture, and interact with a device through your web browser in real
time. There are a number of situations where real-time interaction with a device is useful. For example,
customer service representatives can guide customers through how to use or set up their device. They
can also walk customers through how to use apps running on a specific device. You can install apps on
a device running in a remote access session and then reproduce customer problems or reported bugs.
During a remote access session, Device Farm collects details about actions that take place as you interact
with the device. Logs with these details and a video capture of the session are produced at the end of
the session for your review.
Initially, a limited number of Android and Fire OS devices are supported for remote access. However, the
list of devices will grow during the beta period and as new devices enter the market.

Terminology
Device Farm introduces the following terms that define the way information is organized:
project
A logical workspace that contains runs, one run for each test of a single app against one or more
devices. Projects enable you to organize workspaces in whatever way you choose. For example,
there can be one project per app title, or there can be one project per platform. You can create as
many projects as you need.
run
A specific build of your app, with a specific set of tests, to be run on a specific set of devices. A run
produces a report that contains information about the results of the run. A run contains one or more
jobs. For more information, see Runs (p. 12).
report
Contains information about a run, which is a request for Device Farm to test a single app against
one or more devices. For more information, see Reports (p. 14).

API Version 2015-06-23

2

AWS Device Farm Developer Guide

Setting Up

job
A request for Device Farm to test a single app against a single device. A job contains one or more
suites.
meter
Metering refers to billing for devices, and you may encounter references to "metered devices" or
"unmetered devices" in the documentation and API reference. For more information about pricing,
see AWS Device Farm Pricing.
suite
The hierarchical organization of tests in a test package. A suite contains one or more tests.
test
An individual test within a test package.
session
An interactive session with a single device in the console.

Setting Up
To get set up to use Device Farm, see Setting Up (p. 4).

API Version 2015-06-23

3

AWS Device Farm Developer Guide

Step 1: Sign Up for AWS

Setting Up AWS Device Farm

Before you use Device Farm for the first time, you must complete the following tasks:
Topics
Step 1: Sign Up for AWS (p. 4)
Step 2: Create or Use an IAM User in Your AWS Account (p. 4)
Step 3: Give the IAM User Permission to Access Device Farm (p. 5)
Next Step (p. 6)

Step 1: Sign Up for AWS

Sign up for Amazon Web Services (AWS).
If you do not have an AWS account, use the following procedure to create one.

To sign up for AWS

1.
2.

Open http://aws.amazon.com/ and choose Create an AWS Account.

Follow the online instructions.

Step 2: Create or Use an IAM User in Your AWS

Account
We recommend that you do not use your AWS root account to access Device Farm. Instead, create a
new AWS Identity and Access Management (IAM) user (or use an existing IAM user) in your AWS account,
and then access Device Farm with that IAM user.
To create a new IAM user, see Creating an IAM User (AWS Management Console).

API Version 2015-06-23

4

AWS Device Farm Developer Guide

Step 3: Give the IAM User Permission to Access Device
Farm

Step 3: Give the IAM User Permission to Access

Device Farm
Give the IAM user permission to access Device Farm. To do this, create a new access policy in IAM, and
then assign the access policy to the IAM user, as follows.

Note
The AWS root account or IAM user that you use to complete the following steps must have
permission to create the following IAM policy and attach it to the IAM user. For more information,
see Working with Policies

To create the access policy in IAM

1.
2.
3.

Open the Identity and Access Management (IAM) console at https://console.aws.amazon.com/iam/.

Choose Policies.
Choose Create Policy. (If a Get Started button appears, choose it, and then choose Create Policy.)

4.
5.

Next to Create Your Own Policy, choose Select.

For Policy Name, type a name for the policy (for example, AWSDeviceFarmAccessPolicy).

6.

For Description, type Provides access to all Device Farm actions associated with
the IAM user.

7.

For Policy Document, type the following statement:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"devicefarm:*"
],
"Resource": [
"*"
]
}
]
}

8.

Choose Create Policy.

To assign the access policy to the IAM user

1.
2.
3.

Open the IAM console at https://console.aws.amazon.com/iam/.

Choose Users.
Choose the IAM user to whom you will assign the access policy.

4.
5.

In the Permissions area, for Managed Policies, choose Attach Policy.

Select the policy you just created (for example, AWSDeviceFarmAccessPolicy).

6.

Choose Attach Policy.

API Version 2015-06-23

5

AWS Device Farm Developer Guide

Next Step

Note
Attaching the policy provides the IAM user with access to all Device Farm actions associated
with that IAM user. To learn how to restrict IAM users to a limited set of Device Farm actions,
see Access Permissions Reference (p. 95).

Next Step
You are now ready to start using Device Farm. See Getting Started (p. 7).

API Version 2015-06-23

6

AWS Device Farm Developer Guide

Prerequisites

Getting Started with AWS Device

Farm
This walkthrough shows you how to use Device Farm to test an Android or iOS app. In this walkthrough,
you will use the Device Farm console to create a project, upload an .apk or .ipa file, run a suite of standard
tests, and then view the results.
Topics
Prerequisites (p. 7)
Step 1: Sign in to the Console (p. 8)
Step 2: Create a Project (p. 8)
Step 3: Create and Start a Run (p. 8)
Step 4: View the Run's Results (p. 9)
Next Steps (p. 9)

Prerequisites
Before you begin this walkthrough, make sure you have completed the following requirements:
Complete the steps in Setting Up (p. 4), which include signing up for an AWS account, creating or
using an IAM user in the AWS account, and giving the IAM user permission to access Device Farm.
For Android, you will need an .apk (Android app package) file, and for iOS you will need an .ipa (iOS
app archive) file, which you will upload to Device Farm later in this walkthrough.

Note
Make sure that your .ipa file is built for an iOS device and not for a simulator.
Optionally, you will need a test from one of the test types supported by Device Farm. You will upload
this test package to Device Farm, and then run the test later in this walkthrough. (If you do not have a
test package available, you can specify and run a standard built-in test suite.) For more information,
see Working with Test Types in AWS Device Farm (p. 38).

API Version 2015-06-23

7

AWS Device Farm Developer Guide

Step 1: Sign in to the Console

Step 1: Sign in to the Console

You can use the Device Farm console to create and manage projects and runs for testing. You will learn
about projects and runs later in this walkthrough.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

Step 2: Create a Project

To test an app in Device Farm, you must first create a project.
A project in Device Farm represents a logical workspace in Device Farm that contains runs, one run for
each test of a single app against one or more devices. Projects enable you to organize workspaces in
whatever way you choose. For example, there can be one project per app title, or there can be one project
per platform. You can create as many projects as you need.
1.
2.

Choose Get started or Create a new project.

For Project name, type a name for your project (for example, MyDemoProject).

Note
If you type a project name other than MyDemoProject, be sure to use it throughout this
walkthrough.
3.

Choose Create project.

Step 3: Create and Start a Run

Now that you have a project, you can create and then start a run.
A run in Device Farm represents a specific build of your app, with a specific set of tests, to be run on a
specific set of devices. A run produces a report that contains information about the results of the run. A
run contains one or more jobs. For more information, see Runs (p. 12).
1.
2.
3.
4.

Choose MyDemoProject.
On the Choose your application page, choose Upload.
Browse to and choose your Android or iOS app file. For Android, the file must be an .apk file. For
iOS, the file must be an .ipa file built for a device, not the simulator.
Choose Next step.

5.

On the Configure a test page, choose one of the test suites.

Note
If you do not have any tests available, choose Built-in: Fuzz to run a standard built-in test
suite. For this walkthrough, if you choose Built-in: Fuzz, leave the Event count, Event
throttle, and Randomizer seed boxes at their default values.
6.
7.
8.

If you did not choose Built-in: Fuzz, then choose Upload, and browse to and choose the file that
contains your tests.
Choose Next step.
On the Select devices page, for Device pool, choose Top Devices to select the device pool, and
then choose Next step.
A device pool in Device Farm represents a collection of devices that typically share similar
characteristics such as platform, manufacturer, or model. For more information, see Devices (p. 10).

API Version 2015-06-23

8

AWS Device Farm Developer Guide

Step 4: View the Run's Results

9.

On the Specify device state page, do any of the following:

To provide additional data for Device Farm to use during the run, next to Add extra data, choose
Upload, and then browse to and choose the .zip file.
To install an additional app for Device Farm to use during the run, next to Install other apps,
choose Upload, and then browse to and choose the .apk or .ipa file that contains the app. Repeat
for any additional apps you want to install. You can change the installation order by dragging and
dropping the apps.
To specify whether Wi-Fi, Bluetooth, GPS, or NFC will be enabled during the run, next to Set radio
states, select the appropriate boxes.
To preset the device latitude and longitude for the run, next to Device location, type the coordinates.
To preset the device locale for the run, choose the locale in Device Locale.

10. Choose Review and start run.

11. On the Review and start run page, choose Confirm and start run.
Device Farm should start the run as soon as devices are available, typically within a few minutes. Until
the run starts, Device Farm will display a calendar icon. After the run starts, results will appear as tests
are completed. During this time, Device Farm will display a progress icon.

Step 4: View the Run's Results

You'll know the run is complete when the progress icon changes to a result icon.
To view the run's results, choose the completed run in the Device Farm console. A summary page that
includes the following information is displayed.

The total number of tests, by outcome.

Lists of tests with unique warnings or failures.
A list of devices and test results for each.
Any screenshots captured during the run, grouped by device.

For more information, see Analyze a Report (p. 31).

You have now completed this walkthrough.

Next Steps
To learn more about Device Farm, see Concepts (p. 10).

API Version 2015-06-23

9

AWS Device Farm Developer Guide

Devices

AWS Device Farm Concepts

This section describes important Device Farm concepts.

Devices (p. 10)

Test Types in AWS Device Farm (p. 11)
Runs (p. 12)
Reports (p. 14)
Sessions (p. 14)

Device Support in AWS Device Farm

The following sections contain information above device support in Device Farm.
Topics
Supported Devices (p. 10)
Device Pools (p. 10)
Device Branding (p. 11)
Device Slots (p. 11)
Pre-Installed Device Apps (p. 11)
Device Capabilities (p. 11)

Supported Devices
Device Farm provides support for hundreds of unique, popular Android, iOS, and Fire OS devices and
operating system combinations. The list of available devices grows as new devices enter the market. The
full list of devices can be found here: Device List.

Device Pools
Device Farm organizes its devices into device pools that you can use for your testing. These device pools
contain related devices, such as devices that run only on Android or that run only on iOS. Device Farm
provides curated device pools, such as those for top devices.You can also create your own private device
pools.

API Version 2015-06-23

10

AWS Device Farm Developer Guide

Device Branding

Device Branding
Device Farm runs tests on physical, non-rooted devices that are both OEM- and carrier-branded.

Device Slots
Device slots correspond to concurrency in which the number of device slots you have determines how
many devices you can run in tests or remote access sessions. There are two types of device slots, remote
access device slots and automated testing device slots. An automated testing device slot is one on which
you can run tests concurrently. A remote access device slot is one you can run in remote access sessions
concurrently.
If you have one automated testing device slot, then you can only run tests on one device at a time. If you
purchase additional automated testing device slots, then you can run multiple tests concurrently on multiple
devices to get test results faster. If you have one remote access device slot, you can only run one remote
access session at a time. If you purchase additional remote testing device slots, then you can run multiple
sessions concurrently.
You can purchase device slots based on the device family (Android or iOS devices for automated testing
and Android for remote access). For more information, see Device Farm Pricing.

Pre-Installed Device Apps

Devices in Device Farm include a small number of apps pre-installed by manufacturers and carriers.

Device Capabilities
All devices have a Wi-Fi connection to the Internet. They do not have carrier connections and cannot
make phone calls or send SMS messages.
You can take photos with any device that supports a front- or rear-facing camera. Due to the way the
devices are mounted, photos may look dark and blurry.
Google Play Services is installed on devices that support it, but these devices do not have an active
Google account.

Test Types in AWS Device Farm

AWS Device Farm provides many different built-in and custom test types for Android, iOS, and Web
applications. Built-in tests enable you to test your apps without writing scripts. Custom tests allow you to
test specific flows and business logic within your app. For more information, see Working with Test Types
in AWS Device Farm (p. 38).

Android Test Types

Device Farm provides the following built-in and custom test types for Android devices.
Built-in: Explorer (Android) (p. 82)
Built-in: Fuzz (Android and iOS) (p. 83)
Appium Java JUnit (p. 39)
Appium Java TestNG (p. 44)
Appium Python (p. 48)

API Version 2015-06-23

11

AWS Device Farm Developer Guide

iOS Test Types

Calabash (p. 51)

Instrumentation (p. 53)
UI Automator (p. 54)

iOS Test Types

Device Farm provides the following built-in and custom test types for iOS devices.
Built-in: Fuzz (Android and iOS) (p. 83)
Appium Java JUnit (p. 56)
Appium Java TestNG (p. 60)
Appium Python (p. 65)
Calabash (p. 68)
UI Automation (p. 70)
XCTest (p. 71)
XCTest UI (p. 72)

Web Application Test Types

Device Farm provides the following custom test types for Web applications.
Appium Java JUnit (p. 74)
Appium Java TestNG (p. 76)
Appium Python (p. 78)

Runs in AWS Device Farm

The following sections contain information about runs in Device Farm.
A run in Device Farm represents a specific build of your app, with a specific set of tests, to be run on a
specific set of devices. A run produces a report that contains information about the results of the run. A
run contains one or more jobs.
Topics
Run Configuration (p. 13)
Run Files Retention (p. 13)
Parallel Runs (p. 13)
Instrumenting Apps (p. 13)
Re-Signing Apps in Runs (p. 13)
Obfuscated Apps in Runs (p. 13)
Ads in Runs (p. 13)
Media in Runs (p. 13)
Common Tasks for Runs (p. 14)

API Version 2015-06-23

12

AWS Device Farm Developer Guide

Run Configuration

Run Configuration
As part of a run, you can supply settings Device Farm can use to override current device settings. These
include latitude and longitude coordinates, locale, radio states (such as Bluetooth, GPS, NFC, and Wi-Fi),
extra data (contained in a .zip file), and auxiliary apps (apps that should be installed before the app that
will be tested).

Run Files Retention

Device Farm stores your apps and files for 30 days and then deletes them from its system.You can delete
your files at any time, however.
Device Farm stores your run results, logs, and screenshots for 400 days and then deletes them from its
system.

Parallel Runs
Device Farm runs tests in parallel as devices become available.

Instrumenting Apps
You do not need to instrument your apps or provide Device Farm with the source code for your apps.
Android apps can be submitted unmodified. iOS apps must be built with the iOS Device target instead
of with the simulator.

Re-Signing Apps in Runs

For iOS apps, you do not need to add any Device Farm UUIDs to your provisioning profile. Device Farm
replaces the embedded provisioning profile with a wildcard profile and then re-signs the app. If you provide
auxiliary data, Device Farm will add it to the app's package before Device Farm installs it, so that the
auxiliary will exist in your app's sandbox. Re-signing the app removes entitlements such as App Group,
Associated Domains, Game Center, HealthKit, HomeKit, Wireless Accessory Configuration, In-App
Purchase, Inter-App Audio, Apply Pay, Push Notifications, and VPN Configuration & Control.
For Android apps, Device Farm re-signs the app. This may break any functionality that depends on the
app's signature, such as the Google Maps Android API, or it may trigger antipiracy or antitamper detection
from products such as DexGuard.

Obfuscated Apps in Runs

For Android apps, if the app is obfuscated, you can still test it with Device Farm if you use ProGuard.
However, if you use DexGuard with antipiracy measures, Device Farm will not be able to re-sign and run
tests against the app.

Ads in Runs
We recommend that you remove ads from your apps before you upload them to Device Farm. We cannot
guarantee that ads will be displayed during runs.

Media in Runs
You can provide media or other data to accompany your app. Additional data must be provided in a .zip
file no more than 4 GB in size.
API Version 2015-06-23
13

AWS Device Farm Developer Guide

Common Tasks for Runs

Common Tasks for Runs

For more information, see Create a Run (p. 23) and Working with Runs (p. 23).

Reports in AWS Device Farm

The following sections contain information about Device Farm reports.
A report in Device Farm contains information about a run, which is a request for Device Farm to test a
single app against one or more devices.
Topics
Report Retention (p. 14)
Report Components (p. 14)
Performance Samples in Reports (p. 14)
Logs in Reports (p. 14)
Common Tasks for Reports (p. 14)

Report Retention
Device Farm stores your reports for 400 days. These reports include metadata, logs, screenshots, and
performance data.

Report Components
Reports in Device Farm contain pass and fail information, crash reports, test and device logs, screenshots,
and performance data.
Reports include both detailed per-device data as well as high-level results, such as the number of
occurrences of a given problem.

Performance Samples in Reports

During a test run, Device Farm captures performance samples every second.

Logs in Reports
Reports include complete logcat captures for Android tests and complete Device Console Logs for iOS
tests.

Common Tasks for Reports

For more information, see Analyze a Report (p. 31).

Sessions in AWS Device Farm

You can use Device Farm to perform interactive testing of Android apps through remote access sessions
in a web browser. This kind of interactive testing helps support engineers on a customer call to walk step
API Version 2015-06-23
14

AWS Device Farm Developer Guide

Supported Devices for Remote Access

by step through the customer's issue. Developers can reproduce a problem on a specific device to isolate
possible sources of the problem. You can use remote sessions to conduct usability tests with your target
customers.
A session in Device Farm is a real-time interaction with an actual, physical device hosted in a web browser.
Topics
Supported Devices for Remote Access (p. 15)
Session Files Retention (p. 15)
Instrumenting Apps (p. 15)
Re-Signing Apps in Sessions (p. 15)
Obfuscated Apps in Sessions (p. 15)

Supported Devices for Remote Access

Device Farm provides support for a number of unique popular Android and Fire OS devices and operating
system combinations. The list of available devices grows as new devices enter the market and will grow
beyond the initial set during the beta period. The current list of Android and Fire OS devices available for
remote access is displayed in the console. For more information about devices, see Devices (p. 10).

Session Files Retention

Device Farm stores your apps and files for 30 days and then deletes them from its system.You can delete
your files at any time, however.
Device Farm stores your session logs and captured video for 400 days and then deletes them from its
system.

Instrumenting Apps
You do not need to instrument your apps or provide Device Farm with the source code for your apps.
Android apps can be submitted unmodified.

Re-Signing Apps in Sessions

For Android apps, Device Farm re-signs the app. This may break any functionality that depends on the
app's signature, such as the Google Maps Android API, or it may trigger antipiracy or antitamper detection
from products such as DexGuard.

Obfuscated Apps in Sessions

For Android apps, if the app is obfuscated, you can still test it with Device Farm if you use ProGuard.
However, if you use DexGuard with antipiracy measures, Device Farm will not be able to re-sign the app.

API Version 2015-06-23

15

AWS Device Farm Developer Guide

Purchase a Device Slot with the Device Farm Console

Purchase a Device Slot in AWS

Device Farm
To purchase a device slot, you can use the Device Farm console, the AWS CLI, or the Device Farm API.

Note
To use the device slots feature, you need to be on a metered test plan that allows for unlimited
testing.
Topics
Purchase a Device Slot with the Device Farm Console (p. 16)
Purchase a Device Slot with the AWS CLI (p. 17)
Purchase a Device Slot with the Device Farm API (p. 19)

Purchase a Device Slot with the Device Farm

Console
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

Choose the Account settings icon in the upper right-hand corner to get to the Account settings
page.

API Version 2015-06-23

16

AWS Device Farm Developer Guide

Purchase a Device Slot with the AWS CLI

Note
If you are currently viewing a project or a Run result, you can also access the Project
settings page by choosing the Project settings icon in the upper right-hand corner. The
screen and options look the same.
3.

4.

On the Account settings page (or the Project settings page), choose the number of automated
testing Android slots, automated testing iOS slots, or remote access Android slots you wish to
purchase.
The text dynamically updates with the amount that will be added to your bill for each device slot
purchased. For more information, see Device Farm Pricing.
Choose Purchase.
The next time you create a test run (p. 23), you can have multiple devices running tests concurrently
or multiple remote access sessions running concurrently, based on the number of device slots
purchased.

Purchase a Device Slot with the AWS CLI

You can run the purchase-offering command to purchase the offering.
To list the device slot offerings available to you, run the list-offerings command. You will see output
similar to the following:
{
"offerings": [
{
"recurringCharges": [
{
"cost": {
"amount": 250.0,
"currencyCode": "USD"
},
"frequency": "MONTHLY"
}
],
"platform": "IOS",
"type": "RECURRING",
"id": "GUID",

API Version 2015-06-23

17

AWS Device Farm Developer Guide

Purchase a Device Slot with the AWS CLI

"description": "iOS Unmetered Device Slot"

},
{
"recurringCharges": [
{
"cost": {
"amount": 250.0,
"currencyCode": "USD"
},
"frequency": "MONTHLY"
}
],
"platform": "ANDROID",
"type": "RECURRING",
"id": "GUID",
"description": "Android Unmetered Device Slot"
},
{
"recurringCharges": [
{
"cost": {
"amount": 250.0,
"currencyCode": "USD"
},
"frequency": "MONTHLY"
}
],
"platform": "ANDROID",
"type": "RECURRING",
"id": "GUID",
"description": "Android Remote Access Unmetered Device Slot"
}
]
}

To get the offering status, run the get-offering-status command. You will see output similar to the
following:
{
"current": {
"GUID": {
"offering": {
"platform": "IOS",
"type": "RECURRING",
"id": "GUID",
"description": "iOS Unmetered Device Slot"
},
"quantity": 1
},
"GUID": {
"offering": {
"platform": "ANDROID",
"type": "RECURRING",
"id": "GUID",
"description": "Android Unmetered Device Slot"
},
"quantity": 1

API Version 2015-06-23

18

AWS Device Farm Developer Guide

Purchase a Device Slot with the Device Farm API

}
},
"nextPeriod": {
"GUID": {
"effectiveOn": 1459468800.0,
"offering": {
"platform": "IOS",
"type": "RECURRING",
"id": "GUID",
"description": "iOS Unmetered Device Slot"
},
"quantity": 1
},
"8980F81C-00D7-469D-8EC6-BCE39477F2F6": {
"effectiveOn": 1459468800.0,
"offering": {
"platform": "ANDROID",
"type": "RECURRING",
"id": "GUID",
"description": "Android Unmetered Device Slot"
},
"quantity": 1
}
}
}

Additional commands for this feature include renew-offering and list-offering-transactions.

For more information about specific operations, see the AWS CLI reference for Device Farm.
For information about using Device Farm with the AWS CLI, see AWS CLI Reference (p. 93).

Purchase a Device Slot with the Device Farm

API

Call the PurchaseOffering operation to purchase the offering.

For information about using the Device Farm API, see API Reference (p. 94).

API Version 2015-06-23

19

AWS Device Farm Developer Guide

Create a Project

Working with Projects in AWS

Device Farm
A project in Device Farm represents a logical workspace in Device Farm that contains runs, one run for
each test of a single app against one or more devices. Projects enable you to organize workspaces in
whatever way you choose. For example, there can be one project per app title, or there can be one project
per platform. You can create as many projects as you need.
You can use the Device Farm console, the AWS CLI, or the Device Farm service API to work with projects.
Create a Project (p. 20)
View the Projects List (p. 21)

Create a Project in AWS Device Farm

To create a project, you can use the Device Farm console, the AWS CLI, or the Device Farm API.
Topics
Prerequisites (p. 20)
Create a Project with the Device Farm Console (p. 20)
Create a Project with the AWS CLI (p. 21)
Create a Project with the Device Farm API (p. 21)

Prerequisites

Complete the steps in Setting Up (p. 4), which include signing up for an AWS account, creating or
using an IAM user in the AWS account, and giving the IAM user permission to access Device Farm.

Create a Project with the Device Farm Console

1.
2.

Make sure you have set up an AWS account and an IAM user to access Device Farm.
Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

API Version 2015-06-23

20

AWS Device Farm Developer Guide

Create a Project with the AWS CLI

3.

If Create a new project or Get started is displayed, then choose it.

Tip
If neither Create a new project nor Get started is displayed, then do one of the following:
On the secondary navigation bar, choose the Device Farm console home button, and
then choose Create a new project.
On the secondary navigation bar, for Projects, choose View all projects, and then choose
Create a new project.

4.
5.

For Project name, type a name for your project.

Choose Create project.

Create a Project with the AWS CLI

1.
2.

Make sure you have set up an AWS account and an IAM user to access Device Farm.
Run the create-project command.

Note
For information about using Device Farm with the AWS CLI, see AWS CLI Reference (p. 93).

Create a Project with the Device Farm API

1.
2.

Make sure you have set up an AWS account and an IAM user to access Device Farm.
Call the CreateProject API.

For information about using the Device Farm API, see API Reference (p. 94).

View the Projects List in AWS Device Farm

To view the list of available projects, you can use the Device Farm console, the AWS CLI, or the Device
Farm API.
Topics
Prerequisites (p. 21)
View the Projects List with the Device Farm Console (p. 22)
View the Projects List with the AWS CLI (p. 22)
View the Projects List with the Device Farm API (p. 22)

Prerequisites

Create at least one project in Device Farm. To create a project, follow the instructions in Create a
Project (p. 20), and then return to this page.

API Version 2015-06-23

21

AWS Device Farm Developer Guide

View the Projects List with the Device Farm Console

View the Projects List with the Device Farm

Console
1.
2.

Make sure that you have completed at least one project.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

3.

If the list of available projects is not displayed, then on the secondary navigation bar, do one of the
following:
Choose the Device Farm console home button.
For Projects, choose View all projects.

View the Projects List with the AWS CLI

1.
2.

Make sure that you have completed at least one project.

To view the projects list, run the list-projects command.

Tip
To view information about a single project, run the get-project command.

Note
For information about using Device Farm with the AWS CLI, see AWS CLI Reference (p. 93).

View the Projects List with the Device Farm API

1.
2.

Make sure that you have completed at least one project.

To view the projects list, call the ListProjects API.

Tip
To view information about a single project, call the GetProject API.

For information about the Device Farm API, see API Reference (p. 94).

API Version 2015-06-23

22

AWS Device Farm Developer Guide

Create a Run

Working with Runs in AWS Device

Farm
A run in Device Farm represents a specific build of your app, with a specific set of tests, to be run on a
specific set of devices. A run produces a report that contains information about the results of the run. A
run contains one or more jobs. For more information, see Runs (p. 12).
You can use the Device Farm console, the AWS CLI, or the Device Farm service API to work with runs.

Create a Run (p. 23)

Stop a Run in AWS Device Farm (p. 25)
View a Runs List (p. 28)
Create a Device Pool (p. 29)
Purchase Device Slots (p. 16)
Analyze a Report (p. 31)

Create a Run in AWS Device Farm

To create a run, you can use the Device Farm console, the AWS CLI, or the Device Farm API.
For information about runs, see Runs (p. 12).
Topics
Prerequisites (p. 23)

Create a Run with the Device Farm Console (p. 24)

Create a Run with the AWS CLI (p. 25)
Create a Run with the Device Farm API (p. 25)
Next Steps (p. 25)

Prerequisites

Create a project in Device Farm. Follow the instructions in Create a Project (p. 20), and then return
to this page.

API Version 2015-06-23

23

AWS Device Farm Developer Guide

Create a Run with the Device Farm Console

Create a Run with the Device Farm Console

1.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

2.
3.

If you see the AWS Device Farm console home page, choose Get started.
If you already have a project, you can upload your tests to an existing project or choose Create a
new project.

4.
5.

If the Create a new run button is displayed, then choose it.

On the Choose your application page, choose either Native application (the Android and Apple
button) or Web application (the HTML5 button).
Upload your application file and provide a Run name when instructed to do so.
On the Configure a test page, choose one of the available test suites.

6.
7.

Note
If you do not have any tests available, then choose Built-in: Fuzz to run a standard test
suite that is built-in to Device Farm. If you choose Built-in: Fuzz, and the Event count,
Event throttle, and Randomizer seed boxes appear, you can change or leave the values
as desired.
For information about the available test suites, see Working with Test Types in AWS Device
Farm (p. 38).
8. If you did not choose Built-in: Fuzz, then choose Upload, and browse to and choose the file that
contains your tests.
9. Choose Next step.
10. On the Select devices page, do one of the following:
To choose a built-in device pool to run the tests against, for Device pool, choose Top Devices
or All Devices.
To create your own device pool to run the tests against, follow the instructions in Create a Device
Pool (p. 29), and then return to this page.
If you created your own device pool earlier, for Device pool, choose your device pool.
For more information, see Devices (p. 10).
11. Choose Next step.
12. On the Specify device state page, do none, some, or all of the following:
To provide any additional data that Device Farm will use during the run, choose Upload next to
Add extra data, and then browse to and choose the .zip file that contains the additional data.
To install an additional app that Device Farm will use during the run, choose Upload next to Install
other apps, and then browse to and choose the .apk, or .ipa, file that contains the app. Repeat
this for any additional apps that you want to install. You can change the apps' installation order by
dragging and dropping the apps after you upload them.

API Version 2015-06-23

24

AWS Device Farm Developer Guide

Create a Run with the AWS CLI

To specify whether Wi-Fi, Bluetooth, GPS, or NFC will be enabled during the run, next to Set radio
states, select the appropriate boxes.
To preset the device latitude and longitude for the run, next to Device location, type the coordinates.
To preset the device locale for the run, choose the locale in Device Locale.

13. Choose Review and start run.

14. On the Review and start run page, choose Confirm and start run.
Device Farm will start the run as soon as devices are available, typically within a few minutes. Until the
run starts, Device Farm will display a calendar icon. After the run starts, results will appear as tests are
completed. During this time, Device Farm will display a progress icon.

Note
If you need to stop the test run, see Stop a Run in AWS Device Farm (p. 25).

Create a Run with the AWS CLI

1.
2.

Make sure that you have completed at least one project.

Run the schedule-run command.

Note
For information about using Device Farm with the AWS CLI, see AWS CLI Reference (p. 93).

Create a Run with the Device Farm API

1.
2.

Make sure that you have created a project. For more information, see prerequisites (p. 21).
Call the ScheduleRun API.

For information about using the Device Farm API, see API Reference (p. 94).

Next Steps
You'll know the run is complete when the progress icon changes to a result icon. A report corresponding
to the run will appear as soon as tests are complete. For more information, see Reports (p. 14).
To use the report, follow the instructions in Analyze a Report (p. 31).

Stop a Run in AWS Device Farm

You may want to stop a run after you have started it. For example, you may notice an issue while your
tests are running and wish to restart the run with an updated test script. This topic describes how to stop
a run and what the implications are for billing.
To stop a run, you can use the Device Farm console, the AWS CLI, or the Device Farm API.
For information about runs, see Runs (p. 12).
Topics
Prerequisites (p. 26)
Stop a Run with the Device Farm Console (p. 26)

API Version 2015-06-23

25

AWS Device Farm Developer Guide

Prerequisites

Stop a Run with the AWS CLI (p. 27)

Stop a Run with the Device Farm API (p. 28)

Prerequisites

To stop a test run, you must have a test run created and actively running. For more information, see
Create a Run (p. 23).

Stop a Run with the Device Farm Console

1.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

2.
3.

From the Device Farm console home page, choose the project where you have an active test run.
On the Run results page, choose the test run.
Your screen should look like the following image (with the in-progress icon to the left of the device
name).

4.

Choose Stop run.

The button changes to Stopping, and after a short time the icon next to the device name also changes
to the Stopping icon (a pulsing orange circle with a square inside it). When completely finished, the
icon changes to an orange square.

Important
If a test has already finished, Device Farm cannot stop it. If a test is in progress, Device
Farm will stop the test and you will see the total minutes for which you will be billed in the
Details page. In addition, you will still be billed for the total minutes that Device Farm takes
to run the Setup Suite and the Teardown Suite. For more information, see Device Farm
Pricing.
The following image shows an example Details page after a test run was successfully stopped.

API Version 2015-06-23

26

AWS Device Farm Developer Guide

Stop a Run with the AWS CLI

Stop a Run with the AWS CLI

You can run the following command to stop the specified test run, where myARN is the Amazon Resource
Name (ARN) of the test run.
$ aws devicefarm stop-run --arn myARN

You will see output similar to the following:

{
"run": {
"status": "STOPPING",
"name": "Name of your run",
"created": 1458329687.951,
"totalJobs": 7,
"completedJobs": 5,
"deviceMinutes": {
"unmetered": 0.0,
"total": 0.0,
"metered": 0.0
},
"platform": "ANDROID_APP",
"result": "PENDING",
"billingMethod": "METERED",
"type": "BUILTIN_EXPLORER",
"arn": "myARN",
"counters": {
"skipped": 0,
"warned": 0,
"failed": 0,
"stopped": 0,
"passed": 0,
"errored": 0,
"total": 0
}
}
}

To get the ARN of your run, use the list-runs command. The output will be similar to the following:

API Version 2015-06-23

27

AWS Device Farm Developer Guide

Stop a Run with the Device Farm API

{
"runs": [
{
"status": "RUNNING",
"name": "Name of your run",
"created": 1458329687.951,
"totalJobs": 7,
"completedJobs": 5,
"deviceMinutes": {
"unmetered": 0.0,
"total": 0.0,
"metered": 0.0
},
"platform": "ANDROID_APP",
"result": "PENDING",
"billingMethod": "METERED",
"type": "BUILTIN_EXPLORER",
"arn": "Your ARN will be here",
"counters": {
"skipped": 0,
"warned": 0,
"failed": 0,
"stopped": 0,
"passed": 0,
"errored": 0,
"total": 0
}
}
]
}

Note
For information about using Device Farm with the AWS CLI, see AWS CLI Reference (p. 93).

Stop a Run with the Device Farm API

Call the StopRun operation to the test run.

For information about using the Device Farm API, see API Reference (p. 94).

View a Runs List in AWS Device Farm

To view a list of available runs for a project, you can use the Device Farm console, the AWS CLI, or the
Device Farm API.
Topics
Prerequisites (p. 29)
View a Runs List with the Device Farm Console (p. 29)
View a Runs List with the AWS CLI (p. 29)
View a Runs List with the Device Farm API (p. 29)

API Version 2015-06-23

28

AWS Device Farm Developer Guide

Prerequisites

Prerequisites

Create at least one run in Device Farm. To create a run, follow the instructions in Create a Run (p. 23),
and then return to this page.

View a Runs List with the Device Farm Console

1.
2.
3.

Make sure that you complete the prerequisites (p. 29), including the creation of at least one run.
Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.
In the list of projects, choose the project that corresponds to the runs list you want to view.

Tip
If the list of available projects is not displayed, then on the secondary navigation bar, do
one of the following:
Choose the Device Farm console home button, and then choose the project.
For Projects, choose View all projects, and then choose the project.

View a Runs List with the AWS CLI

1.
2.

Make sure that you have completed the prerequisites (p. 29), including the creation of at least one
run.
To view a list of runs, run the list-runs command.

Tip
To view information about a single run, run the get-run command.

For general information about using Device Farm with the AWS CLI, see AWS CLI Reference (p. 93).

View a Runs List with the Device Farm API

1.

Make sure that you have completed the prerequisites (p. 29), including the creation at least one run.

2.

To view a list of runs, call the ListRuns API.

Tip
To view information about a single run, call the GetRun API.

For general information about the Device Farm API, see API Reference (p. 94).

Create a Device Pool in AWS Device Farm

To create a device pool, you can use the Device Farm console, the AWS CLI, or the Device Farm API.
Topics
Prerequisites (p. 30)
Create a Device Pool with the Device Farm Console (p. 30)
Create a Device Pool with the AWS CLI (p. 31)

API Version 2015-06-23

29

AWS Device Farm Developer Guide

Prerequisites

Create a Device Pool with the Device Farm API (p. 31)

Prerequisites

Start by creating a run in the Device Farm console. Follow the instructions in Create a Run (p. 23).
When you get to the Select devices page in the Create a new run wizard, continue with the
instructions in this section.

Create a Device Pool with the Device Farm Console

1.
2.
3.

Make sure you have started to create a run and stopped on the Select devices page in the Create
a new run wizard.
On the Select devices page, choose Create new device pool.
For Name, type a name that will make this device pool easy to identify in the future.

4.
5.

For Description, type a description that will make this device pool easy to identify in the future.
If you want to use one or more selection criteria for the devices in this device pool, do the following:
1.
2.

Choose Add rule.

For Field, choose one of the following:
Choose Manufacturer to include devices by their manufacturer name.
Choose Type to include devices by their Type value.

3.

For Operator, choose the following:

Choose EQUALS to include devices where the Field value equals the Operand value.

4.

5.
6.

6.

For Operand, type or choose the value you want to specify for the Field and Operator values.
Note that if you choose Platform for Field, then the only available selections are ANDROID and
IOS. Similarly, if you choose Type for Field, then the only available selections are PHONE and
TABLET.
To add another rule, choose Add rule again.
To delete a rule, choose the X icon next to the rule you want to delete.

After you create the first rule, then in the list of devices, the box next to each device that matches
the rule will be selected. After you create additional rules or change existing rules, then in the list of
devices, the box next to each device that matches those combined rules will be selected. Devices
with selected boxes will be included in the device pool. Devices with cleared boxes will be excluded
from the device pool.
If you want to manually include or exclude individual devices, select or clear the box next to each
device.

Note
You can select or clear the boxes only if you do not have any rules specified.
7.

If you want to include or exclude all displayed devices, select or clear the box in the column header
row of the list.

Important
Although you can use the boxes in the column header row to change the list of displayed
devices, this does not mean that the remaining displayed devices will be the only ones
included or excluded. To confirm which devices will be included or excluded, be sure to
API Version 2015-06-23
30

AWS Device Farm Developer Guide

Create a Device Pool with the AWS CLI

clear the contents of all of the boxes in the column header row. Then browse the boxes to
see which devices will be included or excluded.
8.

Choose Save device pool.

Create a Device Pool with the AWS CLI

Run the create-device-pool command.

For information about using Device Farm with the AWS CLI, see AWS CLI Reference (p. 93).

Create a Device Pool with the Device Farm API

Call the CreateDevicePool API.

For information about using the Device Farm API, see API Reference (p. 94).

Analyze a Report in AWS Device Farm

Use the Device Farm console to analyze a report. For more information, see Reports (p. 14).

Prerequisites

Create a run in Device Farm, and verify the run is complete. Follow the instructions in Create a
Run (p. 23), and then return to this page.

Console Icons
Icon

Description

Green check mark inside of a circle

Finished, Success

Orange exclamation mark inside of a triangle

Warning

Red exclamation mark inside of a circle

Failure

Blue circle with a slash through it

Skipped

Orange square

Stopped

Open a Report with the Device Farm Console

1.

Make sure the run is complete.

2.
3.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the project for the run that corresponds to the report that you want to
access.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, choose the
Device Farm console home button, and then choose the project.
API Version 2015-06-23
31

AWS Device Farm Developer Guide

Analyze a Report's Summary Page with the Device Farm
Console

4.

In the list, choose the run with the finished icon (green check mark) that corresponds to the report
you want to access. The report's summary page is displayed.

To analyze the various parts of the report, follow the instructions in the following sections.

Analyze a Report's Summary Page with the Device

Farm Console
1.

If the report's summary page is not already displayed, follow the instructions in Open a Report with
the Device Farm Console (p. 31).

2.

Near the beginning of the summary page, the total number of tests, by outcome, is displayed.
A green check mark inside of a circle is displayed next to successful tests.

3.

An orange exclamation mark inside of a triangle is displayed next to tests with warnings.
A red exclamation mark inside of a circle is displayed next to failed tests.
A red exclamation mark is displayed next to tests with errors.
A blue circle with a slash through it is displayed next to skipped tests.
An orange square is displayed next to stopped tests.

The Details tab displays a list of test results as follows:

Unique problems lists unique warnings and failures. To analyze unique problems, follow the
instructions in Analyze a Report's Unique Problems with the Device Farm Console (p. 32).
Devices displays the total number of tests, by outcome, for the device.
Next to the device's name, one of the following icons is displayed:
If there is at least one stopped test for the device, an orange square is displayed.
If there is at least one test with errors, a red exclamation mark is displayed.
If there is at least one failed test, a red exclamation mark inside of a circle is displayed.
If there is at least one test with warnings, an orange exclamation mark inside of a triangle is
displayed.
Otherwise, a green check mark inside of a circle is displayed.
To analyze the results by device, follow the instructions in Analyze a Report by Device with the Device
Farm Console (p. 33).

4.

The Screenshots tab displays a list of any screenshots Device Farm captured during the run, grouped
by device.

Analyze a Report's Unique Problems with the

Device Farm Console
1.

If the report's summary page is not already displayed, follow the instructions in Open a Report with
the Device Farm Console (p. 31).

2.

Following the total number of tests by outcome for the run, on the Details tab, for Unique problems,
choose the problem that you want to analyze.
The Details tab displays information about the problem. The Logs section displays any information
Device Farm logged during the test. To analyze this information, follow the instructions in Analyze
Log Information for a Problem, Device, Suite, or Test in a Report with the Device Farm Console (p. 35).

3.

API Version 2015-06-23

32

AWS Device Farm Developer Guide

Analyze a Report by Device with the Device Farm
Console

4.
5.

The Screenshots tab displays a list of any screenshots Device Farm captured during the test.
The Performance tab displays information about any performance data Device Farm generated
during the test. To analyze this performance data, follow the instructions in Analyze Performance
Data for a Problem, Device, Suite, or Test in a Report with the Device Farm Console (p. 35).

Analyze a Report by Device with the Device Farm

Console
1.

If the report's summary page is not already displayed, follow the instructions in Open a Report with
the Device Farm Console (p. 31).

2.
3.

On the Details tab, choose the device whose results you want to analyze.
The Details tab displays information about the suites for the device. For each suite, the following
test results are displayed:
For Test Results, the total number of tests for the suite is displayed by outcome.
Next to the suite's name, one of the following icons is displayed:
An orange square is displayed if there is at least one stopped test for the suite.
A red exclamation mark is displayed if there is at least one test with errors for the suite.
A red exclamation mark inside of a circle is displayed if there is at least one failed test for the
suite.
An orange exclamation mark inside of a triangle is displayed if there is at least one test for the
suite with warnings.
Otherwise, a green check mark inside of a circle is displayed.
To analyze the results by suite, follow the instructions in Analyze a Report by Suite with the Device
Farm Console (p. 33).

4.
5.

6.

The Logs section displays any information Device Farm logged for the device during the run. To
analyze this information, follow the instructions in Analyze Log Information for a Problem, Device,
Suite, or Test in a Report with the Device Farm Console (p. 35).
The Screenshots tab displays a list of any screenshots Device Farm captured during the run for the
device, grouped by suite.
The Performance tab displays information about any performance data Device Farm generated for
the device during the run. To analyze this performance data, follow the instructions in Analyze
Performance Data for a Problem, Device, Suite, or Test in a Report with the Device Farm
Console (p. 35).
The Files tab displays a list of suites for the device and any associated files (such as log files) that
can be downloaded. To download a file, choose the file's link in the list.

Analyze a Report by Suite with the Device Farm

Console
1.

If the report's summary page is not already displayed, follow the instructions in Open a Report with
the Device Farm Console (p. 31).

2.

On the Details tab, choose the device that corresponds to the suite whose results you want to
analyze.
Choose the suite that you want to analyze for results.

3.
4.

The Details tab displays information about the tests for the suite. For each test, the following test
results are displayed:
API Version 2015-06-23
33

AWS Device Farm Developer Guide

Analyze a Report by Test with the Device Farm Console

For Test Results, the outcome for the test is displayed as follows:
If the test succeeded, the number 1 is displayed next to a green check mark inside of a circle.
If the test has warnings, the number 1 is displayed next to an orange exclamation mark inside
of a triangle.
If the test was skipped, the number 1 is displayed next to a blue circle with a slash through it.
If the test failed, the number 1 is displayed next to a red exclamation mark inside of a circle.
If the test has errors, the number 1 is displayed next to a red exclamation mark.
If the test was stopped, the number 1 is displayed next to an orange square.
These icons are also displayed next to the test's name.
To analyze the results by test, follow the instructions in Analyze a Report by Test with the Device
Farm Console (p. 34).

5.
6.

7.

The Logs section displays any information Device Farm logged during the run for the suite. To
analyze this information, follow the instructions in Analyze Log Information for a Problem, Device,
Suite, or Test in a Report with the Device Farm Console (p. 35).
The Screenshots tab displays a list of any screenshots Device Farm captured during the run for the
suite, grouped by test.
The Performance tab displays information about any performance data Device Farm generated
during the run for the suite. To analyze this performance data, follow the instructions in Analyze
Performance Data for a Problem, Device, Suite, or Test in a Report with the Device Farm
Console (p. 35).
The Files tab displays a list of tests for the suite and any associated files (such as log files) that can
be downloaded. To download a file, choose the file's link in the list.

Analyze a Report by Test with the Device Farm

Console
1.
2.
3.
4.

If the report's summary page is not already displayed, open the report by following the instructions
in Open a Report with the Device Farm Console (p. 31).
On the Details tab, choose the device that corresponds to the test you want to analyze for results.
Choose the suite that corresponds to the test you want to analyze for results.
On the Details tab, choose the test whose results you want to analyze.

5.

The Details tab displays information about the test.

If the test was stopped, an orange square is displayed.
If the test contains errors, a red exclamation mark is displayed.
If the test failed, a red exclamation mark inside of a circle is displayed.
If the test had a warning, an orange exclamation mark inside of a triangle is displayed.
Otherwise, a green check mark inside of a circle is displayed .

6.
7.

The Logs section displays any information Device Farm logged during the test. To analyze this
information, follow the instructions in Analyze Log Information for a Problem, Device, Suite, or Test
in a Report with the Device Farm Console (p. 35).
The Screenshots tab displays a list of any screenshots Device Farm captured during the test.
The Performance tab displays information about any performance data Device Farm generated
during the test. To analyze this performance data, follow the instructions in Analyze Performance
Data for a Problem, Device, Suite, or Test in a Report with the Device Farm Console (p. 35).

API Version 2015-06-23

34

AWS Device Farm Developer Guide

Analyze Performance Data for a Problem, Device, Suite,
or Test in a Report with the Device Farm Console

8.

The Files tab displays a list of any of the test's associated files (such as log files) that can be
downloaded. To download a file, choose the file's link in the list.

Analyze Performance Data for a Problem, Device,

Suite, or Test in a Report with the Device Farm
Console
1.

If the Performance tab is not already displayed, follow these instructions:

Analyze a Report's Unique Problems with the Device Farm Console (p. 32)
Analyze a Report by Device with the Device Farm Console (p. 33)
Analyze a Report by Suite with the Device Farm Console (p. 33)
Analyze a Report by Test with the Device Farm Console (p. 34)

2.

The following information is displayed:

The CPU graph displays the percentage of CPU the app used on a single core during the selected
problem, device, suite, or test (along the vertical axis) over time (along the horizontal axis).
The vertical axis is expressed in percentages from 0% to the maximum recorded percentage.
This percentage may exceed 100% if the app used more than one core. For example, if three
cores are at 60% usage, this percentage will be displayed as 180%.
The Memory graph displays the number of MB the app used during the selected problem, device,
suite, or test (along the vertical axis) over time (along the horizontal axis).
The vertical axis is expressed in MB from 0 MB to the maximum number of recorded MB.
The Threads graph displays the number of threads used during the selected problem, device,
suite, or test (along the vertical axis) over time (along the horizontal axis).
The vertical axis is expressed in number of threads from 0 threads to the maximum number of
recorded threads.
In all cases, the horizontal axis is represented, in seconds, from the start and end of the run for the
selected problem, device, suite, or test.

3.

To display information for a specific data point, pause in the desired graph at the desired second
along the horizontal axis.

Analyze Log Information for a Problem, Device,

Suite, or Test in a Report with the Device Farm
Console
1.

If the Logs section is not already displayed, follow these instructions:

Analyze a Report's Unique Problems with the Device Farm Console (p. 32).
Analyze a Report by Device with the Device Farm Console (p. 33)
Analyze a Report by Suite with the Device Farm Console (p. 33)
Analyze a Report by Test with the Device Farm Console (p. 34)

API Version 2015-06-23

35

AWS Device Farm Developer Guide

Analyze Log Information for a Problem, Device, Suite,
or Test in a Report with the Device Farm Console

2.

The following information is displayed:

Source represents the source of a log entry. Possible values include:
Harness represents a log entry Device Farm created. These log entries are typically created
during start and stop events.
Device represents a log entry the device created. For Android, these log entries are
logcat-compatible. For iOS, these log entries are syslog compatible.
Test represents a log entry that either a test or its test framework created.
Time represents the elapsed time between the first log entry and this log entry. The time is
expressed in MM:SS.SSS format, where M represents minutes and S represents seconds.
PID represents the process identifier (PID) that created the log entry. All log entries created by an
app on a device will have the same PID.
Level represents the logging level for the log entry. For example, Logger.debug("This is a
message!") would log a Level of Debug. Possible values include the following:
Alert
Critical
Debug
Emergency
Error
Errored
Failed
Info
Internal
Notice
Passed
Skipped
Stopped
Verbose
Warned
Warning
Tag represents arbitrary metadata for the log entry. For example, Android logcat can use this to
describe which part of the system created the log entry (for example, ActivityManager).
Message represents the message or data for the log entry. For example, Logger.debug("Hello,
World!") would log a Message of "Hello, World!".

3.

To display only a portion of the information, do one or more of the following:

To show all log entries that match a value for a specific column, type the value into the corresponding
column header box. For example, to show all log entries with a Source value of Harness, type
Harness in the Source column header box. Similarly, to show all log entries with a PID value of
969 and a Tag value of ActivityManager, type 969 in the PID column header box, and type
ActivityManager in the Tag column header box.
To show all log entries that contain zero or more unknown characters for a specific column, use
the wildcard character (*) to represent the unknown characters. For example, to show all log entries
with a Source value that contain an es (such as Harness and Test), type *es* in the Source
column header box. Similarly, to show all log entries that start with a Source value of H (such as
Harness) and have a Level value that contains an e (such as Passed), type H* in the PID column
header box, and type *e* in the Level column header box.
To show log entries that contain a choice between one or more known characters for a specific
column, surround the set of choices in parentheses (( )), and use the pipe character (|) to separate

API Version 2015-06-23

36

AWS Device Farm Developer Guide

Analyze Log Information for a Problem, Device, Suite,
or Test in a Report with the Device Farm Console

each choice. For example, to show log entries with a Message value that contains either started
or starting, type *start(ed|ing)* in the Message column header box. Similarly, to show all
log entries with a Log value of Info or Debug, type *(Info|Debug)* in the Log column header
box.
To remove all of the characters from a column header box, choose the X in that column header
box. Removing all of the characters from a column header box is the same as typing * in that
column header box.

4.

To download all of the log information for the device, including all of the suites and tests that were
run, choose Download logs.

Note
Even if you display only a portion of the information, if you choose Download logs, all log
information for the device will be downloaded.

API Version 2015-06-23

37

AWS Device Farm Developer Guide

Built-in Test Types

Working with Test Types in AWS

Device Farm
Device Farm provides support for several automation test types.

Built-in Test Types

Built-in tests enable you to test your apps without writing scripts.
Built-in: Explorer (Android) (p. 82)
Built-in: Fuzz (Android and iOS) (p. 83)

Custom Test Types

Custom tests allow you to test specific flows and business logic within your app.

Custom Android Test Types

Appium Java JUnit (p. 39)
Appium Java TestNG (p. 44)
Appium Python (p. 48)
Calabash (p. 51)
Instrumentation (p. 53)
UI Automator (p. 54)

Custom iOS Test Types

Appium Java JUnit (p. 56)
Appium Java TestNG (p. 60)
Appium Python (p. 65)

API Version 2015-06-23

38

AWS Device Farm Developer Guide

Custom Web Application Test Types

Calabash (p. 68)

UI Automation (p. 70)
XCTest (p. 71)
XCTest UI (p. 72)

Custom Web Application Test Types

Appium Java JUnit (p. 74)
Appium Java TestNG (p. 76)
Appium Python (p. 78)

Working with Android Tests in AWS Device Farm

Device Farm provides support for several automation test types.

Built-in Test Types for Android

There are two built-in test types available for Android devices.
Built-in: Explorer (Android) (p. 82)
Built-in: Fuzz (Android and iOS) (p. 83)

Custom Test Types for Android

The following custom tests are available for Android devices.

Appium Java JUnit (p. 39)

Appium Java TestNG (p. 44)
Appium Python (p. 48)
Calabash (p. 51)
Instrumentation (p. 53)
UI Automator (p. 54)

Working with Appium Java JUnit for Android and

AWS Device Farm
Device Farm provides support for Appium Java JUnit for Android.
Device Farm also provides a sample Android application along with links to working tests in three Android
automation frameworks, including Appium. You can download the Device Farm sample app for Android
on GitHub.
Topics
What Is Appium Java JUnit? (p. 40)
Version Information (p. 40)
Prepare Your Android Appium Java JUnit Tests (p. 40)

API Version 2015-06-23

39

AWS Device Farm Developer Guide

Appium Java JUnit

Upload Your Android Appium Java JUnit Tests (p. 43)

Taking Screenshots in Android Appium Java JUnit Tests (p. 43)
Additional Considerations for Android Appium Java JUnit Tests (p. 43)

What Is Appium Java JUnit?

Appium is an open-source tool for automating native, mobile web, and hybrid applications on platforms
such as Android. For more information, see About Appium.

Version Information
Currently, Device Farm supports Appium version 1.4.16 and Java version Java 8.

Prepare Your Android Appium Java JUnit Tests

Your Android Appium Java JUnit tests must be contained in a .zip file.

Build the Appium Java Test Package

The Appium Java test package you upload to Device Farm must be in .zip format and contain all of the
tests' dependencies. The following instructions will show you how to meet these requirements during the
package stage of a Maven build.
1.

Modify pom.xml to set packaging as a JAR file:

<groupId>com.acme</groupId>
<artifactId>acme-android-appium</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>

2.

Modify pom.xml to use maven-jar-plugin to build your tests into a JAR file.
The following plugin will build your test source code (anything in the src/test directory) into a JAR
file:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>

3.

Modify pom.xml to use maven-dependency-plugin to build dependencies as JAR files.

The following plugin will copy your dependencies into the dependency-jars directory:

API Version 2015-06-23

40

AWS Device Farm Developer Guide

Appium Java JUnit

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.10</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/dependency-jars/</out
putDirectory>
</configuration>
</execution>
</executions>
</plugin>

4.

Save the following XML assembly to src/main/assembly/zip.xml.

The following XML is an assembly definition that, when configured, instructs Maven to build a .zip
file containing everything in the root of your build output directory and the dependency-jars
directory:
<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/as
sembly/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plu
gin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
<id>zip</id>
<formats>
<format>zip</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>./</outputDirectory>
<includes>
<include>*.jar</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>./</outputDirectory>
<includes>
<include>/dependency-jars/</include>
</includes>
</fileSet>
</fileSets>
</assembly>

5.

Modify pom.xml to use maven-assembly-plugin to package tests and all dependencies into a
single .zip file.

API Version 2015-06-23

41

AWS Device Farm Developer Guide

Appium Java JUnit

The following plugin uses the preceding assembly to create a .zip file named
zip-with-dependencies in the build output directory every time mvn package is run:
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<finalName>zip-with-dependencies</finalName>
<appendAssemblyId>false</appendAssemblyId>
<descriptors>
<descriptor>src/main/assembly/zip.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>
</plugin>

6.

Build, package, and verify. For example:

$ mvn clean package -DskipTests=true
$ tree target
.
| acme-android-appium-1.0-SNAPSHOT.jar (this is the JAR containing everything
built from the ./src/main directory)
| acme-android-appium-1.0-SNAPSHOT-tests.jar (this is the JAR containing
everything built from the ./src/test directory)
| zip-with-dependencies.zip (this .zip file contains all of the items)
` dependency-jars (this is the directory that contains all of your depend
encies, built as JAR files)
| com.some-dependency.bar-4.1.jar
| com.another-dependency.thing-1.0.jar
| joda-time-2.7.jar
| log4j-1.2.14.jar
| (and so on...)

7.

Use the Device Farm console to upload the test package.

Tip
If you receive an error saying that annotation is not supported in 1.3, add the following to pom.xml:
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>

API Version 2015-06-23

42

AWS Device Farm Developer Guide

Appium Java JUnit

Upload Your Android Appium Java JUnit Tests

Use the Device Farm console to upload your tests:
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the option next to the project where you want to upload your tests.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project where you want to upload your tests.
To create a new project, follow the instructions in Create a Project (p. 20).
3.
4.
5.

If the Create a new run button is displayed, then choose it.

On the Choose your application page, choose Upload.
Browse to and choose your Android app file. The file must be an .apk file.

6.
7.

Choose Next step.

On the Configure a test page, choose Appium Java JUnit, and then choose Upload.

8.

Browse to and choose the .zip file that contains your tests. The .zip file must follow the format
described in Prepare Your Android Appium Java JUnit Tests (p. 40).
Choose Next step, and then complete the remaining on-screen instructions to select devices and
start the run.

9.

Taking Screenshots in Android Appium Java JUnit Tests

You can take screenshots as part of your Android Appium Java JUnit tests.
When Device Farm runs your Appium Java JUnit test, the service sets several Java system properties
that describe the configuration of the Appium server with which you're communicating. For example,
Device Farm sets the appium.screenshots.dir property to a fully qualified path on the local file system
where Device Farm expects Appium screenshots to be saved. The test-specific directory where the
screenshots are stored is defined at runtime. The screenshots are automatically pulled into your Device
Farm reports automatically.To view the screenshots, in the Device Farm console, choose the Screenshots
section.
The following example shows how to use and consume the appium.screenshots.dir property to
capture an Appium screenshot that is pulled into your Device Farm report.
public boolean takeScreenshot(final String name) {
String screenshotDirectory = System.getProperty("appium.screenshots.dir",
System.getProperty("java.io.tmpdir", ""));
File screenshot = ((TakesScreenshot) driver).getScreenshotAs(Output
Type.FILE);
return screenshot.renameTo(new File(screenshotDirectory,
String.format("%s.png", name)));
}

Additional Considerations for Android Appium Java JUnit

Tests
Device Farm does not modify Android Appium Java JUnit tests.

API Version 2015-06-23

43

AWS Device Farm Developer Guide

Appium Java TestNG

Working with Appium Java TestNG for Android

and AWS Device Farm
Device Farm provides support for Appium Java TestNG for Android.
Device Farm also provides a sample Android application along with links to working tests in three Android
automation frameworks, including Appium. You can download the Device Farm sample app for Android
on GitHub.
Topics
What Is Appium Java TestNG? (p. 44)
Version Information (p. 44)
Prepare Your Android Appium Java TestNG Tests (p. 44)
Upload Your Android Appium Java TestNG Tests (p. 47)
Taking Screenshots in Android Appium Java TestNG Tests (p. 47)
Additional Considerations for Android Appium Java TestNG Tests (p. 48)

What Is Appium Java TestNG?

Appium is an open-source tool for automating native, mobile web, and hybrid applications on platforms
such as Android. For more information, see About Appium.

Version Information
Currently, Device Farm supports Appium version 1.4.16 and Java version Java 8.

Prepare Your Android Appium Java TestNG Tests

Your Android Appium Java TestNG tests must be contained in a .zip file before you upload them to Device
Farm.

Build the Appium Java Test Package

The Appium Java test package you upload to Device Farm must be in .zip format and contain all of the
tests' dependencies. The following instructions will show you how to meet these requirements during the
package stage of a Maven build.
1.

Modify pom.xml to set packaging as a JAR file:

<groupId>com.acme</groupId>
<artifactId>acme-android-appium</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>

2.

Modify pom.xml to use maven-jar-plugin to build your tests into a JAR file.
The following plugin will build your test source code (anything in the src/test directory) into a JAR
file:
<plugin>
<groupId>org.apache.maven.plugins</groupId>

API Version 2015-06-23

44

AWS Device Farm Developer Guide

Appium Java TestNG

<artifactId>maven-jar-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>

3.

Modify pom.xml to use maven-dependency-plugin to build dependencies as JAR files.

The following plugin will copy your dependencies into the dependency-jars directory:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.10</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/dependency-jars/</out
putDirectory>
</configuration>
</execution>
</executions>
</plugin>

4.

Save the following XML assembly to src/main/assembly/zip.xml.

The following XML is an assembly definition that, when configured, instructs Maven to build a .zip
file containing everything in the root of your build output directory and the dependency-jars
directory:
<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/as
sembly/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plu
gin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
<id>zip</id>
<formats>
<format>zip</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>./</outputDirectory>
<includes>

API Version 2015-06-23

45

AWS Device Farm Developer Guide

Appium Java TestNG

<include>*.jar</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>./</outputDirectory>
<includes>
<include>/dependency-jars/</include>
</includes>
</fileSet>
</fileSets>
</assembly>

5.

Modify pom.xml to use maven-assembly-plugin to package tests and all dependencies into a
single .zip file.
The following plugin uses the preceding assembly to create a .zip file named
zip-with-dependencies in the build output directory every time mvn package is run:
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<finalName>zip-with-dependencies</finalName>
<appendAssemblyId>false</appendAssemblyId>
<descriptors>
<descriptor>src/main/assembly/zip.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>
</plugin>

6.

Build, package, and verify. For example:

$ mvn clean package -DskipTests=true
$ tree target
.
| acme-android-appium-1.0-SNAPSHOT.jar (this is the JAR containing everything
built from the ./src/main directory)
| acme-android-appium-1.0-SNAPSHOT-tests.jar (this is the JAR containing
everything built from the ./src/test directory)
| zip-with-dependencies.zip (this .zip file contains all of the items)
` dependency-jars (this is the directory that contains all of your depend
encies, built as JAR files)
| com.some-dependency.bar-4.1.jar
| com.another-dependency.thing-1.0.jar
| joda-time-2.7.jar

API Version 2015-06-23

46

AWS Device Farm Developer Guide

Appium Java TestNG

| log4j-1.2.14.jar
| (and so on...)

7.

Use the Device Farm console to upload the test package.

Tip
If you receive an error saying that annotation is not supported in 1.3, add the following to pom.xml:
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>

Upload Your Android Appium Java TestNG Tests

Use the Device Farm console to upload your tests:
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the option next to the project where you want to upload your tests.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project where you want to upload your tests.
To create a new project, follow the instructions in Create a Project (p. 20).
3.
4.
5.
6.
7.
8.

If the Create a new run button is displayed, then choose it.

On the Choose your application page, choose Upload.
Browse to and choose your Android app file. The file must be an .apk file.
Choose Next step.
On the Configure a test page, choose Appium Java TestNG, and then choose Upload.
Browse to and choose the .zip file that contains your tests. The .zip file must follow the format
described in Prepare Your Android Appium Java TestNG Tests (p. 44).

9.

Choose Next step, and then complete the remaining on-screen instructions to select devices and
start the run.

Taking Screenshots in Android Appium Java TestNG Tests

You can take screenshots as part of your Android Appium Java TestNG tests.
When Device Farm runs your Appium Java TestNG test, the service sets several Java system properties
that describe the configuration of the Appium server with which you're communicating. For example,
Device Farm sets the appium.screenshots.dir property to a fully qualified path on the local file system
where Device Farm expects Appium screenshots to be saved. The test-specific directory where the
screenshots are stored is defined at runtime. The screenshots are automatically pulled into your Device
Farm reports automatically.To view the screenshots, in the Device Farm console, choose the Screenshots
section.
The following example shows how to use and consume the appium.screenshots.dir property to
capture an Appium screenshot that is pulled into your Device Farm report.

API Version 2015-06-23

47

AWS Device Farm Developer Guide

Appium Python

public boolean takeScreenshot(final String name) {

String screenshotDirectory = System.getProperty("appium.screenshots.dir",
System.getProperty("java.io.tmpdir", ""));
File screenshot = ((TakesScreenshot) driver).getScreenshotAs(Output
Type.FILE);
return screenshot.renameTo(new File(screenshotDirectory,
String.format("%s.png", name)));
}

Additional Considerations for Android Appium Java TestNG

Tests
Device Farm does not modify Android Appium Java TestNG tests.

Working with Appium Python for Android

Applications and AWS Device Farm
Device Farm provides support for Appium Python for Android apps.
Topics
What Is Appium Python? (p. 48)
Version Information (p. 48)
Prepare Your Android Application Appium Python Tests (p. 48)
Build the Appium Python Test Package (p. 48)
Upload Your Android Application Appium Python Tests (p. 50)
Taking Screenshots in Android Appium Python Tests (p. 51)
Additional Considerations for Android Appium Python Tests (p. 51)

What Is Appium Python?

Appium is an open-source tool for automating native, mobile web, and hybrid applications on platforms
like Android. For more information, see About Appium.

Version Information
Currently, Device Farm supports Appium version 1.4.16 and Python version 2.7.6 (pip version 1.5.4).

Prepare Your Android Application Appium Python Tests

The Appium Python tests for your Android application must be contained in a .zip file.

Build the Appium Python Test Package

The Appium Python test packages you upload to Device Farm must be in .zip format and contain all the
dependencies of your test. The following instructions show you how to meet these requirements.

Note
The instructions below are based on Linux x86_64 and Mac. In the currently supported scheme,
Device Farm requires that the packaging of your Appium Python tests be done on Linux x86_64
if your tests contain non-universal Python wheels dependencies. For the platform on which you
execute a command, the wheels tools gather your .whl dependent files under the wheelhouse/
API Version 2015-06-23
48

AWS Device Farm Developer Guide

Appium Python

folder. When you execute the Python wheel command on any platform other than Linux x86_64,
you would gather the flavor of a non-univesral wheel dependency for that particular platform and
may cause undesired effects. This would most likely lead to errors when executing your tests
on Device Farm.
1.

We strongly recommend that you set up Python virtualenv for developing and packaging tests so
that unnecessary dependencies are not included in your app package.

Tip
Do not create a Python virtualenv with the --system-site-packages option, because
it will inherit packages from /usr/lib/pythonx.x/site-packages or wherever your global
site-packages directory is. This can lead to you including dependencies in your virtual
environment that are not needed by your tests.
You should also verify that your tests do not use dependencies that are dependent on
native libraries, as these native libraries may or may not be present on the instance where
these tests run.

2.

Install py.test in your virtual environment.

An example flow of creating a virtual environment using Python virtualenv and installing pytest in
that virtual environment would look like the following:
$
$
$
$

3.

virtualenv workspace
cd workspace
source bin/activate
pip install pytest

Store all Python test scripts under the tests/ folder in your work space.
workspace
tests/ (your tests go here)

4.

Make sure you have py.test installed in your virtual environment and test cases are discoverable by
the following command, which you should run from your virtual environment workspace folder.
$ py.test --collect-only tests/

5.

Make sure the output of py.test command shows you the tests that you want to execute on Device
Farm.
Go to your work space and run the following command to generate the requirements.txt file:
$ pip freeze > requirements.txt

6.

Go to your work space and run the following command to generate the wheelhouse/ folder:
$ pip wheel --wheel-dir wheelhouse -r requirements.txt

7.

You can use the following commands to clean all cached files under your tests/ folder:

API Version 2015-06-23

49

AWS Device Farm Developer Guide

Appium Python

$
$
$
$

8.

find
find
find
find

.
.
.
.

-name
-name
-name
-name

'__pycache__'
'*.pyc' -exec
'*.pyo' -exec
'*~' -exec rm

-type
rm -f
rm -f
-f {}

d -exec rm -r {} +
{} +
{} +
+

Zip the tests/ folder, wheelhouse/ folder, and the requirements.txt file into a single archive:
$ zip -r test_bundle.zip tests/ wheelhouse/ requirements.txt

Your work space will eventually look like this:

workspace
tests/
test_bundle.zip
requirements.txt
wheelhouse/

Upload Your Android Application Appium Python Tests

Use the Device Farm console to upload your tests.
1.
2.
3.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

If you see the AWS Device Farm console home page, choose Get started.
If you already have a project, you can upload your tests to an existing project or choose Create a
new project.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project where you want to upload your tests.
To create a project, follow the instructions in Create a Project (p. 20).
4.
5.

If the Create a new run button is displayed, choose it.

On the Choose your application page, choose Native Application (the Android and Apple logos).

API Version 2015-06-23

50

AWS Device Farm Developer Guide

Calabash

6.

Next, choose Upload to upload your .apk file.

7.

Device Farm processes your .apk file before continuing.

In the Run name field, type a name for your run.

Tip
Give the run a name that will help you identify a specific build of your app (for example,
Beta-0.1). For more information, see Working with Runs (p. 23).
8. Choose Appium Python to configure your test,
9. To add your Appium test scripts to the test run, choose Upload.
10. Choose Next step, and then complete the instructions to select devices and start the run.

Taking Screenshots in Android Appium Python Tests

You can take screenshots as part of your Android Appium Python tests.
When Device Farm runs your Appium Python test, the service sets several system properties that describe
the configuration of the Appium server with which you're communicating. For example, Device Farm sets
the SCREENSHOT_PATH property to a fully qualified path on the local file system where Device Farm
expects Appium screenshots to be saved. The test-specific directory where the screenshots are stored
is defined at runtime. The screenshots are pulled into your Device Farm reports automatically. To view
the screenshots, in the Device Farm console, choose the Screenshots section.
The following example shows how to use and consume the SCREENSHOT_PATH property to capture an
Appium screenshot that is pulled into your Device Farm report.
screenshot_folder = os.getenv('SCREENSHOT_PATH', '')
self.driver.save_screenshot(screenshot_folder + "/screenshot.png")

Additional Considerations for Android Appium Python Tests

Device Farm does not modify Android Appium Python tests.

Working with Calabash for Android and AWS

Device Farm
Device Farm provides support for Calabash for Android.
Device Farm also provides a sample Android application along with links to working tests in three Android
automation frameworks, including Calabash.You can download the Device Farm sample app for Android
on GitHub.

API Version 2015-06-23

51

AWS Device Farm Developer Guide

Calabash

What Is Calabash?
Calabash is a mobile testing framework that enables automated user interface acceptance tests written
in Cucumber to be run on Android apps. For more information, see the Welcome to Calabash for Android
repository on GitHub.

Note
AWS Device Farm currently works with Calabash version 0.5.8.

Version Information
Currently, Device Farm supports Calabash version 0.5.8.

Prepare Your Android Calabash Tests

Your Android Calabash tests must be contained in a .zip file before you upload them to Device Farm.
This .zip file must contain the following structure:
my-zip-file-name.zip
`-- features (directory)
|-- my-feature-1-file-name.feature
|-- my-feature-2-file-name.feature
|-- my-feature-N-file-name.feature
|-- step_definitions (directory)
|
`-- (.rb files)
|-- support (directory)
|
`-- (.rb files)
`-- (any other supporting files)

Upload Your Android Calabash Tests

Use the Device Farm console to upload your tests:
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the option next to the project where you want to upload your tests.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project where you want to upload your tests.
To create a new project, follow the instructions in Create a Project (p. 20).
3.

If the Create a new run button is displayed, then choose it.

4.
5.
6.

On the Choose your application page, choose Upload.

Browse to and choose your Android app file. The file must be an .apk file.
Choose Next step.

7.
8.

On the Configure a test page, choose Calabash, and then choose Upload.
Browse to and choose the .zip file that contains your tests. The .zip file must follow the format
described in Prepare Your Android Calabash Tests (p. 52).
Choose Next step, and then complete the remaining on-screen instructions to select devices and
start the run.

9.

Taking Screenshots in Android Calabash Tests

You can take screenshots as part of your Android Calabash tests.

API Version 2015-06-23

52

AWS Device Farm Developer Guide

Instrumentation

Android Calabash provides a set of predefined steps for taking screenshots. For details, see the
"Screenshots" section of the Canned steps page in the Calabash Android repository on GitHub.
Alternatively, you can define a custom step inside of a Ruby (.rb) file to call the screenshot_embed
function, which creates a screenshot and saves it to a directory you define. For example, the following
code example creates a screenshot and saves it to the /my/custom/path directory with a file name of
screenshot_seconds-since-Epoch:
screenshot_embed(:prefix => "/my/custom/path", :name => "screen
shot_#{Time.now.to_i}")

Additional Considerations for Android Calabash Tests

Device Farm replaces some Calabash hooks so that Android Calabash tests will run on devices in Device
Farm, but it does not modify Android Calabash tests.

Working with Instrumentation for Android and

AWS Device Farm
Device Farm provides support for Instrumentation (JUnit, Espresso, Robotium, or any
Instrumentation-based tests) for Android.
Device Farm also provides a sample Android application along with links to working tests in three Android
automation frameworks, including Instrumentation (Espresso). You can download the Device Farm
sample app for Android on GitHub.
Topics
What Is Instrumentation? (p. 53)
Upload Your Android Instrumentation Tests (p. 53)
Taking Screenshots in Android Instrumentation Tests (p. 54)
Additional Considerations for Android Instrumentation Tests (p. 54)

What Is Instrumentation?
Android instrumentation enables you to invoke callback methods in your test code. This allows you to run
through the lifecycle of a component step by step, as if you were debugging the component. For more
information, see Instrumentation in the Testing Fundamentals section of the Android Developer Tools
documentation.

Upload Your Android Instrumentation Tests

Use the Device Farm console to upload your tests:
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the option next to the project where you want to upload your tests.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project where you want to upload your tests.
To create a new project, follow the instructions in Create a Project (p. 20).
3.
4.

If the Create a new run button is displayed, then choose it.

On the Choose your application page, choose Upload.

API Version 2015-06-23

53

AWS Device Farm Developer Guide

UI Automator

5.
6.

Browse to and choose your Android app file. The file must be an .apk file.
Choose Next step.

7.
8.
9.

On the Configure a test page, choose Instrumentation, and then choose Upload.
Browse to and choose the .apk file that contains your tests.
Choose Next step, and then complete the remaining on-screen instructions to select devices and
start the run.

Taking Screenshots in Android Instrumentation Tests

You can take screenshots as part of your Android Instrumentation tests.
To take screenshots, call one of the following methods:
For Robotium, call the takeScreenShot method (for example, solo.takeScreenShot();).
For Spoon, call the screenshot method, for example:
Spoon.screenshot(activity, "initial_state");
/* Normal test code... */
Spoon.screenshot(activity, "after_login");

During a test run, Device Farm automatically gets screenshots from the following locations on the devices,
if they exist, and then adds them to the test reports:
/sdcard/robotium-screenshots
/sdcard/test-screenshots
/sdcard/Download/spoon-screenshots/test-class-name/test-method-name
/data/data/application-package-name/app_spoon-screenshots/test-class-name/test-method-name

Additional Considerations for Android Instrumentation Tests

Device Farm supports frameworks, such as Robotium, that have record-and-playback scripting tools.

Working with UI Automator for Android and AWS

Device Farm
Device Farm provides support for UI Automator for Android.

Note
This framework is currently in preview and may not work with all scripts and apps.
Topics
What Is UI Automator? (p. 55)

Prepare Your Android UI Automator Tests (p. 55)

Upload Your Android UI Automator Tests (p. 55)
Taking Screenshots in Android UI Automator Tests (p. 55)
Additional Considerations for Android UI Automator Tests (p. 55)

API Version 2015-06-23

54

AWS Device Farm Developer Guide

UI Automator

What Is UI Automator?
The UI Automator testing framework provides a set of APIs to build user interface tests that perform
interactions on user and system apps for Android. The UI Automator APIs allow you to perform operations
such as opening the Settings menu or the app launcher in a test device. For more information, see UI
Automator in the Testing Support Library section of the Android Developer Tools documentation.

Prepare Your Android UI Automator Tests

The Android UI Automator tests must be contained in a single JAR file before you upload them to Device
Farm. The package name in this file must match the package name used by the related Android app. For
example, if the Android app's package name is com.my.android.app.MyMobileApp, then the Android
UI Automator tests must be in a package named com.my.android.app.

Upload Your Android UI Automator Tests

Use the Device Farm console to upload your tests:
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the option next to the project where you want to upload your tests.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project where you want to upload your tests.
To create a new project, follow the instructions in Create a Project (p. 20).
3.
4.
5.
6.
7.
8.
9.

If the Create a new run button is displayed, then choose it.

On the Choose your application page, choose Upload.
Browse to and choose your Android app file. The file must be an .apk file.
Choose Next step.
On the Configure a test page, choose uiautomator, and then choose Upload.
Browse to and choose the JAR file that contains your tests. Make sure the Android tests are organized
according to the instructions in Prepare Your Android UI Automator Tests (p. 55).
Choose Next step, and then complete the remaining on-screen instructions to select devices and
start the run.

Taking Screenshots in Android UI Automator Tests

You can take screenshots as part of your Android UI Automator tests.
To take a screenshot, call the takeScreenshot method (for example,
takeScreenshot("/sdcard/uiautomator-screenshots/home-screen-1234.png");).

Note
All screenshots must be stored in the /sdcard/uiautomator-screenshots directory. You
must specify the full path (including the file name) of the screenshot to be stored.
The takeScreenshot method works for API Levels 17 and higher only. For API Level 16, UI
Automator is supported, but screenshots are not supported.

Additional Considerations for Android UI Automator Tests

Device Farm re-signs Android UI Automator test packages, but it does not modify Android UI Automator
tests.

API Version 2015-06-23

55

AWS Device Farm Developer Guide

iOS Tests

Working with iOS Tests in AWS Device Farm

Device Farm provides support for several automation test types for iOS devices.

Built-in Test Types for iOS

There is currently one built-in test type available for iOS devices.
Built-in: Fuzz (Android and iOS) (p. 83)

Custom Test Types

The following custom tests are available for iOS devices.
Appium Java JUnit (p. 56)

Appium Java TestNG (p. 60)

Appium Python (p. 65)
Calabash (p. 68)
UI Automation (p. 70)
XCTest (p. 71)
XCTest UI (p. 72)

Working with Appium Java JUnit for iOS and AWS

Device Farm
Device Farm provides support for Appium Java JUnit for iOS. The following information describes how
to use this test framework with Device Farm test types.
Topics
What is Appium Java JUnit? (p. 56)
Version Information (p. 56)
Prepare Your iOS Appium Java JUnit Tests (p. 57)
Upload Your iOS Appium Java JUnit Tests (p. 59)
Taking Screenshots in iOS Appium Java JUnit Tests (p. 60)
Additional Considerations for iOS Appium Java JUnit Tests (p. 60)

What is Appium Java JUnit?

Appium is an open-source tool for automating native, mobile web, and hybrid applications on platforms
such as iOS. For more information, see About Appium.

Version Information
Currently, Device Farm supports Appium version 1.4.16 (1.3.6 for iOS 6-7) and Java version Java 8.

API Version 2015-06-23

56

AWS Device Farm Developer Guide

Appium Java JUnit

Prepare Your iOS Appium Java JUnit Tests

Before you upload your iOS Appium Java JUnit tests to Device Farm for testing, make sure that your iOS
Appium Java JUnit tests are contained within an .ipa file.

Build the Appium Java Test Package

The Appium Java test package you upload to Device Farm must be in .zip format and contain all of the
tests' dependencies. The following instructions will show you how to meet these requirements during the
package stage of a Maven build.
1.

Modify pom.xml to set packaging as a JAR file:

<groupId>com.acme</groupId>
<artifactId>acme-android-appium</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>

2.

Modify pom.xml to use maven-jar-plugin to build your tests into a JAR file.
The following plugin will build your test source code (anything in the src/test directory) into a JAR
file:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>

3.

Modify pom.xml to use maven-dependency-plugin to build dependencies as JAR files.

The following plugin will copy your dependencies into the dependency-jars directory:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.10</version>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/dependency-jars/</out
putDirectory>
</configuration>

API Version 2015-06-23

57

AWS Device Farm Developer Guide

Appium Java JUnit

</execution>
</executions>
</plugin>

4.

Save the following XML assembly to src/main/assembly/zip.xml.

The following XML is an assembly definition that, when configured, instructs Maven to build a .zip
file containing everything in the root of your build output directory and the dependency-jars
directory:
<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/as
sembly/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plu
gin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
<id>zip</id>
<formats>
<format>zip</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>./</outputDirectory>
<includes>
<include>*.jar</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>./</outputDirectory>
<includes>
<include>/dependency-jars/</include>
</includes>
</fileSet>
</fileSets>
</assembly>

5.

Modify pom.xml to use maven-assembly-plugin to package tests and all dependencies into a
single .zip file.
The following plugin uses the preceding assembly to create a .zip file named
zip-with-dependencies in the build output directory every time mvn package is run:
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>

API Version 2015-06-23

58

AWS Device Farm Developer Guide

Appium Java JUnit

<finalName>zip-with-dependencies</finalName>
<appendAssemblyId>false</appendAssemblyId>
<descriptors>
<descriptor>src/main/assembly/zip.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>
</plugin>

6.

Build, package, and verify. For example:

$ mvn clean package -DskipTests=true
$ tree target
.
| acme-android-appium-1.0-SNAPSHOT.jar (this is the JAR containing everything
built from the ./src/main directory)
| acme-android-appium-1.0-SNAPSHOT-tests.jar (this is the JAR containing
everything built from the ./src/test directory)
| zip-with-dependencies.zip (this .zip file contains all of the items)
` dependency-jars (this is the directory that contains all of your depend
encies, built as JAR files)
| com.some-dependency.bar-4.1.jar
| com.another-dependency.thing-1.0.jar
| joda-time-2.7.jar
| log4j-1.2.14.jar
| (and so on...)

7.

Use the Device Farm console to upload the test package.

Tip
If you receive an error saying that annotation is not supported in 1.3, add the following to pom.xml:
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>

Upload Your iOS Appium Java JUnit Tests

To run your iOS Appium Java JUnit tests on a set of iOS devices in Device Farm, you upload your tests
with the Device Farm console as follows:
1.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

2.

In the list of projects, choose the option next to the project that you want to upload your tests to.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project that you want to upload your tests to.
To create a new project, follow the instructions in Create a Project (p. 20).
3.

If the Create a new run button is displayed, then choose it.

API Version 2015-06-23
59

AWS Device Farm Developer Guide

Appium Java TestNG

4.
5.

On the Choose your application page, choose Upload.

Browse to and choose your iOS app file. The file must be an .ipa file.

Note
Make sure that your app file is built for an iOS device and not for a simulator.
6.
7.
8.

Choose Next step.

On the Configure a test page, choose Appium Java JUnit, and then choose Upload.
Choose Next step, and then complete the remaining on-screen instructions to select the devices to
run your tests on and to then start the run.

Taking Screenshots in iOS Appium Java JUnit Tests

You can take screenshots as part of your iOS Appium Java JUnit tests.
When Device Farm runs your Appium Java JUnit test, the service sets several Java system properties
that describe the configuration of the Appium server with which you're communicating. For example,
Device Farm sets the appium.screenshots.dir property to a fully qualified path on the local file system
where Device Farm expects Appium screenshots to be saved. The test-specific directory where the
screenshots are stored is defined at runtime. The screenshots are automatically pulled into your Device
Farm reports automatically.To view the screenshots, in the Device Farm console, choose the Screenshots
section.
The following example shows how to use and consume the appium.screenshots.dir property to
capture an Appium screenshot that is pulled into your Device Farm report.
public boolean takeScreenshot(final String name) {
String screenshotDirectory = System.getProperty("appium.screen
shots.dir", System.getProperty("java.io.tmpdir", ""));
File screenshot = ((TakesScreenshot) driver).getScreenshotAs(Out
putType.FILE);
return screenshot.renameTo(new File(screenshotDirectory,
String.format("%s.png", name)));
}

Additional Considerations for iOS Appium Java JUnit Tests

Device Farm does not modify iOS Appium Java JUnit tests.

Working with Appium Java TestNG for iOS and

AWS Device Farm
Device Farm provides support for Appium Java TestNG for iOS. The following information describes how
to use this test framework with Device Farm test types.
Topics
What is Appium Java TestNG? (p. 61)
Version Information (p. 61)
Prepare Your iOS Appium Java TestNG Tests (p. 61)
Upload Your iOS Appium Java TestNG Tests (p. 64)
Taking Screenshots in iOS Appium Java TestNG Tests (p. 64)
Additional Considerations for iOS Appium Java TestNG Tests (p. 64)

API Version 2015-06-23

60

AWS Device Farm Developer Guide

Appium Java TestNG

What is Appium Java TestNG?

Appium is an open-source tool for automating native, mobile web, and hybrid applications on platforms
such as iOS. For more information, see About Appium.

Version Information
Currently, Device Farm supports Appium version 1.4.16 (1.3.6 for iOS 6-7) and Java version Java 8.

Prepare Your iOS Appium Java TestNG Tests

Before you upload your iOS Appium Java TestNG tests to Device Farm for testing, make sure that your
iOS Appium Java TestNG tests are contained within an .ipa file.

Build the Appium Java Test Package

The Appium Java test package you upload to Device Farm must be in .zip format and contain all of the
tests' dependencies. The following instructions will show you how to meet these requirements during the
package stage of a Maven build.
1.

Modify pom.xml to set packaging as a JAR file:

<groupId>com.acme</groupId>
<artifactId>acme-android-appium</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>

2.

Modify pom.xml to use maven-jar-plugin to build your tests into a JAR file.
The following plugin will build your test source code (anything in the src/test directory) into a JAR
file:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>

3.

Modify pom.xml to use maven-dependency-plugin to build dependencies as JAR files.

The following plugin will copy your dependencies into the dependency-jars directory:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.10</version>
<executions>

API Version 2015-06-23

61

AWS Device Farm Developer Guide

Appium Java TestNG

<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/dependency-jars/</out
putDirectory>
</configuration>
</execution>
</executions>
</plugin>

4.

Save the following XML assembly to src/main/assembly/zip.xml.

The following XML is an assembly definition that, when configured, instructs Maven to build a .zip
file containing everything in the root of your build output directory and the dependency-jars
directory:
<assembly
xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/as
sembly/1.1.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plu
gin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
<id>zip</id>
<formats>
<format>zip</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<fileSets>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>./</outputDirectory>
<includes>
<include>*.jar</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.build.directory}</directory>
<outputDirectory>./</outputDirectory>
<includes>
<include>/dependency-jars/</include>
</includes>
</fileSet>
</fileSets>
</assembly>

5.

Modify pom.xml to use maven-assembly-plugin to package tests and all dependencies into a
single .zip file.
The following plugin uses the preceding assembly to create a .zip file named
zip-with-dependencies in the build output directory every time mvn package is run:

API Version 2015-06-23

62

AWS Device Farm Developer Guide

Appium Java TestNG

<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<finalName>zip-with-dependencies</finalName>
<appendAssemblyId>false</appendAssemblyId>
<descriptors>
<descriptor>src/main/assembly/zip.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>
</plugin>

6.

Build, package, and verify. For example:

$ mvn clean package -DskipTests=true
$ tree target
.
| acme-android-appium-1.0-SNAPSHOT.jar (this is the JAR containing everything
built from the ./src/main directory)
| acme-android-appium-1.0-SNAPSHOT-tests.jar (this is the JAR containing
everything built from the ./src/test directory)
| zip-with-dependencies.zip (this .zip file contains all of the items)
` dependency-jars (this is the directory that contains all of your depend
encies, built as JAR files)
| com.some-dependency.bar-4.1.jar
| com.another-dependency.thing-1.0.jar
| joda-time-2.7.jar
| log4j-1.2.14.jar
| (and so on...)

7.

Use the Device Farm console to upload the test package.

Tip
If you receive an error saying that annotation is not supported in 1.3, add the following to pom.xml:
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>

API Version 2015-06-23

63

AWS Device Farm Developer Guide

Appium Java TestNG

Upload Your iOS Appium Java TestNG Tests

To run your iOS Appium Java TestNG tests on a set of iOS devices in Device Farm, you upload your
tests with the Device Farm console as follows:
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the option next to the project that you want to upload your tests to.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project that you want to upload your tests to.
To create a new project, follow the instructions in Create a Project (p. 20).
3.
4.
5.

If the Create a new run button is displayed, then choose it.

On the Choose your application page, choose Upload.
Browse to and choose your iOS app file. The file must be an .ipa file.

Note
Make sure that your app file is built for an iOS device and not for a simulator.
6.
7.
8.

Choose Next step.

On the Configure a test page, choose Appium Java TestNG, and then choose Upload.
Choose Next step, and then complete the remaining on-screen instructions to select the devices to
run your tests on and to then start the run.

Taking Screenshots in iOS Appium Java TestNG Tests

You can take screenshots as part of your iOS Appium Java TestNG tests.
When Device Farm runs your Appium Java TestNG test, the service sets several Java system properties
that describe the configuration of the Appium server with which you're communicating. For example,
Device Farm sets the appium.screenshots.dir property to a fully qualified path on the local file system
where Device Farm expects Appium screenshots to be saved. The test-specific directory where the
screenshots are stored is defined at runtime. The screenshots are automatically pulled into your Device
Farm reports automatically.To view the screenshots, in the Device Farm console, choose the Screenshots
section.
The following example shows how to use and consume the appium.screenshots.dir property to
capture an Appium screenshot that is pulled into your Device Farm report.
public boolean takeScreenshot(final String name) {
String screenshotDirectory = System.getProperty("appium.screenshots.dir",
System.getProperty("java.io.tmpdir", ""));
File screenshot = ((TakesScreenshot) driver).getScreenshotAs(Output
Type.FILE);
return screenshot.renameTo(new File(screenshotDirectory,
String.format("%s.png", name)));
}

Additional Considerations for iOS Appium Java TestNG Tests

Device Farm does not modify iOS Appium Java TestNG tests.

API Version 2015-06-23

64

AWS Device Farm Developer Guide

Appium Python

Working with Appium Python for iOS Applications

and AWS Device Farm
Device Farm provides support for Appium Python for iOS apps.
Topics
What Is Appium Python? (p. 65)
Version Information (p. 65)
Prepare Your iOS Application Appium Python Tests (p. 65)
Build the Appium Python Test Package (p. 65)
Upload Your iOS Application Appium Python Tests (p. 67)
Taking Screenshots in iOS Appium Python Tests (p. 68)
Additional Considerations for Android Appium Python Tests (p. 68)

What Is Appium Python?

Appium is an open-source tool for automating native, mobile web, and hybrid applications on platforms
like web applications. For more information, see About Appium.

Version Information
Currently, Device Farm supports Appium version 1.4.16 (1.3.6 for iOS 6-7) and Python version 2.7.6 (pip
version 1.5.4).

Prepare Your iOS Application Appium Python Tests

The Appium Python tests for your iOS application must be contained in a .zip file.

Build the Appium Python Test Package

The Appium Python test packages you upload to Device Farm must be in .zip format and contain all the
dependencies of your test. The following instructions show you how to meet these requirements.

Note
The instructions below are based on Linux x86_64 and Mac. In the currently supported scheme,
Device Farm requires that the packaging of your Appium Python tests be done on Linux x86_64
if your tests contain non-universal Python wheels dependencies. For the platform on which you
execute a command, the wheels tools gather your .whl dependent files under the wheelhouse/
folder. When you execute the Python wheel command on any platform other than Linux x86_64,
you would gather the flavor of a non-univesral wheel dependency for that particular platform and
may cause undesired effects. This would most likely lead to errors when executing your tests
on Device Farm.
1.

We strongly recommend that you set up Python virtualenv for developing and packaging tests so
that unnecessary dependencies are not included in your app package.

Tip
Do not create a Python virtualenv with the --system-site-packages option, because
it will inherit packages from /usr/lib/pythonx.x/site-packages or wherever your global
site-packages directory is. This can lead to you including dependencies in your virtual
environment that are not needed by your tests.

API Version 2015-06-23

65

AWS Device Farm Developer Guide

Appium Python

You should also verify that your tests do not use dependencies that are dependent on
native libraries, as these native libraries may or may not be present on the instance where
these tests run.

2.

Install py.test in your virtual environment.

An example flow of creating a virtual environment using Python virtualenv and installing pytest in
that virtual environment would look like the following:
$
$
$
$

3.

virtualenv workspace
cd workspace
source bin/activate
pip install pytest

Store all Python test scripts under the tests/ folder in your work space.
workspace
tests/ (your tests go here)

4.

Make sure you have py.test installed in your virtual environment and test cases are discoverable by
the following command, which you should run from your virtual environment workspace folder.
$ py.test --collect-only tests/

5.

Make sure the output of py.test command shows you the tests that you want to execute on Device
Farm.
Go to your work space and run the following command to generate the requirements.txt file:
$ pip freeze > requirements.txt

6.

Go to your work space and run the following command to generate the wheelhouse/ folder:
$ pip wheel --wheel-dir wheelhouse -r requirements.txt

7.

You can use the following commands to clean all cached files under your tests/ folder:
$
$
$
$

8.

find
find
find
find

.
.
.
.

-name
-name
-name
-name

'__pycache__'
'*.pyc' -exec
'*.pyo' -exec
'*~' -exec rm

-type
rm -f
rm -f
-f {}

d -exec rm -r {} +
{} +
{} +
+

Zip the tests/ folder, wheelhouse/ folder, and the requirements.txt file into a single archive:
$ zip -r test_bundle.zip tests/ wheelhouse/ requirements.txt

Your work space will eventually look like this:

workspace
tests/

API Version 2015-06-23

66

AWS Device Farm Developer Guide

Appium Python

test_bundle.zip
requirements.txt
wheelhouse/

Upload Your iOS Application Appium Python Tests

Use the Device Farm console to upload your tests.
1.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

2.
3.

If you see the AWS Device Farm console home page, choose Get started.
If you already have a project, you can upload your tests to an existing project or choose Create a
new project.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project where you want to upload your tests.
To create a project, follow the instructions in Create a Project (p. 20).
4.
5.

If the Create a new run button is displayed, choose it.

On the Choose your application page, choose Native Application (the Android and Apple logos).

6.

Next, choose Upload to upload your .ipa file.

7.

Device Farm processes your .ipa file before continuing.

In the Run name field, type a name for your run.

Tip
Give the run a name that will help you identify a specific build of your app (for example,
Beta-0.1). For more information, see Working with Runs (p. 23).
8.

Choose Appium Python to configure your test.

9.

To add your Appium test scripts to the test run, choose Upload.

API Version 2015-06-23

67

AWS Device Farm Developer Guide

Calabash

10. Choose Next step, and then complete the instructions to select devices and start the run.

Taking Screenshots in iOS Appium Python Tests

You can take screenshots as part of your iOS Appium Python tests.
When Device Farm runs your Appium Python test, the service sets several system properties that describe
the configuration of the Appium server with which you're communicating. For example, Device Farm sets
the SCREENSHOT_PATH property to a fully qualified path on the local file system where Device Farm
expects Appium screenshots to be saved. The test-specific directory where the screenshots are stored
is defined at runtime. The screenshots are pulled into your Device Farm reports automatically. To view
the screenshots, in the Device Farm console, choose the Screenshots section.
The following example shows how to use and consume the SCREENSHOT_PATH property to capture an
Appium screenshot that is pulled into your Device Farm report.
screenshot_folder = os.getenv('SCREENSHOT_PATH', '')
self.driver.save_screenshot(screenshot_folder + "/screenshot.png")

Additional Considerations for Android Appium Python Tests

Device Farm does not modify iOS Appium Python tests.

Working with Calabash for iOS and AWS Device

Farm
Device Farm provides support for Calabash for iOS. The following information describes how to use this
test framework with Device Farm test types.
Topics
What is Calabash? (p. 68)
Version Information (p. 68)
Prepare Your iOS Calabash Tests (p. 69)
Upload Your iOS Calabash Tests (p. 69)
Taking Screenshots in iOS Calabash Tests (p. 69)
Additional Considerations for iOS Calabash Tests (p. 70)

What is Calabash?
Calabash is a mobile testing framework that enables automated user interface acceptance tests that are
written in Cucumber to be run on iOS apps. For more information, see the Welcome to Calabash iOS
repository on GitHub.

Note
AWS Device Farm currently works with Calabash version 0.14.3.

Version Information
Currently, Device Farm supports Calabash version 0.14.3.

API Version 2015-06-23

68

AWS Device Farm Developer Guide

Calabash

Prepare Your iOS Calabash Tests

Before you upload your iOS Calabash tests to Device Farm for testing, make sure that your iOS Calabash
tests are contained within a .zip file. This .zip file must contain the following structure:
my-zip-file-name.zip
`-- features (directory)
|-- my-feature-1-file-name.feature
|-- my-feature-2-file-name.feature
|-- my-feature-N-file-name.feature
|-- step_definitions (directory)
|
`-- (.rb files)
|-- support (directory)
|
`-- (.rb files)
`-- (any other supporting files)

Upload Your iOS Calabash Tests

To run your iOS Calabash tests on a set of iOS devices in Device Farm, you upload your tests with the
Device Farm console as follows:
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the option next to the project that you want to upload your tests to.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project that you want to upload your tests to.
To create a new project, follow the instructions in Create a Project (p. 20).
3.
4.
5.

If the Create a new run button is displayed, then choose it.

On the Choose your application page, choose Upload.
Browse to and choose your iOS app file. The file must be an .ipa file.

Note
Make sure that your .ipa file is built for an iOS device and not for a simulator.
6.
7.
8.
9.

Choose Next step.

On the Configure a test page, choose Calabash, and then choose Upload.
Browse to and choose the .zip file that contains your tests. The .zip file must follow the format as
described in Prepare Your iOS Calabash Tests (p. 69).
Choose Next step, and then complete the remaining on-screen instructions to select the devices to
run your tests on and to then start the run.

Taking Screenshots in iOS Calabash Tests

You can take screenshots as part of your iOS Calabash tests.
iOS Calabash provides a predefined step for taking screenshots. For details, see the "Screenshots"
section of the Predefined steps page in the Calabash iOS repository on GitHub.
Alternatively, you can define a custom step inside of a Ruby (.rb) file to call the screenshot_embed
function, which creates a screenshot and saves it to a directory that you define. For example, the following
code example creates a screenshot in PNG format and saves it to the /my/custom/path directory with
a file name of screenshot_seconds-since-Epoch:

API Version 2015-06-23

69

AWS Device Farm Developer Guide

UI Automation

screenshot_embed(:prefix => "/my/custom/path", :name => "screen

shot_#{Time.now.to_i}")

Additional Considerations for iOS Calabash Tests

Device Farm currently does not support Calabash configuration profiles or tags.
Device Farm does replace certain Calabash hooks so that iOS Calabash tests will run on devices in
Device Farm, but Device Farm does not modify iOS Calabash tests themselves.

Working with UI Automation for iOS and AWS

Device Farm
Device Farm provides support for UI Automation for iOS. The following information describes how to use
this test framework with Device Farm test types.
Topics
What is UI Automation? (p. 70)
Upload Your iOS UI Automation Tests (p. 70)
Taking Screenshots in iOS UI Automation Tests (p. 71)
Additional Considerations for iOS UI Automation Tests (p. 71)

What is UI Automation?
You can use the Automation instrument to automate user interface tests in your iOS app through test
scripts that you write. These scripts run outside of your app and simulate user interaction by calling the
UI Automation API, a JavaScript programming interface that specifies actions to be performed in your
app as it runs in the simulator or on a connected device. For more information, see About Instruments in
the Instruments User Guide section of the iOS Developer Library.

Upload Your iOS UI Automation Tests

To run your iOS UI Automation tests on a set of iOS devices in Device Farm, you upload your tests with
the Device Farm console as follows:
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the option next to the project that you want to upload your tests to.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project that you want to upload your tests to.
To create a new project, follow the instructions in Create a Project (p. 20).
3.

If the Create a new run button is displayed, then choose it.

4.
5.

On the Choose your application page, choose Upload.

Browse to and choose your iOS app file. The file must be an .ipa file.

Note
Make sure that your .ipa file is built for an iOS device and not for a simulator.
6.

Choose Next step.

7.
8.

On the Configure a test page, choose UI Automation, and then choose Upload.
Browse to and choose the .js file for a single test.

API Version 2015-06-23

70

AWS Device Farm Developer Guide

XCTest

9.

Choose Next step, and then complete the remaining on-screen instructions to select the devices to
run your tests on and to then start the run.

Taking Screenshots in iOS UI Automation Tests

You can take screenshots as part of your iOS UI Automation tests.
To take a screenshot, call the captureScreenWithName function, for example:
target.captureScreenWithName(lang + "_home");, where lang is the current language name.

Additional Considerations for iOS UI Automation Tests

Device Farm adds logging hooks so that it can monitor the execution flow of iOS UI Automation tests,
but Device Farm does not modify iOS UI Automation tests themselves.

Working with XCTest for iOS and AWS Device

Farm
Device Farm provides support for XCTest (including KIF) for iOS, written both Objective-C and Swift. The
following information describes how to use this test framework with Device Farm test types.
Topics
What is XCTest (and KIF)? (p. 71)
Prepare Your iOS XCTest Tests (p. 71)
Upload Your iOS XCTest Tests (p. 71)
Taking Screenshots in iOS XCTest Tests (p. 72)
Additional Considerations for iOS XCTest Tests (p. 72)

What is XCTest (and KIF)?

XCTest is the new testing framework introduced with Xcode 5. XCTest is a modernized reimplementation
of OCUnit, the previous-generation testing framework. For more information, see XCTestthe Xcode
Testing Framework and Transitioning from OCUnit to XCTest in the Testing with Xcode section of the
iOS Developer Library.
KIF (which stands for Keep It Functional) is a related iOS integration test framework. It allows for easy
automation of iOS apps by leveraging the accessibility attributes that the operating system makes available
for those with visual disabilities. KIF builds and performs the tests using a standard XCTest testing target.
For more information, see the KIF iOS Integration Testing Framework repository on GitHub.

Prepare Your iOS XCTest Tests

Before you upload iOS XCTest tests to Device Farm for testing, make sure that your iOS XCTest tests
are contained within a .zip file. This .zip file must contain your my-project-name.xctest directory at
the root of the .zip file. The actual iOS XCTest bundle must be located within this
my-project-name.xctest directory.

Upload Your iOS XCTest Tests

To run your iOS XCTest tests on a set of iOS devices in Device Farm, you upload your tests with the
Device Farm console as follows:.

API Version 2015-06-23

71

AWS Device Farm Developer Guide

XCTest UI

1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the option next to the project that you want to upload your tests to.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project that you want to upload your tests to.
To create a new project, follow the instructions in Create a Project (p. 20).
3.
4.
5.

If the Create a new run button is displayed, then choose it.

On the Choose your application page, choose Upload.
Browse to and choose your iOS app file. The file must be an .ipa file.

Note
Make sure that your .ipa file is built for an iOS device and not for a simulator.
6.
7.

Choose Next step.

On the Configure a test page, choose XCTest, and then choose Upload.

8.

Browse to and choose the .zip file that contains your iOS XCTest tests. In this .zip file, make sure
that the contents are organized according to the instructions as described in Prepare Your iOS XCTest
Tests (p. 71).
Choose Next step, and then complete the remaining on-screen instructions to select the devices to
run your tests on and to then start the run.

9.

Taking Screenshots in iOS XCTest Tests

Device Farm currently supports taking screenshots as part of your iOS XCTest tests using KIF. By default,
KIF will automatically capture screenshots after any failed steps during your tests, and these will be
included in your Device Farm report. If you wish to take on-demand screenshots within your tests, you
should call the captureScreenshotWithDescription method.

Additional Considerations for iOS XCTest Tests

Device Farm supports any version of KIF that is based on OCUnit or XCTest.
Device Farm supports XCTest tests that are written in Objective-C and Swift.

Working with XCTest UI Testing Framework for

iOS and AWS Device Farm
Device Farm provides support for XCTest UI testing framework for iOS, written in both Objective-C and
Swift. The following information describes how to use this test framework with Device Farm test types.
Topics

What is XCTest UI Testing Framework? (p. 73)

Prepare Your iOS XCTest UI Tests (p. 73)
Upload Your iOS XCTest UI Tests (p. 73)
Taking Screenshots in iOS XCTest UI Tests (p. 73)

Additional Considerations for iOS XCTest UI Tests (p. 73)

API Version 2015-06-23

72

AWS Device Farm Developer Guide

XCTest UI

What is XCTest UI Testing Framework?

XCTest UI Framework is the new testing framework introduced with Xcode 7. XCTest UI framework
extends XCTest with UI testing capabilities. For more information, see User Interface Testing in the
Testing with Xcode section of the iOS Developer Library.

Prepare Your iOS XCTest UI Tests

Before you upload iOS XCTest UI tests to Device Farm for testing, make sure that your iOS XCTest UI
test runner bundle is contained within a properly formatted .ipa file. To create an .ipa file, you can place
your my-project-nameUITest-Runner.app bundle in an empty Payload directory. Next, archive the
Payload directory into a .zip file and then change the file extension to .ipa. The *UITest-Runner.app
bundle is produced by Xcode when you build your project for testing, and it can be found in the Products
directory for your project.

Upload Your iOS XCTest UI Tests

To run your iOS XCTest UI tests on a set of iOS devices in Device Farm, you upload your tests with the
Device Farm console as follows:.
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the option next to the project that you want to upload your tests to.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project that you want to upload your tests to.
To create a new project, follow the instructions in Create a Project (p. 20).
3.
4.
5.

If the Create a new run button is displayed, then choose it.

On the Choose your application page, choose Upload.
Browse to and choose your iOS app file. The file must be an .ipa file.

Note
Make sure that your .ipa file is built for an iOS device and not for a simulator.
6.
7.
8.

9.

Choose Next step.

On the Configure a test page, choose XCTest UI, and then choose Upload.
Browse to and choose the .ipa file that contains your iOS XCTest UI test runner. In this .ipa file, make
sure that the contents are organized according to the instructions as described in Prepare Your iOS
XCTest UI Tests (p. 73).
Choose Next step, and then complete the remaining on-screen instructions to select the devices to
run your tests on and to then start the run.

Taking Screenshots in iOS XCTest UI Tests

XCTest UI tests capture screenshots automatically for every step of your tests. These screenshots will
be displayed in your Device Farm test report automatically. No additional code is required.

Additional Considerations for iOS XCTest UI Tests

Device Farm supports XCTest UI tests that are written in Objective-C and Swift.

API Version 2015-06-23

73

AWS Device Farm Developer Guide

Web App Tests

Working with Custom Web App Tests in AWS

Device Farm
Device Farm provides support for the following test types for working with Web applications.
Appium Java JUnit (p. 74)
Appium Java TestNG (p. 76)
Appium Python (p. 78)

Rules for Metered and Unmetered Devices

Metering refers to billing for devices. By default, Device Farm devices are metered and you are charged
per minute after the free trial minutes are used up. You can also choose to purchase unmetered devices,
which allow unlimited testing for a flat monthly fee. For more information about pricing, see AWS Device
Farm Pricing.
If you choose to start a run with a device pool that contains both iOS and Android devices, there are rules
for metered and unmetered devices. For example, if you have 5 unmetered Android devices and 5
unmetered iOS devices, your Web test runs will use your unmetered devices.
Here is another example: suppose you have 5 unmetered Android devices and 0 unmetered iOS devices.
If you select only Android devices for your Web run, your unmetered devices will be used. If you select
both Android and iOS devices for your Web run, the billing method will be metered, and your unmetered
devices will not be used.

Working with Appium Java JUnit for Web

Applications and AWS Device Farm
Device Farm provides support for Appium Java JUnit for Web apps.
Topics
What Is Appium Java JUnit? (p. 74)
Version Information (p. 74)
Prepare Your Web Application Appium Java JUnit Tests (p. 75)
Upload Your Web Application Appium Java JUnit Tests (p. 75)
Taking Screenshots in Web Application Appium Java Junit Tests (p. 76)
Additional Considerations for Web Application Appium Java JUnit Tests (p. 76)

What Is Appium Java JUnit?

Appium is an open-source tool for automating native, mobile web, and hybrid applications on platforms
such as Web applications. For more information, see About Appium.

Version Information
Currently, Device Farm supports Appium version 1.4.16 and Java version Java 8.

API Version 2015-06-23

74

AWS Device Farm Developer Guide

Appium Java JUnit

Prepare Your Web Application Appium Java JUnit Tests

Your Web Application Appium Java JUnit tests must be contained in a .zip file.

Upload Your Web Application Appium Java JUnit Tests

Use the Device Farm console to upload your tests:
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

If you see the AWS Device Farm console home page, choose Get started.

3.

If you already have a project, you can upload your tests to an existing project or choose Create a
new project.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project where you want to upload your tests.
To create a new project, follow the instructions in Create a Project (p. 20).
4.
5.

If the Create a new run button is displayed, then choose it.

On the Choose your application page, choose Web application (the HTML5 button).

6.

Provide a name for your run in the Run name field.

Tip
Name the run something that helps you easily identify a specific build of your app (for
example, Beta-0.1. For more information, see Working with Runs (p. 23).
7.
8.

Configure your test by choosing Appium Java JUnit.

Next, choose Upload to upload your .zip file.
Device Farm processes your .zip file before continuing.

API Version 2015-06-23

75

AWS Device Farm Developer Guide

Appium Java TestNG

9.

Choose Next step, and then complete the remaining on-screen instructions to select devices and
start the run.

Taking Screenshots in Web Application Appium Java Junit

Tests
You can take screenshots as part of your Web Application Appium Java JUnit tests.
When Device Farm runs your Appium Java JUnit test, the service sets several system properties that
describe the configuration of the Appium server with which you're communicating. For example, Device
Farm sets the SCREENSHOT_PATH property to a fully qualified path on the local file system where Device
Farm expects Appium screenshots to be saved. The test-specific directory where the screenshots are
stored is defined at runtime. The screenshots are automatically pulled into your Device Farm reports
automatically. To view the screenshots, in the Device Farm console, choose the Screenshots section.
The following example shows how to use and consume the SCREENSHOT_PATH property to capture an
Appium screenshot that is pulled into your Device Farm report.
public boolean takeScreenshot(final String name) {
String screenshotDirectory = System.getProperty("appium.screenshots.dir",
System.getProperty("java.io.tmpdir", ""));
File screenshot = ((TakesScreenshot) driver).getScreenshotAs(Output
Type.FILE);
return screenshot.renameTo(new File(screenshotDirectory,
String.format("%s.png", name)));
}

Additional Considerations for Web Application Appium Java

JUnit Tests
Device Farm does not modify Web application Appium Java JUnit tests.

Working with Appium Java TestNG for Web

Applications and AWS Device Farm
Device Farm provides support for Appium Java TestNG for Web applications.
Topics
What Is Appium Java TestNG? (p. 76)
Version Information (p. 77)
Prepare Your Web Application Appium Java TestNG Tests (p. 77)
Upload Your Web Application Appium Java TestNG Tests (p. 77)
Taking Screenshots in Web Application Appium TestNG Tests (p. 78)
Additional Considerations for Web Application Appium TestNG Tests (p. 78)

What Is Appium Java TestNG?

Appium is an open-source tool for automating native, mobile web, and hybrid applications on platforms
such as Web applications. For more information, see About Appium.

API Version 2015-06-23

76

AWS Device Farm Developer Guide

Appium Java TestNG

Version Information
Currently, Device Farm supports Appium version 1.4.16 and Java version Java 8.

Prepare Your Web Application Appium Java TestNG Tests

Your Web application Appium Java TestNG tests must be contained in a .zip file before you upload them
to Device Farm.

Upload Your Web Application Appium Java TestNG Tests

Use the Device Farm console to upload your tests:
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

If you see the AWS Device Farm console home page, choose Get started.

3.

If you already have a project, you can upload your tests to an existing project or choose Create a
new project.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project where you want to upload your tests.
To create a new project, follow the instructions in Create a Project (p. 20).
4.
5.

If the Create a new run button is displayed, then choose it.

On the Choose your application page, choose Web application (the HTML5 button).

6.

Provide a name for your run in the Run name field.

Tip
Name the run something that helps you easily identify a specific build of your app (for
example, Beta-0.1. For more information, see Working with Runs (p. 23).

API Version 2015-06-23

77

AWS Device Farm Developer Guide

Appium Python

7.
8.

Configure your test by choosing Appium Java TestNG.

Next, choose Upload to upload your .zip file.

9.

Device Farm processes your .zip file before continuing.

Choose Next step, and then complete the remaining on-screen instructions to select devices and
start the run.

Taking Screenshots in Web Application Appium TestNG

Tests
You can take screenshots as part of your Web Application Appium TestNG tests.
When Device Farm runs your Appium TestNG test, the service sets several system properties that describe
the configuration of the Appium server with which you're communicating. For example, Device Farm sets
the SCREENSHOT_PATH property to a fully qualified path on the local file system where Device Farm
expects Appium screenshots to be saved. The test-specific directory where the screenshots are stored
is defined at runtime.The screenshots are automatically pulled into your Device Farm reports automatically.
To view the screenshots, in the Device Farm console, choose the Screenshots section.
The following example shows how to use and consume the SCREENSHOT_PATH property to capture an
Appium screenshot that is pulled into your Device Farm report.
public boolean takeScreenshot(final String name) {
String screenshotDirectory = System.getProperty("appium.screenshots.dir",
System.getProperty("java.io.tmpdir", ""));
File screenshot = ((TakesScreenshot) driver).getScreenshotAs(Output
Type.FILE);
return screenshot.renameTo(new File(screenshotDirectory,
String.format("%s.png", name)));
}

Additional Considerations for Web Application Appium

TestNG Tests
Device Farm does not modify Web application Appium TestNG tests.

Working with Appium Python for Web Applications

and AWS Device Farm
Device Farm provides support for Appium Python for web applications.
Topics
What Is Appium Python? (p. 79)
Version Information (p. 79)
Prepare Your Web Application Appium Python Tests (p. 79)
Build the Appium Python Test Package (p. 79)
Upload Your Web Application Appium Python Tests (p. 80)
Taking Screenshots in Web Application Appium Python Tests (p. 81)
Additional Considerations for Web Application Appium Python Tests (p. 82)

API Version 2015-06-23

78

AWS Device Farm Developer Guide

Appium Python

What Is Appium Python?

Appium is an open-source tool for automating native, mobile web, and hybrid applications on platforms
like web applications. For more information, see About Appium.

Version Information
Currently, Device Farm supports Appium version 1.4.16 and Python version 2.7.6 (pip version 1.5.4).

Prepare Your Web Application Appium Python Tests

The Appium Python tests for your web application must be contained in a .zip file.

Build the Appium Python Test Package

The Appium Python test packages you upload to Device Farm must be in .zip format and contain all the
dependencies of your test. The following instructions show you how to meet these requirements.

Note
The instructions below are based on Linux x86_64 and Mac. In the currently supported scheme,
Device Farm requires that the packaging of your Appium Python tests be done on Linux x86_64
if your tests contain non-universal Python wheels dependencies. For the platform on which you
execute a command, the wheels tools gather your .whl dependent files under the wheelhouse/
folder. When you execute the Python wheel command on any platform other than Linux x86_64,
you would gather the flavor of a non-univesral wheel dependency for that particular platform and
may cause undesired effects. This would most likely lead to errors when executing your tests
on Device Farm.
1.

We strongly recommend that you set up Python virtualenv for developing and packaging tests so
that unnecessary dependencies are not included in your app package.

Tip
Do not create a Python virtualenv with the --system-site-packages option, because
it will inherit packages from /usr/lib/pythonx.x/site-packages or wherever your global
site-packages directory is. This can lead to you including dependencies in your virtual
environment that are not needed by your tests.
You should also verify that your tests do not use dependencies that are dependent on
native libraries, as these native libraries may or may not be present on the instance where
these tests run.

2.

Install py.test in your virtual environment.

An example flow of creating a virtual environment using Python virtualenv and installing pytest in
that virtual environment would look like the following:
$
$
$
$

3.

virtualenv workspace
cd workspace
source bin/activate
pip install pytest

Store all Python test scripts under the tests/ folder in your work space.
workspace
tests/ (your tests go here)

API Version 2015-06-23

79

AWS Device Farm Developer Guide

Appium Python

4.

Make sure you have py.test installed in your virtual environment and test cases are discoverable by
the following command, which you should run from your virtual environment workspace folder.
$ py.test --collect-only tests/

5.

Make sure the output of py.test command shows you the tests that you want to execute on Device
Farm.
Go to your work space and run the following command to generate the requirements.txt file:
$ pip freeze > requirements.txt

6.

Go to your work space and run the following command to generate the wheelhouse/ folder:
$ pip wheel --wheel-dir wheelhouse -r requirements.txt

7.

You can use the following commands to clean all cached files under your tests/ folder:
$
$
$
$

8.

find
find
find
find

.
.
.
.

-name
-name
-name
-name

'__pycache__'
'*.pyc' -exec
'*.pyo' -exec
'*~' -exec rm

-type
rm -f
rm -f
-f {}

d -exec rm -r {} +
{} +
{} +
+

Zip the tests/ folder, wheelhouse/ folder, and the requirements.txt file into a single archive:
$ zip -r test_bundle.zip tests/ wheelhouse/ requirements.txt

Your work space will eventually look like this:

workspace
tests/
test_bundle.zip
requirements.txt
wheelhouse/

Upload Your Web Application Appium Python Tests

Use the Device Farm console to upload your tests.
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

If you see the AWS Device Farm console home page, choose Get started.

3.

If you already have a project, you can upload your tests to an existing project or choose Create a
new project.

API Version 2015-06-23

80

AWS Device Farm Developer Guide

Appium Python

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project where you want to upload your tests.
To create a project, follow the instructions in Create a Project (p. 20).
4.
5.

If the Create a new run button is displayed, choose it.

On the Choose your application page, choose Web application (the HTML5 button).

6.

In the Run name field, type a name for your run.

Tip
Give the run a name that will help you identify a specific build of your app (for example,
Beta-0.1). For more information, see Working with Runs (p. 23).
7.
8.

Choose Appium Python to configure your test.

Next, choose Upload to upload your .zip file.
Device Farm processes your .zip file before continuing.

9.

Choose Next step, and then complete the instructions to select devices and start the run.

Taking Screenshots in Web Application Appium Python

Tests
You can take screenshots as part of your Appium Python tests for your web application.
When Device Farm runs your Appium Python test, the service sets several system properties that describe
the configuration of the Appium server with which you're communicating. For example, Device Farm sets
the SCREENSHOT_PATH property to a fully qualified path on the local file system where Device Farm
expects Appium screenshots to be saved. The test-specific directory where the screenshots are stored
is defined at runtime. The screenshots are pulled into your Device Farm reports automatically. To view
the screenshots, in the Device Farm console, choose the Screenshots section.

API Version 2015-06-23

81

AWS Device Farm Developer Guide

Built-in Tests

The following example shows how to use and consume the SCREENSHOT_PATH property to capture an
Appium screenshot that is pulled into your Device Farm report.
screenshot_folder = os.getenv('SCREENSHOT_PATH', '')
self.driver.save_screenshot(screenshot_folder + "/screenshot.png")

Additional Considerations for Web Application Appium

Python Tests
Device Farm does not modify Appium Python tests for your web application.

Working with Built-in Tests in AWS Device Farm

Device Farm provides support for several automation test types.

Built-in Test Types

Built-in tests enable you to test your apps without writing scripts.
Built-in: Explorer (Android) (p. 82)
Built-in: Fuzz (Android and iOS) (p. 83)

Working with the Built-in Explorer Test for Device

Farm
Device Farm provides a built-in explorer test type.

What Is the Built-in Explorer Test?

The built-in explorer test crawls your app by analyzing each screen and interacting with it as if it were an
end user. It takes screenshots as it explores, and you can provide Device Farm with credentials so the
test can log in.
Parameters
Username (Optional). Specifies a user name the explorer will use if it encounters a login screen within
your app. If no user name is provided, Device Farm will not insert a user name.
Password (Optional). Specifies a password the explorer will use if it encounters a login screen within
your app. If no password is provided, Device Farm will not insert a password.

Use the Built-in Explorer Test Type

Use the Device Farm console to run the built-in explorer test:
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the option next to the project where you want to run the built-in explorer
test.

API Version 2015-06-23

82

AWS Device Farm Developer Guide

Built-in: Fuzz (Android and iOS)

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project where you to run the built-in explorer test.
To create a new project, follow the instructions in Create a Project (p. 20).
3.

If the Create a new run button is displayed, then choose it.

4.
5.

On the Choose your application page, choose Upload.

Browse to and choose your app file where you want to run the built-in explorer test.

6.
7.
8.

Choose Next step.

On the Configure a test page, choose Built-in: Explorer.
Choose Next step, and then complete the remaining on-screen instructions to select devices and
start the run.

Working with the Built-in Fuzz Test for Device Farm

Device Farm provides a built-in fuzz test type.

What Is the Built-in Fuzz Test?

The built-in fuzz test randomly sends user interface events to devices and then reports results.

Use the Built-in Fuzz Test Type

Use the Device Farm console to run the built-in fuzz test:
1.
2.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

In the list of projects, choose the option next to the project where you want to run the built-in fuzz
test.

Tip
If the list of projects is not displayed, then on the secondary navigation bar, for Projects,
choose the name of the project where you to run the built-in fuzz test.
To create a new project, follow the instructions in Create a Project (p. 20).
3.
4.
5.
6.
7.
8.

If the Create a new run button is displayed, then choose it.

On the Choose your application page, choose Upload.
Browse to and choose your app file where you want to run the built-in fuzz test.
Choose Next step.
On the Configure a test page, choose Built-in: Fuzz.
If any of the following settings appear, you can either accept the default values or specify your own:
Event count: Specify a number between 1 and 10,000, representing the number of user interface
events for the fuzz test to perform.
Event throttle: Specify a number between 1 and 1,000, representing the number of milliseconds
for the fuzz test to wait before performing the next user interface event.
Randomizer seed: Specify a number for the fuzz test to use for randomizing user interface events.
Specifying the same number for subsequent fuzz tests ensures identical event sequences.

9.

Choose Next step, and then complete the remaining on-screen instructions to select devices and
start the run.

API Version 2015-06-23

83

AWS Device Farm Developer Guide

Create a Session

Working with Remote Access in

AWS Device Farm
Remote access allows you to swipe, gesture, and interact with a device through your web browser in real
time in order to test functionality and reproduce customer issues. You interact with a specific device by
creating a remote access session with that device.
A session in Device Farm is a real-time interaction with an actual, physical device hosted in a web browser.
A session displays the single device you select when you start the session. A user can start more than
one session at a time with the total number of simultaneous devices limited by the number of device slots
you have. You can purchase device slots based on the device family (for example, Android devices). For
more information, see Device Farm Pricing.
Device Farm captures video of each remote access session and generates logs of activity taking place
during the session. These results include any information you provide during a session.

Note
For security reasons, we recommend that you avoid providing or entering sensitive information
such as account numbers, personal login information, and other details during a remote access
session.
Create a Session (p. 84)
Use a Session (p. 87)
Get Session Results (p. 88)

Create a Remote Access Session in AWS Device

Farm
For information about remote access sessions, see Sessions (p. 14).
Prerequisites (p. 85)
Create a Run with the Device Farm Console (p. 85)
Next Steps (p. 86)

API Version 2015-06-23

84

AWS Device Farm Developer Guide

Prerequisites

Prerequisites
Create a project in Device Farm. Follow the instructions in Create a Project (p. 20), and then return to
this page.

Create a Session with the Device Farm Console

1.
2.
3.

Sign in to the Device Farm console at https://console.aws.amazon.com/devicefarm.

If you see the AWS Device Farm console home page, choose Get started.
Choose a project from the Projects drop-down or choose Create a new project.

4.
5.

Choose the Remote access tab.

Choose the Start a new session button.

6.

Choose a device for your session. You can choose from the list of available devices or search for a
device using the fields at the top of the list. You can search by:

Name
Platform
Operating system
Form factor

API Version 2015-06-23

85

AWS Device Farm Developer Guide

Next Steps

7.
8.

Type a name for the session in Session name.

Choose Confirm and start session to begin the session.

Next Steps
Device Farm will start the session as soon as the requested device is available, typically within a few
minutes. The Device requested dialog box appears until the session starts. To cancel the session request,
choose Cancel request.
After a session starts, if you should close the browser or browser tab without stopping the session or if
the connection between the browser and the Internet is lost, the session remains active for five minutes
from that time. After that, Device Farm ends the session automatically. Your account will be charged for
the idle time, however.

After the session begins, you can start interacting with the device in the web browser.

API Version 2015-06-23

86

AWS Device Farm Developer Guide

Use a Session

Use a Remote Access Session in AWS Device

Farm
For information about sessions, see Sessions (p. 14).
Prerequisites (p. 87)
Use a Session in the Device Farm Console (p. 87)
Next Steps (p. 88)

Prerequisites
Create a session. Follow the instructions in Create a Session (p. 84), and then return to this page.

Use a Session in the Device Farm Console

As soon the device you requested for a remote access session becomes available, the console displays
the device screen. The session has a maximum length of 60 minutes. The time remaining in the session
appears below the menu bar on the right side of the console.

Installing an Application
To install an application on the session device, in Install applications, choose Upload, and then choose
the .apk file for the Android application you want to install. Applications you run in an remote access
session don't require any test instrumentation or provisioning.

API Version 2015-06-23

87

AWS Device Farm Developer Guide

Next Steps

Controlling the Device

You can interact with the device displayed in the console as you would the actual physical device using
your mouse for touch and the device's on-screen keyboard. There are buttons in View controls that
function just as the Home and Back buttons on an Android device do. You can also switch between
applications running on the device by choosing Recent apps.

Next Steps
Device Farm will continue the session until you stop it manually or until the sixty-minute time limit is
reached. To end the session, choose the Stop session button. After it stops, you can access captured
video and logs generated by the session. For more information about getting session results, see Get
Session Results (p. 88).

Get Results of an Remote Access Session in

AWS Device Farm
For information about sessions, see Sessions (p. 14).
Prerequisites (p. 88)
Viewing Session Details (p. 88)
Downloading Session Video or Logs (p. 89)

Prerequisites
Complete a session. Follow the instructions in Use a Session (p. 87), and then return to this page.

Viewing Session Details

When a remote access session ends, the Device Farm console displays a table containing details about
activity that took place during the session. For more information about the details displayed, see Analyzing
Log Information (p. 35).

To return to the details of a session at a later time:

1.

Choose the project you want to review from the Project drop-down list.

API Version 2015-06-23

88

AWS Device Farm Developer Guide

Downloading Session Video or Logs

2.

Choose the session you want to review from the list or View all sessions from the Runs & sessions
drop-down list then choose the session you want to review from the list of sessions displayed.

Downloading Session Video or Logs

When a remote access session ends, the Device Farm console provides access to a video capture of
the session along with logs about activity in the session. In the session results, choose the Files tab for
a list of links to the session video and logs. You can view these files in the browser or save them locally.

API Version 2015-06-23

89

AWS Device Farm Developer Guide

Device Farm Information in CloudTrail

Logging AWS Device Farm API

Calls by Using AWS CloudTrail
Device Farm is integrated with CloudTrail, a service that captures API calls made by or on behalf of
Device Farm in your AWS account and delivers the log files to an Amazon S3 bucket you specify. Examples
of these API calls include creating a new project or run in Device Farm. CloudTrail captures API calls
from the Device Farm console or the Device Farm APIs. Using the information collected by CloudTrail,
you can determine which request was made to Device Farm, the source IP address from which the request
was made, who made the request, when it was made, and so on. To learn more about CloudTrail, including
how to configure and enable it, see the AWS CloudTrail User Guide.

Device Farm Information in CloudTrail

When CloudTrail logging is enabled in your AWS account, API calls made to Device Farm actions are
tracked in log files. Device Farm records are written together with other AWS service records in a log file.
CloudTrail determines when to create and write to a new file based on a time period and file size.
All of the Device Farm actions are logged and documented in the AWS CLI Reference (p. 93) and the
API Reference (p. 94). For example, calls to create a new project or run in Device Farm generate entries
in CloudTrail log files.
Every log entry contains information about who generated the request. The user identity information in
the log helps you determine whether the request was made with root or IAM user credentials, with
temporary security credentials for a role or federated user, or by another AWS service. For more
information, see the userIdentity field in the CloudTrail Event Reference.
You can store log files in your bucket for as long as you want, but you can also define Amazon S3 lifecycle
rules to archive or delete log files automatically. By default, your log files are encrypted by using Amazon
S3 server-side encryption (SSE).
If you want to take quick action upon log file delivery, you can have CloudTrail publish Amazon SNS
notifications when new log files are delivered. For more information, see Configuring Amazon SNS
Notifications.
You can also aggregate Device Farm log files from multiple AWS regions and multiple AWS accounts
into a single Amazon S3 bucket. For more information, see Aggregating CloudTrail Log Files to a Single
Amazon S3 Bucket.

API Version 2015-06-23

90

AWS Device Farm Developer Guide

Understanding Device Farm Log File Entries

Understanding Device Farm Log File Entries

CloudTrail log files can contain one or more log entries where each entry is made up of multiple
JSON-formatted events. A log entry represents a single request from any source and includes information
about the requested action, any parameters, the date and time of the action, and so on. The log entries
are not guaranteed to be in any particular order. That is, they are not an ordered stack trace of the public
API calls.
The following example shows a CloudTrail log entry that demonstrates the Device Farm ListRuns action:
{
"Records": [
{
"eventVersion": "1.03",
"userIdentity": {
"type": "Root",
"principalId": "AKIAI44QH8DHBEXAMPLE",
"arn": "arn:aws:iam::123456789012:root",
"accountId": "123456789012",
"accessKeyId": "AKIAIOSFODNN7EXAMPLE",
"sessionContext": {
"attributes": {
"mfaAuthenticated": "false",
"creationDate": "2015-07-08T21:13:35Z"
}
}
},
"eventTime":"2015-07-09T00:51:22Z",
"eventSource": "devicefarm.amazonaws.com",
"eventName":"ListRuns",
"awsRegion":"us-west-2",
"sourceIPAddress":"203.0.113.11",
"userAgent":"example-user-agent-string",
"requestParameters": {
"arn":"arn:aws:devicefarm:us-west-2:123456789012:project:a9129b8c-df6b4cdd-8009-40a25EXAMPLE"},
"responseElements": {
"runs": [
{
"created": "Jul 8, 2015 11:26:12 PM",
"name": "example.apk",
"completedJobs": 2,
"arn": "arn:aws:devicefarm:us-west-2:123456789012:run:a9129b8cdf6b-4cdd-8009-40a256aEXAMPLE/1452d105-e354-4e53-99d8-6c993EXAMPLE",
"counters": {
"stopped": 0,
"warned": 0,
"failed": 0,
"passed": 4,
"skipped": 0,
"total": 4,
"errored": 0
},
"type": "BUILTIN_FUZZ",
"status": "RUNNING",
"totalJobs": 3,
"platform": "ANDROID_APP",

API Version 2015-06-23

91

AWS Device Farm Developer Guide

Understanding Device Farm Log File Entries

"result": "PENDING"
},
... additional entries ...
]
}
}
}
]
}

API Version 2015-06-23

92

AWS Device Farm Developer Guide

AWS CLI Reference for AWS

Device Farm
To use the AWS Command Line Interface (AWS CLI) to run Device Farm commands, see the AWS CLI
Reference for AWS Device Farm.
For general information about the AWS CLI, see the AWS Command Line Interface User Guide and the
AWS Command Line Interface Reference.

API Version 2015-06-23

93

AWS Device Farm Developer Guide

API Reference for AWS Device

Farm
Use HTTP to call the Device Farm APIs. For more information, see the AWS Device Farm API Reference.

API Version 2015-06-23

94

AWS Device Farm Developer Guide

User Access Permissions for AWS

Device Farm
You can use IAM to enable IAM users in your AWS account to perform only certain actions in Device
Farm. You may want to do this, for example, if you have a set of IAM users that you want to allow to list,
but not create, resources in Device Farm; you may have another set of IAM users you want to allow to
list and create new resources; and so on.
For example, in the Setting Up (p. 4) instructions, you attached an access policy to an IAM user in your
AWS account that contains a policy statement similar to this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"devicefarm:*"
],
"Resource": [
"*"
]
}
]
}

The preceding statement allows the IAM user in your AWS account to perform actions in Device Farm
to which your AWS account has access. In practice, you may not want to give the IAM users in your AWS
account this much access.
The following information shows how you can attach a policy to an IAM user to restrict the actions the
IAM user can perform in Device Farm.

API Version 2015-06-23

95

AWS Device Farm Developer Guide

Create and Attach a Policy to an IAM User

Create and Attach a Policy to an IAM User

To create and attach an access policy to an IAM user that restricts the actions the IAM user can perform
in Device Farm, do the following:
1.

Sign in to the Identity and Access Management (IAM) console at https://console.aws.amazon.com/

iam/.

2.

Choose Policies, and then choose Create Policy. (If a Get Started button appears, choose it, and
then choose Create Policy.)
Next to Create Your Own Policy, choose Select.
For Policy Name, type any value that will be easy for you to refer to later, if needed.

3.
4.
5.

For Policy Document, type a policy statement with the following format, and then choose Create
Policy:
{
"Version": "2012-10-17",
"Statement" : [
{
"Effect" : "Allow",
"Action" : [
"action-statement"
],
"Resource" : [
"resource-statement"
]
},
{
"Effect" : "Allow",
"Action" : [
"action-statement"
],
"Resource" : [
"resource-statement"
]
}
]
}

In the preceding statement, substitute action-statement as needed, and add additional statements
as needed, to specify the actions in Device Farm the IAM user can perform. (By default, the IAM
user will not have the desired permissions unless a corresponding Allow statement is explicitly
stated.) The following section describes the format of allowed actions for Device Farm.

Note
Currently, the only allowed value for resource-statement in the preceding example is
the asterisk character (*). This means that while you can restrict the actions an IAM user
can perform in Device Farm, you cannot also restrict the Device Farm resources the IAM
user can access.
6.

Choose Users.

7.
8.

Choose the IAM user to whom you want to attach the policy.
In the Permissions area, for Managed Policies, choose Attach Policy.

9.

Select the policy you just created, and then choose Attach Policy.

API Version 2015-06-23

96

AWS Device Farm Developer Guide

Action Syntax for Performing Actions in Device Farm

Action Syntax for Performing Actions in Device

Farm
The following information describes the format for specifying actions an IAM user can perform in Device
Farm.
Actions follow this general format:
devicefarm:action

Where action is an available Device Farm action:

An asterisk character (*), which represents all of the available Device Farm actions.
One of the available Device Farm actions, as described in the AWS Device Farm API Reference.
A combination of an available Device Farm action prefix and an asterisk character (*). For example,
specifying List* enables the IAM user to perform all available Device Farm actions that begin with
List.
Some example action statements include:
devicefarm:* for all Device Farm actions.
devicefarm:Get* for only the Device Farm actions that begin with Get.
devicefarm:ListProjects for just the ListProjects Device Farm action.
For example, the following policy statement gives the IAM user permission to get information about all
Device Farm resources that are available to the user's AWS account:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"devicefarm:Get*",
"devicefarm:List*",
],
"Resource": [
"*"
]
}
]
}

API Version 2015-06-23

97

AWS Device Farm Developer Guide

Limits in AWS Device Farm

The following list describes current Device Farm limits.
The maximum app file size you can upload is 4 GB.
There is no limit to the number of devices you can include in a test run. However, the maximum number
of devices that Device Farm will simultaneously test during a run is 5. (This number can be increased
upon request).
There is no limit to the number of runs you can schedule.
There is a sixty-minute limit to length of a remote access session.

API Version 2015-06-23

98

AWS Device Farm Developer Guide

Jenkins CI Plugin

Tools and Plugins for AWS Device

Farm
This section contains links and information about working with AWS Device Farm tools and plugins. You
can find Device Farm plugins on AWS Labs on GitHub.
If you are an Android developer, we also have an AWS Device Farm sample app for Android on GitHub.
You can use the app and example tests as a reference for your own Device Farm test scripts.
Topics
AWS Device Farm Integration with Jenkins CI Plugin (p. 99)
AWS Device Farm Gradle Plugin (p. 104)

AWS Device Farm Integration with Jenkins CI

Plugin
This plugin provides AWS Device Farm functionality from your own Jenkins continuous integration (CI)
server. For more information, see Jenkins (software).

Note
To download the Jenkins plugin, go to GitHub and follow the instructions in Step 1: Install the
Plugin (p. 101).
This section contains a series of procedures to set up and use the Jenkins CI plugin with AWS Device
Farm.
Topics
Step 1: Installing the Plugin (p. 101)
Step 2: Creating an AWS Identity and Access Management User for your Jenkins CI Plugin (p. 102)
Step 3: First-time configuration instructions (p. 103)
Step 4: Using the Plugin in a Jenkins Job (p. 103)
Dependencies (p. 104)
The following images show the features of the Jenkins CI plugin.
API Version 2015-06-23
99

AWS Device Farm Developer Guide

Jenkins CI Plugin

API Version 2015-06-23

100

AWS Device Farm Developer Guide

Step 1: Install the Plugin

The plugin can also pull down all the test artifacts (logs, screenshots, etc.) locally:

Step 1: Installing the Plugin

There are two options for installing the Jenkins continuous integration (CI) plugin for AWS Device Farm.
You can search for the plugin from within the Available Plugins dialog in the Jenkins Web UI, or you
can download the hpi file and install it from within Jenkins.

API Version 2015-06-23

101

AWS Device Farm Developer Guide

Step 2: Create an IAM User

Install from within the Jenkins UI

1.

Find the plugin within the Jenkins UI by choosing Manage Jenkins, Manage Plugins, and then
choose Available.

2.
3.
4.

Search for aws-device-farm.

Install the AWS Device Farm plugin.
Ensure that the plugin is owned by the Jenkins user.

5.

Restart Jenkins.

Download the Plugin

1.

Download the hpi file directly from http://updates.jenkins-ci.org/latest/aws-device-farm.hpi.

2.

Ensure that the plugin is owned by the Jenkins user.

3.

Install the plugin using one of the following options:

Upload the plugin by choosing Manage Jenkins, Manage Plugins, Advanced, and then choose
Upload plugin.
Place the hpi file in the Jenkins plugin directory (usually /var/lib/jenkins/plugins).

4.

Restart Jenkins.

Step 2: Creating an AWS Identity and Access

Management User for your Jenkins CI Plugin
We recommend that you do not use your AWS root account to access Device Farm. Instead, create a
new AWS Identity and Access Management (IAM) user (or use an existing IAM user) in your AWS account,
and then access Device Farm with that IAM user.
To create a new IAM user, see Creating an IAM User (AWS Management Console). Be sure to generate
an access key for each user and download or save the user security credentials. You will need the
credentials later.

Give the IAM User Permission to Access Device Farm

To give the IAM user permission to access Device Farm, create a new access policy in IAM, and then
assign the access policy to the IAM user as follows.

Note
The AWS root account or IAM user that you use to complete the following steps must have
permission to create the following IAM policy and attach it to the IAM user. For more information,
see Working with Policies

To create the access policy in IAM

1.

Open the Identity and Access Management (IAM) console at https://console.aws.amazon.com/iam/.

2.
3.
4.

Choose Policies.
Choose Create Policy. (If a Get Started button appears, choose it, and then choose Create Policy.)
Next to Create Your Own Policy, choose Select.

5.

For Policy Name, type a name for the policy (for example, AWSDeviceFarmAccessPolicy).

6.

For Description, type a description that helps you associate this IAM user with your Jenkins project.

API Version 2015-06-23

102

AWS Device Farm Developer Guide

Step 3: First-time configuration instructions

7.

For Policy Document, type the following statement:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DeviceFarmAll",
"Effect": "Allow",
"Action": [ "devicefarm:*" ],
"Resource": [ "*" ]
}
]
}

8.

Choose Create Policy.

To assign the access policy to the IAM user

1.
2.
3.
4.
5.
6.

Open the IAM console at https://console.aws.amazon.com/iam/.

Choose Users.
Choose the IAM user to whom you will assign the access policy.
In the Permissions area, for Managed Policies, choose Attach Policy.
Select the policy you just created (for example, AWSDeviceFarmAccessPolicy).
Choose Attach Policy.

Step 3: First-time configuration instructions

The first time you run your Jenkins server, you will need to configure the system as follows.

Note
If you are using device slots (p. 16), the device slots feature is disabled by default.
1.
2.
3.
4.

Log into your Jenkins Web user interface.

On the left-hand side of the screen, choose Manage Jenkins.
Choose Configure System.
Scroll down to the AWS Device Farm header.

5.

Copy your security credentials from Step 2: Create an IAM User (p. 102) and paste your Access Key
ID and Secret Access Key into their respective boxes.
Choose Save.

6.

Step 4: Using the Plugin in a Jenkins Job

Once you have installed the Jenkins plugin, follow these instructions to use the plugin in a Jenkins job.
1.
2.
3.

Log into your Jenkins web UI.

Click the job you want to edit.
On the left-hand side of the screen, choose Configure.

4.
5.

Scroll down to the Post-build Actions header.

Click Add post-build action and select Run Tests on AWS Device Farm.

6.

Select the project you would like to use.

API Version 2015-06-23

103

AWS Device Farm Developer Guide

Dependencies

7.
8.

Select the device pool you would like to use.

Select whether you'd like to have the test artifacts (such as the logs and screenshots) archived locally.

9. In Application, fill in the path to your compiled application.

10. Select the test you would like run and fill in all the required fields.
11. Choose Save.

Dependencies
The Jenkins CI Plugin requires the AWS Mobile SDK 1.10.5 or later. For more information and to install
the SDK, see AWS Mobile SDK.

AWS Device Farm Gradle Plugin

This plugin provides AWS Device Farm integration with the Gradle build system in Android Studio. For
more information, see Gradle.

Note
To download the Gradle plugin, go to GitHub and follow the instructions in Building the Device
Farm Gradle Plugin (p. 104).
The Device Farm Gradle Plugin provides Device Farm functionality from your Android Studio environment.
You can kick off tests on real Android phones and tablets hosted by Device Farm.
This section contains a series of procedures to set up and use the Device Farm Gradle Plugin.
Topics
Step 1: Building the AWS Device Farm Gradle Plugin (p. 104)
Step 2: Setting up the AWS Device Farm Gradle Plugin (p. 105)
Step 3: Generating an IAM User (p. 106)
Step 4: Configuring Test Types (p. 107)
Dependencies (p. 109)

Step 1: Building the AWS Device Farm Gradle

Plugin
This plugin provides AWS Device Farm integration with the Gradle build system in Android Studio. For
more information, see Gradle.

Note
Building the plugin is optional. The plugin is published through Maven Central. If you wish to
allow Gradle to download the plugin directly, skip this step and jump to Setting up the Device
Farm Gradle Plugin (p. 105).

To build the plugin

1.

Go to GitHub and clone the repository.

2.

Build the plugin using gradle install.

The plugin is installed to your local maven repository.

API Version 2015-06-23

104

AWS Device Farm Developer Guide

Setting up the Device Farm Gradle Plugin

Next step: Setting up the Device Farm Gradle Plugin (p. 105)

Step 2: Setting up the AWS Device Farm Gradle

Plugin
If you haven't done so already, clone the repository and install the plugin using the procedure here:
Building the Device Farm Gradle Plugin (p. 104).

To configure the AWS Device Farm Gradle Plugin

1.

Add the plugin artifact to your dependency list in build.gradle.

buildscript {
repositories {
mavenLocal()
mavenCentral()
}
dependencies {
classpath 'com.android.tools.build:gradle:1.3.0'
classpath 'com.amazonaws:aws-devicefarm-gradle-plugin:1.0'
}
}

2.

Configure the plugin in your build.gradle file. The following test specific configuration should
serve as your guide:
apply plugin: 'devicefarm'
devicefarm {
projectName "My Project" // required: Must already exists.
devicePool "My Device Pool Name" // optional: Defaults to "Top
Devices"
useUnmeteredDevices() // optional if you wish to use your un-metered
devices

authentication {
accessKey "aws-iam-user-accesskey"
secretKey "aws-iam-user-secretkey"
// or
roleArn "My role arn" // Optional, if role arn is specified, it
will be used.
// Otherwise use access and secret keys
}
// optional block, radios default to 'on' state, all parameters op
tional

API Version 2015-06-23

105

AWS Device Farm Developer Guide

Generating an IAM user

devicestate {
extraDataZipFile file("relative/path/to/zip") // default null
auxiliaryApps [file("path1"), file("path2")] // default empty
list
wifi on
bluetooth off
gps off
nfc on
latitude 47.6204 // default
longitude -122.3491 // default
}

// Configure test type, if none default to instrumentation

// Fuzz
// fuzz { }
// Instrumentation
// See AWS Developer docs for filter (optional)
// instrumentation { filter "my-filter" }
// Calabash
calabash {
tests file("path-to-features.zip")
}
}

3.

Run your Device Farm test using the following task: gradle devicefarmUpload.
The build output will print out a link to the Device Farm console where you can monitor your test
execution.

Next step: Generating an IAM user (p. 106)

Step 3: Generating an IAM User

AWS Identity and Access Management (IAM) helps you manage permissions and policies for working
with AWS resources. This topic walks you through generating an IAM user with permissions to access
AWS Device Farm resources.
If you haven't done so already, complete steps 1 and 2 before generating an IAM user.
We recommend that you do not use your AWS root account to access Device Farm. Instead, create a
new IAM user (or use an existing IAM user) in your AWS account, and then access Device Farm with
that IAM user.

Note
The AWS root account or IAM user that you use to complete the following steps must have
permission to create the following IAM policy and attach it to the IAM user. For more information,
see Working with Policies.

API Version 2015-06-23

106

AWS Device Farm Developer Guide

Configuring Test Types

To create a new user with the proper access policy in IAM

1.

Open the Identity and Access Management (IAM) console at https://console.aws.amazon.com/iam/.

2.
3.

Choose Users.
Choose Create New Users.

4.

Enter the user name of your choice.

For example, GradleUser.

5.

Choose Create.

6.

Choose Download Credentials and save them in a location where you can easily retrieve them
later.
Choose Close.
Choose the user name in the list.

7.
8.

9. Under Permissions, expand the Inline Policies header by clicking the down arrow on the right.
10. Choose Click here where it says, There are no inline policies to show. To create one, click here.
11. On the Set Permissions screen, choose Custom Policy.
12. Choose Select.
13. Give your policy a name, such as AWSDeviceFarmGradlePolicy.
14. Paste the following policy into Policy Document.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DeviceFarmAll",
"Effect": "Allow",
"Action": [ "devicefarm:*" ],
"Resource": [ "*" ]
}
]
}

15. Choose Apply Policy.

Next step: Configuring Test Types (p. 107).
For more information, see Creating an IAM User (AWS Management Console) or Setting Up (p. 4).

Step 4: Configuring Test Types

By default, the AWS Device Farm Gradle plugin runs the Instrumentation (p. 53) test. If you want to run
your own tests or specify additional parameters, you can choose to configure a test type. This topic
provides information about each available test type and what you need to do to configure it for use from
within Android Studio. For more information about the available test types in Device Farm, see Working
with Test Types in AWS Device Farm (p. 38).
If you haven't done so already, complete steps 1 3 before configuring test types.

Note
If you are using device slots (p. 16), the device slots feature is disabled by default.

API Version 2015-06-23

107

AWS Device Farm Developer Guide

Configuring Test Types

Appium
Device Farm provides support for Appium Java JUnit and TestNG for Android.
Appium Java JUnit (p. 39)
Appium Java TestNG (p. 44)
You can choose useTestNG() or useJUnit(). JUnit is the default and does not need to be explicitly
specified.
appium {
tests file("path to zip file") // required
useTestNG() // or useJUnit()
}

Built-in: Explorer
Device Farm provides a built-in app explorer to test user flows through your app without writing custom
test scripts. You can specify a user name and password to test scenarios that require a login. Here is
how you configure user name and password:
appexplorer {
username "my-username"
password "my-password"
}

For more information:

Built-in: Explorer (Android) (p. 82)

Built-in: Fuzz
Device Farm provides a built-in fuzz test type, which randomly sends user interface events to devices
and then reports the results.
fuzz {
eventThrottle 50 // optional default
eventCount 6000 // optional default
randomizerSeed 1234 // optional default blank
}

For more information, see Built-in: Fuzz (Android and iOS) (p. 83).

Calabash
Device Farm provides support for Calabash for Android. To learn how to prepare your Android Calabash
tests, see Calabash (p. 51)

API Version 2015-06-23

108

AWS Device Farm Developer Guide

Dependencies

calabash {
tests file("path to zip file") // required
tags "my tags" // optional calabash tags
profile "my profile" // optional calabash profile
}

Instrumentation
Device Farm provides support for instrumentation (JUnit, Espresso, Robotium, or any Instrumentation-based
tests) for Android. For more information, see Instrumentation (p. 53).
When running an instrumentation test in Gradle, Device Farm uses the .apk file generated from your
androidTest directory as the source of your tests.
instrumentation {
filter "test filter per developer docs" // optional
}

UI Automator
Upload your app, as well as your UI Automator-based tests, packaged in a .jar file.
uiautomator {
tests file("path to uiautomator jar file") // required
filter "test filter per developer docs" // optional
}

For more information, see UI Automator (p. 54).

Dependencies
Runtime
The Device Farm Gradle Plugin requires the AWS Mobile SDK 1.10.15 or later. For more information
and to install the SDK, see AWS Mobile SDK.
Android tools builder test api 0.5.2
Apache Commons Lang3 3.3.4
For Unit Tests
Testng 6.8.8
Jmockit 1.19
Android gradle tools 1.3.0

API Version 2015-06-23

109

AWS Device Farm Developer Guide

Document History
The following table describes the important changes to the documentation since the last release of this
guide.
API version: 2015-06-23
Latest documentation update: April 19, 2016

Change

Description

Date Changed

Remote Access Sessions

You can now remotely access and interact with a single device April 19, 2016
in the console. Learn more about Working with Remote Access (p. 84).

Device Slots Self-Ser- You can now purchase device slots using the AWS Manage- March 22,
vice
ment Console, the AWS Command Line Interface, or the API. 2016
Learn more about how to Purchase Device Slots (p. 16).
How to stop test runs

You can now stop test runs using the AWS Management
March 22,
Console, the AWS Command Line Interface, or the API. Learn 2016
more about how to Stop a Run in AWS Device Farm (p. 25).

New XCTest UI test

types

You can now run XCTest UI custom tests on iOS applications. March 8, 2016
Learn more about the XCTest UI (p. 72) test type.

New Appium Python

test types

You can now run Appium Python custom tests on Android and January 19,
iOS applications, as well as Web applications. Learn more
2016
about Test Types in AWS Device Farm (p. 11).

Web Application test

types

You can now run Appium Java JUnit and TestNG custom tests November 19,
on Web applications. Learn more about Working with Custom 2015
Web App Tests in AWS Device Farm (p. 74).

AWS Device Farm

Gradle Plugin

Learn more about how to install and use the Device Farm
Gradle Plugin (p. 104).

September 28,
2015

New Android Built-in

Test: Explorer

Learn more about Built-in: Explorer (Android) (p. 82). The explorer test crawls your app by analyzing each screen as if it
were an end user and takes screenshots as it explores.

September 16,
2015

API Version 2015-06-23

110

AWS Device Farm Developer Guide

Change

Description

iOS support added

Learn more about testing iOS devices and running iOS tests August 4, 2015
(including XCTest) in Working with Test Types in AWS Device
Farm (p. 38).

Initial public release

This is the initial public release of the AWS Device Farm Developer Guide.

API Version 2015-06-23

111

Date Changed

July 13, 2015

AWS Device Farm Developer Guide

AWS Glossary
For the latest AWS terminology, see the AWS Glossary in the AWS General Reference.

API Version 2015-06-23

112