Automating GUI Testing for Android Applications

Cuixiong Hu Iulian Neamtiu

Department of Computer Science and Engineering
University of California, Riverside, CA, USA
{huc,neamtiu}@cs.ucr.edu

ABSTRACT alone [23, 22, 24]. A major draw of any smartphone is its
Users increasingly rely on mobile applications for computa- ability to run applications, thus users are increasingly rely-
tional needs. Google Android is a popular mobile platform, ing on smartphones for computing needs, rather than using
hence the reliability of Android applications is becoming laptops or desktops. This leads to an increasing impetus for
increasingly important. Many Android correctness issues, ensuring the reliability of mobile applications. Reliability
however, fall outside the scope of traditional verification is particularly important for sensitive mobile applications
techniques, as they are due to the novelty of the platform such as online banking, business management, health care,
and its GUI-oriented application construction paradigm. In or military domains.
this paper we present an approach for automating the testing In this paper we focus on ensuring the reliability of mo-
process for Android applications, with a focus on GUI bugs. bile applications running on the Google Android platform.
We first conduct a bug mining study to understand the na- According to Fall 2010 reports, Android is the second most
ture and frequency of bugs affecting Android applications; popular mobile OS, surpassing BlackBerry and iPhone OS,
our study finds that GUI bugs are quite numerous. Next, we and will be tied for number one with Nokia’s Symbian by
present techniques for detecting GUI bugs by automatic gen- 2014 [29, 24]; Android is in fact the only mobile OS plat-
eration of test cases, feeding the application random events, form to gain market share since Q4’09 [23, 22, 24]. The
instrumenting the VM, producing log/trace files and analyz- Android ecosystem includes the Android Market, which cur-
ing them post-run. We show how these techniques helped to rently lists more than 220,000 applications, 12,316 of which
re-discover existing bugs and find new bugs, and how they were added in December 2010 alone, and an estimated 2.6
could be used to prevent certain bug categories. We believe billion downloads [17].
our study and techniques have the potential to help devel- Many tools and techniques exist for automating the test-
opers increase the quality of Android applications. ing of mature, well-established applications, such as desktop
or server programs. However, the physical constraints of mo-
bile devices (e.g., low-power CPU, small memory, small dis-
Categories and Subject Descriptors play), as well as developers’ unfamiliarity with mobile plat-
D.2.4 [Software Engineering]: Software/Program Verifi- forms (due to their novelty), make mobile applications prone
cation—Reliability; D.2.5 [Software Engineering]: Test- to new kinds of bugs. For example, an Android application
ing and Debugging—Testing tools,Tracing is structured around activities (GUI windows), broadcast re-
ceivers, services and content providers; this is different from
General Terms standard server applications, or from an event-based system
used in a desktop GUI application. The tendency of mobile
Reliability, Verification
applications to have bugs is evidenced by their high defect
density: a study by Maji et al. [27] has found that Android
Keywords applications can have defect densities orders of magnitude
Test automation, Mobile applications, Google Android, GUI higher than the OS.
testing, Test case generation, Empirical bug studies In this paper we aim to bring novel, Android-specific classes
of bugs to light, and show how to construct an effective test
1. INTRODUCTION automation approach for addressing such bugs, especially
GUI bugs, and ensuring the reliability of Android applica-
Smartphones are becoming pervasive, with more than 195
tions. First, we conduct a bug collection and categorization
million sold worldwide in the first three quarters of 2010
on 10 popular open source Android applications (Section 2).
We found that, while bugs related to application logic are
still present, the remaining bugs are Android-specific, i.e.,
Permission to make digital or hard copies of all or part of this work for due to the activity- and event-based nature of Android ap-
Permission
personal or to make digital
classroom use isorgranted
hard copies
withoutof fee
all provided
or part ofthat
thiscopies
work are
for plications. We categorized all confirmed bugs in the bug
personal
not madeororclassroom
distributeduse
for is granted
profit without feeadvantage
or commercial provided and
thatthat
copies are
copies
not made or distributed for profit or commercial advantage and that copies database based on our observations. To detect and fix these
bear this
bear this notice
notice and
and the
the full
full citation
citation on
on the
the first
first page.
page. To To copy
copy otherwise,
otherwise, to
to categories of bugs, we employ an automated test approach
republish, to
republish, to post
post on
on servers
servers oror to
to redistribute
redistribute to
to lists,
lists, requires
requires prior
prior specific
specific
permission and/or
and/or aa fee.
fee.
(Section 4). Our approach uses a combination of techniques.
permission
AST ’11,May
AST’11, May23-24,
23-24,2011,
2011,Waikiki,
Waikiki,Honolulu,
Honolulu,HI, HI,USA USA First, we employ test and event generators to construct test
Copyright 2011 ACM 978-1-4503-0592-1/11/05 ...$10.00 ...$10.00.

77
cases and sequences of events. We then run these test cases renamed to Robotic Space Rock) is an OpenGL-based 3D
(and feed the events, respectively) to the application. Once game. GuessTheNumber [6] is a number guessing game.
a test case is running, we record detailed information about The results of the study are presented in Table 1. The first
the application in the system log file; after each test case run, column contains the application name, the second column
we perform a log file analysis to detect potential bugs. To shows the first release we analyzed, and the third column
demonstrate the effectiveness of our approach, in Section 5 shows the last release we analyzed; source code was col-
we present an evaluation on the open source applications lected from Google Code [13] for each application. The rest
that form the object of our bug study. We generated test of the columns (4–11) show the bug counts for each applica-
cases for all projects used in the bug study and compared tion, categorized by bug type; the bug reports were retrieved
bugs we found with bugs reported by users. We detected from Google Code [13]. The grey-background columns (4–6)
most bugs reported, and found new bugs which have never are the bugs we focus on in this paper. We now provide a
been reported. description of each bug type.
In summary, our work tackles the challenges of verifying Activities are the main GUI components of an Android
mobile applications and makes two contributions: application; an activity error (column 4) usually occurs due
to incorrect implementations of the activity protocol, as ex-
1. A bug study and categorization of Android-specific plained in detail in Section 4.4.1. Event errors (column 5)
bugs that shows an important number of Android bugs occur when the application performs a wrong action as a re-
manifest themselves in a unique way that is different sult of receiving an event (details in Section 4.4.2). Dynamic
from traditional, e.g., desktop/server application bugs. type errors (column 6) arise from runtime type exceptions
(details in Section 4.4.3). Unhandled exceptions (column
2. An effective approach for detecting Android GUI bugs, 7) are exceptions the user code does not catch and lead to
based on a combination of test case and event genera- an application crash. API errors (column 8) are caused by
tion with runtime monitoring and log file analysis. incompatibilities between the API version assumed by the
application and the API version provided by the system.
2. ANDROID BUGS: A STUDY I/O errors (column 9) stem from I/O interaction, e.g., file
or card access errors. Concurrency errors (column 10) oc-
To identify the most frequent Android bug categories, we
cur due to the interaction of multiple processes or threads.
performed an empirical study (bug collection and categoriza-
Bugs categorized as other (column 11) are due to errors in
tion) on 10 popular applications in the Android Market—the
the program logic.
official repository for Android applications. In selecting the
The bug categories are non-overlapping, i.e., a bug listed
applications for our study, we used several criteria: applica-
in the table cannot belong to more than one category. The
tions had to be popular, have a long lifetime, have a detailed
only exception to this rule was a bug in ConnectBot 1.46,
bug history and have the source code available. The 10 ap-
which can be categorized as both a type error and an API er-
plications that form the target of our bug study are available
ror (we have categorized it as a type error). As we can see in
for free in Android Market, have high download counts, and
Table 1, many errors are program logic related errors, some
cover most of application categories, which ensures we get a
of which can be found using standard techniques such as
broad range of representative bugs.
static analysis or model checking. However, these techniques
The time frame of our analysis, for each application, is
cannot always be applied directly to mobile applications—
shown in columns 2 and 3 in Table 1. As Android OS has
their structure and libraries differ substantially from stan-
only been available for 2.5 years (since August 2008), these
dard applications; also, building or extracting a model for
applications have had a relatively long lifespan, which has
each application could be time-consuming. Therefore, we
given developers and users a chance to detect and report
prefer an automated approach that can detect a variety of
issues; hence it allows us to observe a wide range of bugs.
bugs, without any per-application effort. We illustrate our
We now provide a brief description of each application.
approach by showing how it can be used to detect activity,
Opensudoku [8] is a popular Sudoku game which allows users
event, and type errors. In the remainder of the paper we
to download and create their own puzzle in the game; the
present the tools and techniques we used to find such errors,
aim of Opensudoku was to create a framework which can be
and illustrate how our techniques have re-discovered some
used for any kind of game, and provide as much functional-
known issues, as well as found new bugs.
ity and flexibility as possible. Skylight1 [10] is a Java mobile
projects framework and collection of Android mobile appli-
cations and demos. CMIS [2] is a browser, which enables 3. ANDROID OVERVIEW
the user to browse and search CMIS (Content Management We now proceed to presenting an overview of the Android
Interoperability Services) repositories. Delicious [3] (our ab- platform and the components of an Android application. As
breviation for Android delicious bookmarks) allows users to shown in Figure 1, the Android platform is composed of 4
save bookmarks to the Delicious social bookmarking service layers: Applications at the top, an Application Framework
from the Android web browser. ConnectBot [4] is a Secure layer that provides services to applications, e.g., controlling
Shell client, which allows Android users to securely connect activities or providing data access, a Library/VM layer, and,
to remote servers. DealDroid [5] is a small application for at the bottom, the Linux kernel.
Android devices that continuously watches for new deals on Applications run at the very top of the platform. Services
deal sites; it runs in the background and produces Android for applications, e.g., the Activity Manager, which controls
notifications when new items become available. Rokon [9] activities for each application, or Content Providers which
is a 2D game engine, intended as a flexible game creation load the content provider defined by each application while
framework with several demo games embedded. Andoku [1] restricting data accessibility across applications are located
is Sudoku-type puzzle game. MonolithAndroid [7] (recently in the Application Framework layer. The Library/VM layer

78
 Application First Last Bug category
analyzed analyzed Activity Event Type Unhandled API I/O Concur- Other
release release exception rency
Skylight1 Aug. 2009 July 2010 3 2 0 1 0 0 0 4
CMIS Jan. 2010 Apr. 2010 0 0 0 2 0 0 0 6
Delicious Feb. 2009 June 2010 0 0 0 0 1 0 0 4
ConnectBot Aug. 2008 July 2010 2 8 2 5 1 3 1 57
DealDroid Mar. 2009 May 2009 1 1 0 0 0 0 0 8
Rokon Sep. 2009 July 2010 0 6 2 3 0 4 0 14
Andoku July 2009 July 2010 0 0 0 0 0 0 0 1
Opensudoku Apr. 2009 July 2010 1 1 0 0 0 0 0 5
GuessTheNumber Feb. 2009 Nov. 2009 1 1 0 0 0 0 0 0
MonolithAndroid Dec. 2008 Jan. 2010 0 2 0 0 2 0 0 3
Total 8 21 4 11 4 7 1 102

Table 1: Android applications: study time frame, bug categories and bug counts.

application. A Content Provider manages data for a cer-

tain application and controls the accessibility of the data;
for example, an email client may make email addresses in
its database accessible to other applications. Broadcast Re-
ceivers listen and react to broadcast announcements. For
example, an email client may receive a notification that the
battery is low and, as a result, proceed to saving email drafts.
Though we believe our approach is general enough to facil-
itate bug detection for all component classes, in this paper
we focus on GUI bugs related to activities and GUI events.

4. APPROACH
Our dynamic analysis approach combines several tech-
niques, from automatic test case and event generation tools
to log file analysis. In Figure 2 we provide an overview.
Starting from an application’s source code, we first use JU-
nit [15], a Java test case generation tool, to generate test
cases. Since most applications in the Android Market are
Figure 1: Architecture of Android platform. GUI-based, for each test case, we may need to add some
events (simulating user interaction) to make the application
move from one state to another. Therefore, we use Mon-
contains static libraries and the Android runtime environ- key [16], an automatic event generation tool, to produce
ment. Static libraries provide common system and applica- events in both random and deterministic ways and feed these
tion libraries for applications. The Android runtime environ- events to the application. Once a test case is running, we
ment is composed of core runtime libraries and the Dalvik record detailed information about the application in the sys-
virtual machine (VM)—an optimized Android-specific Java tem log file; after each test case run, we perform a log file
virtual machine. Finally, the Linux kernel completes the analysis to detect potential bugs.
OS and the software stack. Each Android application runs
with a unique user ID, in its own copy of the Dalvik virtual 4.1 Test Case Generation
machine, which ensures separation between applications and JUnit is a testing framework for Java applications, in-
provides protection. tegrated in the Android development environment. JU-
Our work covers the top three layers in Figure 1. To test nit can generate several classes of test cases based on the
programs running in the Application layer, we use system application source code. Since activities are the main en-
services from the Application Framework layer and instru- try points and control flow drivers in Android applications,
mentation tools in the Dalvik VM. our test case generation is based on activities. We first
Android applications can be composed of four component identify all activities in an application and then use the
categories: Activity, Broadcast Receiver, Content Provider Activity Testing class in JUnit to generate test cases for
and Service. Activities are focused windows in which the each activity. Activity Testing is shipped with the Android
user interaction takes place; only one activity can be active SDK, works in conjunction with JUnit and provides three
at a time. Each activity is a class in the source code and features:
should perform according to events generated by users and
system. Services run in the background, e.g., an email client • Initial condition testing tests whether the activity is
may check for new mails while users are running another created properly.

79
 Figure 2: Overview of our approach.

• GUI testing tests whether the activity performs ac- has an associated “pattern,” as explained next. These pat-
cording to the GUI specification. terns can indicate proper operation, or they can indicate
a bug. Apart from automatic bug detection, log files are
• State management testing tests whether the applica- also useful in debugging—since the log file contains method
tion can properly enter and exit a state. and event traces, leading to the bug, developers can use our
We used all three features for identifying activity bugs. framework to reconstruct the method sequence that lead to
For more effective GUI tests, we used an event generation a bug.
tool, explained next.

4.2 Automatic Event Generation 4.4.1 Detecting Activity Bugs

Automatic event generation is a powerful technique for Activities are window containers derived from an Activ-
verifying GUI applications. GUI bugs are revealed by event ity superclass; their implementations consist of responding
sequences that fall outside the set of permissible events as- to events generated by users and the system. Activity bugs
sociated with the current state of the GUI application. To stem from incorrect implementation of the Activity class,
help generate GUI events, we use the Monkey event gen- e.g., one activity might be created or destroyed in the wrong
erator, which comes with the Android SDK. Monkey can way so that it will make the application crash. In general,
generate random or deterministic event sequences and feed activity bugs occur either because developers are not suffi-
these events to an application. To discover a wide range of ciently familiar with the activity- and event-based applica-
issues, in our work we use random sequences: we generate tion model in Android, or because the implementation fails
these sequences using Monkey, and feed the sequences to the to obey the activity state machine. In practice, almost every
application under test. application we analyzed has activity bugs because it is hard
to check whether each base function of the base class has
4.3 Trace Generation been properly implemented.
Once the test cases are generated, we run them on the ap- An activity has a life cycle described by a state machine,
plication through the Dalvik VM. To monitor the execution hence violations of this state machine lead to activity bugs.
of test cases, we configure the VM to log the details of each A simplified version of the state machine is shown in Fig-
test case into a trace file. Our traces capture three kinds of ure 3; the full state machine can be found on the Android
events: GUI events, method calls, and exceptions. We also developer website [12]. Each activity can be in one of five
monitor the VM operation to detect application bugs that states: Active, Pause, Stop, Restore or Destroy. If an activ-
cause the VM to shut down prematurely. ity occupies the screen’s foreground, it is running, hence in
the Active state. If another non-full screen or transparent
4.4 Log File Analysis and Bug Detection activity overlaps the current activity, the current activity
Once test cases are generated for a certain application, will be moved into the Pause state. An activity is in state
we run the application on these test cases and log the per- Stop once it is fully covered by another activity. Activities
formance of each test case so that we can detect errors. in states Stop or Pause can be killed by system if memory
With the log file at hand, we use patterns to identify po- is needed elsewhere. If the activity is killed and the user has
tential bugs. Each class of errors (activity, event or type) restarted it again after some time, that activity will be in

80
 Program Activity Event Type
Pause bugs bugs errors
Old New Old New Old New
Skylight1 3 0 2 3 0 0
CMIS 0 0 0 0 0 0
Restore Active Destroy Quit Delicious 0 0 0 0 0 0
ConnectBot 2 2 6 2 2 0
DealDroid 1 0 0 0 0 0
Rokon 0 0 6 0 2 0
Start
Stop Andoku 0 1 0 0 0 0
Opensudoku 1 0 1 1 0 0
GuessTheNumber 1 0 1 0 0 0
Figure 3: Simplified state machine of an Android MonolithAndroid 0 0 2 0 0 0
activity. Total 8 3 18 6 4 0

Table 2: Results: old (re-discovered) bugs and new

(not previously reported) bugs.

4.4.2 Detecting Event Bugs

Android applications should be prepared to receive events,
and react to events, in any state of an activity. If developers
fail to provide proper implementations of event handlers as-
sociated with certain states, the application can either enter
an incorrect state or crash outright. Figure 5(b) is a log file
example extracted from ConnectBot release 80, which shows
how the application crashes when Monkey feeds it with an
unhandled event (HOME button click).

4.4.3 Detecting Type Errors

Detecting type errors is quite simple: once the type error
has been triggered, a ClassCastException entry will appear in
the log file. A type error in ConnectBot release 236 is shown
in Figure 5(c).

5. RESULTS
Our verification techniques turned out to be effective in
Figure 4: Screenshot of ConnectBot activity failure. practice. In Table 2 we report the number of bugs we found
using our approach. For each class of bugs we were able
to re-discover bugs already reported (the Old columns) as
state Restore and then Active. Once an activity needs to be well as new bugs that have not been reported yet (the New
killed, it will be in the Destroy state. columns). For assurance, we took each event sequence our
To ensure a correct state sequence, e.g., Start→ Active→ approach has found automatically and played it manually,
Pause→ Restore→ Active→ Destroy, the corresponding user- through GUI interaction, to make sure the bug can actually
defined activity methods should be called in a valid order as be reproduced in practice. We have reported the new bugs to
specified by the state machine, in this case: onCreate()→ the developers; two of the bugs have been confirmed, while
onPause()→ onResume()→ onDestroy(). We use the state others are in the process of confirmation.
machine as a specification and match method calls from log
file entries against it. Violations of the state machine are 5.1 Activity Bugs
then flagged as potential bugs. We were able to detect all 8 activity errors that have al-
For example, in ConnectBot release 256 we found a new ac- ready been reported (Table 1), as well as 3 new activity
tivity bug, indicated by the log file entries shown in Figure 5 errors in Andoku and Connectbot.
(a). The bug in Figure 5(a) manifests itself as an onCreate()
on line 1 without a subsequent onPause() preceding line 3, 5.2 Event Bugs
which is a violation of the state machine specification. The Our technique has detected 24 event errors (18 existing
bug corresponds to a situation where the user sets up a de- bugs and 6 new bugs). However, Table 1 contains 21 event
fault shell host beforehand and then starts the application, errors; the three errors we could not detect could not be
which crashes the application. Figure 4 is a screen shot of reproduced by other users either, so we suspect they might
the application crash when the scenario described above un- be spurious bug reports.
folds.

81
(a) Activity bug in ConnectBot release 256

1 E/AndroidRuntime( 190): at org.connectbot. SettingsActivity .onCreate( SettingsActivity . java :29)

2 E/AndroidRuntime( 190): at android.app.ActivityThread .performLaunchActivity(ActivityThread . java:2364)
3 I/ActivityManager( 52): Process org .connectbot (pid 190) has died .
4 D/AndroidRuntime( 211): Shutting down VM W/dalvikvm( 211):
5 threadid=3: thread exiting with uncaught exception (group=0x4001aa28)
6 E/AndroidRuntime( 211): Uncaught handler: thread main exiting due to uncaught exception

(b) Event bug in ConnectBot release 80

1 I/ Starting activity : Intent {action=android.intent . category .HOME}

2 ...
3 D/:Shutting down VM

(c) Type error in ConnectBot release 236

1 D/PackageParser( 63): Scanning package: /data/app/vmdl57744.tmp W/Resources(

2 63): Converting to int : TypedValue{t=0x3/d=0x11 ”Donut” a=1} W/PackageParser(
3 63): /data/app/vmdl57744.tmp W/PackageParser( 63):
4 java . lang .NumberFormatException: unable to parse ’Donut’ as integer
5 ...
6 W/PackageParser( 63): at android.os.HandlerThread.run(HandlerThread.java:60)
7 E/AndroidRuntime( 1555):java.lang.RuntimeException:java.lang . ClassCastException : java . lang .Long;
8 D/AndroidRuntime: Shutting down VM

Figure 5: Bug manifestation: examples of log file entries for each bug category.

5.3 Type Errors data misuse during runtime for an Android application. In
We successfully detected all type errors in Table 1, but we their work they tested 30 popular applications and found
could not find any new ones—we suspect that, since type several cases of information leak, e.g., leaking user contacts
errors are critical and cause the applications to crash imme- information. Chaudhuri [18] presented a formal study of An-
diately, they are more likely to be observed, reported and droid security. Their work introduced a core typed system
fixed quickly. for describing Android applications, and reasoning about
their data flow security properties. Any violation of rules
described in the core typed system and operational seman-
6. FUTURE WORK tics may result in security concerns of applications. These
The last column of Table 1 shows bugs that do not fall approaches are deployed at the OS level and focus on secu-
into the activity/event/type categories. Some examples in- rity, whereas we focus on providing developers with a toolset
clude unhandled exceptions, API errors, I/O errors, or con- for detecting GUI errors.
currency errors. We plan to pursue a two-prong approach
for detecting these classes of bugs. First, we will model the GUI testing. Kervinen et al. [26] present a formal model
correct invocation of certain I/O and concurrency primitives and architecture for testing concurrently running applica-
as state machines [21], which will allow us to compare ap- tions. The behavior of the system under test is specified
plication logs with the model and find I/O and concurrency as a labeled transition system. Their specification model is
errors. Second, we plan to graft our model-based verification more rigorous and powerful than ours, as our model is se-
onto Java static analysis tools, e.g., WALA [11], to permit quential, rather than concurrent. They used their system to
pattern and state-machine violations at compile-time, rather test mobile applications running on the Symbian platform
than via automated testing. and found 6 bugs in those applications.
GUITAR [14] is a GUI testing framework for Java and Mi-
7. RELATED WORK crosoft Windows applications. Hackner and Memon [25] de-
scribe the architecture and overview of the GUITAR tool—
Android verification. Most of the prior work on verifica- the components of GUITAR and their functionality. It is un-
tion of mobile applications has focused on security. Enck, clear to us whether GUITAR can be applied directly to An-
Ongtang, and McDaniel’s [20] describe Kirin, a logic-based droid, because of Android’s application development model.
tool for Android that ensures permissions needed by a cer- Yuan and Memon [30] generate event-sequence based test
tain Android application are met by global safety invari- cases for GUI applications. They proposed a model-based
ants. These invariants ensure that data flow of each appli- approach for testing GUI-based applications. Their tech-
cation can be passed safely among applications. Ongtang, nique can generate test cases automatically using a struc-
McLaughlin, Enck and McDaniel [28] have proposed a sim- tural event generation graph. Their approaches target Java
ilar tool called Saint, which enforces OS-level data flow se- desktop applications, which are quite different from the An-
curity. Enck, McDaniel, Jung and Chun [19] provide a real- droid mobile environment.
time data flow monitor called TaintDroid, which can check

82
Android bug studies. Maji et al. [27] performed a failure [13] Google code, April 2010. http://code.google.com/.
characterization study on two mobile platforms, Android [14] Guitar – a gui testing framework, August 2010.
and Symbian. They collected data on bugs in the OS, mid- http://guitar.sourceforge.net/index.shtml.
dleware/library, and development tools/core applications for [15] JUnit, May 2010. http://www.junit.org/.
these two platforms. They found that defect density tends [16] Monkey UI/Application Exerciser, May 2010.
to be lowest in the OS, higher in the middleware, and highest http://developer.android.com/guide/developing/
in development tools/core applications. They also computed tools/monkey.html.
the cyclomatic complexity for the two code bases and found [17] Androlib. Number of New Applications in Android
the values to be comparable across all layers. They stud- Market by month, September 2010.
ied the code fixes and found that many of the fixes required http://www.androlib.com/appstats.aspx.
just a few lines of code. Our study is centered around a
[18] A. Chaudhuri. Language-based security on android. In
complementary set of applications (third-party, non-core),
PLAS ’09, pages 1–7.
since this is the type of applications prevalent on the An-
droid Market. Their study is focused on bug location (which [19] W. Enck, P. Gilbert, B. gon Chun, L. P. Cox, J. Jung,
subsystem contains the bug) as well as the fix (what source P. McDaniel, and A. N. Sheth. Taintdroid: An
code change was made to fix the bug), whereas our bug cat- information-flow tracking system for realtime privacy
egorization looks at the semantic nature of the bug (e.g., monitoring on smartphones. 2010.
activity, event, type error, concurrency, etc.). Our approach [20] W. Enck, M. Ongtang, and P. McDaniel.
is oriented on dynamic, rather than static verification. We Understanding Android security. IEEE Security and
do not study fixes and source code, but rather provide an Privacy, 7(1):50 –57, Jan.-Feb. 2009.
automatic testing framework that can be used to discover [21] D. Engler and K. Ashcraft. Racerx: effective, static
activity bugs, event bugs and type errors. detection of race conditions and deadlocks. SOSP,
pages 237–252, 2003.
[22] Gartner Corporation. Gartner Says Worldwide Mobile
8. CONCLUSIONS Device Sales Grew 13.8 Percent in Second Quarter of
The number of mobile applications and mobile applica- 2010, But Competition Drove Prices Down, August
tion users are growing rapidly, which creates an impetus for 2010.
researchers and developers to come up with effective verifica- http://www.gartner.com/it/page.jsp?id=1421013.
tion techniques to ensure the reliability of these applications. [23] Gartner Corporation. Gartner Says Worldwide Mobile
Towards this goal, we perform a bug study to understand the Phone Sales Grew 17 Per Cent in First Quarter 2010,
nature and possible remedies for bugs in mobile applications, May 2010.
and construct an automated testing framework for Android http://www.gartner.com/it/page.jsp?id=1372013.
applications. Our framework combines automatic event and [24] Gartner Corporation. Gartner Says Worldwide Mobile
test case generation with runtime monitoring and log file Phone Sales Grew 35 Percent in Third Quarter 2010;
analysis. Our techniques have proved effective for activity, Smartphone Sales Increased 96 Percent, November
event, and type errors: we have been able to re-discover ex- 2010.
isting bugs while finding some new bugs. We believe our http://www.gartner.com/it/page.jsp?id=1466313.
framework can be easily extended to find a broader range of
[25] D. R. Hackner and A. M. Memon. Test case generator
bugs in Android applications.
for guitar. In ICSE Companion ’08, pages 959–960.
[26] A. Kervinen, M. Maunumaa, T. Pääkkönen, and
9. REFERENCES M. Katara. Model-based testing through a gui. In
[1] andoku. http://code.google.com/p/andoku/. Formal Approaches to Software Testing, volume 3997,
[2] android-cmis-browser. pages 16–31. 2006.
http://code.google.com/p/android-cmis-browser/. [27] A. Kumar Maji, K. Hao, S. Sultana, and S. Bagchi.
[3] android-delicious-bookmarks. http://code.google. Characterizing failures in mobile oses: A case study
com/p/android-delicious-bookmarks/. with android and symbian. In Software Reliability
[4] connectbot. Engineering (ISSRE), 2010 IEEE 21st International
http://code.google.com/p/connectbot/. Symposium on, pages 249–258.
[5] dealdroid. http://code.google.com/p/dealdroid/. [28] M. Ongtang, S. Mclaughlin, W. Enck, and
P. Mcdaniel. Semantically rich application-centric
[6] guessthenumber.
security in android. In ACSAC’09: Annual Computer
http://code.google.com/p/guessthenumber/.
Security Applications Conference, 2009.
[7] monolithandroid.
[29] M. Swift. Android operating system is expected to
http://code.google.com/p/monolithandroid/.
surge past rivals. Los Angeles Times, Sep 11 2010.
[8] opensudoku-android.
[30] X. Yuan and A. M. Memon. Generating event
http://code.google.com/p/opensudoku-android/.
sequence-based test cases using gui runtime state
[9] rokon. http://code.google.com/p/rokon/. feedback. IEEE Trans. on Software Engineering,
[10] skylight1. http://code.google.com/p/skylight1/. 36:81–95, 2010.
[11] T. J. Watson Libraries for Analysis (WALA). http://
wala.sourceforge.net/wiki/index.php/Main_Page.
[12] Android activity lifecycle, May 2010.
http://developer.android.com/reference/
android/app/Activity.html#ActivityLifecycle.

83