Vol. 2, Issue 3, May 2007, pp.

112-132

Adapting Usability Investigations for

Agile User-centered Design
Desirée Sy
Autodesk, Inc. (formerly Alias) Abstract
210 King Street East When our company chose to adopt an Agile
development process for new products, our User
Toronto, ON, M5A 1J7 Canada
Experience Team took the opportunity to adjust, and
[email protected] consequently improve, our user-centered design (UCD)
practices.
Our interface design work required data from
contextual investigations to guide rapid iterations of
prototypes, validated by formative usability testing.
This meant that we needed to find a way to conduct
usability tests, interviews, and contextual inquiry—both
in the lab and the field—within an Agile framework.

To achieve this, we adjusted the timing and granularity

of these investigations, and the way that we reported
our usability findings. This paper describes our main
adaptations.
We have found that the new Agile UCD methods
produce better-designed products than the “waterfall”
versions of the same techniques. Agile communication
modes have allowed us to narrow the gap between
uncovering usability issues and acting on those issues
by incorporating changes into the product.

Keywords
Permission to make digital or hard copies of all or part of this work usability method, Agile, XP, iterative development,
for personal or classroom use is granted without fee provided that
copies are not made or distributed for profit or commercial
software, case study, field study, contextual inquiry,
advantage and that copies bear this notice and the full citation on ethnography, formative usability testing, user-centered
the first page. To copy otherwise, or republish, to post on servers design, iterative design.
or to redistribute to lists, requires prior specific permission and/or a
fee. Copyright 2006, ACM.
 113

Context Agile development: a brief primer for

usability practitioners
As more organizations adopt Agile development
practices, usability practitioners want to ensure that the Agile programming methods are a family of software
resulting products are still designed with users in mind development processes [15,16,17] that share a
[1,2,3,4,5,6,7]. This was the situation of our User common philosophy (more fully described by the Agile
Experience Team at Alias (now Autodesk) in 2002. Our Manifesto at http://agilemanifesto.org/). Examples of
group’s goal-directed rapid design process was Agile principles include valuing the following:
successful, and we wanted to continue to design
ƒ individuals and interactions over processes and tools
usable, engaging and innovative user experiences while
“working Agile.” ƒ working software over comprehensive
Members of our team, along with developers, were documentation
trained in Agile software development practices by the ƒ customer collaboration over contract negotiation
Cutter Consortium [8]. The method for gathering user
feedback taught Agile team members to conduct a ƒ responding to change over following a plan.
focus group after a feature was implemented, In a traditional development lifecycle (sometimes called
demonstrate the product, and ask for users’ opinions. waterfall lifecycle), requirements gathering for all
This method was insufficient for our design process, features in a release leads to a design phase, which is
which relies heavily on observing detailed user behavior then followed by coding and quality assurance testing.
with interactive usability investigation methods, such as The entire process for a release is measured in months,
formative usability testing [9,10] and contextual inquiry if not years.
[11] before implementation.
In contrast, the Agile development lifecycle is
Our initial approaches to conducting Agile usability characterized as a series of incremental mini-releases.
investigations had similarities to methods described by (See Figure 1.) Each mini-release, with a subset of the
Holtzblatt, Wendell, and Wood [12] and some of the features for the whole release, has its own
case studies reported by McInerney and Maurer [5], but requirements analysis, design, implementation, and
with modifications. The differences were previously quality assurance phases, and is called a working
described by Miller [13] (but for an audience of version. Each working version must be complete and
developers unfamiliar with usability methods), and by stable, which makes it possible for the product release
Sy [14] (but with emphasis on the design iterations date to coincide with that of any working version.
rather than the investigation methodology). Working versions are created at regular intervals, called
iteration cycles or sprints, which are generally two to
This paper describes our adaptations to the timing,
four weeks long. Cycle end dates are fixed; features
granularity, and reporting used for Agile interactive
that cannot be completed are moved to the next
usability investigations, with an intended audience of
working version.
usability practitioners.
 114

Figure 1. In a waterfall development cycle, analysis, design, coding, and quality assurance testing are separate stages of a software
release that spans months or years. In Agile development, each of a set of incremental mini-releases (each created in 2-4 weeks) has
these stages. Adapted from Cutter Consortium [8].

At the beginning of each iteration cycle, the full, cross- determine when that feature is complete, and also
functional Agile team meets to do cycle planning. They includes a time estimate for completion.
determine the theme, or user story, of the next
The Agile team meets daily at a short, stand-up
working version, and the features to put in it. Future
meeting (sometimes called a scrum), where team
cycles remain more loosely planned, since each cycle
members each describe what they are working on, and
planning session is based on the most current
any blocking issues. Scrums, through face-to-face
information.
communication, take the place of detailed documents
Cycle planning is guided by an overall vision or plan for to guide project planning.
the release. At Alias, Agile teams did release-level
Working versions are periodically delivered to users or
planning during a brief 4- to 6-week phase called Cycle
to customers to validate the acceptance criteria for the
Zero. The first iteration immediately follows Cycle Zero.
feature cards. Their feedback influences current cycle
Each feature (the smallest development component, as implementation, and directs future cycle planning.
defined by developers) is described on an index card
Note that in Agile terminology, a customer is not a
called a feature card. Feature cards are grouped into
person external to the product team who purchases or
iteration cycle clusters, and displayed in a public space
uses the product, but a role filled by one or more
for the whole Agile team as a communication artifact, in
members of the product team. The duties of the Agile
lieu of more traditional planning documents. Each
customer include acting as the voice of the end-user on
feature card describes the acceptance criteria that
the development team, and helping to prioritize and
 115

plan cycles and releases. Miller [18] suggests that Changes to the timing of usability
interaction designers who are willing to understand and investigations
accept Agile development concepts are well-suited to
Problems with the timing of waterfall UCD
take on the Agile customer role.
Previously, in waterfall development projects, the User
This is particularly relevant to Agile usability Experience Team conducted usability investigations as
investigations because Agile projects are highly early as we could during the development cycle.
feedback-driven, yet product teams often rely on user
We performed contextual inquiry (sometimes combined
opinion in situations where observation is more
with field usability testing of the prior release) before or
appropriate (such as the focus group elicitation strategy
at the onset of a project, often during a market
described earlier). Usability practitioners can be the
validation for a release. During the design phase, we
best-suited members of an Agile team to prevent this
would rapidly iterate on key designs for a release, using
type of data bias because of their skills in gathering and
formative in-house usability testing to direct the re-
analyzing user experience data. On the Agile projects
design of prototypes [19,20]. We would then describe
that our User Experience Team works on, interaction
the validated designs as feature specifications, and
designers assume the role of the Agile customer.
pass them on to development to be incorporated into
The iterative and incremental lifecycle of Agile the implemented code.
development methods described in Figure 1 are similar
In theory, the analysis and design phases preceded the
to those of other iterative development processes (such
implementation (see Figure 2). However, in practice,
as the Rational Unified Process). They differ mainly in
developers would begin coding at the onset of a project
the length of the iteration timeboxes (in Agile,
without waiting for feature specifications. The result
measured in weeks, rather than months), the fixed
was that the implementation of some features would
nature of the cycle end dates, and the highly
begin before they were designed. To combat this
collaborative and document-light form of project
tendency, we investigated, designed, and validated well
planning and implementation. (There are other
in advance, often conducting usability investigations
differences as well, but these affect developers rather
almost a full release ahead. However, this led to writing
than usability practitioners.)
many unused or out-of-date feature specifications,
Because of the similarities in their development since we could not anticipate all planning issues (such
lifecycles, the adaptations to usability investigations as business goals).
described in this paper may also benefit usability
practitioners working on projects using an iterative
development process.
 116

Figure 2. In the perfect theoretical version of waterfall development, usability investigations contributing to the analysis and design
phases were supposed to precede coding, but in reality developers would begin coding immediately.

Also, since months often would pass between when we We were caught between wanting to push our
specified a design and the time it was coded, requirements gathering as late as possible, so that the
implementation would sometimes drift from the design most timely information would inform the product
intent. direction, and yet not leaving it so late that too many
undesigned features started being implemented [13].
Finally, development for all features going into a
release would begin simultaneously, with one developer Just-in-time design
working on each feature. Because there were more In contrast, on Agile projects the Agile team only
developers than interaction designers, the result was focuses on a few new features at a time. This means
that some features in a release were designed while that the User Experience Team does not have to work
other features were not. Furthermore, because work on on all the designs in a release at the same time.
all features was partially implemented, waterfall Instead, we can focus on the most important designs, a
products often shipped with incomplete features, few at a time.
despite delaying the release date to try and
At any given time during the Agile development cycle
accommodate feature completion.
for a release, we conduct usability activities for only
those key designs. We then work closely with
 117

developers to ensure that implementations of the Two parallel tracks: iterating the design and
designs do not drift from the validated design intent. implementation separately, but simultaneously
A key principle of our User Experience Team’s UCD
Because developers are working on only a subset of
process is design iteration; we need to be able to catch
features at one time, and interaction designers are
design failures early, change designs as many times as
designing the same subset, this also means that any
needed, and then incorporate the design fixes [19].
features that require careful UCD work receive it. Since
everything in the product must be done, traces of half- Therefore, we not only need to conduct formative
complete features don't impede the user experience. usability tests to check our prototypes, but we need to
do so before coding begins, while the design is still
In waterfall UCD, field investigation data would usually
malleable. Because coding begins immediately in Agile
not have a visible product impact until the next release.
development, we needed to find a way to separate
Just-in-time design spreads out contextual inquiry and
design iterations from implementation iterations.
field work through the whole development process,
instead of concentrating those activities at the To do this, the UCD work was done in an Interaction
beginning of the lifecycle (or in the lifecycle of the Designer Track while developers worked in a separate
previous release). Consequently, the data that we bring and parallel Developer Track [13,14]. The Agile UCD
to Agile projects during cycle planning is up-to-the- parallel tracks for development and interaction design
minute. This allows product improvements to be are illustrated in Figure 3.
implemented within the current release—sometimes
even in the next working version.
 118

Figure 3. To allow the User Experience Team to iterate on designs, we usability tested prototypes at least one cycle ahead of
developers, and then passed on the validated designs to be implemented. We would also conduct contextual inquiry for workflows at
least two cycles ahead, and usability test the implemented working version to check for design drift.

Usability investigation activities in Cycle Zero exploratory designs for market validation. Based on
Cycle Zero is the brief requirements-gathering phase at these data, deriving the design principles that
the start of the project. Usability investigation activities inform and guide design decisions for the product.
in Cycle Zero depend on whether the product is the
ƒ (For an ongoing release) Analyzing and summarizing
next release of an existing product or completely new.
prior contextual inquiry and usability test data.
They can include the following activities:
Based on these data, elucidating release-level
ƒ Gathering data to refine or hone product- and design goals to inform and guide design decisions
release-level goals. Facilitating the alignment of all through all iterations.
team members’ understanding of these goals, so ƒ (For a completely new market or capability)
they constitute a shared vision. Developing brief and vivid descriptions of target
ƒ (For a completely new product) Interviewing or users and workflows (light personas and scenarios)
conducting contextual inquiry during customer site from investigations.
visits for market validation. Preparing high-level
 119

For example, we conducted an investigation during Usability investigation activities in iteration cycles
Cycle Zero for SketchBook Pro v2.0, a digital sketching In Cycle 1, usability investigation activities can include:
application developed specifically for use with tablet
ƒ Designing prototypes for Cycle 2, and conducting
input devices. Version 1.0 of the product had a free
trial version. The release goal of v2.0 was “to improve rapid formative usability testing to refine their
the rate of conversion of trial users to paid users by design.
removing barriers to purchase.” The User Experience ƒ Conducting contextual inquiry and interviews to
Team helped to focus the feature set by conducting a investigate designs for Cycle 3.
survey targeted at people who had downloaded
During the first few early cycles, to give interaction
SketchBook v1.0, but who had not purchased it. These
designers time to do these usability investigations,
data helped refine the feature set for v2.0 from over
developers work on coding software architecture (which
100 potential features to five major workflows. The
requires no user interface design) or important features
release goal also informed design prioritization during
that need only minor design.
cycle planning throughout the release lifecycle.
For example, during Cycle 1 for SketchBook Pro v2.0
Usability activities for Cycle Zero of the first release of
developers worked on adding Adobe Photoshop export.
Autodesk Showcase (a real-time automotive 3D
This was identified as a key issue that affected users’
visualization product) were different. We helped the
purchasing decisions. It required significant
team prepare for a market validation trip to Europe,
development effort, but had a very simple UI (adding
and also traveled there with the project manager and
an item to the File Save As type list).
subject matter expert. We interviewed potential
purchasers about their work activities for the areas that In Cycle 2, the designs from Cycle 1 are presented to
the new product would support. We then reported developers, who begin coding them. Interaction
these back to the larger team. Also, these data were designers work closely with developers to answer
the foundation for the design principles we wrote for questions about the design as it is implemented.
Autodesk Showcase that allowed us to make Cycle 2 usability investigation activities can include:
prioritization and design decisions as development
progressed. ƒ Prototyping and usability testing for Cycle 3 designs,
using the requirements information gathered in
Cycle Zero usability activities are those that most
Cycle 1.
closely resemble their waterfall UCD counterparts.
However, they occur in weeks rather than months. ƒ Contextual inquiry to investigate designs for Cycle 4.

ƒ Usability testing the implemented working version

from Cycle 1.
 120

This pattern of designing at least one cycle ahead of usability test design chunks in the Interaction Designer
developers, and gathering requirements at least two Track, carrying the progressively built design forward in
cycles ahead, continues until the product is released. this track until it is complete. Then, we pass the
finished design to the Developer Track for
In cycles later in the release, while we continue to
implementation [14].
focus on checking the implementation of designs, we
can also begin some contextual investigations to Interaction designers are trained to consider
prepare for the Cycle Zero of the next release [13,14]. experiences holistically, so breaking designs into
pieces—especially into pieces that do not initially
Changes to the granularity of usability support workflows—can be difficult at first, but it is a
investigations skill that comes with practice. Design chunking yields
Problems with the size of the problems to investigate many benefits in Agile UCD, which we will describe in
The parallel tracks allowed the User Experience Team later sections.
to iterate designs before they were implemented. To deconstruct a large design into smaller pieces, it is
However, we still had to deal with the reality of cycles essential to start with well-defined design goals and to
that were only two to four weeks long. We could understand the high-level design intent. Our design
complete small designs in this timeframe, but complex goals are derived from observation, which is why
designs required more than four weeks to finish. We contextual inquiry plays a critical role in our Agile UCD
needed to figure out how to do usability investigations process. Each design chunk lets us progressively
for designs spanning more than one Agile cycle. achieve a subset of the design goals.
Furthermore, the overall speed of Agile UCD was much The priority and sequence of the design chunks is
faster than when we were doing waterfall UCD. We had determined by what we can validate at any given time
to move much more quickly toward design solutions in the product lifecycle. We examine the full list of
with a fewer number of usability tests within a release. design goals, and decide which we can attain with the
Design chunking: Breaking designs apart into current resources within a cycle’s length. There is also
cycle-sized pieces an ordering dependency. In Agile projects, components
We looked to the Agile model of implementation for build on one another, so early design chunks must be
hints about how to approach this dilemma. Working low-level and fundamental—design attributes that will
versions are implemented mini-releases that not change as more design chunks are added on top of
incrementally build on each other. Based on the same them. For example, for SketchBook Pro v2.0, we
principles, we decided to create mini-designs that needed to design the ability to move, rotate, and scale
incrementally build on each other. a selected area in a canvas. Contextual investigation
during usability testing of the prior release told us that
We break large designs into small, cycle-sized pieces these functions were experienced as one single high-
called design chunks that incrementally add elements level activity (positioning and fitting a selection).
to the overall design. We investigate, prototype, and
 121

Other observations were that users didn’t tend to need To evaluate these design alternatives, we asked
this functionality in the first few hours of use in the internal users (quality assurance, training, support, or
application, and that it was often used with 2D layers. subject matter experts) to use high-fidelity prototypes
to rotate images, and observed the interaction. The
A few of the design goals that we derived from these
usability acceptance criteria for the design chunk
investigations included:
included checking whether users could rotate the
ƒ Cursor travel should be minimized when swapping canvas accurately without instruction, and whether the
between move, rotate, and scale. interaction felt smooth to them. We could get feedback
from each tester in less than two minutes—we did not
ƒ The interaction should feel natural and smooth.
even have to set up informal usability test sessions to
ƒ Users should be able to figure out how to position a achieve this. We could just drop by someone’s desk
selected area with out-of-box materials, but not with a tablet PC that had the prototypes installed, and
necessarily in the first hour of use. ask for a few minutes of their time.

ƒ The function should work seamlessly with layers.

These goals suggested an order of approach. We could
not validate the interaction of positioning a selection
with layers, for example, until users could work with a
high-fidelity prototype that incorporated the new
functionality with the existing layer feature. This clearly
was a much later stage of design for this feature, so we
designated it as a late design chunk.

It was possible, however, to design and validate the

kinesthetic qualities for the move, rotate, and scale
modes right away, within a 2-week period using only
internal resources. This was the earliest design chunk
that we worked on for this functionality.
For example, we examined two possible dragging Figure 4. An early design chunk for Move/Rotate/Scale
algorithms for the Rotate interaction. In one, dragging selection investigated two ways that dragging the cursor could
the cursor along the virtual canvas was like putting work for the Rotate mode. In one, dragging acted like a virtual
your hand on it and spinning it. In the other, dragging hand on the canvas. In the other, dragging along an axis
rotated the canvas.
the cursor along an axis would rotate the canvas—this
was like pulling along an invisible Rotate slider.
(Figure 4)
 122

The interaction for the Move, Rotate, and Scale modes chunk, we could address other more interdependent
were good choices as early design chunks for the design problems.
following reasons: In contrast, the types of design chunks that we usually
complete in later cycles include the following:
ƒ We could prototype them very quickly. Our student
intern coded more than a dozen variations for this ƒ Prototypes that require an implementation or
design chunk within a cycle. technology that hasn’t been completed yet.
ƒ It was easy to usability test them. In-house testers ƒ Design chunks that provide workflow-level, rather
were sufficient to evaluate the usability criteria, and than operation-level, functionality.
the prototypes for all modes took less than 15
minutes to test. ƒ Design chunks to support any investigation of a
discoverability or learning goal, such as the design
ƒ We anticipated that some set-up or explanation of how a user will access a new function. Since
would be needed to run the prototypes, and we these are designs that depend on the first
knew that the testing tasks would be highly artificial, experience of a user, you need to replicate that
and operation-level, rather than workflow-level. For experience to test them. These prototypes should be
example, we told our testers that we had two incorporated into a copy of a working version to
different algorithms for rotating (without explaining avoid task bias.
how each one worked), and we had to intervene to
switch between the two prototypes. This was fine for ƒ Design chunks that are hubs for other designs. For
internal users, but it would be inappropriate and example, many different designs converged in the
confusing for an external end-user to evaluate these Brush Palette, which is why it was one of the last
prototypes. designs that we completed for SketchBook Pro v2.0.
At the same time that we break big designs into these
ƒ The interaction was a fundamental, low-level design
small chunks, we are still completing small designs to
component. There could be many different ways
pass to development for the next iteration cycle. Key to
that we would approach how to position a selected
our success as interaction designers on Agile teams is
area, but no matter what happened for later design
that we keep ahead of development, feeding a steady
chunks, we would need to decide on the drag
stream of designs into the Developer Track. For this
interaction.
reason, we only use design chunking for a few key
ƒ The design problem could be validated in isolation. large designs per interaction designer in a release.
For the design goal we were looking at (a natural
All of the illustrating examples in this article are slightly
and smooth interaction), we didn’t have to evaluate
simplified for clarity. It is possible to chunk more
the three modes in combination, so prototypes for
complex designs than the one described. This is
each mode could be quickly built separately. Once
because design chunks are not complete designs. They
all the modes were combined in a later design
are simply design components that can be prototyped,
 123

iterated, and validated within Agile timeframes. By contextual information as we progress. In essence,
design chunking, we do not ignore high-level design we need to mine more ore while drilling fewer holes.
considerations; instead, we work toward them in cycle- To overcome these hurdles, we took the Agile idea of
sized steps. progressive mini-iterations one step further. In addition
Progressive refinement of protocols: breaking usability to design chunking, we also progressively performed
testing, contextual inquiry, and recruiting into cycle- the following activities:
sized pieces
ƒ defined test protocols
Agile UCD presents particular challenges in protocol
design for usability investigations, because of two ƒ recruited test participants
considerations:
ƒ conducted contextual investigations.
ƒ The progressively incremental character of Just as working versions are mini-implementations that
both implementation and design. It is one thing get progressively closer to the final release, and design
to decide to design in chunks that build chunks are mini-designs that get progressively closer to
incrementally, but how is it possible to validate and the final designs, we use usability testers who get
investigate many small pieces for different designs progressively closer to our end-users, ask them to let
simultaneously? It is impossible to usability test us observe them doing work that gets progressively
early-release design chunks with external users, and closer to their real-world activities, and then ask them
seemingly impossible to conduct meaningful to do those activities as usability test tasks [21].
contextual investigations to understand the work
Because usability investigations are a limited resource,
users might do with them. Yet we needed to explore
we need to both maximize the information we collect
our users’ work domains to derive relevant
per session, and to hoard the validation commodity of
activities, both to design later design chunks and
our external users. We reserve external users to test
also to provide real-world (and hence, unbiased)
only mid- to late-stage design chunks, and the focus of
validation activities.
those usability tests is on validating design goals that
ƒ The fixed number of usability investigations can only be determined by an actual user.
that fit within the timeframe of a cycle. Because
For the earliest design chunks, as described in the
Agile development is faster than waterfall, the time
previous section, we use in-house users who share
to create a release is briefer. Thus, we have fewer
some characteristics with our end-users (that is, people
usability tests per design that we did in waterfall
who are not developers, and with the same domain
UCD, and in particular, we have fewer opportunities
knowledge as our users). We ask them to do operation-
to test full workflows before they are implemented.
level tasks that would probably not occur in isolation in
Yet, we need to uncover more information during
real-world work (such as arbitrarily rotating an image
each investigation, since we need to collect
to different angles, for no reason).
 124

In later design chunks that have prototypes capable of students, or students with digital sketching experience
evaluating more holistic activities, we refine usability from industrial design programs or art colleges.
test activities. Beginning with our internal testers, we
We continue to refine our usability test tasks with these
prepare very artificial tasks. For example, as shown in
participants, asking them whether the activities we ask
Figure 5, we asked SketchBook Pro testers to position
them to do represent how they might really use the
and resize the shapes to fit within the appropriate
tool, and adjust the protocols for later testers.
boundary lines. This was a completely arbitrary and
unnatural task. We admitted this to internal usability Finally, when late-stage design chunks are available
testers, and then asked for a more realistic example of that can emulate partial in-product workflows, we bring
when they might use the function. We used their these workflow prototypes to actual end-user sites for
examples as our next iteration of the usability test task. usability testing. We call these users Design Partners.
They commit to evaluating a longitudinal series of
workflow prototypes, and sometimes also working
versions. They act as expert usability testers, and also
as observable beta testers. (Users who cannot commit
to a series of visits are used to test mid-stage design
chunks at our lab.)

For usability testing of workflow prototypes by Design

Partners, we start with the more realistic test tasks
derived from external usability testers. We often
progress to using real work files and true work
activities.
There are several progressive stages of contextual
investigations that we use for Agile UCD work, but all
Figure 5. We asked usability testers to fit the shapes within are based on the idea of examining a user’s work
the boundary lines to test a mid-release design chunk. context for a specific activity in retrospect. These
methods are described more fully elsewhere [21].
With these newer test tasks, we then recruit usability
testers external to our company, and bring them in- Stages of contextual investigation include the following
house to our usability lab. Because the actual users of activities:
our products who are accessible to us are in limited
ƒ Interviewing and watching internal users. This
supply, and we want to reserve them for late-stage
provides us with the relevant vocabulary and
design chunks, we find people who could be (but are
framework to structure our interviews with external
not) users. For example, these testers sometimes
users.
include people studying in our users’ work domains. For
SketchBook Pro, we recruited computer animation
 125

ƒ Asking external users to bring work artifacts Here are three hypothetical examples of usability
(identified by a remote interview) to our company. investigation sessions for different design chunks of
This does not yield any environmental data about several complex designs:
our users, but can still provide us workflow context.
ƒ (Early design cycle) In-house usability test,
ƒ Asking Design Partners to set up work artifacts at with internal users. A 15-minute session where a
their site. We visit them to watch both a high-level QA person evaluates 6-8 very low-level prototypes
walkthrough of the artifacts, and a compressed for two designs by performing operation-level tasks.
workflow demonstration to view detailed During the session, we ask for a better activity-level
interactions. task for each tool.
ƒ Installing a working version at a Design Partner’s ƒ (Mid-release design cycle) In-house usability
site, and then watching an artifact walkthrough and test, with external users. A one-hour session.
compressed workflow demonstration (on- or off- Before the test, we interview the tester by telephone
site). about a workflow for a future design (two cycles
This last stage—the ability to observe how using the later), and ask her to bring some relevant files on a
actual implemented product changes the work behavior USB drive. We begin the session with the contextual
of users—is unique to Agile contextual investigations. investigation, watching her walk us through the files
This contextual data can inform future designs within and demonstrate key interactions. We also usability
the timeframe of a release. Comparable waterfall test four prototypes exploring different stages for
contextual inquiry sessions could only guide designs for two design chunks we are designing in the current
the following release. cycle. During the session, we check that our test
activities are representative. If needed, we will
Mixing design chunks: studying more than one design
adjust the tasks for the tester coming in the next
problem at a time
day.
There seem to be an overwhelming number of usability
investigation activities for any given cycle in the ƒ (Late design cycle) Usability investigation, at a
Interaction Designer Track described in Figure 3. Design Partner’s site. A 2-hour session, the
second in a series of longitudinal visits to the same
Design chunking is what gives us the freedom to solve
user. Before our visit, we ask the tester to set up
the problem of how to do so much at the same time in
some files to show us what type of work he was
fewer sessions. We combine smaller-scaled
doing with the working version that we left with him
investigations for different design chunks into single
on our last visit. We also interview him concerning a
usability investigation sessions.
workflow for a future design (for the next release),
and ask if he can show us the relevant files. When
we arrive, we observe as he walks us through the
files and demonstrates the interactions. We note
 126

that he is using an implemented feature in an We need to communicate the following information to

unexpected way, and he gives us permission to the Agile team:
bring back the files so that we can demonstrate this
ƒ which designs we are working on, and
new workflow to the Agile team. We have brought
two workflow prototypes on a laptop, and he approximately when developers should expect to
usability tests them for us. We install the new receive them
working version on his system, and watch a usability ƒ usability test results and high-level progress for
test of the two new functions that have been added late-stage design chunks
since our last visit.
ƒ recommendations and fixes for working versions
Breaking designs down into cycle-sized chunks gives us
the freedom to mix and match different types of ƒ user and task information from external users,
usability investigations into the same session, which especially from field visits
enables us to juggle more than one design, and more
ƒ the user interface design to be implemented.
than one type of usability investigation at the same
time. Design chunks are what allow us to elicit more Examining this list, we realized that this information
data from fewer users. was only of value to the Agile team once it was in the
product, not when we wrote it down. In a sense, it was
Changes to the reporting of usability interim data and did not need to be captured for
investigations posterity. In fact, since its purpose was to instigate
action, written text was not the best medium to achieve
Oral storytelling: showing versus writing
this. We needed to convey it to the team in a more
Agile processes value “working software over
timely, vivid, and specific manner.
comprehensive documentation” [22], and yet
traditional deliverables from usability investigations We patterned our solution on how other Agile team
(such as formal UI specifications, use case scenarios, or members communicate. For developers and project
usability test reports) are formal documents. To deliver managers on an Agile team, documents such as
on the prized Agile quality of rapid response, our User marketing and product requirements documents are
Experience Team had to figure out how to reduce the largely replaced by daily scrums and cycle planning
number of detailed documents that we wrote. We re- sessions, with the displayed feature cards acting as a
evaluated the content and form of our communication reference artifact for these discussions, supplemented
with the project team to re-align it with Agile principles. by brief documents. We do the same.
Design Cards: communicating to plan
We give high-level progress reports on upcoming
designs at the scrum. Upcoming designs are
represented on the planning boards as design cards,
 127

which are blue to differentiate them from feature cards, ƒ feature requests
and have no implementation time estimates [14].
ƒ major usability problems with design prototypes
In addition to keeping in touch with the whole Agile
team through the daily scrum, we work with developers ƒ bugs in working versions.
very closely throughout design and development. With these stories and demos, we replace personas
Although the dual tracks depicted in Figure 3 seem with people, and scenarios with workflows and sample
separate, in reality, interaction designers need to work files.
communicate every day with developers. This is not
We use index cards as a reporting artifact for these
only to ensure that designs are being implemented
data, so the team is reminded of the presentation
correctly, but also so that we have a thorough
during later cycles. To prepare for this, on the way back
understanding of technical constraints that affect
from a Design Partner site, the interaction designers
design decisions [13].
write out individual usability issues, feature requests,
Issue Cards: communicating to persuade and bugs on issue cards [14].
We report information gathered from our Design
After we present the information to the Agile team,
Partners the day after the visit in a review meeting that
team members decide what to do about feature
happens after the scrum. Any interested Agile team
requests or unexpected uses of the product. Sometimes
members can stay for the presentation, and it’s usually
an issue card is moved immediately into the cycle
the full team. This is far more people than those who
planning board, and becomes a feature or design card.
read our reports in the waterfall UCD days, and a wider
Any bugs are logged in our bug-tracking database.
cross-section of the team than those who used to listen
to our debrief presentations. It often includes, for The remaining issue cards are tracked on a User
example, technical writers and QA people. Experience board in the same public space as the cycle
planning board. (See Figure 6.) On each issue card, the
We present the following kinds of information in the
interaction designers note any fixes for a design issue,
form of verbal stories, supplemented where necessary
the usability testers who found it, and the iteration
by a demonstration of the prototypes or working
progress.
versions to represent the observed interactions:

ƒ successes with design prototypes or working

versions, and examples of our users’ work in the
application

ƒ contextual information about users’ workflows,

especially unexpected uses of the product
 128

If the final prototype is a high-fidelity one, we also pass

the code to the developer as part of the specification.
With the developers, we then work out what feature
cards will represent the implementation of the design.
For a simple design, this may be only one or two cards.
We ensure that usability criteria are defined as part of
the acceptance criteria for the feature card before it is
added to the cycle planning board.
For a complex design, we help ease the transition to
feature cards by breaking the whole design into
suggested implementation chunks. Whereas the order
of design chunks depends on what can be validated
Figure 6. Issue cards are posted on a User Experience board. with the current resources, the order of implementation
As prototypes are fixed, the cards move from the Fix in chunks depends on creating working versions that
prototype area to the Fixed & works area of the board.
enable a user to complete a workflow. The interim
Issue cards move on the board from Usability issues to workflows will not be ideal, or even allow users to do
Fix in prototype when the prototype design changes. If what they want to, but working versions should enable
the change solves the problem, then the card moves to users to complete actions that they can evaluate.
the Fixed and works area of the board.
We work with developers to finalize the implementation
The User Experience board also tracks feature requests order, and they compose feature cards based on this.
from Design Partners that are not moved to the cycle Since they are responsible for completing the work on
planning boards as feature cards. feature cards, and for creating the time estimates,
developers own them. Once again, we ensure that
Moving to Feature Cards: communicating to implement
usability criteria are part of the acceptance criteria.
Finally, when designs are completed, the interaction
designers pass them to developers. Our Agile design A factor that has allowed our User Experience Team to
deliverables are also different than the comprehensive modify our deliverables in this manner is that we were
feature specifications that we used to write for waterfall in the same building as the rest of the Agile team. A
development [14]. future challenge we may face is how to capture the
immediacy and vividness of our storytelling with team
We present the design in person to the developer(s)
members in other locations.
who will implement it. We demonstrate a typical
workflow on the last available prototype using data Documents are for the design team
from a Design Partner. We explain in detail any Although the User Experience Team writes far fewer
additions or changes to the design from this prototype. documents working on Agile projects than we did for
waterfall projects, we have not eradicated the written
 129

word entirely. However, we had a key insight: we ƒ As working versions are completed, the design
ourselves are the primary readers of design documents. chronology is extended to include additional
We can, therefore, write the shortest possible workflow information and links to bugs related to
documents that still capture the information that we changes in design, or unexpected uses.
need to reference [14]. To give an idea of the length of this type of light
This information is the design history for a feature or specification, the Move/Rotate/Scale selection feature
set of features. The purpose of the record is principally for SketchBook Pro was written as two documents (one
to avoid “design thrash,” where design decisions are for each implementation chunk). One described the
accidentally re-visited, particularly between versions, or look and interaction behavior for the Move/Rotate/Scale
if a new User Experience Team member is added to a UI widget, and the other described the interaction of
project. this widget within a selected area of the canvas. The
description sections of these documents (excluding the
We generally write one document for each
design chronology and feature cards) were,
implementation chunk. The document is written in a
respectively, 1,215 and 735 words long.
medium that is easy to update (such as a Wiki page).
Design history documents are available to the whole
Information in a design history document can include
Agile team, but we have found that few team members
the following:
read them, preferring the immediacy of face-to-face
ƒ Design goals and a brief description of the problems conversation to clarify issues.
that the design addresses.
Reflections
ƒ A high-level description of the design, including
Five years ago, our User Experience Team faced the
rough sketches, and a pointer to the last design
challenge of adjusting our practices. We didn’t
prototype.
anticipate it then, but now we prefer Agile user-
ƒ Links to related design history documents. centered design for the following reasons:

ƒ A reverse chronology of the design iterations, ƒ More of the product is designed than before.
including the reasons for the design changes, and
ƒ Usability investigations are conducted throughout
design limitations and constraints defined by
usability investigation data as design work proceeds. the entire product release lifecycle, rather than
Relevant usability investigation data are recorded in clustered at the front end of a release, or in the
this chronology. This section of the document is prior release.
written as design cycles progress. The oldest entry ƒ The most important designs are worked on first, and
in this design chronology describes technology there is no effort wasted writing unused designs.
constraints.

ƒ The names of the associated feature cards.

 130

ƒ Product changes suggested by usability testing and Practitioners’ take-away

contextual inquiry investigations of the actual
Here are some practical implications of the concepts
product can be implemented in the current release.
discussed in this paper:
ƒ Just as formative usability test results allow us to
ƒ For our User Experience Team, Agile user-centered
iterate on the design of a product, now the Agile
design resulted in better-designed software than
team’s responsiveness to contextual inquiry results
waterfall user-centered design. Agile communication
allow us to iterate on the requirements for a
modes narrowed the gap between gathering
product.
usability data and acting on it.
ƒ Implementations of designs are more true to the
ƒ Because Agile development is highly feedback-
design intent. The overall quality of the user
driven, product teams may rely on user opinion in
experience in our Agile products is higher than in
situations where user observation is more
the waterfall products we worked on.
appropriate. Usability practitioners can be the best-
Much of the power that drives user-centered design is suited members of an Agile team to mitigate this
generated from observational data gathered in usability bias because of their skills in gathering and
investigations. By adjusting the way that we derived analyzing user experience data.
and reported these data, we could harness its power to
the Agile characteristics of speed, responsiveness, and ƒ It is possible to use the familiar arsenal of usability
high implementation quality to better fuel the UCD investigation methods on Agile (and other iterative
engine. development) projects, including formative usability
testing, user and task analysis, interviews, and even
Acknowledgements field-based work like contextual inquiry. This is
achieved by changing the timing and granularity of
When working Agile, it’s best to make haste slowly and
the investigations, and how results are reported.
take a moment for reflection. I thank Autodesk for
giving me some of that time. The other key ƒ Just as Agile (and iterative) development implement
contributors on our User Experience Team to the software as incremental mini-releases, usability and
described process innovations were John Schrag and design activities can be scoped as incremental mini-
our fearless leader, Lynn Miller. They, Veronica Meuris, designs. Different validation and elicitation activities
Stephen Gaebel, Amy Kidd, Peter Liepa, and Catherine can be blended within single sessions conducted at a
Courage read early drafts, and helped me to clarify the usability lab or in the field. Design activities occur at
article. However, all errors and omissions are my own. least one Agile cycle or sprint ahead of the
Illustrations in the article are by Marsha Leverock. development team in an Interaction Designer Track
separate from the Developer Track. Developers
receive validated designs.
 131

ƒ Prototype demonstrations and daily conversation [9] Dumas, J., and Redish, J. (1999). A Practical Guide
have largely replaced detailed documents, such as to Usability Testing (revised edition). Bristol, U.K.:
Intellect.
usability test reports and UI specifications, when
communicating with the product team. Documents [10] Rubin, J. (2001). Handbook of Usability Testing:
How to Plan, Design, and Conduct Effective Tests.
are now written for interaction designers, to record
New York, NY: John Wiley & Sons.
a history of design decisions.
[11] Holtzblatt, K., and Beyer, H. (1996). Contextual
References design: defining customer-centered systems. San
Francisco, CA: Morgan Kaufmann Publishers.
[1] Constantine, L. L. (2002). Process Agility and
Software Usability. Information Age, August 2002. [12] Holtzblatt, K., Wendell, J.B., and Wood, S. (2005).
Rapid Contextual Design. San Francisco, CA:
[2] Beck, K. and Cooper, A. (2002). Extreme Morgan Kaufman/Elsevier.
Programming vs. Interaction Design. Retrieved on
December 8, 2006 from [13] Miller, L. (2005). Case Study of Customer Input For
www.fawcette.com/interviews/beck_cooper/ a Successful Product. Proceedings of Agile 2005.
Denver: Agile Alliance.
[3] Patton, J. (2004). Interaction Design Meets Agility.
Retrieved on December 8, 2006 from [14] Sy, D. (2005). Strategy & tactics for Agile design: a
www.agilealliance.org/system/article/file/1368/file. design case study. Proceedings of UPA 2005.
pdf Montréal: Usability Professionals’ Association.
[4] Pearson, G., and Pearsall, S. (2005). Becoming [15] Highsmith, J. A. III (2000). Adaptive Software
Agile: Usability and Short Project Cycles. User Development: A Collaborative Approach to
Experience, 4(4), 2005. Managing Complex Systems. New York, NY: Dorset
House Publishing Co., Inc.
[5] McInerney, P., and Maurer, F. (2005). UCD in Agile
Projects: Dream Team or Odd Couple?, [16] Beck, K. (2000). Extreme Programming Explained:
interactions, 12(6), November + December 2005, Embrace Change. Boston, MA: Addison-Wesley
19-23. Professional.
[6] Lee, J.C. (2006). Embracing Agile Development of [17] Schwaber, K., and Beedle, M. (2002). Agile
Usable Software Systems. Proceedings of CHI Software Development with Scrum. Upper Saddle
2006. Montréal: ACM. River, NJ: Prentice Hall.
[7] Lievesley, M.A., and Yee, J.S.R. (2006). The Role of [18] Miller, L. (2006). Interaction Designers and Agile
the Interaction Designer in an Agile Software Development: A Partnership. Proceedings of UPA
Development Process. Proceedings of CHI 2006. 2006. Denver/Broomfield: Usability Professionals’
Montréal: ACM. Association.
[8] Highsmith, J. A. III (2002). Agile Software [19] Schrag, J. (2006). Using Formative Usability
Development (course notes). Arlington, MA: Cutter Testing as a Fast UI Design Tool. Proceedings of
Consortium. UPA 2006. Denver/Broomfield: Usability
Professionals’ Association.
 132

[20] Medlock, M., Terrano, M., Wixon, D. (2002) Using

the RITE Method to Improve Products: A Definition
and a Case Study. Proceedings of UPA 2002. Desirée Sy has been crafting useful
Orlando: Usability Professionals’ Association. and enjoyable user experiences for
[21] Sy, D. (2006). Formative usability investigations over 16 years, the last 11 of them at
for open-ended tasks. Proceedings of UPA 2006. Autodesk (formerly Alias). Customers
Denver/Broomfield: Usability Professionals’ and trade reviewers have described the
Association. user interfaces she’s designed as both
[22] Beck, K. et al. (2001). Manifesto for Agile Software innovative and a pleasure to use. She
Development. Retrieved on December 7, 2006 from has presented and taught on topics, such as Agile user-
agilemanifesto.org centered design, gathering good usability data,
[23] Highsmith, J. A. III (2004). Agile Project usability investigations for open-ended tasks, and low-
Management. Boston, MA: Addison-Wesley/Pearson cost usability investigations at UPA and ACM SIGDOC.
Education.