Evaluating

Prototypes
P
rototyping is a fast, low-cost, low-risk, learning-rich This document describes 5 steps to the evaluation of
approach to surface and test promising responses prototypes:
to tough challenges. The approach originated in the
technical fields of engineering, software design, and manu- 1. Confirm the Prototype
facturing, and has since spread to the arena of social inno- 2. Develop Questions
vation where diverse groups are experimenting with new 3. Design Methods
ways to tackle vexing social and environmental challenges. 4. Implement & Adapt
5. Make Decisions
Prototyping precedes, rather than replaces, conventional
pilot projects. Unlike pilots, where a promising intervention These five steps are iterative. It is a framework, not a
is “fixed” for a longer period of time and assessed through recipe, that can guide social innovators and evaluators
thorough evaluative techniques, prototyping can be used to in designing an evaluation for their prototype. It is also
quickly and inexpensively develop and test ideas that may a work in progress: it will be constantly updated based
warrant eventual pilot testing. In some unusual cases, the on the experiences of The Natural Step organization in
results of prototype tests may be sufficiently robust that developing its Sustainability Transition Lab model over
innovators can skip a full-fledged pilot study altogether. the course of 2016 and 2017.

Social innovators are now using prototyping processes in an

impressive variety of settings:

„„ Different types of interventions (e.g., technology, busi-

ness processes, policies)
„„ Diverse domains (e.g., criminal justice, environmental
sustainability)
„„ Unique processes (e.g., human-centered design, quality
improvement, replicating ‘best practices’)
While the art and science of prototyping is growing, com-
paratively little attention has been paid to how to evaluate
the prototypes that emerge from the process.

1
Step 1: Confirm the Prototype

T
he first task in evaluating a prototype is settling on the prototype you
want to evaluate. This is trickier than you might think and requires
social innovators and evaluators to answer three basic questions:

Do you actually have a prototype?

A prototype is a physical or experiential representation of an idea and how
it might play out in the real world. A description of it on a white board, ipad,
Members of a management team use lego clip chart or memo will not do. Even very rough prototypes that are tangible
to create multiple rough prototypes of a new enough to evoke feedback from would-be users is sufficient to kickstart the
business process
evaluation process.

What type of prototype is this?

There are a lot of different types of prototype, each with a different purpose
and features. As the table on page 4 shows, these include rapid versus slow
prototypes, exploratory versus developmental prototypes, one-off or throw-
away prototypes versus evolutionary and incremental ones.

This will influence how extensive an evaluation you will carry out. For ex-
The urbanistas at the Urban Block Founda- ample, the creators of a lego representation of a new business process can
tion create a temporary “European” block
benefit from rapid and light-weight feedback from their peers in a room. By
of trees, cafes and bike lanes in an industrial
neighborhood in Dallas to engage residents contrast, the creators of an elaborate mock-up of a new pedestiran-friendly
in a conversation about urban design. urban neighborhood will require something more planned and sophisticated.

What parts of the prototype do you want

to test?
Social innovators should be clear if they want to test the whole protoype or
just parts of it. The creators of a new small -scale wind turbine may want to
get feedback on the entire process of producing, marketing, distributing and
servicing the product. Alternatively, they may want to focus just how it might
be set up and maintained by the consumers that buy it.
An independent inventor creates a small-
scale wind turbine to demonstrate how
home owners can create their own Social innovators and evaluators should not proceed to the next phase of the
micro-generator.
evaluation process until they can agree on the answers to these questions.

2
TABLE 1: THE CONTINUUM OF EXPERIMENTS
IDEA RAPID PROTOTYPE FIELD PROTOTYPE PILOT DECISION

Surfacing new Making ideas Testing the A full, robust, A decision to adopt,
ideas tangible “manifestations” of longer term test scale, or let go of an
an idea in the field of the idea idea

• Concept • Story Boards • Simulations • Pilot projects • (In)formal adoption

Paper • Client walk • Working samples • Demonstration of policies, regula-
• Presentations through of all or parts of projects tion, technologies,
• Role playing new model • Trials practices and be-
• Lego or paper haviors that survive
• Simulations the earlier phases.

Example from the Better Block Foundation’s Work on

Urbanizing Forgotten Dallas Neighborhoods (www.betterblock.org)

The group The group sketch- The group creates The group elects City Council change
surfaces ideas es out what ideas a mock up block to carry out a score of ”an-
on how to might look like with trees, bike more in-depth ti-street life” reg-
make Dallas through vignettes lanes, cross walks, experiments on ulations, the com-
neighborhood and diagrams. patios and stories key prototypes. munity introduced
more vibrant. in forgotten neigh- bike path, and
borhood. local entrepreneur
opened business
based on prototype
mock up.

3
TABLE 2: DIFFERENT TYPES OF PROTOTYPING

TYPE DESCRIPTION EXAMPLE

A process to help innovators make ideas A new group of professionals and clients of the family
tangible, learn by doing, and quickly test justice system develop and sketch out a half-dozen dif-
Rapid
whether they hold promise for further de- ferent ways that the legal services can be made more
velopment. affordable for low-income families.

A process to develop and test a new idea or An oil and gas company diversifying into other forms
model that requires changes in the capacity of energy technology (e.g., solar, thermal, wind) must
Slow
or culture of the innovating organization in also build new skills, networks and knowledge in order
order to succeed. to be successful in those industries.

Developing and testing an idea primarily to An environmental group develops a clean technology
help groups learn more about the complex prototype to surface the kind of regulatory barriers
Probing (aka system in which they are operating and/or and opportunities involved with introducing new tech-
Throwaway) trying to change. Once the experiment is nology to the market.
over, the prototype is ‘thrown away’ in the
sense it may not be further developed.

A group of local foundations interested in testing a

An early form of prototype, usually a
Exploratory collaborative granting process create a story board
visualization of a concept or idea, tested
(aka Proof of that describes each step in detail. It then tests the
to determine whether it warrants further
Concept Testing) feasibility of the process with a role-playing exercise
development and testing in the field.
with prospective grantees.

A local transit authority expanding its light rail transit

An applied or field test of an idea, either the
Developmental line tests the compatibility of its new signalling system
entire model or a specific component, in the
(aka Live) with the signalling systems of the older line using
real world.
several types of relational software.

A construction company interested in building “net

A building block approach where an inno- zero” ready homes develops, tests and refines dif-
vator adds and tests components of a more ferent elements of the housing unit over time (e.g.,
Incremental
complex model or solution incrementally water, electrical, heating). It adds and integrates each
over time. element as it emerges until the house is considered
“net zero” ready.

Built in an iterative and incremental manner

Evolutionary A cell phone company develops successive versions of
in an effort to more fully develop and refine
a new cell phone to test customers’ responses to an
a model or intervention. May result in pro-
ever more sophisticated set of features.
totype 1.0, 1.1. 1.2, 2.0, 3.0, etc.

4
Step 2: Develop Questions

Q
uestions form the foundation of an evaluation of a a prototype. Questions determine
which methods and indicators will be used to test the prototypes. The table below contains
some typical questions asked of prototypes, with a list of illustrative responses generated
by a series of “potential users and supporters” of a hypothetical wind turbine, developed by
social innovators interested in options for locally-owned micro-generation.

Question Example
To what extent is this prototype likely to be effective People agree that the only way that this thing would work is if we put it
in achieving what we want to achieve?  20 metres tall or higher.

To what extent is this prototype likely to be feasible These devices are easy to build and they seem as easy to set up as
in the real world?   satellite dishes.
The Prototype

There appears to be very little in the way of municipal regulation of this

To what extent is this prototype likely to be viable in
type of technology: it’s a pretty grey area, which may or may not work
the current context (economic, political, social, etc.)?
in our favour.

To what extent is this prototype likely to be support- We did not test this in any meaningful way – we should do so in our
ed by key stakeholders? next iteration and round of testing.
To what extent is this prototype scalable for bigger The executive director of a large environmental organization estimates
impact? there is potential for 50% market penetration in the city: let’s consider
  doing a market analysis.
Alot of the people we talked to said that they were looking for ways for
The Challenge

What are we learning about the challenge we are

homeowners to get involved in micro-generation in general, not just
trying to address?
this windmill. They gave us three new ideas for micro-generation.

What are we learning about the broader context and/ Local community leagues have a lot of influence with City Hall. We
or systems in which our challenge and opportunity should consider engaging them if we choose to develop the next
are embedded? prototype.

What did we learn about the capacity of our innova- Boy, our team is more risk adverse than I thought: we were really ner-
The Team

tion team? vous about developing this idea.

What has changed in the working relationship Everyone on the team agreed that we worked well together and that
amongst our team members? we should consider doing some new prototypes together.

It is important for social innovators to develop clear questions. “I just want to know what people
think!” or “What are the chances this will work?” are a good start. However, they require more
crafting in order to get the kinds of response required to truly test the prototype, and provide guid-
ance in the development of more sophisticated methods for assessment.

The National Science Foundation User-Friendly Handbook for Mixed Method Evaluation provides
a template for prioritizing and eliminating evaluation questions. See http://betterevaluation.org/
resources/guide/design-process_mixed_method_evaluations
5
Step 3: Design Methods

T
here is no cookie cutter set of indicators or methods to eval-
uate prototypes. The evaluation design for the urban char-
rette, new business model, and client services highlighted on
page 2 of the document each require a customized set of tools and
indicators.

There are, however, principles to guide social innovators and evalua-

tors when crafting an evaluation design:

1. Relevant. The methods must be designed to answer the key

questions of the prototype team.

2. Credible. The evaluation design should be considered legitimate Social innovators might be able to draw on
by the people using the evaluation feedback. (For example, social these principles to craft evaluations on their
innovators may feel that feedback from engineers on a new car- own – particularly for evaluating rapid pro-
bon-capture technology is more appropriate than the opinions of totypes. For prototypes that require a more
municipal administrators.) sophisticated design, they may require the sup-
port of an evaluator.
3. Appropriate burden of Proof. The methods should generate
data that meets an appropriate burden of proof. For example, Whether simple or sophisticated, all evaluation
the burden of proof in an exploratory prototype is much lower design for prototypes has to answer the follow-
than in a pilot project where the stakes and risks are higher. (See ing four questions:
next page for more details.) „„ What are the preferred data sources and
method for each evaluation question?
4. Quality. The evaluation design should meet the basic quality „„ When do the innovators require the feed-
standards of the Canadian Evaluation Society. back?
„„ What things should be considered in mak-
5. Timely. The results of the assessment should be made available ing the evaluation useful?
in ‘real time,’ that is, provided in such a way that social innova- „„ Who is the for coordinating and managing
tors can incorporate the feedback into their decisions about the the evaluation?
next steps for the prototype.

The Canadian Evaluation Society lays out clear standards for quality evaluation
http://www.evaluationcanada.ca/evaluation-standards 6
 An Escalating Burden of Proof

Rapid Prototypes Field Prototypes

Evaluating rapid prototypes should employ quick and As the investment, risks and stakes of the innovation
light-weight feedback methods, reflecting the low risk increase, so does the need to complement user feed-
stage and stakes at this stage of the process, and focus back with more structured research on key questions.
primarily on getting rapid feedback from would-be users While direct user feedback is still central in proto-
or participants about a new idea. types, the questions and methods tend to be more
robust and intensive.
Example Methods
1. Appreciative Inquiry: a way of providing real-time Example Studies
feedback, organized around positively framed ques- 1. Feasibility Analysis: A study to test select the
tions: e.g., I like this part of the prototype: have you technical, operational, economic and political
thought about [concern, question, critique]? feasibility of a new idea, venture or model.

2. Ritual Assent-Dissent: a structured process by 2. Process Evaluation: An exploration of the dif-

which two teams provide rounds of positive and ferent ways of designing and delivering a new
then negative feedback. model, surfacing challenges and how they might
be resolved.
3. Red Team-Blue Team: a role-playing game used in
military and security fields in which two groups are 3. Outcome Evaluation: A focus on testing the likely
organized to “defend or attack” an idea, model or outcomes of the model in the real world.
approach.
Higher
Effort, Risk & Stakes

More Efforts, Risk & Stakes = Higher Evaluation Burden of Proof

Lower Burden of Proof

7
An Example of Escalating Burden of Proof
CARBON CAPTURE
TECHNOLOGY & BUSINESS

A
hypothetical account of how an industrial firm developed and tested the (real) idea for
a new carbon capture technology, through different experimental stages, from rapid
prototype, to slow prototype to pilot project.

RAPID PROTOTYPE FIELD PROTOTYPE PILOT

Multiple rapid feedback ses- A series of technical assessments A series of more detailed tests
sions with peers, potential to test select features of the carbon to assess operations, partner
customers and government capture technology to determine capacity and the market for
regulators on a variety of whether it might work well enough to the product of an ‘end to end’
schematics, drawings and warrant a full pilot experiment. demonstration plant, a last step
models of the process. to determine whether to build a
commercial plant.

8
Step 4: Implement & Adapt

I
mplementing an evaluation design is rarely as simple as
‘plan the work, work the plan.’ In fact, there are three
major reasons that social innovators and evaluators
may need to adapt the design and implementation of their
evaluation plan:

„„ The original design is insufficient: A group realizes

quickly that adjustments are required to provide good „„ The prototype evolves: Social innovators are always
answers to the question. (“We thought we could test upgrading their prototypes and even little adjust-
this with a focus group of municipal administrators, ments may open up the need for new methods
but it turns out that we need to talk to the planning and data. (“We had not considered the option of
department as well, and it’s best done with one-on-one creating neighborhood-based cooperatives for this
interviews.”) technology. How would that sell?”)
„„ Deeper questions emerge: The very act of testing
a prototype will result in new questions about that As a general rule, the evaluation for any prototype is a
prototype. (“We realize that that there are a lot more work-in-progress whose design co-evolves as quickly or
regulatory barriers that we first anticipated. How are as slowly as the questions and data expectations of the
we going to get a handle on them all?”) social innovators.

W
hile rapid prototypes are usually developed Plan-Do-Study-Act (PDSA) Cycle
and evaluated in a single meeting or session, This highly structured methodology for experimentation
the work of preparing and assessing field originated in Total Quality Management. Since then it
prototypes takes time. Here are three things to draw has been adapted to support the creation Plan
upon to design your own process. and testing of new ideas.
Act Do
Plan
Lean Start-up Methodology
A build-measure-learn process that begins with developing
Act Do Study
a minimum viable product (MVP) and then testing and Plan
learning from it as quickly as possible.
Act Do Study

Study

Rapid Results Campaign

An idea popularized by the Rapid Results
Institute. It encourages social innovators
to set micro “stretch goals,” e.g., reduce
homelessness in a city by 2.5% – in an
iterative series of 100 Day Campaigns.

9
Evaluation Worksheet for Prototypes
STEP ONE STEP TWO STEP THREE

What part of Key Things to

Evaluation Preferred Data When Feedback
the prototype is Consider to Make Lead/Logistics
Questions Sources/Methods Required
being tested? Test Useful

A possible Do you think The management Immediately A facilitator Janice, the

“five stages our staff will team will assume after the first needs to lead facilitator,
of customer understand the role of rapid prototypes document the will need ipod
engagement” this proposed different front- are produced feedback from camera to film
process for new business line staff and try during the design each prototype the event, with
a software process? to anticipate how session. and give it back assistant to write
company. each will react to to the team up notes that
each of the five ASAP. afternoon.
phases.

A mock up Where do our Have municipal During a Keep the Christopher, our
“European prototypes managerial and neighborhood walkthrough light government
block” in our test have friction front line staff walk through the and fun, allowing relations
neighborhood with municipal from 3 planning site facilitated by participants to Director, will lead
in Dallas by policies and departments visit Foundation staff. offer discrete walk through
Better Block regulations? the mock up and feedback by and Kimberly
Foundation. write up problem email after the will transcribe
areas on sticky session as well. and analyze the
notes, and leave notes.
them on site.

The feasibility Does installing Have one on one By mid- Ensure that the Janusz will design
and interest micro-generators interviews with December’s installers in our the whole test
of satellite require extra ten different management test group work project and work
installation skills? Under installers, from meeting: this with a variety of with Beatrice –
experts to install what conditions different parts means tests have different types our installation
our micro-wind are installers of the city, after to be complete in of satellites to experts – to
generators on willing to do it? a demonstration October. ensure we get run the
residential roof of installation a diversity of demonstrations.
tops, as part of by our company expertise.
their expanded staff.
set of services.

10
Step 5: Decisions

T
he purpose of an evaluating a prototype is to help the 3. Graduate to Pilot: The test results are sufficiently
innovators make a data-informed decision about the positive and unambiguous, that the prototype team
future of the prototype. feels the prototype should be more fully tested
through a formal pilot.
There are five broad decisions the team can make after review-
ing the data and analysis of a prototype evaluation: 4. Go to Scale: The test results are so positive and
unambiguous, and the risks so low or manageable,
1. Discard: The team has decided that the prototype is not that the prototype team is convinced that it is
worth continued investment. worth scaling without any further testing.

2. Evolve: Adapt the prototype in some new way: 5. Keep Testing: The results of the evaluation were
„„ Test the current prototype in a new setting or context. not strong enough to make a decision at this time.
„„ Adapt the current prototype (and evaluation design) Upgrade the evaluation design and try again.
based on new learnings.
„„ Upgrade the design brief and generate new proto-
types.

11
Resources Type in ‘prototyping’ in a google search engine and it turns up over
18,000,000 entries. These five sites are among the most useful of the bunch.

http://www.nesta.org.uk/
The world’s premier social innovation organization, with a half-dozen tools on
prototyping – and a special focus on public services and public policy.

http://www.servicedesign-
tools.org/
http://www.designkit.org An open-source website
with 60 design tools,
The organization that popularized design
many of them (e.g., client
thinking in general and created a first-rate
journeys, user profiles,
kit for human-centered design.
sketches, simulations)
useful for developing and
testing prototypes.

http://www.leanchange.net/blog/
A webpage with a variety of resources –
including a few on evaluation -- organized
around the lean start-up methodology, de-
veloped in the private sector and adapted
for use in social change initiatives.

http://diytoolkit.org/media/Prototype-
Testing-Plan-Size-A4.pdf
A little known group, specializing in inter-
national development. DIY produced the
Prototyping Testing Plan. http://betterevaluation.org
The most comprehensive web-based evaluation resource
in the world with a step-by-step approach to evaluation
design and dozens of tools.
What We Know So Far is a
series of documents that
summarize some of the latest
Here to There Consulting Inc.is licensed under a Creative
thinking or developments in
Commons Attribution-NonCommercial 3.0 Unported License.
the field of social innovation
and community change.

12