Int. J. Engng Ed. Vol. 22, No. 3, pp.

519532, 2006
Printed in Great Britain.

0949-149X/91 $3.00+0.00
# 2006 TEMPUS Publications.

What Constitutes Good Design? A Review

of Empirical Studies of Design Processes*
MATTHEW M. MEHALIK and CHRISTIAN SCHUNN
Learning Research and Development Center, University of Pittsburgh, 3939 O'Hara Street, Pittsburgh,
PA 15260 USA. Email: [email protected] and [email protected]
The authors conducted a meta-analysis in which they sampled over 40 journal articles of empirical
studies of the design process. The articles were selected from several journals that are indexed in
PsychInfo. They were reviewed and indexed according to whether or not those articles mentioned
and/or focused one or several of 15 common design process elements. These elements consist of a
bootstrapped framework that the authors constructed from past experience in studying the design
process and from validation from literature. The coding was conducted by an additional expert in
design, and the results were shown to have reliability greater than 80%. The results show which
elements of the design process have been studied most frequently, and of those studied, which were
documented to contribute to an effective design outcome. Both the framework and the coding
outcomes contribute to a stronger understanding of the overall design process that relies upon
existing, more fine-grained empirical studies of design.
Keywords: design processes; meta-analysis; design outcomes.

organizations that specialize in innovative design,

for which new contexts, markets, and products
emerge and shape worldwide practice.
Second, an integrated, meta-perspective on
design based on previous empirical studies can
focus attention onto understudied areas in the
design process. Not all aspects of the design
process have been studied or documented empirically at equal degrees of depth or breadth. As it
may turn out, the landscape of empirical studies of
design can point to areas in which further exploration can generate more understanding of effective
design practice.
Third, a more integrated, meta-perspective on
design can point to ways for which design can be
used as an effective vehicle for learning. Designbased learning has been explored as a way to
engage students to enhance their abilities to solve
real-life problems and to reflect on their learning
processes. This style of active learning is an extension of project-based learning, which is argued to
enable students to relate problems to science
concepts [1, 2]. Design-based learning differs
from project-based learning in that, in addition
to constructing and building, students engage in a
design and planning process that follows engineering design. The ability for learning science
researchers to anchor their evaluations of designbased projects can be informed by having a validated framework for design that considers multiple, integrated aspects of the overall design
process. Such a study can also permit learning
science evaluations of design to consider how
different learning styles [3] can be engaged at
various stages of the design process from an
overall design process perspective.

BACKGROUND
WHAT CONSTITUTES good design? For this
article we surveyed and reviewed over 40 journalpublished, empirical studies and case studies of
design in order to derive a common set of reported
design process elements. Further, we explored this
same literature from the perspective of which
elements are reported to be associated with good
(effective) design practice. The intentions of this
effort are to offer a common set of effective design
process steps and to focus attention onto areas of
design that require additional exploration or investigation. The point of identifying the design
process elements in this paper is to take a metaperspective on empirical approaches that have
looked at design and have documented observable,
describable design activities.
What are some reasons for taking such a metaperspective to understanding good design? First,
there are a number of published empirical studies
on various parts of the design process. Such
empirical approaches typically focus in-depth on
a particular aspect of design and attempt to situate
that aspect in a larger design context. Studies of
this type lend themselves to providing quality data
about these particular aspects. The value of these
studies can be extended if their findings can be
assembled into a broad, integrated perspective.
The end result of this exercise ought to be a
common set of effective design stages that can
be useful for practitioners of design and for
organizations whose practices rely upon effective
design processes. Such is the case especially for
* Accepted 22 November 2005.

519

520

M. Mehalik and C. Schunn

Finally, in terms of engineering education,

design has become explicitly recognized as an
important outcome of what undergraduate engineers need to experience and to achieve [4]. Specifically, Criterion 3 of accreditation standards
states that their program must demonstrate that
their graduates must have:
b) an ability to design and conduct experiments, as
well as to analyze and interpret data;
c) an ability to design a system, component, or
process to meet desired needs [4].
In addition, Criterion 4 specifies that an engineering
student's professional development must include a
major design experience that integrates much of
their earlier skills, coursework, and standards:
b) One and one-half years of engineering topics,
consisting of engineering sciences and engineering design appropriate to the student's field of
study. . . . Engineering design is the process of
devising a system, component, or process to
meet desired needs. It is a decision-making
process (often iterative), in which the basic
sciences, mathematics, and the engineering
sciences are applied to convert resources optimally to meet these stated needs [4].
An examination of previous empirical design
aspects in order to identify which contribute to
good design can serve as a way for engineering
professions to re-examine design education in
order to emphasize the aspects of design that
lead to effective design.
A meta-analysis of a design process can also
highlight in greater specificity what has been
documented to show the uniqueness of the type
of thinking that is associated with design. Herb
Simon emphasized that design mainly involves
thinking about the artificial in a process involving
mostly synthesis, whereas other types of thinking,
such as scientific inquiry, focus more on analytical
thinking. Design is also a highly goal-directed
enterprise in the pursuit of a normative outcome.
What designers produce is the result of making
clear an outcome that embodies some attributes
that can be specified but also have not yet been
achieved. Design places more emphasis on the
functional nature of what is being produced. It
also involves the functional decomposition of an
overall design outcome, so that these functional
aspects can be examined separately and later
integrated into a functional whole. Finally,
design involves a large degree of symbolic manipulation and representation in order to explore
complex relationships among various aspects of a
particular design [5].
FRAMEWORKS ON DESIGN
One place to start a process of constructing
evaluating a set of design elements involves looking at some other frameworks that discuss the

design process. Simon [5] proposed what knowledge needs to be taught so that a true science of
design can emerge:
1. Frameworks for rational choice among given
alternatives (e.g. Utility theory, decision analysis, statistical modeling)
2. Procedures for optimizing among alternatives
(e.g. linear programming)
3. Effective methods for searching space of solutions for `satisficing' goals and constraints
4. Effective, or satisfactory, allocation of
resources for searching among alternatives,
including partial paths of searches [5].
In addition, Simon advocated exploring alternative
representations of design problems. The authors
have considered these elements of design when
they began compiling a list of common design
elements.
Techniques and frameworks from systems
analysis and design also provide some past insights
into key stages in designing large-scale systems.
For example, Gibson [6] wrote that the process of
systems analysis and design consists of seven stages
that rely upon interaction and iteration among
these stages:
1.
2.
3.
4.
5.
6.
7.

Determine the goals of the system

Construct meaningful indices of performance
Consider alternate candidates for solutions
Rank alternatives
Validate solutions
Iterate
Take action [6].

These two frameworks reflect some of the disciplinary origins of the authors, who have pooled
their expertise in the areas of studying the cognitive
science of design and systems analysis and engineering in order to begin a bootstrapping process
for identifying and validating a common set of
design elements.
The order in which the design process elements
identified in this paper are presented should not be
taken as a descriptive or normative representation
of the way in which the design process proceeds in
practice or as observed in an educational task
setting. As Mawson [7] argues, sequential models
of the design process tend to be misrepresentations
of the way practitioners actually do design, and
they do not offer a useful structure for the way that
pupils learn design; instead, such models, which
tend to be oversimplified, linear representations of
design, serve as a useful `administrative' function
for teachers who may lack expertise in design
activities. Portraying and teaching design as
following a linear, or algorithmic structure therefore serves the wrong constituency in education.
The point of identifying the design process
elements in this paper is to take a meta-perspective
on empirical approaches that have looked at
design and have documented observable, describable design activities. Although they are the
result of clustering similarities, the categories are

What Constitutes Good Design? A Review of Empirical Studies of Design Processes

nevertheless representative of design activities.
Identifying these elements serves to highlight
which activities get the attention of researchers
reporting on case studies and empirical studies of
design.
METHODOLOGY
The authors began the process of identifying and
clustering design process elements/stages from
their past experiences in cognitive studies of
design and systems engineering design. The discussion of earlier frameworks by Simon [5] and
Gibson [6] reflect some of the disciplinary origins
of their initial thinking. These past experiences led
them to consider sampling articles from several
journals, mostly in Design Studies and Cognitive
Science, to check the feasibility of identifying
empirical studies of design and being able to
characterize some of the design stages discussed
in such articles. The authors next recorded a series
of elements that they believed characterized the
design process, an initial framework of design
activity descriptors. This framework was used to
code an initial set of five empirical journal articles
in Design Studies and Cognitive Science. The framework was adjusted and re-clustered after extensive
discussions on whether the identified elements
were adequately descriptive, comprehensive, and
distinct.
After this initial validation, additional articles
were identified through a database search of
PsychInfo using these keywords: design; engineering design; design process; design methodology;
and cognitive analysis of design. The searches
using these keywords located articles between the
years 1993 and 2003.
From this initial set, the articles were screened
briefly according to some basic criteria for inclusion. Articles that were included must be from a
peer-reviewed journal and must involve an empirical investigation of design. The article needed to
come from a journal in which a review of a design
process represents a major interest in the community that would read such an article. The empirical
analysis needed to emphasize the cognitive aspects
of design, not personality issues.
COMMON DESIGN PROCESS
ELEMENTS/STAGES
The bootstrapping procedure discussed in the
previous section resulted in a set of 15 elements/
stages commonly found in the design process. Each
step represents a distinct aspect of a design activity.
Each is mutually exclusive in terms of documenting a specific describable aspect of the design
process. However, because a designer can be
engaged in activities that have multiple goals and
meanings, it is possible for more than one criterion
to be applicable at a time; for example, if a

521

designer is working on framing what problem

needs to be solved (Explore Problem Representation) while using a sketch (Use Graphical Representation), two design aspects are applicable. The
design aspects are thus mutually exclusive in terms
of defining specific aspects of design, but they are
not necessarily mutually exclusive with respect to a
particular event during the design process.
Explore problem representation
This design criterion refers to how designers go
about framing the design task or problem. Framing the design task can involve defining or exploring a problem, issue, or artifact that needs to be
analyzed, synthesized, investigated, or constructed.
The way in which designers construe their task
can have an impact on what aspects of a design a
designer emphasizes, on what solutions paths
designers choose, and on which goals and
constraints designers meet. This aspect of design
can occur early in the design process when a
designer or design team deliberates what their
task really is or is about. However, designers can
also decide in the midst of exploring a particular
design that what he or she actually wants to make
or solve is different from the way they previously
construed it.
For example, Guenther et al. [8], reported that
one of the differences between experienced
designers, who had no university educational
training or training in design methodology, and
designers who did have university training in
design methodology was that the former group
generally pursued shorter design processes with
narrower objectives and singular solutions. The
latter group tended to spend more time exploring,
clarifying, and changing the problem.
Another characteristic associated with exploring
the representation of the problem involves devising
or discovering when to stop pursuing the design or
the problem; in other words, when the solution or
design fits how the problem or design has been
framed. Simon [5] has characterized the design
process as an open-ended search process with no
universal stopping rule at optimality; instead,
design involves a process of `satisficing', or deciding when a configuration is good enough to be
called complete. Therefore, evidence that a
designer is working on the problem representation
involves instances in which the designer decides
what the end product will be.
Explore graphical representation/visualization
Designers often use a strategy of representing
some aspect of a design or design problem using
graphics or visual media. The main distinction of
this category involves the use of visual means to
construct a representation, contrasting with a
verbal representation (such as a list of requirements), a quantitative representation (such as a
list of measurements), or other alternative forms of
representation (such as kinesthetic, as in the case of
a prototype or mock-up). Designers often sketch

522

M. Mehalik and C. Schunn

their ideas or use graphics software, such as a CAD

program.
This criterion differs from the problem representation criterion in that framing the problem
represents a broader focus on conceptual issues
that can take form in various modes of representation, whereas the graphical representation criterion
focuses on specific dimensions of a design task that
have largely been constructed within a conceptual
framework. Representing something graphically
does not necessarily involve a stopping condition
associated with final design requirements.
A designer often attempts to visualize details of
a design in order to explore the design's overall
configuration, the design's relationship to its
context, or to explore some feature of the design
in greater detail. Visual representations also can
facilitate communication on a design team, and
they also can structure interaction with end users
who may be involved in providing input into the
design process.
Use functional decomposition
Instead of focusing on what the designed artifact
needs to do overall, a designer may decide to break
down a design into several more-detailed aspects to
investigate how they perform, interact, and contribute to the overall functionality.
The use of functional decomposition serves to
simplify complex, confounding aspects of a design
by isolating them to discover which have the most
impact on the overall design's functionality.
Gary Bradshaw [9] has written about how the
Wright Brothers used functional composition in
their pursuit of designing aircraft and has argued
that their strategic use of functional decomposition
accounts for how they solved the problem of flight
before others did. Instead of designing and building entire glider configurations and testing them,
the Wright Brothers broke down the problem of
flight into components such as providing lift
through wing configurations, providing thrust
using a powerplant and propeller combination,
and providing stability and control using wing
warping and movable control surfaces. The
Wright Brothers were able to design and test
each component separately and then reassemble
their results into an overall design.
Explore engineering facts
A designer who explores engineering facts delves
into a specific knowledge domain regarding some
property of an aspect of a design.
The designer may do this in order to doublecheck what can be taken as a given; to pursue a
desired, documented outcome; to articulate some
common principles on which the design can be
based or communicated to others; and to come to
grips with what others have investigated related to
the design.
For example, when Jay Forrester was designing
the first electronically programmable memory
systems for project Whirlwind, he explored the

magnetic properties of magnetic-ceramic materials

whose induced magnetic poles represented 1's or
0's of writable and retrievable information bits
[10].
Explore issues of measurement
This criterion involves examining the way
quantitative information is gathered relating to
some aspect of a design.
Designers may question or explore methodologies they use to quantify aspects of their design to
discover or determine whether more effective strategies for collecting quantitative information could
improve their ability to make what they want to
make.
For example, when engineers were designing the
first stage F-1 engines for the Saturn V rocket, a
design that leapfrogged previous designs in terms
of thrust significantly, the engineers needed to
design new test stands and acoustical equipment
for eliminating combustion instability due to the
resonant properties of the engine's geometry [11].
Build normative model
Designers may spend time articulating what the
desired, ideal outcome of their design ought to
look like. Such ideal outcomes constitute a normative representation or model of their designwhat
their design might look like if they were not
constrained or limited. The term `model' in this
criterion includes multiple forms of representationsverbal, visual, tactual, physical, etc . . .
This criterion is different than `Explore Problem
Representation,' which can involve descriptions of
any current conditions of the context of the design
as well as desired outcomes, whereas `Build
Normative Model' involves only ideal or (imagination-based) optimal, not merely desired, outcomes
for the design.
Normative model building can also apply to
design process management, not just artifact
representation. The fact that the criterion includes
`model' means that the designer(s) make some
attempt at formalizing the desired outcomes.
Formalizing requires some attempt at comparing,
critiquing, and testing, in addition to articulating.
Problem framing does not necessarily require this
level of formality.
This criterion is important because oftentimes
designers limit their solutions narrowly to what
they believe are inviolable constraints and limitations. Building a normative model is different from
redefining constraints or exploring the scope of
constraints because the normative model involves
the suspension of constraints in constructing an
ideal outcome.
An example of a case that involved the building
of a normative model includes Susan Lyons', Albin
Kaelin's and William McDonough's design plans
for a textile product and production system that
eliminates the concept of waste by making the
product benignly compostable and the production
system with zero effluents. Such a conception

What Constitutes Good Design? A Review of Empirical Studies of Design Processes

proved valuable for this design team for constructing a successful product that achieved many goals
at minimizing (and in many cases eliminating)
environmental impacts [12].
Explore scope of constraints
After a designer or a design team articulates and
accepts a particular frame of the design problem
and a working set of goals for the design, the
designer(s) typically face factors, or constraints,
that limit how a design can fulfill these goals within
that problem frame. These constraints may be
conceptual or physical. Oftentimes, a designer
may decide to investigate to what extent a
constraint or set of constraints become salient.
The activity of a designer spending effort to learn
more about how constraints are affecting the
design, falls under this `explore scope of
constraints' criterion.
This criterion is different from exploring the
problem representation because exploring the
scope of constraints does not involve a possible
redefinition of the overall design goal or goals,
whereas exploring the problem definition does.
Instead, exploring the scope of constraints refers
to a situation where goals and most constraints are
fixed; only a few are explored.
For example, breaking the sound barrier for
controlled, piloted aircraft was an effort that
required a great deal of effort and design readjustments. The sound barrier represented the main
constraint on controlled flight, and designers
explored multiple configurations in order to determine which designs were able to meet the
constraint of controlled supersonic flight.
Redefine constraints
This criterion describes the activity when a
designer decides to investigate further what is
involved with a constraint or set of constraints to
reconfigure the way that that constraint or set
becomes salient for the design. Instead of modifying the goals of the design and constraints (which
would be the case when a designer is exploring the
problem representation) or instead of focusing on
one constraint to see to what extent it shapes a
design, a designer may redefine a constraint in
order to achieve an original goal, but by using a
design that may not otherwise conform to the
original constraint.
Ball [13] has summarized previous research
efforts that investigated the way designers balance
a structured design approach (by sticking with a
plan of design goals and constraints) and an
opportunistic approach (by switching the set of
salient constraints to take advantage of an unexpected configuration or new information).
Conduct failure analysis
Not all designs fulfill their goals or desired
outcomes. Designers oftentimes pay attention to
when their designs fall short of their goals or do
not meet performance expectations. They can be

523

deliberate (systematic and purposeful) in their

investigation of what the reasons were that
caused a design to fail, or they can look at failures
on a more ad-hoc basis to learn about them. Either
mode counts as a failure analysis activity in design
in this paper.
In addition, a design may not be failing with
respect to a global set of performance criteria
there may be only a few dimensions of underperformance. Designers may also investigate failures with respect to this smaller set as part of
failure analysis activity. In either case, the focus
of this activity is to gather knowledge associated
with what produces the failure. Treating failed
designs as disconfirming cases of performance
can be a learning strategy in this aspect of design.
There are many examples of analyses of failed
designs. Doerner [14] reports that subjects who test
more hypotheses and look at more failed designs
tended to produce more satisfactory outcomes
over time. Chiles [15] chronicles a long list of
large-scale engineering failures, ranging from the
Hindenburg to Apollo 13 to Three Mile Island and
how designers can learn from such failures.
Validate assumptions and constraints
The flip side of learning from a disconfirming
case of failure can be an investigation of success in
order to gain more confidence in a design.
Designers oftentimes test their designs in order to
confirm that they are falling within constraints as
expected and that the assumptions they made
about the design appear to be holding true. Validation can also involve ensuring that the representation of the user's, or other stakeholder's
expectations for the design appear to be met by
engaging them in the design process at various
stages.
Search the space (evaluate design alternatives)
This criterion describes a designer's actions to
use some framework of performance criteria (goals
and constraints) to search for and evaluate potential solutions for the design. Designers may use
various strategies and procedures to guide this
process.
Simon [5] characterizes design as a goal-directed,
heuristic search process in which the goals are not
necessarily fixed. Part of this search activity
involves considering what alternative configurations can meet the current state of which goals
and constraints are applicable. Ball [13]
summarizes the debate on whether the search
process consists of a structured process of
breadth-first followed by depth searches for
solutions.
Thomas Edison, in his quest to construct a
lighting system that used high-resistance filaments
in incandescent bulbs, searched an enormous
number of alternative filament configurations
and materials to find a filament that was stable,
provided enough light, was economical, and was
possible to manufacture. He eventually designed a

524

M. Mehalik and C. Schunn

lampblack carbon filament as the outcome of his

search for a viable filament [16].
Examine existing designs/artifacts
This criterion describes a subset of activities of
the previous criterion. Sometimes designers
borrow solution ideas from designs that already
exist and that relate to the design problem. They
may also look at what has been done previously in
order to improve upon those designs in various
design dimensions. This criterion describes the
activity of when designers spend effort looking at
what alternatives exist relating to their design
problem.
Follow interactive/recursive/iterative design
methodology
This criterion describes cases in which designers
can be documented to follow a design process that
involves the purposeful jumping around through
various aspects of design activities instead of
sticking to a fixed sequential model of design.
Oftentimes the jumping around comes from
following a process that encourages interactions
with other designers, other stakeholders, or other
processes that initiates and/or structures the shifts
in activities.
Explore user perspective(s)
In this design activity designers involve users in
various aspects of the design process in order to
capture their requirements and needs for the
design's performance. Designers can engage users
using past user knowledge and experience,
structured interviews, mock-ups, simulations,
prototypes, questionnaires, or final design configurations, among others.
Encourage reflection on design process
(self-reflect)
Designers may spend time reflecting upon their
own process that they used to achieve their
production goal(s). Self-reflection can occur in
the midst of the design activity or after the activity
has been completed.
Donald Norman [17] has written about the
importance of self-reflection in the process of
solving problems and in developing expertise.
Engaging in self-reflection initiates a process
of restructuring and transforming experiential
knowledge.
REVIEW PROCEDURE
The primary author conducted the review and
coding of the journal articles. He discussed his
results with the second author, who posed questions to clarify the way the articles were coded.
After completing the complete set of over 40
articles, an additional expert in engineering
design reviewed a sample of 11 articles from the
set of 40 to validate the coding. This additional

expert reviewed one sample article (not included in

the database) as practice after which he was given
feedback through discussion with the authors. The
additional reviewer then proceeded to code two
articles, after which the reviewer and the authors
discussed the procedure once again in order to
clarify the task and check the degree of correspondence. The additional reviewer then completed
a review of ten randomly selected articles from the
database, and the degree of correspondence of
coding was shown to exceed 80 percent across all
coding dimensions.
The coding procedure consisted of two main
parts: characterizing the study and characterizing
the findings relating to design. Both parts were
accomplished using a spreadsheet template. For
each article the reviewer was permitted to write
down whatever notes that the reader thought
would help him for the coding. The reader was
permitted to re-read the article as many times the
reader wished. The reader was permitted to fill out
the template during the reading or at the end of
reading. The reader was permitted to revise the
ratings as desired.
Part 1 of the procedure/template consisted of
items related to classifying the type of study in the
article. The dimensions, along with possible values,
consisted of:
Level of expertise in design
The minimum criterion of qualifying as an
expert designer was 57 years. Novices typically
constituted subjects with design experience ranging
from college graduates in design to practitioners
having 5 years of experience. Beginners were
undergraduates who have not completed their
degrees or others who had no formal training or
experience in design.
Level of expertise in domain
The criteria for experts, novices, and beginners
were the same as the experience levels in the
Expertise in Design dimension. The actual coding
of expertise in design and domain used the following conventions:
. Beginner 01 years of prior experience;
includes students just starting their educational
exposure.
. Novice 25 years of prior experience; includes
senior students who are in their fourth year/
senior year of education.
. Expert people with more than 5 years of
expertise.
. Expert/novice a contrast study in which
experts and novices are compared.
. Various a study in which there are participants
with multiple levels of expertise.
Type of task examined
The types of activities included in the survey
spanned many different categories. Some of the

What Constitutes Good Design? A Review of Empirical Studies of Design Processes

more common tasks included software and new
consumer product development. The actual coding
used the following conventions:
. Engineering: includes mechanical, electrical,
aeronautical, nuclear, civil, and other non-computer related engineering design tasks.
. Computer programming and software: includes
software design and software usability tests;
programming and theory of programming; and
computer hardware design.
Real/artificial task?
A real task refers to a case in which the subjects
of the studies were engaged in designing something
to meet an actual need or outcome. For example,
one of the software design tasks involved designing
groupware for chemical engineering processes that
would be used at a company. An artificial task
refers to cases in which subjects were studied while
engaged in a simulated activity. The coding
followed this convention:
. Real an actual design task in a practitioner
setting.
. Artificial a simulated design task or a problem
constructed in an educational setting.
Type of study/methodology
This dimension captured the data gathering and
processing strategies that the surveyed articles
reported. The studies used such techniques as
video, verbal, and/or written protocols, detailed
case studies, ethnographic analysis, user interviews, among others. The ways in which the various protocols were encoded and processed were
noted if any special techniques were used during
these stages of data processing. The actual coding
protocol consisted of:
. Case study a documented reporting of a
design process that does not involve specific
protocol analysis.
. Verbal protocol only voice conversation was
recorded, coded and analyzed.
. Verbal Other voice and some other (video,
written) type of protocol was used.
. Ethnography a description that occurs at a
greater level of detail than a case study, but
does not involve recording participant actions
beyond the writing of the researcher's observations, which become the raw data for analysis
instead of a recording of the participants' voices,
video, etc . . .
. Interviews (structured/unstructured) the researcher interviews subjects to get them to
report their experiences post-hoc.
. Video protocol only recording of participants'
actions on video tape and constructing a classification protocol from that record for analysis.
. Written protocol only examining the drawings
and sketches of participants using a classification scheme.

525

Group or individual?
If the subject engaged in their design task alone,
then that activity was counted as an individual
activity. If the subjects acted in a concerted effort,
then they performed their design task as a group.
Some of the studies examined designers working
alone and within a group setting on the same task.
These instances counted as having both individual
and group activities.
Which part of design cycle?
Constructing this category required multiple
revisions and reclassifications, because there is no
accepted, standardized, meaningful and defined
design cycle. The categories for this dimension
were generated by using the design process designations that the empirical studies themselves
mentioned as their focus. This strategy produced
a wide span of categories, ranging from conceptual
design, requirements definition, goal analysis,
preliminary design, problem definition, and redesign, among other categories:
. Various several parts of the design task.
. Preliminary design involves working on the
framing of the problem and the conceptual
aspects of a design, including basic modeling.
. Preliminary design through coding all the
stages between working on framing the problem
through writing computer code.
. Preliminary design through detail design all
the stages between working on framing the
problem through working on the more specific
features and/or aspects of a design.
. Preliminary design through prototype construction all the stages between working on framing
the problem, detailed design, through constructing a prototype.
. Preliminary design through release all the
stages between working on framing the problem
through what it takes to get the product into the
marketplace.
. Redesign taking an old design and redesigning
it to meet new specifications.
. Software usability evaluating software using
usability protocols or involving users in improving the product's performance.
. Requirement construction and testing process
for eliciting, defining, validating, and testing
user requirements.
. Preliminary design through product testing all
the stages between working on framing the
problem through testing a product.
Part 2 of the procedure/template involved coding
what the study focused on and what the study
reported as significant. There are four possible
options for each of the fifteen design process
elements discussed in the previous section. The
four options are:
. 1 factor reported as significant for good
design. This means that the study focused on
that particular design criterion and reported that

526

M. Mehalik and C. Schunn

Table 1. Journals represented in the review database

Table 2a. Levels of expertise in design

International Journal of Human-Computer Interaction

Design Studies
Cognitive Science
International Journal of Man-Machine Studies
Behaviour & Information Technology
International Journal of Human-Computer Studies
International Journal of Technology and Design
Education
Journal of Engineering Design
Applied Ergonomics
Ergonomics
Human-Computer Interaction
International Journal of Intelligent Systems
Journal of Applied Psychology
Learning and Instruction
Proc Instn Mech Engers
Thinking and Reasoning

10
8
3
3
2
2
2

Total

40

2
1
1
1
1
1
1
1
1

it is significant in terms of a positive impact on

the design process.
. 0 factor mentioned in study, but not reported
as significant for good design. This code was
used for factors mentioned in passing in the
article, even if the factors were not the primary
focus of the study.
. 1 factor found not to correspond with good
design (i.e., be actively bad). This code was used
when a design factor was a focus of the paper
and when the paper reported that the particular
design element diminished good design practice.
. n/a factor not mentioned in study
RESULTS
Overall, there were 40 articles that were reviewed
and included in the analysis. The 40 articles
represented 16 different journals, as is brokendown in Table 1.
Experts in both design and various domain areas
were more heavily represented in the database and
analysis. Tables 2a and 2b show the distribution of
articles as broken down by levels of expertise.
Nearly half of the types of design tasks examined in the database consisted of engineering
design tasks in mechanical, electrical, structural,
civil, automotive, and other types of engineering.
Nearly 40% consisted of design tasks associated
with computer programming and software design
tasks. The remaining articles consisted of various
design tasks in architecture, branding, marketing,
and other non-engineering or computer-related
areas.
Of the 40 articles, 24 of them involved studies
for which the subjects of study were engaged in a
real design task for their analysis; 15 of the 40
articles involved subjects working on artificial
tasks; and 1 article consisted of a study that
included both real and artificial tasks.
Table 3 displays the distribution of the type of
methodologies the empirical studies used for their

Level of Expertise in Domain

Beginner
Novice
Expert
Expert/Novice
Various

Number of Articles
1
5
21
5
8

Table 2b. Levels of expertise in domain areas

Level of Expertise in Design
Beginner
Novice
Expert
Expert/Novice
Various

Number of Articles
1
4
24
5
6

analysis. The database has nearly equal representation of case studies; verbal protocol studies only;
and verbal, video, and written protocol studies.
The remaining quarter of the database is spread
among ethnographic studies, interviews, video
only, and written only protocols.
About half (21 out of 40) of the studies involved
subjects working on individual tasks. 14 articles
involved group tasks, and 5 of the articles involved
studies using both individual and group tasks.
The articles disproportionately studied parts of
the design process that are more often associated
with the earlier stages of design. Mentioned earlier,
the process of design in terms of thinking processes
does not follow a linear path Mawson [7]; however,
when examined from a perspective that traces how
products make it from the idea stage through
release to the marketplace, a linear representation
can serve as a rough approximation of how such
processes operate.
The results of the coding of the fifteen different
design elements were categorized according to
three different dimensions. First the results were
categorized according to the frequency with which
they were reported in the set of articles. This
dimension consisted of three categories: high
reporting, moderate reporting, and low reporting.
A design element was included in the `high reporting' category if it was mentioned as a focus in more
than 50% of the articles included in the database.
The element was considered to be in the `moderate
reporting' category if it was mentioned as a focus
in 25% to 50% of the articles. Finally, elements that
Table 3. Methodologies used in the database articles
Case Study
Verbal Protocol Only
Verbal Other
Ethnography
Interviews (Structured/Unstructured)
Video Protocol Only
Written Protocol Only

11
9
9
4
3
2
2

Total

40

What Constitutes Good Design? A Review of Empirical Studies of Design Processes

527

Fig. 1. Stages of the design cycle represented in the database.

were reported in fewer than 25% of the articles

were considered to fall in the `low reporting'
category.
Second, of the elements for which reporting
occurred as a specific focus (not just mentioned
in passing) and the element was noted as being a
factor for good/effective design, the articles were
categorized according to whether the factor should
be considered as significantly good for design,
maybe good for design, or unlikely good for
design. Articles that were mentioned as being
good for design in over 75% of the times they
were mentioned at all were considered to fall into
the `significantly good' for design category. Design
elements that were mentioned as being good for
design between 25% to 75% of the times they were
mentioned at all were categorized as `maybe good'
for design. Finally, elements that were mentioned
as good for design less than 25% of the time they
were mentioned at all were categorized as unlikely
good for design.
Third, of the elements for which reporting
occurred as a focus, if the element was noted as
being actively bad for design more often than good
for design, then the elements was categorized as
`bad for design.' Not all combinations of this
conditional structure eventually had elements in
every possible category. Tables A1 and A2
summarize the results of the coding.
Tier 1: Design elements significant for good design
(high reporting frequency)
The design elements that were reported with a
high degree of frequency and were also categorized
to be significant for good design are:
. Explore problem representation.
. Use interactive/iterative design methodology.
. Search the space (explore alternatives).
Based on this analysis, these three items were most
often recorded in empirical studies that showed
that positive design outcomes were associated with
people who used at least one of these three
strategies in their design process. An implication

of this finding is that such design process steps

should be emphasized in the process of becoming
an expert (or in the actual expert practice) of
design. Because the sample of journal articles
included more experts in their empirical studies
means that likely these three design elements are
associated with expert design processes that are
effective.
Tier 2: Design elements may be significant for
good design (high reporting frequency)
One design element was categorized as may be
significant for good design was:
. Use functional decomposition
This strategy was mentioned frequently in the
database of articles; however, when it was
mentioned, it was not always mentioned as associated with good design. There are several interpretations of this finding. One could be that
functional decomposition is a necessary aspect of
design (hence mentioned frequently); however, the
strategy is not perceived to make a large difference
in achieving what is thought to be good design.
Another interpretation is that the studies included
in the articles tended to not focus primarily on this
particular design stage in an evaluation of the
design processthe focus may have been elsewhere. Because functional decomposition occurs
often in design, however, it still gets mentioned in
the studies.
Tier 3: Design elements significant for good design
(moderate reporting frequency)
Of the 15 design process elements, 6 were
reported to be significantly good; however, the
certainty of this designation is less certain because
they were mentioned less frequently overall in the
database of empirical studies. These 6 elements
are:
. Explore graphic representation.
. Redefine constraints.
. Explore scope of constraints.

528

M. Mehalik and C. Schunn

. Validate assumptions and constraints.

. Examine existing designs.
. Explore user perspective.
The items in this category can be further sorted if
the sampling size of studies were increased. These
items appear to be promising for what is associated
with effective design. Confidence in this categorization can likely be increased if the database
included a larger number of articles.
Tier 4: Design elements may be significant for
good design (moderate reporting frequency)
One of the fifteen design elements was mentioned
less frequently as associated with good design when
mentioned, and was mentioned fewer times overall:
. Build normative model.
Perhaps the nature of design itself makes one
automatically think that it is necessary to project
a normative outcome or model of what is being
designed so that this type of thinking gets less
emphasized when thinking about good design.
Tier 5: Items requiring further study (low
reporting frequency)
The low reporting frequency category reveals
which design elements in the database may be
valid candidates for further empirical study
because of their low level of being mentioned in
the database of articles. These elements are:
.
.
.
.

Explore engineering facts.

Explore issues of measurement.
Conduct failure analysis.
Encourage reflection on process.

In the database, there tended to be less emphasis

on engineering content knowledge/facts in the
empirical studies. Also measurement techniques
were not often mentioned. The low reporting of
these two items may be due to the types of journals
considered in this analysis. In general, more indepth engineering content and measurement technique knowledge tends to be reported in engineering journals for specific disciplines rather than as a
focus in a design process study. However, this
finding suggests that there is an opportunity for
more in-depth studies for how deep domain knowledge impacts the effectiveness of design outcomes.
In addition, few studies tend to focus on the
importance of design failures for improving design
outcomes or on the way that design practitioners
reflect on what they do as a meta-cognitive strategy
for improving the quality of design process
outcomes. These two areas represent possible fruitful areas for additional empirical studies for how
effectiveness in the design process can be achieved.

CONCLUSIONS
What do the results of this meta-analysis contribute to answering the question of what constitutes

good design? First, the analysis began by

constructing a framework of design process
elements using a process that characterizes the
design process into 15 different stages. Having
such a framework provides a method for linking
together separate empirical analyses of design that
focus on specific parts of the design process, not on
the whole. Thus, other excellent sources of data on
parts of the design process can provide linked
insights into a broader view of the design process.
The use of the 15-element framework permits the
question of what constitutes good design to be
refined into what specific parts of design do
empirical analyses of design tend to associate
with good design. This more specific question has
been answered in the results section of the paper,
which points to specific elements of the design
process that these empirical studies have most
often identified as being associated with good
design.
Second, the decomposition into 15 specific
design stages also permits an assessment of which
areas tend to receive more frequent levels of
attention than others. Answering the question of
what constitutes good design requires being able to
point to areas in which there has not been enough
research attention in order to answer that question
unconditionally. Thus, the results in this paper
distinguish between which aspects of design the
existing literature tends to focus on, and which
aspects can benefit from further attention in
empirical study.
Third, from an educational standpoint, teaching
of design ought to focus on elements of design that
have been documented to have higher levels of
impact for achieving effective design. Results from
this study point to the fact that it might be better to
emphasize the importance of framing a design
problem, using an iterative strategy, and spending
time searching for alternative solutions to try than
to spend time on teaching the importance of
articulating normative outcomes for a design
process. However, this advice should be taken
with caution, given the fact that so many of the
design process elements had a low level of reporting in the literature. An alternative perspective
based on these findings could also point to areas
of design that have not received enough attention
and therefore may be invisible in terms of cognitive
recognition of their importance, but present and
important nonetheless. Education about design
could therefore involve spending time on learning
about these less-reported aspects of design as a
possible way to make the invisible more visible and
perhaps make refinement of these areas possible.
Finally, none of the 15 design process elements
were found to be actively bad in terms of their
impact/role in the design process. In fact, there
have been very few studies that have documented
what aspects of common design practice impact
the design process negatively. Understanding both
the positive and negative dimensions of influences
on design process outcomes can lead to a more

What Constitutes Good Design? A Review of Empirical Studies of Design Processes

comprehensive perspective on achieving effective
design.
AcknowledgementsThe authors wish to thank Professor
Mary Besterfield-Sacre and her team of undergraduate students

529

at the University of Pittsburgh for their contributions to helping

us refine the design element categories. In addition, special
thanks to Alex Burns of the University of Pittsburgh for his
insights and contributions.

REFERENCES
1. M. Prince, Does active learning work? A review of the research. J. Eng. Educ., 93(3), 2004,
pp. 223232.
2. W. J. Thomas, A review of research on project-based learning, in The Autodesk Foundation, San
Rafael CA, (2000).
3. H. Gardner, Multiple Intelligences, Basic Books (1993).
4. ABET, Criteria for Accrediting Engineeering Programs, Engineering Accreditation Commission:
Baltimore, MD (2003), pp. 12.
5. H. Simon, The Sciences of the Artificial, 3rd edn, MIT Press (1996).
6. J. E. Gibson, How to Do A Systems Analysis. Charlottesville, Virginia: University of Virginia
Department of Systems Engineering (1992).
7. B. Mawson, Breaking the Shackles: Beyond the Design Process, Auckland, New Zealand: Center
for Technology Education (2003).
8. J. a K. E. Guenther, Comparing designers from practice and designers with systematic design
education, Design Studies, 20(5), 1999, pp. 439450.
9. G. F. Bradshaw, The airplane and the logic of invention, Cognitive Models of Science, ed. R. N.
Giere, Univ. of Minnesota Press (1992), pp. 239250.
10. T. P. Hughes, Rescuing Prometheus. New York: Pantheon Books (1998).
11. R. Bilstein, Stages to Saturn, NASA SP-4206, Washington, DC: NASA (1980).
12. M. M. Mehalik, Sustainable network design: a commercial fabric case study, Interfaces: Special
Edition on Ecologically Sustainable Business Practices, 2000.
13. L. J. Ball, St. B. T. Evans and I. Dennis, Cognitive processes in engineering design: a longitudinal
study, Ergonomics, 37(11), 1994, pp. 17531786.
14. D. Doerner, The Logic of Failure, Perseus Books Group (1996).
15. J. Chiles, Inviting Disaster, Harper Business (2002).
16. T. P. Hughes, Networks of Power, The Johns Hopkins University Press (1983).
17. D. Norman, Things that Make us Smart, Addison Wesley Publishing Company (1994).

APPENDIX A
List of articles included in database

Fricke, G. (1999), Successful approaches in dealing with differently precise design problems, Design Studies 20(5) pp. 417429.
Guenther, J. a. K. E. (1999), Comparing designers from practice and designers with systematic design education, Design Studies, 20(5)
pp. 439450.
Adams, R. S., Jennifer Turns and Cynthia J. Atman (2003), Educating effective designers: the role of reflective practice, Design Studies,
24(3) pp. 275294.
Baird, F., C. J. Moore and A. P Jagodzinski (2000), An ethnographic study of engineering design teams at Rolls Royce, Design Studies,
21(4) pp. 333355.
Lloyd, P. (2000), Storytelling and the development of discourse in the engineering design process, Design Studies, 21(4) pp. 357373.
Jagodzinski, P., F. J. M. Reid, P. Culverthouse, R. Parsons and I. Phillips (2000), A study of electronics design teams, Design Studies,
21(4) pp. 375402.
Ball, L. J. a. T. C. O. (2000), Applying ethnography in the analysis and support of expertise in engineering design, Design Studies, 21(4)
pp. 403421.
Ball, L. J., Johnathan St. B. T. Evans and Ian Dennis (1997), Problem-solving strategies and expertise in engineering design, Thinking
and Reasoning, 3(4) pp. 247270.
Lee, A. a. N. P. (1994), the effects of paradigm on cognitive activities in design, International Journal of Human-Computer Studies, 40
pp. 577601.
Sobiesiak, R., Robert J. Jones and Scott M. Lewis (2002), DB2 Universal Database: A Case Study of a Successful User-Centered
Design Program, International Journal of Human-Computer Interaction, 14(3&4) pp. 279306.
Sawin, D. A., Kazuhiko Yamazaki and Atsushi Kamaki (2002), Putting the D in UCD: User-Centered Design in the ThinkPad
Experience Development, International Journal of Human-Computer Interaction, 14(3&4) pp. 307334.
Ominsky, M., Kenneth R. Stern and James R. Rudd (2002), User-Centered Design at IBM Consulting, International Journal of HumanComputer Interaction, 14(3&4) pp. 349368.
Healy, V. a. R. H. (2002), A Walk-Up-and-Use Information System for the Sydney Olympics: A Case Study in User-Centered Design,
International Journal of Human-Computer Interaction, 14(3&3) pp. 335347.
Kamper, R. J. (2002), Extending the Usability of Heuristics for Design and Evaluation: Lead, Follow, and Get Out of the Way,
International Journal of Human-Computer Interaction, 14(3&4) pp. 447462.
Gruen, D., Thyura Rauch, Sarah Redpath, and Stephan Ruettinger (2002), The Use of Stories in User Experience Design, International
Journal of Human-Computer Interaction, 14(3&4), pp. 503534.
Wolf, M., Christian Folz, Christopher Schlick, and Holger Luczac (2002), Development and Evlaution of a Groupware Ssytem to
Support Chemical Design Processes, International Journal of Human-Computer Interaction, 14(2) pp. 181198.
Moreno-Munoz, A., Adolfo Plaza-Alonso, Carlos de Castro-Lozano, and Sebastian Dormido-Bencomo (2002), Hypermedia
Design Methodology in World Wide Web Applications, International Journal of Human-Computer Interaction, 14(2),
pp. 251270.

530

M. Mehalik and C. Schunn

Hall, R. (2001), Prototyping for Usability of New Technology, International Journal of Human-Computer Studies, 55, pp. 485501.
McFadden, E., Danny R. Hager, Claude J. Elie, and Jason M. Blackwell (2002), Remote Usability Evaluation: Overview and Case
Studies, International Journal of Human-Computer Interaction, 14(3&4), pp. 489502.
Rist, R. (1995), Program Structure in Design, Cognitive Science, 19(4), pp. 507562.
Schraagen, J. M. (1993), How Experts Solve a Novel Problem in Experimental Design, Cognitive Science, 17(2), pp. 285309.
Suwa, M., Terry Purcell, and John Gero (1998), Macroscopic Analysis of Design Processes Based on a Scheme for Coding Designers
Cognitive Actions, Design Studies, 19(4), pp. 455483.
Goel, V. a. P. P. (1992), The Structure of Design Problem Spaces, Cognitive Science, 16, pp. 395429.
Visser, W. (1990), More or Less Following a Plan During Design: Opportunistic Deviations in Specification, International Journal of
Man-Machine Studies, 33(3), pp. 247278.
Guindon, R. (1990), Knowledge Exploited by Experts During Software System Design, International Journal of Man-Machine Studies,
33(3), pp. 279304.
Adelson, B. a. E. S. (1986), A Model of Software Design, International Journal of Intelligent Systems, 1, pp. 195213.
Guindon, R. (1990), Designing the Design Process: Exploiting Opportunistic Thoughts, Human Computer Interaction, 5, pp. 305344.
Rist, R. (1990), Variability in Program Design: The Interaction of Process with Knowledge, International Journal of Man-Machine
Studies, 33, pp. 305322.
Ball, L. J., St. B. T. Evans and I. Dennis (1994), Cognitive Processes in Engineering Design: A Longitudinal Study, Ergonomics, 37(11),
pp. 17531786.
Hill, A. M. (2001), Comparisons and Contrasts Between Elementary/Primary School Situated Design and Workplace Design in
Canada and England, International Journal of Technology and Design Education, 11, pp. 111136.
Davies, S. P. (1991), Characterizing the Program Design Activity: Neither Strictly Top-Down or Globally Opportunitistic, Behaviour &
Information Technology, 10(3), pp. 173190.
Burns, C. M. a. K. J. V. (2000), A Participant-Observer Study of Ergonomics in Engineering Design: How Constraints Drive Design
Process, Applied Ergonomics, 31(1), pp. 7382.
Etelaepelto, A. (2000), Contextual and Strategic Knowledge in the Acquisition of Design Expertise, Learning and Instruction, 10,
pp. 113136.
Hill, A. M. (1998), Problem Solving in Real-Life Contexts: An Alternative for Design in Technology Education, International Journal
of Technology and Design Education, 8, pp. 203220.
Busby, J. S. a. K. P. (1998), The Situated Nature of Judgment in Engineering Design Planning, Journal of Engineering Design, 9(3),
pp. 273293.
Sonnentag, S. (1996), Planning and Knowledge about Strategies: Their Relationship to Work Characteristics in Software Design,
Behaviour & Information Technology, 15(4), pp. 213225.
Sonnentag, S. (1998), Expertise in Professional Software Design: A Process Study, Journal of Applied Psychology, 83(5), pp. 703715.
Dwarakanath, S. a. K. M. W. (1995), Decision-making in Engineering Design: Observation from Design Experiments, Journal of
Engineering Design, 6(3), pp. 191207.
Busby, J. S. a. K. P. (2001), Characterizonf Failures in Design Activity, Proc. Instn. of Mechanical Engineers, 215(Part B),
pp. 14171424.
Lai, J., Stella Mitchell, Marisa Viveros, David Wood and Kwan Min Lee (2002), Ubiquitous Access to Unified Messaging: A Study of
Usability and the Use of Pervasive Computing, International Journal of Human-Computer Interaction, 14(3&4), pp. 385404.

What Constitutes Good Design? A Review of Empirical Studies of Design Processes

Table A1. Coding of the database articles

531

532

M. Mehalik and C. Schunn

Table A2. Coding of the database articles, Part 2