Acting with Creative Confidence: Developing

a Creative Agency Assessment Tool

Adam Royalty, Lindsay Noelle Oishi, and Bernard Roth

Abstract Universities around the world are quickly introducing new learning
models aimed at developing creativity and innovation in students. A leading
model is the experiential teaching of design thinking as a creative problem solving
process aimed at enhancing students’ creative confidence. Although these programs
exist, little is known about student outcomes. Furthermore, the criteria by which we
evaluate student “success” is not well defined because these programs almost
uniformly have ambiguously stated learning objectives. This research uses qualita-
tive and quantitative data to capture and categorizes successful outcomes by
examining alumni of these programs. Based on these data is a scale that measures
creative agency, a fundamental outcome of teaching design thinking.

1 Introduction

This research focuses on a new model of teaching creativity and innovation through
a design process, often called design thinking (Cross 2007). In particular, this paper
focuses on one of the leading institutions in this field, The Hasso Plattner Institute
of Design at Stanford University (“d.school”). Stanford University and the d.school
have garnered a reputation for producing successful entrepreneurs and innovators
according to a recent Stanford Entrepreneurship Survey, and there is widespread
interest in replicating its educational methodology. For example, universities in
Japan, Chile, Malaysia and China have founded or are in the process of creating
educational programs modeled on the d.school. Corporations are also interested in
teaching design and creativity; in Europe, Germany’s design group Palomar 5 and
PICNIC in The Netherlands have created ambitious programs inspired by the
d.school’s educational methods. While the outcomes of these endeavors have not

A. Royalty (*) • L.N. Oishi • B. Roth

Stanford University, Stanford, USA
e-mail: [email protected]; [email protected]; [email protected]

H. Plattner et al. (eds.), Design Thinking Research, Understanding Innovation, 79

DOI 10.1007/978-3-319-01303-9_6, © Springer International Publishing Switzerland 2014
80 A. Royalty et al.

yet been fully evaluated, previous educational trends have demonstrated that
expensive, culturally grounded practices rarely work as expected when imported
to new international contexts.
One reason that imported educational methods tend to “fail” in new contexts is
that the expected results are often imperfectly understood and specified (e.g., “math
achievement” or “innovation”). To help those interested in the d.school and design
thinking avoid this pitfall, we first sought to describe the actual outcomes of the d.
school’s educational practices using qualitative and quantitative research methods
rather than anecdotal reports or case studies alone. Such impact questions are often
motivated by goals of program evaluation, that is, identifying what is pedagogically
effective for the sake of promoting or propagating it, and what is ineffective for the
purpose of improving it. At this level, only what works is important; why it works is
unimportant. On a more theoretical level, however, we were also interested in the
psychological mechanisms underlying whatever learning happens within an educa-
tional setting and curriculum such as the d.school. Our research, therefore, occupies
the intersection of psychological research and program evaluation: we wished to
identify the psychological constructs relevant to effective learning experiences
within a particular program (the d.school), with the goal of designing evaluations
that would reflect a common learning framework that would apply across educa-
tional settings.
Naturally, describing the d.school’s impact is an impractically large research
question. Any reasonably complex educational institution has myriad impacts and
innumerable contributing variables. Furthermore, as institutions like the d.school are
non-traditional and interdisciplinary, they often do not have specific or easily mea-
surable target outcomes for students. This is quite different from a graduate program
in Physics that teaches students to do research in one of several subfields, each with a
known set of standards and criteria with which to determine alumni achievement. At
the d.school, the only concrete outcome identified in the mission statement is the
production of “future innovators” (d.school 2004). Therefore, a first step in
investigating the impact of the d.school is selecting targeted outcomes for which
research will be appropriate and useful. One outcome of the d.school, for example,
could be graduates’ romantic relationships; however, from an educational perspective
this is of relatively minor interest. Given that the prevailing motivations for studying
and imitating the d.school stem from its apparent contributions to business and the
economy (see, for example, the Design Ladder developed by the Danish Design
Centre), we chose to focus on students’ professional outcomes after graduation.
A second step in designing research for understanding the impact of the d.school
is narrowing the field of variables we examine as characteristic of the institution.
The space of potential variables is vast, including, for example, geographical
location, student population, instructor characteristics, material resources, and
individual course curricula. Because the d.school is inherently concerned with
learning and teaching (Beckman and Joyce 2012), and because its unique pedagogy
is its most salient feature, we chose to study student outcomes related to the d.
school’s signature problem-solving approach, design thinking.
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 81

Fig. 1 Model of creative

confidence (Rauth
et al. 2010)

2 Background

While design thinking has been defined in different ways (Buchanan 1992), (Cross
2007), d.school courses are based on a common pedagogy that focuses on an overall
process with five core constructs: Empathy, Define, Ideate, Prototype and Test. On
a more specific instructional level, nearly all d.school courses involve techniques
such as interviewing, brainstorming, and rapid prototyping. Design thinking as it is
taught at the d.school goes beyond explicit pedagogy. The goal of the d.school is to
develop future innovators, who, in addition to performing discrete observable skills,
also have a characteristic set of attitudes and dispositions that propel them toward
creative activity and achievement.
According to a framework of a d.school teaching model (Fig. 1), mindsets and
an overall sense of creative confidence are built on top of repeated practice and
success with discrete techniques (Rauth et al. 2010) such as the design thinking
process (dt.process) and its various associated methods. These mindsets include a
bias towards action, radical collaboration, and being human centered. Other
dispositions, not specified in the model but commonly promoted at the d.school,
include constant reframing and rapid iteration.
The literature about design thinking outcomes is fairly sparse. A great amount of
design methodology research focuses on engineering or design teams and how they
perform (Eris 2004), (Brereton et al. 1996). Because the d.school does not train
designers, but rather helps students from all disciplines work in a more creative
way, the existing design research is not a sufficient base.
Kolb’s learning model (Beckman and Barry 2007) connects how the design
process helps students develop a sense of integrated thinking. This is very valuable,
but we are interested in learning how deeper behavioral aspects are affected, and a
more holistic view of how individual skills and techniques learned work together to
create the overall problem-solving-approach of design thinking. There is also
research exploring teaching models that are similar to design thinking, such as
Leifer’s (1996) work evaluating Product-Based-Learning Education, Gerber’s
examination of Design-Based Learning (Gerber 2011), and the Cambridge-MIT
Institute (Lucas and Cooper 2004; Lucas and Cooper 2005). Though such work
82 A. Royalty et al.

sheds valuable light on how the d.school teaching mechanism affects students, it
does not focus particularly on student outcomes beyond the learning experience.
In deciding on a research design, we also kept in mind that the outcomes believed to
be related to design thinking as it is taught at the d.school are based on the d.school
founders’ and instructors’ a priori intuitions and beliefs. It is quite likely that
graduates’ skills, dispositions and attitudes are not fully specified by either the
traditional design thinking model or that shown in Fig. 1. Therefore, the studies
described here exhibit a tension between our a priori research focus on graduates’
professional outcomes as they relate to the d.school pedagogy, and our desire to
remain open to discovering new variables of interest, in both outcomes/impact and
contributing factors.
We chose individual alumni as the unit of analysis versus organizations that have
d.school graduates because the d.school’s mission is that of personal behavior change
and developing individual attitudes, not redefining how companies operate. That said,
knowing how individuals who apply d.school learnings within a corporate context
does inform the impact of design thinking on a company (Gerber and Carroll 2012).
As confidence to think and act creatively came up frequently in the qualitative data,
an initial psychological construct we used to characterize this phenomenon was self-
efficacy (Bandura 1977). Self-efficacy refers to an individual’s belief in his or her
abilities within a particular domain, in this case, creative problem-solving. Another
design thinking institution has done some work on how self-efficacy may be related to
design pedagogy (Jobst et al. 2012). Self-efficacy, however, is only one part of a larger
construct, agency, which Bandura (1982) defined as the means by which “people can
effect change in themselves and their situations through their own efforts,” (p. 1175).
Agency includes not only self-efficacy, but also beliefs about the world, context,
physical and emotional states, social support, and other factors. Though it is more
complex, we felt that agency better reflected the multifarious nature of the creative
competencies that many d.school graduates exhibited. We defined creative agency as
individuals’ capacity to effect change in themselves and their situations to support
successful creative problem-solving.
Given that there is little research on the outcomes of programs that involve teaching
design thinking through problem-based learning methodologies, we decided to take a
wide-to-narrow approach starting with exploratory qualitative studies (an open-ended
survey with follow-up interviews) and an initial conceptual model based on our
observations. To test the model’s key psychological construct, agency, we did a
pilot test using a new scale, revised it based on the data, and then tested a second
version of the scale across multiple populations (study 3).

3 Study 1: Alumni Survey

Although we reported the background, methodology and initial results for Study
1 in last year’s report (Royalty et al. 2012), for this year, we analyzed additional
data and applied a new coding scheme to several questions. The preliminary goal of
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 83

this continued work on last year’s data was to generate a foundational set of
descriptive results regarding d.school alumni, so that stakeholders, prospective
students and the larger design education community can better understand the
alumni population. The second, more research-oriented purpose of these additional
analyses was to address the following new research questions:
1. To what extent does the d.school affect alumni professional outcomes (i.e. career
choice)?
2. To what extent are alumni professional activities related to d.school pedagogy?
As this study was exploratory in nature, there were no a priori hypotheses. For
the sake of completeness, we re-report the methods in the following section, but
interested readers can go straight to the results.

3.1 Method

3.1.1 Participants

We defined a d.school alumnus as any person who had taken at least one d.school
course, where a course was defined as a quarter (ten weeks) of unit-bearing
instruction. Based on student records, the participants we surveyed were a close
approximation of the entire population (approximately 670 alumni). The response
rate was 28 % for a final sample of 184. Among those reporting gender, 56 % were
men (n ¼ 73). Nearly all participants had been graduate students at Stanford
University (some had finished their degrees and others had not) and were affiliated
with various schools and programs, including Business, Law, Arts & Sciences,
Education, Medicine and Engineering. As the d.school officially offered courses
beginning in 2006, participants had been out of school and employed full-time from
zero to a maximum of 5 years. The majority (83 %) had graduated in 2008 or after.

3.1.2 Procedure

Given the large number of potential participants in this study, for ease of data
collection and analysis, we chose a survey methodology and used online survey
software (Qualtrics). Alumni were identified through student records and contacted
via personal email as well as a general Facebook announcement (it is possible that
some people received more than one notification or invitation to participate in the
study). Invitations were sent in May 2011 and the survey was open for 2 weeks.
Participants were free to take the survey at a time and place of their convenience.
There was no explicit incentive to participate; however, respondents had the
opportunity to join a mailing list that would notify them of upcoming alumni
events.
84 A. Royalty et al.

Table 1 Selected survey questions on outcomes of d.school pedagogy

Item Response type
What are the most important effects or Short paragraph
outcomes from your experience from the d.
school?
In the last month, how often did you apply 5-point Likert scale: not in the last month, once or
what you learned through the d.school in twice, once/week, 2–3 times/week, almost
your professional life? every day
What are some examples of how you applied Short paragraph
what you learned through the d.school in
your professional life in the last month?

3.1.3 Materials

All of the questions on the survey were designed specifically for this research.
Participants were asked about the current and previous occupations and occupa-
tional plans, how they applied what they learned at the d.school in their professional
lives (Table 1), and demographic questions. So as not to bias respondents towards
reporting specific techniques or general dispositions, we did not use the term
“design thinking” in any questions; instead, we referred to “what you learned at
the d.school”. The survey questions also did not offer any definition or examples of
“applying” what was learned at the d.school; respondents were free to respond
based on their own interpretation of this term.

3.2 Results

3.2.1 Alumni Professional Outcomes and Their d.school Experiences

By comparing participants’ self-reported planned occupation when they began their

graduate studies with their current occupation, we found that among those who had
a planned occupation, a majority experienced a career change. Table 2 shows the
frequency breakdown of all respondents, including those who had career plans and
those who did not when they began graduate study. Looking at only those
participants who did have a planned employment field and/or role when they
began at Stanford (n ¼ 122), 62.3 % of these (n ¼ 76) reported a difference
between the job/career they had planned on when they began their graduate
study, and their current employment.
A subset of participants (n ¼ 70) reported a career change and also answered the
question, “Thinking about the difference (if any) between your expected occupation
when you started at Stanford, and your current occupation, how much did your
experience at the d.school play a role in this change?” The mean response was 2.47
(SD ¼ 1.46), which corresponds to approximately halfway between “A moderate
amount” (scale point 2) and “A lot” (scale point 3).
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 85

Table 2 Categories and frequencies of career plans and outcomes for d.school alumni
Percent of total Mention d. Percent of category
Category Frequency (%) school mentioning d.school (%)
Career change 76 41.50 26 34.20
No career change 46 25.10 6 13.00
No original plan 59 32.20 3 5.10
No current job given 2 1.10 0 –
Total 183 100.00 35 19.10

Of the 76 career changers, 34.2 % (n ¼ 26) explicitly (and spontaneously)

mentioned their experiences at the d.school as a reason for this outcome. Even
among those who did not report a career change or did not have an original plan,
some spontaneously mentioned their d.school experience as having an impact on
their professional path.
To understand any potential role that the d.school had on alumni career choice
and professional outcomes, responses were coded from the question, “If your
current job is different from what you expected to do when you started at Stanford,
what factors played a role in this change?” Among participants who reported
working in an occupation that they had not planned on when they matriculated,
the most common reason was a change in professional interests or desires (Table 3).
Furthermore, approximately half of those who reported a change in interests or
desires for their job/employment (19 out of 37) explicitly mentioned the d.school as
a reason for this change.
For those who mentioned the d.school as a reason for their change in occupation,
the overall theme was that various aspects of the d.school experience (e.g., a
particular class, teamwork) enhanced their self-understanding or their understand-
ing of professional roles. Table 4 gives some examples of these open-ended
responses.
To understand how specific pedagogical elements of the d.school’s instructional
methods may have affected alumni professional outcomes, we focused on the open-
ended questions; “What are the most important effects or outcomes of your
experience at the d.school?” and “What are some examples of how you applied
what you learned at the d.school in your professional life in the last month?”. As not
all participants responded to these questions, the number of respondents for this
analysis was 131.
From these responses, we coded for elements of the d.school design process,
defined a priori. We also included teamwork as an a priori coding category, given its
centrality to the design thinking learning process. We also coded for dispositions
that correspond to key tenets of the d.school curriculum. “Creative confidence” is
a new term that we used to describe a theme of feeling comfortable with
creative endeavors and a sense of ability and self-efficacy in the creative domain
(Table 5).
86 A. Royalty et al.

Table 3 Reasons for occupational change among d.school alumni

Reason for occupational change Frequency (%) Number mentioning d.school (%)
Change in interests/desires 37 (20.2 %) 19 (51.4 %)
No response/vague response 21 (11.5 %) 3 (14.3 %)
Could not get desired job 9 (4.9 %) 1 (11.1 %)
Unexpected opportunity 4 (2.2 %) 2 (50 %)
Wanted to make impact 3 (1.6 %) 1 (33.3 %)
Money/necessity 2 (1.1 %) 0
Total (all career changers) 76 (100 %) 26 (34.2 %)

Table 4 Example responses to question about change in occupation relationship of alumni

professional activities to d.school pedagogy
Theme
Self-understanding “It just feels like I can bring more of myself to work and after I had
a taste of that experience at the d.school, it was hard to go
back”
“I was taught a method of problem solving that fit better who I am
(a creative person) than the traditional engineering way of
solving problems”
Specific d.school experience “I went in looking to play a business role (i.e., strategy) on
creative teams. . . I ended up playing more of a hybrid role.
Time on teams at the d.School gave me an opportunity to play
exactly the type of role that I wanted”
“I took Needfinding, a class in Product Design, and found my
passion in applying empathy to understand problems and
finding new opportunities”
Understanding of profes- “I was introduced to the possibility of designing products that help
sional opportunities impoverished people by the d.school”

3.3 Discussion

Re-analyzing the survey data with a particular focus on career changes and how
alumni viewed the contribution of d.school pedagogy revealed moderate evidence
for the claim that educational methods at the d.school have an impact on graduates’
professional outcomes. Across all participants, nearly twenty percent brought up
the d.school as a factor in their professional paths and current occupation. It is
important to note that these participants mentioned the d.school’s influence without
prompting – the question about why they were working in their current occupations
did not mention the d.school, and was placed before (and on a separate page from)
the question that specifically asked about the role of the d.school. Furthermore, the
results show that alumni perceive their d.school experience as a causally
contributing factor in their occupational choices.
As participants varied in what they found most significant about their experience
at the d.school, we cannot conclude that it was, in fact, d.school pedagogy rather
than other factors that drove these outcomes. The data on alumni use of
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 87

Table 5 Frequency of d.school alumni mentioning d.school pedagogies

Frequency Percentage of all respondents
Component of d.school pedagogy (n) mentioning (n ¼ 131) (%)
Empathy 56 43
Define 32 24
Ideate 48 37
Prototype/test 49 38
Teamwork 24 19
Creative confidence 33 25
Comfort with risk, ambiguity, 32 24
change, or failure
Bias towards action 14 11

fundamental design thinking methods, however, do support a hypothesis that

alumni apply d.school lessons in professional settings. Of the six a priori categories
that we coded for, empathy, ideation and prototyping were the most commonly
mentioned. This is unsurprising as these techniques are in sharp contrast with more
traditional work methods, and apply in many situations. The methodology of
“defining” a problem was mentioned less frequently, probably because it is a
more subtle, procedural step that can be difficult to describe. It was, however,
somewhat surprising that only a fifth of participants talked about teamwork, as this
is central to nearly all d.school experiences.
Arguably the most important outcome of the survey was the emergence of
several new coding categories. We view these as overall psycho-behavioral
“dispositions”; in particular, the themes of creative confidence, comfort with
seemingly negative states (such as failure or ambiguity), and having a bias towards
action (rather than intellectual reflection or rationalization) were frequently
observed. In order to better understand how alumni may have developed these
holistic dispositions, in addition to more explicit design thinking/problem-solving
skills, we decided to do in-depth follow-up interviews with a subset of the alumni
surveyed in Study 1.

4 Study 2: Alumni Interviews

As Study 2 was described in the previous report (2011), we will just briefly review
the methodology and major findings in order to describe the larger research arc.
Sixteen d.school alumni (ten women) from Study 1 were interviewed in person or
via teleconference (Skype). Seven participants currently worked in business
(including healthcare, technology and entertainment), three were self-employed
or entrepreneurs, three worked in education, two in consulting or research, and one
was in engineering. The interviews were semi-structured and were 45–100 min.
88 A. Royalty et al.

The interviews probed how “successful” alumni viewed what they had learned at
the d.school and how that may have influenced both their career choices and their
professional activities. By “successful”, we merely meant those who said that they
chose to, and were able to, use what they learned at the d.school (which they defined
for themselves) in their working life. On a high level, we wanted to see how former
d.school students described their learning path and their subsequent behaviors in
order to better understand how the d.school may be contributing the formation of
“future innovators”. More specifically, we focused on their observable behaviors
and dispositions, and how alumni perceived the d.school as shaping these
outcomes.
The overall result of the interviews was a better understanding of how some d.
school alumni develop and demonstrate creative confidence. Beyond simply
learning basic design thinking processes and techniques, participants who reported
using what they learned at the d.school had a strong desire and confidence to
actually apply skills and methods in their workplace. This can be a daunting and
risky endeavor in some traditional work contexts, where normal or known
behaviors are expected and job responsibilities are tightly codified. In the strongest
case – seen in several interviewees – creative confidence in the professional realm
even extended to using the design thinking process to launch or change one’s entire
career. For example, one participant left her job in engineering to pursue teaching,
and explicitly stated that it was what she learned at the d.school that gave her the
confidence to “try out” or prototype a new career, even though it meant taking
significant professional risks.
Another, unexpected example of how alumni demonstrated creative confidence
that came up in multiple interviews was how they built creative environments
around themselves at work and in their personal lives. This often took the form of
manipulating physical space to allow for more creative working styles, and in
some cases, making the space similar to the interior of the d.school. Some
participants also reported teaching colleagues about design thinking as a way to
enable better collaborations and facilitate their own creativity. Across most of the
alumni we interviewed, the rich descriptions of how they used what they learned
at the d.school went far beyond simple applications of basic techniques or tools
such as brainstorming or rapid prototyping. Instead, their stories reflected an
overall approach to work and life that was informed by dispositions to solve
problems in an innovative manner, and propelled by the confidence to do so in
multiple ways.
Successful alumni seemed to share a set of behavioral patterns and dispositions
that went beyond what we had captured in Study 1. Although we could observe
those characteristics in interviews, we wanted to define a single psychological
construct that could be measured accurately and efficiently. In Study 3, we
attempted to narrow and quantify what was observed in the survey and
interviews, by creating a scale that attempted to measure the new construct of
creative agency.
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 89

5 Study 3: Competency-Based Creative Agency Scale

(CBCA)

In developing a scale measuring the impact of a design-thinking based pedagogy,

we revisited the data from the surveys and interviews and coded for key
competencies frequently demonstrated by successful alumni. Again, we defined
“successful” as those who reported remembering and using “what they learned at
the d.school” (self-defined) the most within our sample. We then created the scale
directly from those factors. The questions were worded in an abstract manner rather
than by referring to specific scenarios, in order to limit the influence of previous
experience or gender differences. The key factors that emerged were:
Sources (gathering information from external sources)
Comfort (with ambiguity)
Mastery (of one’s own creative process)
Environment (developing creative environments)
Anti-perfectionism (reducing a sense that everything must be perfect)
Prototyping (developing a culture of prototyping)
Perseverance (increased in the face of failure)
Facilitation (confidence to lead a creative process)
Openness (to changes in thought, direction, beliefs, et cetera)
Process (being able to describe one’s own creative process)
Creative Output (solving problems in creative ways)

5.1 Method

5.1.1 Participants and Procedure

1. Current d.school students (General). The scale was implemented in the Autumn
quarter of 2011 with current d.school students drawn from two classes. The first
was a large, introduction to design methodology course (n ¼ 72) (95 % of class)
and the second was a more specialized course exploring the intersection of
design and society (n ¼ 13) (76 % of class). The questionnaire was given at
the beginning (September) and end (December) of the academic quarter and no
incentives were offered for participation. Of the 85 pretest participants,
68 completed the posttest.
2. Current d.school students (Education). Thirteen students in a small, Education-
focused d.school course took the questionnaire as a pretest and posttest at the
beginning and end of the Spring 2012 academic quarter. The pretest was given at
the beginning of (April) and the end (June) of the Spring quarter; eight students
took the posttest. A $15 gift card was offered for participating.
90 A. Royalty et al.

Table 6 Items on the competency-based creative agency scale

Item Construct
Find sources of creative inspiration not obviously related to a Creative idea sourcing
given problem
Effectively work on a problem that does not have an obvious Comfort with ambiguity
solution
Change the definition of a problem you are working on Openness
Shape or change your external environment to help you be more Building creative
creative environments
Share your work with others before it is finished Anti-perfectionism
Try an approach to a problem that may not be the final or best Prototyping
solution
Continue work on a problem after experiencing a significant Perseverance after failure
failure
Help others be more creative Creativity facilitation in others
Identify and implement ways to enhance your own creativity Mastery of creative process
Explicitly define or describe your creative process Knowledge of creative process
Solve problems in ways that others would consider creative Successful creative problem-
solving

3. Non-d.school students (Education). Twelve Education students who were not

taking the Education-related or any other d.school course prior to or during
Spring 2012 were recruited as a control group for the d.school/Education group.
These students completed the questionnaire online.
4. Workshop participants: Business executives. Participants in a 3-day design
thinking workshop for business executives received an email with a link to an
online questionnaire, 2 weeks prior to the workshop in March 2012.
57 participants took the pretest (98 % response rate). To date, six participants
have taken the posttest, which was also provided via email, 10 weeks after the
workshop. Those who took the post-test received a t-shirt.
5. Workshop participants: Business executives and teachers. Participants in a
3-day design thinking workshop for in-service teachers and business executives
took the questionnaire on paper, at the beginning and end of the workshop
(n ¼ 62) (52 % response rate at pretest), which took place in May, 2012.

5.1.2 Materials

There were 11 to 14 Likert-scale questions on the CBCA scale; all were created for
this study. Eleven items concerned self-assessed confidence related to a set of
general competencies in the area of creative problem-solving (Table 6). The
CBCA-related Likert-scale items were presented as a group and preceded by the
question, “How confident are you that you could. . .”. Unlike the items in Study
3, the new questions were written to be sufficiently general so as to apply to nearly
any situation or professional domain. There were five response categories: “Not at
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 91

50

Competency-based Creative Agency

45

(Range: 11-55) Men (n=30)

40

Women
(n=33)
35

30
Pre Post
Time

Fig. 2 Mean competency-based creative agency, by time and gender

all confident” (scale point 1), “A little confident” (2), “Moderately confident” (3),
“Very confident” (4), and “Completely confident” (5). There were no “Don’t
Know” or “Not Applicable” response choices; however, participants were free to
leave any item blank.
After implementing the 11-item questionnaire with the first group of partici-
pants, all of whom were d.school students, we added three new items on topics
relatively unrelated to competency-based creative agency. These items concerned
comfort with technology, artistic ability, and goal achievement. They were added
to provide data on whether any apparent outcome of d.school study was specific
to the design thinking competencies we found in earlier studies, or a more general
positive impact.

5.2 Results

While there was no overall effect of gender, there was some evidence of a trend for
gender to interact with time, such that women started out with higher CBCA but
ended at a similar mean score, F(1, 53) ¼ 3.04, p ¼ .087 (Fig. 2). The effect size
for this possible interaction was, however, quite small, with a partial eta-squared of
.05.
There were too few participants in the non-d.school student “control” group for
statistical analysis; however, among the eight who took both pretest and posttest,
there was no apparent change in mean CBCA. At pretest, the control group mean
was 35.25 (SD ¼ 5.31); at posttest, the mean was 36.12 (SD ¼ 3.68). Figure 3 plots
the means and standard errors for the d.school students versus non-school students
at pretest and posttest.
92 A. Royalty et al.

48

Competency-based Creative
46

Confidence (Range: 11-55)

44
42
40 d. school (n=55)
38
36 --------- control (n=8)
34
32
30
Pre Post

Fig. 3 Mean competency-based creative agency, by time and group

50

40

30

20

10
d.school students teachers/ executives non-d.school students
(n=55) (n=23) (n=8)

Pre Post

Fig. 4 Mean competency-based creative agency, by sample and time

For the 23 teacher/executive workshop participants for whom we have both

pretest and posttest scores, mean CBCA at pretest was 32.91 (SD ¼ 7.13) and mean
CBCA at posttest was 38.65 (SD ¼ 8.50). A paired-samples t-test showed a
significant effect of time, t(22) ¼ 4.50, p < .001. Figure 4 shows the pretest
and posttest scores for each participant – only three decreased in their CBCA
scores, while the remaining 20 increased.

5.3 Discussion

Study 3 tested a new scale, competency-based creative agency, with three main
samples: d.school students, similar students not taking d.school classes, and
teachers and executives who took a 3-day design thinking workshop. As the scale
items were based on empirical research on the applied skills of successful creative
problem-solvers who were d.school alumni, it was hypothesized that d.school
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 93

experience would be associated with greater CBCA (see Fig. 4). The results
provided evidence for this relationship, as d.school students increased in CBCA
from the beginning to the end of the course, across two different academic quarters
and in three different d.school classes. In one of those quarters, a small group of
non-d.school students also took the scale; they did not show any significant change
in CBCA. While a larger control group would, of course, provide better support for
the hypothesis, these data do suggest that the increase in CBCA seen among d.
school students was not merely due to maturation or demand characteristics of the
scale items.
Results from the teacher/executive sample also supported the hypothesis that d.
school experience increases CBCA. Although the design thinking workshop was
only 3 days, participants reported a significant increase in CBCA from the begin-
ning to the end. Data from this sample also showed that the scale could be used with
a non-students sample and retain its internal validity as well as its usefulness in
showing change over time.
It is interesting to note that the average pre-test score of the teachers and
executives was slightly lower than the average pre-test score of the non-d.school
students, which in turn was marginally lower than the average pre-test score for d.
school students. There was, however, an apparent difference between the mean
starting CBCA for d.school students and teachers/executives, with d.school
students reporting higher CBCA at pretest. Taking a design thinking course
shows willingness to take risks by studying a topic that is relatively new and not
a regular part of any degree program; a higher pretest score among these students,
then, is not surprising.

6 Implications

The three studies reported here have led us toward a larger model of how individual
creativity develops and is subsequently expressed. One of the original concerns of
our research was to investigate how the d.school and other educational institutions
can foster the kinds of skills that lead to real-world innovations. Our initial model of
creative outcomes, as observed in our research, includes four constructs: self-
efficacy, agency, output and impact (Fig. 5).
Creative self-efficacy is belief in one’s own creative abilities, and it is developed
in many ways, such as mastery experiences (successfully completing creative
activities) and vicarious experiences (learning from others who are creative).
Self-efficacy is the most important part of creative agency, which goes beyond
mere belief in oneself and extends to real-life application of creative problem-
solving. Those who have high creative agency are capable of producing creative
output – real work, artifacts, and ideas. Creative output determines creative impact,
this is the change made due to a creative effort.
94 A. Royalty et al.

Creative Outcomes
Creative Self- Creative Agency: Creative Output: Creative Impact:
Efficacy: Applying one’s Manifestations of Effects of creative
Belief in one’s creative abilities applying creative actions
creative abilities ability

Internal External

Fig. 5 A model of multiple creative outcomes

This model is not intended as a linear or temporal path, as there is multi-path

feedback among the nodes. Self-efficacy naturally leads to greater agency, which
may result in greater output, which strengthens both self-efficacy and agency. This
can result in more creative output, which can strengthen agency that can lead to
even more creative self-efficacy. The model is also not intended to be exhaustive:
given the work being done to better understand structures of the creative brain, we
may soon include segments to the left of creative self-efficacy, representing more
internal structures. Likewise, there may be multiple segments to the right of
creative impact, when one considers the impacts of not just individuals, but also
institutions and other large systems.

7 Work in Progress

We are currently in the early pilot stages of three studies aimed at the creative
output node of our model, that is, the ideas, work, artifacts, procedures and other
“deliverables” that creative agents make.

8 Conclusions and Future Work

Understanding the impact of any educational institution or method is a daunting

task. By focusing on the unique problem-solving approach taught at the d.school,
design thinking, and grounding our work in an understanding of the characteristics
that d.school alumni report, we have begun to provide evidence for the impact of d.
school pedagogy on a newly identified construct, creative agency. Instead of
focusing on more traditional outcome measures, such as salaries, patents, or awards,
we chose to look at the competencies, capacities and habits that those demonstrated
by alumni who say that they use what they learned at the d.school in their work and
lives.
What emerged from our qualitative research is a sense that many d.school
alumni are equipped with a strong sense of their own ability to be creative in
Acting with Creative Confidence: Developing a Creative Agency Assessment Tool 95

problem-solving contexts (self-efficacy), and that this propels them to act to change
their own thinking and their environments in order to perform better as creative
problem-solvers (creative agency). These actions are characterized by multiple
competencies, first learned and practiced at the d.school, and then carried into the
workplace. These include beliefs (anti-perfectionism, failure as opportunity),
knowledge (of one’s creative process), dispositions (towards prototyping, radical
collaboration), and skills (teaching others, shaping one’s own environment). We
also observed that these capacities were largely domain-agnostic, and were reported
by teachers, engineers, entrepreneurs, and even chefs in equal measures.
In our quantitative research, we moved towards an efficient and accurate way to
represent and measure the outcome we observed among the alumni, by designing
and testing two scales. While the first did not function well as a singular-construct
scale, we did see interesting results, such as, more experienced d.school students
said that they were more comfortable starting a company than students with less d.
school experience. Our second scale, which built upon the lessons of the first, was
more successful at providing a valid and reliable measure of the impact of d.school
pedagogy. By testing it with several populations, we found initial support for its use
beyond the d.school. In the near future, we hope to repeat our studies with larger
and more robust control groups, and implement the scale in other educational
programs and institutions, both similar to and different from the d.school.
With more evidence for the validity and reliability of the Competency-based
Creative Agency scale, we would like to eventually expand the research to its
potential predictive validity, its neurological correlates, and ways to use it in
controlled experiments. For example, using the CBCA in the workplace may
yield insights into how workers with high creative confidence are recognized for
their creativity and the benefit they provide to their companies (predictive validity).
We are also collaborating with another research team on fMRI studies that incor-
porate the CBCA scale. Finally, to truly test the effects of the d.school pedagogy,
we would need to run experimental studies that manipulate what we believe are key
aspects of design thinking as it is learned at the d.school, and then measure how
these components affect CBCA. This will take us closer to understanding the
contribution of the d.school to educational methods, as well as how the successes
of the d.school may be not only repeated, but also enhanced, at other organizations
throughout the world.

References

Bandura A (1977) Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev
84(2):191–215
Bandura A (1982) Self-efficacy mechanism in human agency. Am Psychol 37(2):122–147
Beckman S, Barry M (2007) Innovation as a learning process: embedding design thinking. Calif
Manage Rev 50(1):25–56
Beckman S, Joyce CK (2012) Reflections on teaching design thinking to MBA students. In:
Business as an agent of world benefit conference, 2–5 June 2009
96 A. Royalty et al.

Brereton MF, Cannon DM, Mabogunje A, Leifer L (1996) Characteristics of collaboration in

engineering design teams: mediating design progress through social interaction. In: Dorst K,
Christiaans H, Cross N (eds) Analyzing design activity. Wiley, Chichester
Buchanan R (1992) Wicked problems in design thinking. Des Issues 8(2):5–21
Cross N (2007) Designerly ways of Knowing. Birkhauser Verlag AG, Boston
d.school (2004) dschool.stanford.edu. Retrieved 10 July 2013 from https://dschool.stanford.edu/
Eris O (2004) Effective inquiry for innovative engineering design. Kluwer, Boston
Gerber E (2011) Extracurricular design-based learning: preparing students for careers in
innovation. Int J Eng Edu 28(2):317–324, 2012
Gerber E, Carroll M (2012) The psychological experience of prototyping. To appear in design
studies (forthcoming)
Jobst B, Köppen E, Lindberg T, Rhinow H, Moritz J, Meinel C (2012) The faith-factor in design
thinking: creative confidence through education? In: Design thinking research studying
co-creation in practice. Springer, Heidelberg, pp 35–46
Leifer L (1996) Evaluating product-based learning education. White Paper. Center for Design
Research, Stanford University, Palo Alto. Retrieved 29 Feb 2008
Lucas WA, Cooper SY (2004) Enhancing self-efficacy to enable entrepreneurship: the case of
CMI’s connections. MIT Sloan school of management working paper, 4489–04
Lucas WA, Cooper SY (2005) Measuring entrepreneurial self-efficacy. In: EDGE conference:
bridging the gap: entrepreneurship in theory and practice, Singapore, 11–13 July 2005
Rauth I, Köppen E, Jobst B, Meinel C (2010) An educational model towards creative confidence.
1st Proc. ICDC, Kobe
Royalty A, Oishi L, Roth B (2012) ‘I Use it Everyday’: pathways to adaptive innovation after
graduate study in design thinking. In: Design thinking research measuring performance in
context. Springer, Heidelberg, pp 95–105