Liao 2020

Human–Computer Interaction
ISSN: 0737-0024 (Print) 1532-7051 (Online) Journal homepage: https://www.tandfonline.com/loi/hhci20
A framework of artificial intelligence augmented

design support
Jing Liao, Preben Hansen & Chunlei Chai
To cite this article: Jing Liao, Preben Hansen & Chunlei Chai (2020): A framework of artificial
intelligence augmented design support, Human–Computer Interaction
To link to this article: https://doi.org/10.1080/07370024.2020.1733576
Published online: 30 Apr 2020.
Submit your article to this journal
Article views: 5
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=hhci20
HUMAN–COMPUTER INTERACTION
https://doi.org/10.1080/07370024.2020.1733576
A framework of artificial intelligence augmented design support

a
Jing Liao , Preben Hansenb, and Chunlei Chaia
a
College of Computer Science and Technology, Zhejiang University, Hangzhou, China; bDepartment of Computer and
System Sciences, Stockholm University, Stockholm, Sweden
ABSTRACT ARTICLE HISTORY

Recent advances in Artificial Intelligence raise interest in its participation in Received 20 June 2019
design activity, which is commonly considered to be complex and human- Revised 19 February 2020
dominated. In this work, we aim to examine AI roles in early design stages. Accepted 19 February 2020
The human ideation components and design tools related to AI are discussed KEYWORDS
in a framework of AI-augmented design support. The framework develops a Human computer
hierarchy of design cognition (basis), approaches and principles. The cogni- interaction theory; design
tive models are constructed in an empirical study of 30 designers (26 for supports; artificial
analysis, 4 for pilot study) by concurrent Think-Aloud protocol and behavior intelligence
analysis. The process of producing new design ideas is explained by a
transparent analysis of designers’ language and behaviors. Three strategies
to organize cognitive activities in design ideation are summarized: develop
structured consideration, relate to a scenario, and stick-to designing. These
strategies suggest AI could act as (1) representation creation, (2) empathy
trigger and (3) engagement, in principles of “knowledge-driven” and
“decompose-and-integrate”. The design support with AI provides new per-
spectives on computer-based design tools that limit to well-defined design
variables. The framework is built on a generic notion of design activity and
“mimic” human design rationales, expected to benefit research of domain-
independent computational design supports and cognitive supports.
1. Introduction
Artificial Intelligence (AI) is experiencing the third tide. AI has evolved capabilities that are
comparable to humans, such as perceiving objects in images, understanding and responding to
human conversations, playing games, and translating languages. Looking into this prospect, we raise
an interesting question: Could AI plays roles in design ideation, the generic creative process in
design activities?
Previous AI researches are interested in design process models simulating with human reasoning
and problem-solving. Many studies (Fallman, 2003; Gero & Kannengiesser, 2004; Gero, 2006; Kim &
Ryu, 2014; Zimmerman & Forlizzi, 2014) associate design process with an “interact- centric” design
optimization framework (Gero & Kannengiesser, 2006). The design optimization framework con-
siders interaction as a pivot connecting design tools, designers, design problems and design out-
comes (artifacts). Naturally, the connected factors could be explained by the “wicked” nature of
design (Buchanan, 1992; Rittel & Webber, 1973), which points toward certain objectivity (involving
complicated domains, materials, activities, and environments) in human experience. Interactions in
creative activity (such as design tasks) involve multivariate design situations related to the design
process, the designers, and the artifacts (Lubert, 2005; Visser, 2009). The multivariate factors of
design activity can pose great challenges in designing interactions, particularly those aiming for
increasing creative insights (Lubert, 2005). Studies uncovered limitations of traditional
CONTACT Chunlei Chai [email protected] Institute of Industry Design, College of Computer Science and Technology,
Zhejiang University, 502, Zetonglou, 38, Zhedalu, Xihuqu, Hangzhou 310058
© 2020 Taylor & Francis Group, LLC
2 J. LIAO ET AL.
computational tools for these challenges e.g., exploration of tacit knowledge, the framing of problems
(Bernal et al., 2015; Mothersill & Bove, 2018). It is still challenging to support design thinking, where
designers raise questions, change existing solutions and beliefs, and generate new ideas.
Nowadays, AI has entered a “deep learning” age. Powerful computation models contribute to an
novel research area where non-human aspects could promote inspiration (Chen et al., 2019), and co-
creation of novel and meaningful contents (Oh et al., 2018; Fan et al., 2019; Davis et al., 2016).
Recent studies suggest a collaboration with AI could help discover “unexpected design reference”
(Chen et al., 2019; Fan et al., 2019; McDonald, n.d.; Schmitt, n.d.) and provide a content experience
(Oh et al., 2018). These practices suggest the design possibilities of emergent AI technologies in the
HCI context. Nevertheless, we have identified several gaps:
● only a few studies examined computational support for early design stages, where the core
value of design activity inhabits. The origin of creative insights and the sense-making process
remain to be explored.
● the traditional computer-aided design paradigm fails to consider the nature of design problems
(Bernal et al., 2015). Most design methods and tools are not developed in open, dynamic design
situations that reflect the uniqueness and essence of design activity. These methods and tools
suffer from “invariants” of design problems (features of a structured model do not occur in
design problem space).
● Design practices demand theoretical investigations concerning designers and design activity,
more than design and evaluation of AI applications. Computational support for ideation is
dependent on multi–variant factors (Lubert, 2005). An optimal solution for all design situations
may never exist because of the complexity of the cognitive process and individual diversity
(Lubert, 2005).
Broadly, the deep learning AI technology is a method to achieve machine learning (ML). Machine
learning increasingly plays an important role in shaping how users interact and experience technol-
ogy products (Yang, 2018). Both User experience (UX) practitioners and researchers have recently
become more interested in design opportunities involving ML technologies (Dove et al., 2017). As
such ML poses an opportunity in the sense that we could explore how ML might function as
a creative material (Gillies et al., 2016; Kuniavsky et al., 2017). This emergent strand of work within
HCI has spawned the notion of ML as a design material (Dove et al., 2017).
This article focuses on the potentials of AI in early design activities involving creativity and
decision-making. These activities include setting, framing, reframing problems, and initiating, devel-
oping and evaluating design ideas and concepts. We provide a framework describing explicit roles of
AI in design ideation as representation creation, empathy trigger, and engagement. The framework
suggests approaches to assist cognitive patterns in design process. The cognitive patterns of design
representations and design rationales are investigated via an empirical study of 30 designers (although
we only include 26 participants in the end) with concurrent Think-Aloud protocols and behavior
analysis. The study discovers opportunities for AI to support human creativity and decision-making:
AI could provide inspiration, inform design scope and request design actions. The principles and
challenges brought by design complexity and variables of design individuals are also discussed.
The following section will introduce research gaps in an interdisciplinary field of design and AI.
These gaps motivate a framework of AI for design ideation described in Section 3. Section 4 presents
an observational study of designers’ cognitive processes, which provides empirical evidence relating
to the framework. Significant characteristics in design rationales are identified. Section 5 introduces
three key principles in developing new design support using AI. Section 6 discusses challenges in the
AI-augmented design framework and Section 7 suggests limitations and future work.
HUMAN–COMPUTER INTERACTION 3
2. Gaps in AI-augmented design

The research gaps are related to the nature of design problems and design progress models. Firstly,
existing design support overlooks some critical design stages. Design ideation (referred to as
“conceptual design” in design research) involves creative and reflective thinking. It is challenging
to provide external support for this high-order thinking. Secondly, design ideation involves compli-
cated cognitive processes and has a broad ontology context. Significant design factors need to be
considered in the process of designers developing parallel and nontrivial paths to a problem,
however, they have not been well examined in design situations.
2.1. Background and progress

On reviewing early studies discussing roles of AI in design activity (Campbell et al., 1998; Gero,
1990; Simon, 1996) and logical elements of design (Campbell et al., 1998; Dorst & Cross, 2001; Gero,
1996, 1987; Visser, 2009), we find these researches share a common view, that is, designers,
environment and artifacts mutually interact with each other in the design process. Design is
a reciprocal process where the designer’s actions, and choices of tools affect each other
(Dalsgaard, 2017). It is not an isolated mental process but connects both the designers, design
artifacts, as well as the design goals (Dalsgaard, 2017; Gero & Kannengiesser, 2006, 2004). Empirical
studies suggested effective computational tools can help designers make better-informed decisions
(Hay et al., 2017), refine needs and specifications of design artifacts when design move toward its
goal (Bernal et al., 2015; Hay et al., 2017). And the design process is an ongoing crafting process:
designers gain insights into how people experience things in the process of making models, scenarios
and prototypes (Koskinen et al., 2013).
Recent AI algorithms echo interactive, incremental creation and processing of design materials
(Figure 1), suggesting an interesting direction toward “intelligent” design support (Davis et al., 2016).
Image translation (Chai et al., 2018) as shown in Figure 1a, image synthesis (Zhang et al., 2018) as in
Figure 1b and style colorization (Gatys et al., 2016) as in Figure 1e could inspire novel insights,
intuitive design understandings and paths to desirable solutions; image editing (Zhu et al., 2016) as
in Figure 1c and image creation (Reed et al., 2016) as in Figure 1d allow flexible development of
ideas and imaginary modeling and simulation to facilitate design evaluation and iteration.
Collaborative creation with AI may promote a pleasing design experience (Oh et al., 2018).
Nevertheless, these studies are oriented to particular tools and are limited to the design of
symbolic and visual communication. Design includes a wide range of domains: materials, services,
complex systems, working/learning/playing. As such, design indicates multiple meanings to
designers and design tools vary between design fields. Moreover, although studies proposed novel
ontologies of intelligent design tools (Mothersill & Bove, 2018) and design guidelines for AI
(Amershi et al., 2019; Burgess, 2018), the potential scope of “intelligent” design support has not
been examined.
Figure 1. Potential AI techniques in supporting design. (a) pixel-to-pixel image translations (b) text-to-image synthesis (c) sketch-
based image editing (d) drawing by what-and-where (e) deep neural style transfer.
4 J. LIAO ET AL.
2.2. In-improving design problem

Understanding design cognition is essential for human-oriented computer support for creative
activities (Bonnardel & Marme`che, 2005; Lubert, 2005). Studies of design processes point to three
major stages in human cognition: the formulation of problems, the generation of solutions, and the
utilization of design process strategies (Cross, 2001). Influential studies in early design activities e.g.,
“design as reflective practice” (Schön, 1987; Schon & DeSanctis, 1986) emphasized the significance of
repetitive framing of design problems and things to be designed under a unique, value-laden, and
uncertain situation (Stumpf & McDonnell, 2002).
However, existing computational tools overlooked the pivotal framing process that yields tacit
design knowledge for such distinct and implicit situations (Figure 2). Previous AI-driven design
focuses on algorithm and cognitive inference in design activity (Campbell et al., 1998; Gero, 1996,
1990, 1987), including agent-based design systems for facilitating design actions (Campbell et al.,
1998), knowledge-based systems aimed at design knowledge management and reuse
(Chandrasegaran et al., 2013; Liu & Lim, 2011; Park, 2011). Creativity, the core of ill-defined,
complex and iterative design creation (Howard et al., 2008) also integral to human intelligence, is
hardly assisted by computers (Lubert, 2005). Computer-aided Design (CAD) tools have sufficiently
explored support for solution-finding, but the critical phase of sense-making and decision- making is
still under-examined.
Tacit design knowledge is usually generated in the early stages of design activity. In design
practice, computational support shows limited success in early design activities (Bernal et al.,
2015; Gero & Kannengiesser, 2004) due to “invariants” of design problem space. The “invariants”
refer to features of a structured information-processing model that do not all occur in problem space
(Goel & Pirolli, 1992). In other words, design problem spaces exhibit major invariants across design
problem-solving situations and major variants across non-design problem-solving situations in
which computational models can fail. For example, a design-agent only works if the conditions of
a well-defined design task are specified and developed, i.e. in the late design process where concrete
forms and solutions are being generated. But not all requirements of a design-agent are known at the
outset of a design task (Gero & Kannengiesser, 2004), and requirements of explicit design descrip-
tion are not available in the early design process (Mothersill & Bove, 2018).
These researches suggest insufficient HCI examination for problem (re)framing processes. Most
computational tools focus on a well-developed design situation of analysis (Pourmohamadi et al.,
2011) and design knowledge reuse (Liu & Lim, 2011; Park, 2011), which is less dependent on
creativity and thinking that involves in very complicated and mutual factors. The gap would further
increase due to in inconsistency between the adoption and advance of digital technology. Design
tools in practical usage are still dominated by physical objects (Hay et al., 2017), although digital
artifacts can affect forms and functions of workplaces, communication and culture in the design
process (Fallman, 2003; Löwgren & Stolterman, 2004; Zimmerman et al., 2007).
Figure 2. Limitations of traditional computational tools in design phase of generating tacit knowledge, Bernal et al. (2015).
2.3. Structured design process model

The in-improving design problem determines the design process is intrinsically iterative (Council,
2006), parallel (Hay et al., 2017) and strategically complicated (Cross, 2001). A key research problem
in AI-based design is design process modeling, aiming to address the following aspects of the design
process: the state of the design; the goal structure of the design process; design decisions; rationales
for design decisions; control of the design process; and the role of learning in design (Mostow, 1985).
Early AI design models contribute to the formalization of design progress and optimization of a set
of variables defined by designers and related through an explicitly understood and code-able
algorithmic logic (Loukissas, 2012; Mothersill & Bove, 2018). Particularly, the FBS model uncovered
designers’ cognitive processes move across things to be designed with problem formulation and
design tools (Gero, 2006), providing understandings about designing in an open, dynamic world
(Howard et al., 2008).
The structured design process models have a common assumption that design process is worked
via transitions of design states (Gero, 1990; Maher, 1990; Reich, 1991). Design states in structured
models are presumed to cover all situations in design space and design moves are represented by the
transition path of states. But this assumption fails to comply with features invariant of design
problem discussed in section 2.2., such as being mentally parallel and simultaneous (Kim, 2011).
Most design process models presumed the iterative and parallel exploration of design rationality is
formalized and explainable (Howard et al., 2008). However, interactions in design process (Figure 3)
can vary in different design situations defined by design problems, the designers, the design tools,
and the artifacts (Gero & Kannengiesser, 2006). Accordingly, some major problem features can
locate out of these models (Goel & Pirolli, 1992) and the design rationality lacks explanations for
practical situations. Some intellectual abilities and key cognition processes could be dismissed. The
core of design, generating insights and reflection are overlooked.
To examine underlying patterns of design process in a situation, it requires understanding of
design rationality (why design is designed in a way) (Kim & Ryu, 2014), while the intra-personal
and inter-personal differences in human intelligence, character, and experience contribute to sub-
stantial HCI challenges (Lubert, 2005). Designers differ greatly in their profile of abilities, preferred
working styles and personality traits (Lubert, 2005). Also, novel and experienced designers work
differently in ideation process (Bonnardel & Marme`che, 2005). The diversity can result from
multivariate human creativity: the intellectual abilities (e.g., analogical thinking, divergent thinking,
Figure 3. An interactive view of design optimization (Gero, 2006).

6 J. LIAO ET AL.
selective combination skills) and domain-relevant knowledge, thinking styles (e.g., being more or less
detail-oriented), personality traits (e.g., risk-taking, tolerance of ambiguity and uncertainty, open-
ness, perseverance), as well as other factors interact to yield creative potential (Lubert, 2005).
The next section will introduce a framework of AI-augmented design support, which discusses cognitive
strategies in problem formulation, design approaches and principles with the perspective of design
rationality. We associate with designers’ cognitive activities in an empirical study of design ideation. The
study explains and reflects on the framework, and contributes to implications, design principles and
challenges of the framework in a broad context. The details of the study will be introduced in Section 4.
3. A framework for design ideation with AI

The framework for design ideation with AI (Figure 4) is built on the design optimization framework
(Figure 3) and a generic-design hypothesis context (Goel & Pirolli, 1992) independent of design
domains (Visser, 2009). The design optimization framework represents fundamental processes of
designing in a dynamic world. The generic-design hypothesis characterizes complex design “situa-
tions” on three main dimensions, the design process, the designers, and the artifact (Visser, 2009).
These dimensions are explained with cognitive strategies identified in an observational study of
a conceptual design task of engineering design.1
We will start with three notions of designers’ cognitive activities and introduce cognitive patterns
in design ideation. In each notion of cognitive activities, the explicit roles of AI will be discussed with
two typical stages of human design ideation (Figure 6): discovery (gather, sort information) and
reframe/define (generate hypotheses, identify novel directions) (Council, 2006). The opportunities
for AI as design support (Figure 7) are explained with an ontology of AI design mechanisms
(Mothersill & Bove, 2018).
More specifically, in the discovery phase, AI’s computational mechanism is triggered by designer’s
selection and parse initial information (e.g., design research notes, interview transcripts, inspirational
Figure 4. The framework of AI-augmented design ideation supports.
The engineering design emphasizes “close to users” and is much oriented to HCI insights compared with other design domains of
1
materials, architecture, etc.

images) into insightful representations (e.g., entity patterns, logical representations). In reframe/define
phase, suggestions and references are produced to juxtapose the information from the discovery phase,
for example, informing potential design states by AI’s predictions. Designers are encouraged to imagine
unfamiliar concepts and novel design goals, variables, feelings, and experience.
3.1. The origin of inspirations

Designer’s cognitive activities orient toward three notions: Construction of Representations (Visser,
2009), Problem-Solving (Simon, 1996), and Reflective-in-Action (Schon & DeSanctis, 1986). We
associate the three notions and outline critical cognitive activities in design ideation (Figure 4a,c).
The critical cognitive activities are identified by the empirical study (Section 4.).
3.1.1. Elicit psychological representations by concept structure

The concept structure refers to the explicit representations of mental notions created and processed
during design ideation (Kim, 2011). The mental categorization of information processing has critical
effects in design ideation according to the model of designers’ cognitive and affective processes (Kim,
2011). Experienced designers can develop a habit of managing their design activity with structured
concepts, particularly in the generative phase where to find the “good form” of design concepts. For
example, they would create a map of design outcomes and its candidates during the ideation process
or a list of semantic descriptions of design concepts, issues or precedents to derive at the beginning
(as shown in Figure 10).
The findings of empirical observation support Kim’s model (Kim, 2011) that creating a concept
structure of own thinking benefits early design generation. We observed a burst of ideas after the
designer outlined consideration of proposed ideas. In some cases, idea prompts come highly abstract,
e.g., as a keyword. The keywords are probably not be developed into concrete ideas immediately;
they could be clustered and inter-related to each other to generate novel insights. The abstract
concepts, as well as associations among them, are structures of belief, perception, and appreciation,
within which designers construct a view of the problem and attempt to solve it (Stumpf &
McDonnell, 2002).
Interestedly, some ideas are inspired by the initial irrelevant pieces of stuff. Designers are inspired
in a process of articulating all thoughts in mind. The concept structure may be “unstructured” at
a certain moment. Nevertheless, it will be developed completely and will echo the psychological
representation of creation in the end. The “I can continue to think about … ” example in Self-
Instructions suggests implicit connections to future concepts. Another example, the divide- into-
cases covers a range of aspects to be considered, e.g., comfort, entertainment. Associations among
these aspects are evaluated after and evolving during ideation.
3.1.2. Empathy thinking by scenario descriptions

The scenario description is defined as a semantic descriptor of designers’ memorizing process.
Recalling design scenarios is a crucial element of design activity. It provides frames of reference
shifting, which shift the subjective frame of reference in hypothetical situations (Shah et al., 2003). It
also provides design precedents that change the “invariants” of design problems (Goel & Pirolli,
1992) by relating the current design problem and conditions to previous experience. The scenario
description reflects the ability of empathy thinking. Empathy thinking encourages intuitive design
outcomes that meet real needs rather than assumed needs and thus contribute to innovation
(McDonagh & Thomas, 2010). When designers recall the previous experience, encountered pro-
blems and potential solutions to a similar design problem, design contexts are broadly explored
during into a stage of what they are intended to design.
In design optimization mentioned previously, Gero and Kannengiesser modeled the memorizing
process as a connection between designer’s internal world (Gero & Kannengiesser, 2004), which
contains experience, expected world of design goals and values, and the external world consists of
8 J. LIAO ET AL.
tools, environments, and external information representations. Changes in a world can affect the
others, e.g., problem formulation connects the designer’s memory to the current situation and design
actions are then taken toward goals in the expected world. Accordingly, design optimization is
pushed forward by such dynamic interactive effects of three worlds of designers, design materials
and design goals. In the empirical study (Section 4.), scenario descriptions are found to occur with
writing/drawing action, which suggests designers’ visual thinking strategy to aid external information
processing (Ware, 2010).
3.1.3. Stick to design and tackles of suffering

It is common for a designer exhausts idea when developing thinking from a pre-defined design space
and reflecting on new design possibilities (Schon & DeSanctis, 1986). We observed that some
participants stop to speak and draw when “exhaust minds”, while most productive participants
keep on investigating thoughts, expectations, and experience to endure the situation. The struggles of
facing the design barriers gain interest in examining design strategies in inspiration generation. In
the empirical study, having a “fight” rather “truce” with design barriers are found to result in better
design outcomes. Productive participants (F02, F07, and M11) continue to speak and express their
reference of thinking, expectations, and what they usually do when facing difficulty. This design
strategy is identified to have psychological alignments with the ideation component of random
connection (Shah et al., 2003). Designers are enhanced by incubating connections in a wide range of
domains, not necessarily related to project context (Shah et al., 2003).
Another strategy is to implement details of design solutions. This strategy is found to be
frequently used by mechanical designers (M08, M10, M12, M13, F12). They are also less likely to
fall into the situation where they could not think about anything. The mechanical designers prefer
sketching the very details of design, so they normally have works to do. They find new inspirations
in linear development of ideas (individual, coarse-to-fine), compared with some of the others (e.g.,
industrial designers) may develop many ideas in parallel. The linear development (focus on an idea
and develop it from coarse-to-fine) may lead to very similar ideas such that fewer design concepts
could be generated. Nevertheless, working on details of existing solutions could be a panacea to drive
designers out of a situation that being tracked with confusion (M01, F02) and low self-esteem
(M02, F03).
3.2. Design ideation supports with AI

We have explained cognitive strategies and actions in design ideation. In this section, we introduce
how these strategies inform the explicit roles of AI as design ideation supports (Figure 4b). The roles
include decision-making supports to guide designers to perceive design problems, and action-taking
supports to augment their ability to ideate design concepts (Figure 5). We describe the roles of AI act
in design activity as representation creation, empathy trigger, and engagement (Figure 6). The roles
are categorized into three branches respectively (provide inspiration, widen scope, and call for
action) based on Haritaipan’s taxonomy of 112 design creativity tools in practice (Haritaipan,
2019). To engage interdisciplinary understandings, a summary of AI computational design methods
and HCI relevance are suggested in Figure 7; design creativity tools related to AI techniques are
described by two languages in design and algorithm in Figure 8.
3.2.1. Representation creation: provide inspiration

3.2.1.1. Disclose underlying patterns of design entities. Management of design entities helps
designers more quickly structure their research into constructive categories, identify interesting
targets, and produce insights (Mothersill & Bove, 2018). Traditional CAD systems such as
SolidWorks often require designers to manage large-scale design entities and work on a design
that has already been decomposed into underlying attributes. Understanding the underlying
attributes and sorting entities are human-based, laborious work. With computational supports,
Category Approach Limitation

Provide Random triggers deep image and video generators (McDonald, n.d.)
inspiration Context-related Semantic style transfer and image synthesis
triggers (McDonald, n.d.)
Widen Information visualization of image/image category similarities
scope (Mothersill & Bove, 2018)
Call for Instructions -
design Methods “Machine create, Human produce” (Schmitt, n.d.)
action Empathizing with interactive systems build on user-generated
users contents (W. Park et al., 2019)
Team building interactive team collaborative learning and
problem-solving (Chopade et al., 2018)
Game -
Figure 5. The creativity components and examples of design support with AI. The approach is based on Haritaipan’s taxonomy of
112 design creativity tools in design practice.
Figure 6. The roles of human (hollow arrow) and AI (solid arrow) in design discovery, and (re)framing/definition. Our framework
aligns AI’s creation with human designer’s creation.
design entities could be managed in some ways to facilitate structuring design knowledge.
Random and context-related management is usually handled by probability selection and database
(Hay et al., 2017). Differently, AI allows organizing design entities by mathematical proximity. For
10 J. LIAO ET AL.
Category Method AADS HCI Aspects

Dimension uses many dimensions to Disclose Learnability,
reduction connect all of the information underlying Reliability,
that is abstracted down into a patterns of Trust
smaller, more manageable set of design entities
key features
Association allow integrating analogical Provide design Interoperability
creation information and designer’s stimuli
operations
Information collect information in a way that Create Learnability,
organization allows easy analysis and information Reliability
comparison layout
Representation help designers look for patterns Build upon AI Interoperability,
creation and more quickly structure their Learnability
understandings
Representation use diverse information Link to existing Reliability
mixture representation, i.e. word/image knowledge
to encourage multiple
interpretations
Information use diverse ways of interpreting Drive thinking Surprise
interpretation information to create off the context
unexpected inferences and
illumination
Figure 7. Align AI-augmented design support (AADS) and HCI prominence with mothersills’ ontology of AI–involved computa-
tional tools.
example, a cluster of mathematical representations of entity neighborhood computed by AI could

assist tasks of understanding the underlying attributes of information (Mothersill & Bove, 2018).
The visualization of proximity may engage visual thinking (retrieve and present information in
ways of visual perceptions) (Ware, 2010). It also engages designers to build analogical thinking
(relating an attribute to another) (Ball & Christensen, 2009; Christensen & Schunn, 2007), which is
important to design ideation. By suggesting implicit associations of entities, designers could
interpret current design and find paths to a preferred state. Creative insights are expected to be
inspired by investigating similarity and analogy (Gentner & Markman, 1997) among the related
entities.
3.2.1.2. Create logical representations. The logical representations are typical AI-based design
knowledge. Early AI models have developed logical elements of design (Hatchuel & Weil, 2003;
Gero, 1987, 1990; Maher, 1990). Traditional logical CAD tools often have a very structured way of
automating design optimization, using procedural knowledge and design language which focuses on
geometric relationships and are pre-defined or user prescribed (Bernal et al., 2015; Mothersill &
Bove, 2018). Here, creating logical representations by AI keeps the logical tradition of transmitting
among design states, while variant by enhancing designs with user-guided design inference, i.e. the
structure of design variables is defined by users rather than fixed by computation and algorithms,
and are adjusted during design iterations rather than remain unchanged. Furthermore, if user-
related data become available to AI, it is likely to produce inference from user-generated content
Category Descriptions in Design Language Descriptions in AI Language

Random triggers from a wide range of produce new values of materials,
triggers domains, which are not necessarily not necessarily comply with pre-
related to the project context defined rules or algorithms
Context- triggers related to the project produce materials according to
related context, i.e. keywords, pictures, rules or algorithms that adapted to
triggers items or imaginary scenarios. semantic, contextual or temporal
associations
Information sources of useful information, i.e. provide knowledge examples /
concept explanations, context representations, describing
details, knowledge of behavior variables in current state and goals
maps. to reach in a future state
Instructions provide random advice, asking provide semantic stimuli from a
questions, and ways of thinking set of advice or questions
Methods design methods provide inspirations provide understandings and
and smooth design process in auxiliaries in design phases
ideation, idea evaluation and
selection
Empathizing present persona characters, user provide explanations based on
with users behavior simulation user specifications or inference
from contents generated by users
Team support group cooperation and building intuitive, communicative
building communication and sharable knowledge
representation
Game in the format for entertainment and initiate approaches to change
encourage engagement and emotion states and reach a
creativity as a mood-changer satisfactory situation
Figure 8. Descriptions of Creativity tools category in design language and AI language. The design language is from Haritaipan’s
taxonomy.
such that design paths are led by users. As such, we could track a history of design states involving
user operations, design artifacts, and information fed into AI. The design history could help build
the relationships among previous ideas, and then help designers decide potential ones to implement.
The implicit intents are significant but may be difficult to capture previously. Now, AI may fit this
condition with contextual signals and situation-aware interactions.
3.2.1.3. Link to existing knowledge. AI extends the ability of computational tools assisting in
connecting diverse design concepts, which is a crucial part in early creative process (Mothersill &
Bove, 2018). Today’s very parametric CAD tools such as SolidWorks neither provide the cognitive
supports in search nor highlight new information that the designer might not have thought of
(Bernal et al., 2015). Traditional computational supports that designers use to find information on
their initial prompt is in the form of the ubiquitous semantic search engines. The computational
mechanism of AI extends the metaphorical search capabilities; therefore, it would potentially help
designers structure the search prompts, and find unexpected information in their search activities,
leading to more novel design solutions (Mothersill & Bove, 2018). The AI can act as “random
triggers” (Haritaipan, 2019) – the creativity supports extensively used in analogical thinking (transfer
12 J. LIAO ET AL.
an idea from one domain to another to generate novel solutions) (Hey et al., 2008). The deep neural
networks allow integrating analogical information into computation, which is a key feature to
encourage divergent thought and idea generation (Gero & Maher, 1993; Mothersill & Bove, 2018).
More importantly, AI’s results are abstract and unconventional, they are likely to trigger “unpre-
dictable inferences” that is valuable for early design activities (Bernal et al., 2015). To achieve these,
design examples generated by AI could be related by context, created from parallel manipulations on
existing examples, and crafted with holistic and diversified transformations.
3.2.2. Empathy trigger: widen scope

3.2.2.1. Provide design stimuli. The AI-generated content, speaking from the external form, can be
characterized as design stimuli. Design stimuli are external representations consumed in design-problem
-solving when those are present in the designer’s work environment (Goldschmidt & Smolkov, 2006).
Images produced by AI estimate an existing picture at pixel-level, resulting in similar but not alike to an
existing one. In some cases, the generated images could be dissimilar to any of the materials being used to
train AI models. The presence of different kinds of visual stimuli, as analyzing suggested by research in
cognitive psychology as well as design thinking, can affect practicality, originality and creativity of design
performance (Goldschmidt & Smolkov, 2006). In this way, AI performs stimuli augmentation to
encourage designers to ideate more practical, original and creative design outcomes. On the other
hand, design stimuli could be viewed as visual information presented in a design context. Information
is one of the creativity components (Haritaipan, 2019) and the source of design knowledge and ideas
(Chan et al., 2011). It offers designers creative insights through analyzing design entities across different
domains and in broad context (Chan et al., 2011). From this perspective, AI could provide inspirations
due to the nature of rich information: constitute information relations among existing and evolving one
by computation. Empirical studies also suggest information relations, e.g., analogy (Moreno et al., 2016),
reasoning (Hatchuel et al., 2011) can overcome fixation and enhance design ideation.
3.2.2.2. Drive thinking off the context. Design ideation is not always driven by targeted, logical
processes to solve problems defined in the current context. Great design works need to break the
context, propose new problems and insights out of a box of existing knowledge. The random, aimless
perceiving from users is a significant method to acquire inspirations, creative senses in particular
(Köppen & Meinel, 2015). Although user-related inspirations are difficult to capture, they are
suggested to have close relationships with design empathy (Kouprie & Visser, 2009). Design empathy
is a significant methodology in both design activity (Köppen & Meinel, 2015; Kouprie & Visser, 2009)
and HCI practice (Wright & McCarthy, 2008), where one’s feeling and experience are shared with
observer (i.e. designer, researcher). AI could enhance the emotional perceiveness of the observer by
suggesting intuitive results from awkward materials such as users’ statistics and usage trajectory. The
empathy ability to connect designers’ knowledge and experience with users is expected to increase
(Wright & McCarthy, 2008). The enhanced empathy ability will contribute to practical and creative
design thinking which is significant to the nature of the design (Köppen & Meinel, 2015).
3.2.3. Engagement: call for design action

3.2.3.1. Associate with designer’ operations. Different to problems of a well-defined task, design
problem is co-evolving with finding design solutions (Dorst & Cross, 2001; Hay et al., 2017).
Designers learn new insights in paraphrasing, modeling, and prototyping (Koskinen et al., 2013;
Zimmerman & Forlizzi, 2014) and re-define design problems; the process is known as reframing. AI
can be triggers of design actions to drive designers into reframing. For example, AI may predict if
designers stuck in barriers and then provide oriented instructions or ask questions to help them out
of barriers. As observed in empirical study, idea inspiration related design actions include self-
answering, self-instructions, describe design examples/usage/insights. Designers were very likely
stuck in fixation (Crilly, 2015) if they are uncertain about what they are thinking and doing, and
what they should do to overcome fixation. The self-answering and self-instruction (examples in
Section 4.5.) could engage reflections on the current situation and actions to develop own design
activities in different situations.
Broadly, AI could be incorporated into interactive systems that receive data about what designers
have done, are doing, and recommends actions appropriate in the current design context. The
system should be connected with information sources that designers require to work on a design,
such as design examples, the knowledge base of design features/elements. Apart from WHAT DATA
the system may require, HOW to recommend AI’s prediction can raise interests in the HCI
community. A recent study in Human-AI interaction suggests to passively trigger automatic actions,
provide rich operations, and transfer controls of tools to designers (Oh et al., 2018). AI’s suggestions
can be redundant and tiresome. The paradigm of “user lead, AI suggest with enough details” is
shown to provide a more content user experience while bringing efforts in selecting desired tools
(Oh et al., 2018). The proactive suggestions can reduce redundant actions, but should be carefully
designed to avoid a feeling of intrusion and distracts.
3.2.3.2. Build upon the AI. There are increasingly growing design tools and methods integrated
with AI. Google’s Deep Dream tools enable the user to re-create images and videos based on
individual templates (McDonald, n.d.). Chai et al. show a generative adversarial network (GAN)
can translate a sketch into diverse image contents given design works in the online design commu-
nity (Chai et al., 2018). These examples show AI may reverse the traditional computer-aided
design (CAD) process, where designers create a preliminary idea and CAD tools assist to refine the
idea (“I create, the machine develops”). With AI, the design process can start from AI-generated
content. Designers create a design based on AI-generated content (“Machine creates, I develop”).
The AI-generated results are evaluated and re-designed by designers. A design project of the
paradigm of “machine creates, I develop” is introduced in (Schmitt, n.d.).
4. An empirical study of design rationality

In the previous section, we discussed strategies, approaches and explicit roles of AI in design
ideation. This section introduces an observational study that investigates ways of designers use
and transform internal and external representations of design cognition (Kim, 2011) during design
ideation. The implications of AI-augmented design support on designers are discussed (Figure 4d).
To include significant design factors (e.g., design problem), we examined the cognition of individual
designers in a conceptual design task and the design outcomes were evaluated with both design ideas and
design concepts produced for the conceptual design task. To be noted, a design concept is a collection of
embodiments (design solutions) that completely cover all the requirements of a design situation; it differs
from a design idea that is considered as an embodiment covers only some of the requirements of a design
situation (Salustri, n.d.). The design concept is given much attention in many creative and conceptual
design tasks. Also, the design concept distinguishes our study of design activity from those on general
cognitive activities which probably overlook to evaluate the scope of design outcomes.
4.1. Aim and method

The study aims to discover patterns of designer’s meditation of design solutions: how they generate
new knowledge, what they think and perform and in what patterns they follow during the design
process. The research follows the three-stage approach (Bayazit, 1993) for transparency (Pedgley,
2007), including data collection, data analysis, and knowledge structuring. We use the concurrent
Think-Aloud protocol (Alhadreti & Mayhew, 2018; Jourdenais et al., 1995) to analyze cognitive
processes. Participants are asked to speak out what they are thinking, why and how design is carried
out. We combine concurrent Think-Aloud with Kim and Ryu ’s framework of protocol analysis in
examining design thinking (Kim & Ryu, 2014), such that the collected data can discover designers’
difficulties or expectations in a non-intrusive way. The behavior analysis investigates non-verbal
14 J. LIAO ET AL.
indicators of design cognition patterns. Overall, the analysis is hierarchical and retrospective for
reliability and validity.
4.2. Participants and task

We recruited 30 designers, including 4 participants in a pilot study. The participants enrolled in the pilot
study were set to (1) produce examples of the design task; (2) explore experiment settings and avoiding
oversights; (3) standardize experiment candidates. The rest of the 26 participants in formal experiment
had approximate even distribution on gender and experience in design areas of industrial, product,
mechanical and art design (Figure 9). They were allowed to observe design examples (solutions proposed
in the pilot study) when working on the design task. In the analysis, we invite three design experts to
evaluate the total number of generated design ideas and concepts, as well as the duration of design actions
spent in stages of the creative process. Their backgrounds will be detailed in the next section.
Designers were enrolled in a common engineering design task of “designing a computer desk”.
The engineering design is typical user-centered and would provide insights for the HCI community.
The design task is generic to industrial, product, mechanical and art design. The potential design
Participant Background Experience Ideas Concepts Concepts/Ideas (%)

M01 Industry 4 24 19 79.2
M02 Industry 4 11 9 81.8
M03 Industry 4 11 7 63.6
M04 Industry 4 16 7 43.8
F01 Industry 4 16 13 81.3
F02 Industry 4 32 22 68.8
F03 Industry 4 22 9 40.9
F04 Industry 4 18 16 88.9
F05 Industry 4 13 5 38.5
F06 Industry 3 15 4 26.7
F07 Industry 3 30 26 86.7
F08 Art 4 12 1 8.3
M05 Art 4 11 9 81.8
M06 Art 4 12 4 33.3
M07 Product 4 5 3 60.0
F09 Product 3 19 9 47.4
F10 Product 3 25 13 52.0
F11 Product 3 25 4 16.0
M08 Mechanic 3 25 13 52.0
M09 Mechanic 3 11 10 90.9
M10 Mechanic 3 8 4 50.0
M11 Mechanic 3 23 18 78.3
M12 Mechanic 3 15 4 26.7
M13 Mechanic 3 20 3 15.0
F12 Mechanic 3 23 13 56.5
F13 Mechanic 3 6 4 66.7
Figure 9. Participant specifications and design ideation performance.

subjects appear in the experiment environment. This allows designers to have a similar ability to
investigate design context and work on the design task as they usually conduct in practical design
activities. Because we are interested in how designers form and deliver ideas, we used an explicit
problem expression rather than abstract language e.g., “designing a way of supporting” to build
intuitive and common understandings of idea formation and reduce ambiguity in the analysis. To
inspire creativity, we introduce the “SUSPENDED JUDGMENT” component of ideation (Shah et al.,
2003) in addition to experiment requirements. Participants were asked to propose as many as
solutions considering originality, practicality, and feasibility. We provide a pen, a pencil, an eraser
and papers for sketching, which help express ideas and form design solutions in conceptual design
(Buxton, 2007; Landay & Myers, 2001). The experiment brief including the design task and
requirements (detailed in Appendix) are printed in a paper in A4 size.
4.3. Procedure
The experiment lasts 50 minutes, including 5 minutes instruction to be familiar with the design task
and materials, 5 minutes observation of design examples and 40 minutes designing which is
suggested sufficient for concept generation (Tsenn et al., 2014). We follow Bayazit’s three-stage
approach (Bayazit, 1993) to capture and analyze own design activity for methodology transparency
(Pedgley, 2007). We define three phases of the study: observation, protocol and behavior analysis,
and discussion. In observation, we collect raw materials from the participant and the environment
(Figure 10 shows the camera view). In the analysis, these materials are interpreted by the on-site
instructor and design experts. Finally, the interpretations are evaluated and reflected by all members
in discussions, such that the results manifest underlying perceptions, insights about problem-setting,
and aims, approaches, expectations about problem- solving.
Figure 10. A participant was formulating design concepts in the experiment. The participant wrote down lists of factors of design
solutions to reexamine design issues and investigate new directions.
16 J. LIAO ET AL.
● Phase I (Data Collection) Collect data on designers’ thinking and archive this in an unstruc-
tured and unanalyzed form. We collected sketches produced by 4 participants in the pilot study
as design examples to be used in the formal experiment. The design examples help build
understandings of the design task. The rest of the 26 participants were invited in the same
conceptual design task with formalized instructions. The instructions were appropriate to
concurrent Think-Aloud requirements as well as responses from participants in pilot study.
We recorded the design process and sketches of the 26 participants as raw materials for analysis
(a sketch example is shown in Figure 11).
● Phase II (Protocol Analysis and Behavior Analysis) Analyze data by a hierarchical proce-
dure. We followed the concurrent Think-Aloud protocol in the analysis of participants’
aims, methods, actions in the design process. Participants were asked to speak out every-
thing in mind, including what they are concerning, thinking, doing, and feeling. We
involved two design experts of averaged 7 years’ experience in coding the individual design
process and reflecting how findings (in both pilot and formal experiment) indicate parti-
cipant’s design trajectory. The criteria and examples of design process code are shown in
Figure 12.
● Phase III (Knowledge Structuring) Present findings and discuss their wider validity and
implications. We involved the experiment instructor and a Ph.D. candidate who has 9-years
of design research experience in data interpretation and evaluation. They have access to
protocol analysis results and behavior records produced in Phase II. The instructor parti-
cipated in design experiments and supervised transcription of verbal recordings and align-
ment with videos. The Ph.D. candidate supervised the design process code and its
implications. The results and findings are presented with all members to validate our
understandings.
Figure 11. An example of sketches produced in the experiment. The words recorded when sketching is: “There is a telescopic rod
at the elbow, and the telescopic rod has a groove after it extends. The measurement of pressure can be used to adjust the height
of the context and the angle of placement to support the elbow. It will ease the shoulders and improve comfort ”.
Code Description Example

A Analyze design task For people working on this table, e.g.,
programmers, I think of a survey that describe
their work boring and tedious; and their
repeated/continuous labors lead to health
problems.
G1 Ideate and generate Then, we can focus on to design an interesting
ideas/concepts table to overcome their boredom. As I
mentioned before... (describing and sketching)
G2 Sketching -
E Assess and implement I have seen a very funny example, that is, huh,
ideas/concepts spark. The screen will display sparks falling
from words when they (programmers) typing.
But this technique cannot fundamentally relief
the pains of tedious work. (describing and
sketching)
Figure 12. The criterion of process coding and example code of independent and integral design concept words.
4.4. Materials and notations

All raw data, analytic materials, and results were preserved for collaborative analysis and review
(Zimmerman & Forlizzi, 2014). A camera with a microphone was set above the participant to
capture their words and behaviors. In data evaluation, all sketches were scanned and preserved.
In phase I, we printed participants’ words and annotate with diverse colors of markers. Vague
words, e.g., participants’ murmurs were examined by both design experts. Unavailable words or
conflict judges were dropped out. In phase II, all well-developed concepts were identified; the
concurrent words and actions were coded. The recorded actions were encoded into phases of
Analysis, Generation, Evaluation according to creative process model summarized in (Howard
et al., 2008):
● fact analysis & problem-finding as Analysis (A): exploring and analyzing problems;
● idea-finding as Generation (G), including idea generation G1: propose factors or descriptions of
the idea, and sketching as G2: creating a sketch of the idea;
● idea-developing as Evaluation (E): looking back, verifying and relating the idea to others.
The process code of 26 participants (Figure 14) indicated that the design process can iterate with
certain patterns. In the case study of the most productive participant (F02), we observed three
patterns in the generation of ideas:
(1) a sudden whimsy < G >;

(2) infer by analysis < A, G > or reasoning < E, G >;
(3) infer by connections with previous concept < E, G >.
We classified the first two types of ideas as “new concept”, and the third one as “developed
concept”. Then, the identified ideas were divided into normal ideas and two categories of concepts:
the new concept and the developed concept based on patterns of process code. The associated
previous concepts to a developed concept were annotated in parentheses (Figure 16). In total, the
participant proposed 17 new concepts and 4 developed concepts. We evaluated the number of design
18 J. LIAO ET AL.
concepts rather than the design ideas since a concept is complete to resolve a design situation. The
concept was denoted as Ci where i is the temporal order of concepts (the abandoned concepts were
not labeled). The annotations of the design process and concept generation of F02 are shown in
Figure 16.
4.5. Results
4.5.1. Patterns of design rationality
We are interested in design concept generation and the transformation of design ideas into design
concepts since the research aims for design ideation rather than cognitive ideation. We examined design
rationality patterns by both qualitative and case study. The generic characteristics were found in
qualitative analysis. Design ideation strategies were investigated in the case study. The criteria of
Figure 13. The duration of different design phases in the experiment. The experiment lasts 50 minutes.
Figure 14. The design process of 26 participants annotated by think-aloud protocol and Howard’s model. The blank denotes
a participant remains silent and does not perform any action during an interval.
Figure 15. Ideas and design concepts created by 26 participants. F02 is the most productive participant, proposing 32 design ideas
and 22 design concepts. F07 is the most efficient participant, proposing 30 design ideas and 26 design concepts.
Figure 16. The annotation of the design process (proc.) and concept outputs (out.) for the most productive designer (F02).
The * denotes the generation of the concept involves in describing the usage scenarios.
a design concept differentiate from design ideas was detailed in the description of function, structure or
behavior of a computer table to a specific problem, or a sketch illustrating those descriptions. The most
frequent sentences participants use and provide a burst of inspirations are categorized as follows:
4.5.2. Self-answering
● “What kind of function or experience do I expect?” (F04)
● “Whether there is anything that can be improved” (F04)
● “Can it be like … or?” (M02, F04)
● “What can be combined with a computer table?” (F04)
● “Who will use a computer table?” (M01)
● “How to make this computer table … (e.g., lighter or more convenient)” (M06, M07)
20 J. LIAO ET AL.
● “What else is there on the computer desk itself?” (F01)

● “How do people sit here?” (F11)
4.5.3. Self- instruction

● “I can start with my usual computer use.” (F01)
● “I can continue to think about … ” (F03, F08, F12)
● “I need to start to enter a sky-high and being more innovative” (F11)
● “Aside from what I just thought … ” (M10, M13)
4.5.4. Describe previous example/usage/idea

● “I usually put my computers/phone/limbs/hands here … ” (M09)
● “I think of in a movie/someone mentioned … ” (F03, F04, M11)
● “I remember previously … ” (F04, F05, M11, F11)
● “What I said before is that … then … ”/“What I didn’t consider before is … ” (F01, F03, M07,
F11, M08, M11)
4.5.5. Divide into cases

The productive participants tend to explore the design problem space from coarse and general to
fine and individual level, i.e. creating a list of vague factors and then specifying each factor or
combination of factors to develop mature concepts such as:
● “Regarding age/profession/usage/ergonomics … ” (F02, M02, M06, F11, M08)
Several observations can be concluded from Figures 13 and 14. First, most participants dominantly
prefer sketching, which is consistent with the significance of formulation experience in design
process (Dix & Gongora, 2011). Within 26 participants, 21 of them spend the most time on
sketching. The other participants, M05, F09, M08 presented a dominate of speaking out their
ideas in framing problems, and M02 described and proposed design solutions. In additions, it is
straightforward that mechanical design students (M10, M13, F12, and F13) spent a significant
majority of time on sketching, while M02 and F07 (Industry design) preferred proposing ideas by
words and M05 (Art design), F09 (Product design), and M08 (Mechanical design) show a slight
tendency in analysis phase.
An interesting finding is, a sufficient sketching time does not always lead to good design results.
When comparing the design outcome (Figures 9–15) and period time of design process (Figure 13), we
found that mechanical design participants (ideas: 16.38, concepts: 8.63, transform efficiency: 52.7) spend
the most time on sketching, and their averaged proposed concepts surpass participants in art design
(ideas: 11.67, concepts: 4.67, transform efficiency: 40.0) and product design (ideas: 18.5, concepts: 7.25,
transform efficiency: 39.2). However, design ideas and concepts proposed by mechanical design
participants are not as many as industry design participants (ideas: 18.91, concepts: 12.45, transform
efficiency: 65.8). Considering education of design expertise, it is reasonable since the ability to form
design concepts is considered to be one of the most core skills of an industrial designer, while
mechanical design, product, and art design are interested in the mechanism, functionalities or esthetics.
The solutions proposed by designers in these areas can be described in very detailed, compared with
industrial designers yielding incomplete and abstract design outcomes.
In the analysis of participants’ behaviors and concurrent Think-Aloud documents, we found that
descriptive and imaginary thinking is critical to efficient design ideation. Descriptive thinking refers
to depicting a design situation to find potential problems. The imaginary thinking, for example, can be
brainstorming a map of potential design directions (M01, F02, F04, F07). Besides, aimless thoughts lead
to many “new concepts” (< G >). The moments of idea generation, thinking of “irrelevance” is a feasible
way to explore problems, options, and possibilities of improvements. Participants questioned the ground
of existing understandings, or randomly linked to distant design attributes and evaluate the possibilities.
The process of “aimless” probing may not always contribute to new ideas but can inspire problem
background and underlying factors. This process, known as memory retrieval, is a critical method to gain
new ideas in design ideation from a problem or solution previously learned in daily experience.
In the case study on the most productive designers (F02), we found a great number of ideas coming up
with the scenario description. The most productive participant described the usage scenario in propos-
ing 12 new concepts (C1, C2, C5, C7, C10, C11, C13, C16, C17, C18, C19, C21) and 2 developed concepts (C6,
C9) (Figure 16). The scenario descriptions occurred in each of the concepts that thereafter produced
developed concepts, i.e. C2, C5. Particularly, the concept generation happened in fact analysis and
problem-finding (A) and idea-finding (G). Some examples of scenario descriptions are:
● “ … what if I use a keyboard without using hands, for example, some programmers often use
Ctrl+C (copy operation) and Ctrl+V (paste operation). I can ‘save’ these operations by using
pedals. I step my left foot on (a pedal) is Ctrl+C, and step my right foot on (a pedal) is Ctrl
+V … ”
● “ … how to enhance privacy? That should be (like) other people cannot see the screen. Here are
(screen) lights … I can wear glasses so that I am the only one to access the screen contents …
The screen may not always need to be private (in this way). For example, when I go to the
washroom, a shading board unfolds and comes down … ”
● “ … here is a cuckoo, connected with a telescopic rod. When I use a computer for a long time,
the cuckoo will jump out and call ‘cuckoo, cuckoo, don’t play anymore’ … ”
4.5.6. Sufferings in design ideation

It is common that designers get stuck on an idea and feel unable to generate novel ideas. The
tendency to get stuck on an existing idea, known as design fixation, is prevalent among designers
(Crilly, 2015). In the empirical study, we observed that design fixation results in suffering on
designers: they feel unmotivated, unconfident and lost.
● “I don’t know what I’m thinking about” (M01)

● “I really can’t think of it for a while”/“Thought has dried up” (M02, F02, F03)
● “It would be nice if there is a computer desk can massage my waist” (F01, F02)
● “I am so tired, I usually like to take a break after thinking for a long time.” (F02)
● “I need to develop my thinking again” (F02)
● “I hope that there is a computer in front of me, then I can go searching. I don’t have to search
the computer table. I can search for the aircraft for rockets, search for drones, search for some
(other) products, or (conceptual products such as) Red Dot Awards to see what they have
(designed), and if I could put (existing) things to the computer desk. Then I may feel, wow，
this type of idea is pretty good. I won’t be like face to a blank sheet of paper as now.” (F02)
● “I hope that the desk can do a massage for me now, and then let me listen to a song.” (F02)
● “Now I feel that I need to start unbounded imagination and being more innovative” (F11)
Interestingly, the most productive designer (F02), encountered fixation situations more frequently
than the others. And many ideas and design concepts are created under such a hard situation, during
the designer trying to overcome it. In total, F02 fell into two fixations (4–9 min, 24–26 min) in the
experiment. The participant began to write down a list of factors of design solutions when falling
into design fixation. This helps the participant continue to propose new concepts again. We observed
a burst in idea generation after the designer outlining factors at 27 min (proposing two solutions in
a minute). It seems that the participant explored design solutions by creating a list of vague factors
and then specifying each factor or combination of factors to develop mature concept prototypes.
Similarly, other productive designers were observed to create a mind-map like preliminary solu-
tions back and forth during designing, or keywords at the beginning of the design process. An
example of lists of factors to overcome design barrier is shown in Figure 10.
22 J. LIAO ET AL.
4.6. Implications
Providing stimuli and widening scope could promote design ideation by eliciting and organizing
psychological representations and design scenarios while calling for design actions could overcome
design barriers and contribute to delightful design experience. The externalized psychological
representations allow designers to share and experience feelings of another person and adopt the
current situation from new perspectives of design problems (Köppen & Meinel, 2015). In other
words, these representations echo designers’ empathy of users, which effectively leads to sense and
knowledge of design (Köppen & Meinel, 2015). The patterns of design rationale also show design
ideation is well-organized by designers’ prompts of design operations. For example, designers ask
themselves questions to clarify design goals, potential candidates, and aspects related to users. The
prompts avoid designers stuck in exhausted thinking, increase feelings of confidence and lead more
creative and productive outcomes.
For AI as representation creation, creating a list of factors about design solutions can yield
a burst of ideas and concepts, uncovered by the case study on the most productive designer. Besides,
one of the significant strategies – imaginary thinking should be supported. The research in cognitive
psychology and design thinking has shown that the generation of external representations via
sketching and inner representations by imagination are instrumental in design problem solving
(Goldschmidt & Smolkov, 2006). These lists of factors (can be considered as the “Concept Structure”
in our framework) described imaginary representations of cognitive paths to be developed during
design. With the externalization of cognitive paths, most well-developed design outcomes are created
along a cognitive path during sketching. Designers created Concept Structure at the very beginning
of design, and utilize to guide design optimization. Also, designers are observed to develop Concept
Structure during design to break design barriers and to inform design states. If AI assists in
externalizing such imaginary representations, not only design performance rated by quantity and
quality as well as subjective design experience could be improved.
For AI as empathy triggers, the other strategy – descriptive thinking should be supported.
Describing design scenarios can create design stimuli to inform design ideation when the working
environment lacks external information. Designers try to construct a story rich in potentials and
frame assumptions to break design space limited to the current context. In the study, design
scenarios are found to occur with design empathy thinking when designers encounter design
barriers. Several designers expressed their feeling and hope highly related to problems being
considered and solutions to be proposed, e.g., a computer table that do massage when they tired
of generating ideas. It is likely that designers put themselves into the design task, i.e. via
“meditation-self” (Kim & Ryu, 2014) or “empathy” (Köppen & Meinel, 2015; Wright &
McCarthy, 2008). Also, scenario descriptions suggest the current situation and inform designers
what can be done, helping designers develop ideas again (e.g., F02). On the contrary, participants
(e.g., M02-04, F05, F06 of the same expertise) do not describe scenarios rest till the end of the
experiment when get stuck on existing ideas, contributing much fewer ideas and concepts
compared with F02. These findings suggest that imaging what design could be is more critical
to create a design in an existing scope. The descriptive thinking could be assisted by AI to explore
“what design could be” and “What a design serves for”. This would encourage innovation and
yield more design outcome fitting in real situations rather than assumed ones (McDonagh &
Thomas, 2010).
For AI as engagement, it requires to trade-off design performance with individual diversity in
planning and initiating design actions. Our study suggests design expertise can lead to diverse design
goals, problem definitions, methods to solve problems, in addition to the design experience and
individual differences identified in previous studies (Bonnardel & Marme`che, 2005; Lubert, 2005).
Some participants spent majority time on sketching, but sufficient sketching time does not guarantee
good design quality which suggests sufficient time on completing a design may not result in a better
design outcome. Rather, spending more time on querying about possibilities of design and trying to
develop novel considerations lead to higher transformation efficiency from ideas into design con-
cepts (Figure 9). Meanwhile, we identified substantial expertise differences in design strategy and
style. Even for mechanical design and industrial design that have a common focus on engineering,
designers are different in interests of design elements. Generally, mechanical design prefers struc-
tural analysis and functionalities, while the industrial design is intended for creative thinking and
human factors e.g., friendly usage, ergonomics. AI could learn to satisfy complex design strategies
and styles of designers while supporting to maximize design performance.
5. Design principles with AI

The cognitive characteristics in design ideation are challenging to support. Besides, design practice
with AI cannot be well-solved by the traditional design paradigm. In this section, we emphasize
a novel perspective of design principles to be considered in AI-augmented design support. These
principles would provide holistic understandings to develop design approaches with AI computation
methods and would help to bridge the gaps in design ideation supports.
5.1. Start from scratch

As Lubert suggested, there is no optimal HCI solution for all circumstances of design ideation (Lubert,
2005). Similarly, we found a great variance in designers’ thinking patterns, selection of tools, reacting
to ambiguity and working styles in the empirical study. Accordingly, design supports should be
cautiously designed to include the difference between individual preferences. Another reason for
being cautious is that problems of the creative process are diverse by nature, and the concept of the
“creative process” can vary with the problem type (Lubert, 2005). We may never find a precedent being
more desirable than the solution proposed for problems in the current design situation. Before starting
design, it is better to ask ourselves: WHAT professional fields, design stages, individuals and forms of
outcome we aim for. The answers to these questions will help to construct a specific design context and
appropriate goal, which determines dependent variables to be considered in interacting with computer
systems. An example is, experienced designers and novel designers have a very different creative
process in design ideation, thus need to offer support dependent on their experience level (Bonnardel
& Marme`che, 2005; Kim & Ryu, 2014). Meanwhile, a trade-off between design quality and individual
preferences should be considered. In our study, mechanical designers are dominated by sketching in
design. Encouraging re-thinking about design problems could generally help improve design quality,
while this is not the way of thinking that they are familiar with.
5.2. Knowledge-driven
AI produces outputs independent of domain-relevant rules. The processing of materials, data
representations, and optimization goals used in training an AI model are the only dependent source.
It suggests a path to domain-independent design. Still, AI-augmented design supports are dependent
on AI’s prediction on the given condition e.g., current context, a design target, design entities. One
may question: what condition should be given to AI? How to provide AI with an appropriate
condition in the context of design?
The answer to the first question is related to the knowledge-driven principle. The knowledge
discussed here is not only traditional design knowledge that consists of knowledge representations,
languages of designers, sketches as well as multimedia materials (Manzini, 2009). The outcomes
generated by AI could be a new form of design knowledge: design materials along with computation
and management of them. An example of the new design knowledge is text-to-image synthesis
(Figure 1b) where images can be generated from oral descriptions of design ideas. The AI model
produces knowledge in a reverse direction of inference, where a concrete design example (a picture)
that is traditionally produced in the end, is first vectorized into a more abstract representation
24 J. LIAO ET AL.
aligned with texts. Then, the AI model infers from an uncertain space where creativity normally
generates.
The new understanding of knowledge may suggest unexplored space in design and HCI. Many
concepts in AI considerations, e.g., variable optimization, knowledge discovering are related to pivotal
concepts in HCI and design fields. The knowledge, described as deep, iterative reflections of under-
standing of people, problem, and context around a situation that researchers feel to improve
(Zimmerman & Forlizzi, 2014), specify what designers think and act, and the related changes of design
states. From this perspective, the knowledge-driven could be explained as encouragements of (re)
framing design problems and (re)define the border of contexts in iterations. The optimization toward
a preferred state allows designers to understand the current state and suggests an improved future state
in addressing under-constrained problems (Zimmerman & Forlizzi, 2014; Zimmerman et al., 2007).
A knowledge-driven design approach that cooperates with AI in design is suggested in (Schmitt, n.
d.). Firstly, a neural network is trained with a large scale of in-domain materials (e.g., chair images) to
generate many estimations of the potential desired design outcome. Then, human designers select
several estimations as an archetype – a design example – and recreate archetypes into final concepts
delivered in sketch and physical models. As for the requirements of the initial materials for training,
machine learning community has established large-scale image datasets containing universal knowledge
about the world, e.g., Microsoft Common Objects in Context (MS-COCO) dataset (Lin et al., 2014); for
the narrow knowledge domain such as textile design, a great number of fine-grained chair descriptions
and images are easy-accessed in online design community e.g., Behance (Chai et al., 2018).
Certainly, AI-generated design outcomes can far from “good” solutions produced in the human-
only design process. The generated contents may not contribute to a new design but could be refined
in collaboration with the designer incrementally. At minimal, these outcomes can provide designers
intuitiveness of what a design probably be, as suggested in our framework. The “intuitive” features of
design behavior are frequently found in design activity and are the most effective and relevant to the
intrinsic nature of design (Cross, 2001).
5.3. Decomposition and integration

This principle provides a general approach to resolve the second question: how to provide AI with an
appropriate condition in the context of design? It explains the mechanism of AI algorithms on views
of design artifacts. Design space is decomposed into several potential problem spaces, solve in
a narrow scope, and integrate all solutions in different problem spaces for evaluation. The decom-
position-and-integration principle is associated with divergent-convergent thinking e.g., Geneplore
model (Finke et al., 1992), which explains creative process diverging from a narrow design statement
to a wide range of problems and possible paths, and finally, these problems and approaches are select
and reframed to a narrow scope. Similarly, the general process of AI “create a design with the given
condition” is represented by three stages of re-construction of design representations:
(1) transform an idea/concept into numerical representation, e.g., pixel values of an image. The
numerical representation brings mathematical meanings to a concept or an idea and makes
them computable;
(2) compute the proximity of the idea (compared to human analogical thinking of relating an
aspect/context to another one);
(3) transform the hidden proximity to original representation space (e.g., text, image) which can
be perceived and assessed by human designers.
During the reconstruction process, the design representations are produced at different abstrac-
tion levels (tacit or implicit) and can be used to broaden and narrow design scope in design process.
6. Challenges
AI has the potentials to support conceptual design activities, being able to increase understanding of
design situations but also empower abilities to carry out design actions. However, its adoption and
use in practical scenarios are challenging. The practical usage and attitudes of computation tools are
also central interests in HCI studies of creativity and intellectual ability (Lubert, 2005). We will
discuss the challenges of AI augments on predominant considerations: Trust, Learnability, Usability,
and Acceptance. These considerations are raised when we formulated design space with involve-
ments of AI: (1) individual difference in tolerance of design ambiguity and uncertainty can lead to
variant acceptance; (2) design traits, expertise and experience levels collectively affect the ways of AI
intervention; and (3) design-friendly systems raise higher requirements on adaptation to users.
6.1. Visible AI for trust and learnability

The invisible nature of AI, if not considered, can inhibit designers to perceive design outcomes and
find creative insights. According to the Technology Acceptance Model (Davis, 1989), the adoption
and use of technology are determined by perceived usefulness and perceived ease of use. AI systems
work as a “black box” whose results are currently not well-explained and transparent. The computa-
tion mechanism in producing the results is hardly self-evident, compared with design optimization
by variables, algorithms or logic rules. Thus, designers may question the results and reluctant to
accept them in critical applications (Luhmann, 2018). It could be dilemmatic that AI- augmented
design tools to provide foresight of design paths. The design process has features of both problem-
solving (Hay et al., 2017) and learning (Schön, 1987). Designers’ understandings can evolve with
actions of building solutions. Unfortunately, the goals of existing AI applications are normally fixed
and thus AI can not always generate better solutions that should be able to enhance designers’
understandings, such as to enlarge the design problem space. In many cases, the AI application
produces many different and probably more effective solutions. The better solution may be post-hoc,
determined after designers evaluating all solutions have been created.
To overcome “invisible AI”, an intuitive method is to increase system’s interaction and
visualization, including visualization of computing process, explanation of system behaviors,
response to a context (e.g., recent interactions, current task, states) and suggestions to users.
Regarding extend design in a wide range of possibilities, visualization of computing process
exposes how the system transform, organizes and presents design materials. It will increase
understanding of the characteristics and behaviors of tools, allow users to make causal associations
of system characteristics and variables with their actions, and build a contextual understanding of
a border between what the system can do and cannot. Regarding using AI as practical design tools,
contextual and comparative results are empirically encouraged to be presented to users (Cai et al.,
2019). The context and comparison may engage intuition and design analogy, which is the critical
component in creative design (Hey et al., 2008) and the psychological mechanism of creative
process (Gentner & Markman, 1997). Regarding limitation of fixed goals of AI applications, novel
computation mechanisms that allow flexible learning could be applied to design activities, for
example, reinforce learning that learns knowledge iteratively from an environment by receiving
a reward of trials.
6.2. Controllable AI for interoperability and reliability

AI can be integrated with automating and predictable design process. As explained in section 6.,
functions of AI-augmented design support systems can be affected by multiple variables due to the
nature of design problem complexity and individual diversity. This brings the second challenge of
determining who (user or AI), when and how (passive, proactive or others) to take control of the
26 J. LIAO ET AL.
design situation. With inappropriate automation, designers can be confused to perform design
actions (Oh et al., 2018), and could be constrained to derive creative insights.
Broadly speaking, this challenge is a key in collaboration of human and AI, determining the
criterion of WHO controls in communication between AI and human (Färber, 2016). HCI
researches lead to many solutions to this challenge. The first category is on system awareness,
including guidelines of increasing feedback to provide a global control to users (Amershi et al.,
2019), interactive visualization tools to enhance user’s sense-making of data (Sherkat et al., 2019) and
to increase users’ confidence of the system (Wang et al., 2019).
Another feasible approach is to increase AI’s engagement with users in the system. Engagement is
a significant factor in experiential system design (Jacques, 1995). Generally, a feeling of control
increases the focus on system and efficiency, leading to a satisfactory experience i.e. immersive
working. Besides, design ideation highlights the value in process of associating designers with design
objects and environments (Goel & Pirolli, 1992; Zeisel, 1984). The value of design is generated in
hand-on crafting of materials and formulation of problems, during describing, planning, retro-
spection and analyzing for shifting existing environment and presenting design objects in a form
to guide construction. The experience of design also has particular impacts on design quality
(Zimmerman et al., 2007).
6.3. Adaptive AI for variant backgrounds, attitudes, and expectations

Adaption is an ever-standing problem in designing user-related interactions and is particularly
important in computational supports for creative process (Lubert, 2005). As suggested by our
framework and the empirical study, design experience, design expertise, and inter- and intra-
personal difference in ability, working style, trait and accustom lead to diverse attitudes and potential
results. The adaption will promote AI’s enhancement on human experience and ability, for example,
increase users’ confidence with automatic machine learning system (Wang et al., 2019), and increase
understanding of users’ behavior, task inter-dependencies and interactions within a collaborative
learning environment (Chopade et al., 2018).
To introduce adaption in computation supported design activity, it is crucial to facilitate collabora-
tion between design tools, designers and user-generated content in co-creation with AI. The chAIr
Project (Schmitt, n.d.) explores a design method integrating co-creativity between the designer and AI.
The design method is explained as “reversing the role of human and machine in design process”. They
use a Generative Adversarial Network (GAN) trained on a dataset of “iconic 20th-century chairs” to
learn visual design knowledge of chairs. Then, the GAN generates many images as “visual prompts”.
Designers select several desirable images and create four classic chair designs based on AI’s creation.
To design adaptive AI supports, it is necessary to determine the task allocation between AI and
designers to develop guidelines and strategies of Human-AI coordination. We will introduce the
boundary of what and how well AI could do. The general boundary of AI and human decision-
making has two intrinsic variances:
(1) human perceive and process information by patterns whereas machine by rules, data and
algorithms;
(2) human know and holistically learn object, can figure out even if some parts are missing or
distorted; whereas machines can easily fail in such cases.
The two variances identify basic differences between the human and AI from a perspective of
information processing. For AI related design tasks involving a large amount of external informa-
tion, these variances could help identify potential challenges in schema development of computa-
tional supports as well as design evaluation. Also, having a clear understanding of the boundary will
be helpful to guide empirical investigations in design practice with AI applications. In other words,
the two variances can be considered when designing with a specific focus on design target, problem
type or environment.
7. Conclusions, limitations and future work

Designing with intelligent tools is an interesting and challenging topic in human-computer interaction.
This work aims to discover the possibilities of Artificial Intelligence (AI) in supporting design ideation.
We propose a framework for new design tools incorporating AI. The framework demonstrates three
critical aspects to increase efficacy and provide a pleasant experience in the early stages of design
process: design stimuli, design scenarios and instructing design actions. The framework is built on an
empirical study of 26 designers (of 30 recruited) to discover cognitive patterns and characteristics in
design ideation. We look into insights of designers’ design goals, styles of design reasoning as well as the
ways of dealing with design fixation. New principles on what and how to accommodate AI-augmented
design are summarized. Additionally, challenges in visible, controllable and adaptive AI are explained.
We propose to increase trust, learnability, interoperability, reliability, and knowledge about users’
backgrounds, attitudes, and expectations in designing interactive supports with AI. We hope this
study can inspire researchers on creativity supports and interactive machine learning systems.
7.1. Limitations
The limitations of traditional computational tools (Bernal et al., 2015; Mothersill & Bove, 2018)
indicated a future of design instruments: supporting cognitive moves of design. This future is a place
where opportunities for AI arise. We acknowledge that the framework is limited to computational
methods with texts and images and a major design thinking style. The rationality and principles
behind these methods are discussed in our framework and can be transferable to a broad context in
HCI and design domains.
7.1.1. AI as representation creation

This framework discusses AI models act to provide inspiration, widen design scope or trigger design
actions by suggesting texts or images. Designers usually conduct field study, user study and discuss
with multiple stakeholders to inspire novel insights from not only design experience and feeling.
These traditional design methods provide rich and realistic information in the experience of
designing and designers can obtain good design ideas and design solutions. These design methods
greatly increase cost in time, economic and human resources, etc. As in traditional design methods,
additional costs will be introduced to collect knowledge about the necessary information when AI is
designed to help designers develop logical inference as well as perceptual intuition. On the other
hand, a great number of AI models have been proposed in algorithmic domains but only a few of
them are made into demonstrative or reusable examples for design supports. The missing AI
prototypes may limit our investigation of AI’s potentials in design activities.
7.1.2. AI as empathy trigger

When design scenarios are not familiar with or target users are unclear, designers can seldom arouse
emotional connections and thus develop empathy thinking. In design activity, emotional experience
is significant to personal inspiration (McDonagh & Storer, 2004) but is challenging to support.
Emotional experience tools e.g., a mood board can play five main roles of framing, aligning,
paradoxing, abstracting, and directing in the early stages of the design process (Lucero, 2012). The
roles of emotion experience tools demand high-level human intelligence and abilities which could
beyond the supports of artificial tools. Apart from the emotional arousal of individual designers,
identifying and determining emotional states can be challenging. The human emotions are multi-
dimensional (Mehrabian, 1996), complex in taxonomy (Wierzbicka, 1986), can occur concurrently
28 J. LIAO ET AL.
and interrelated (Russell, 1989), and are affected by the working environment (Spector & Fox, 2005).
Computational tools may not cover all factors of human emotional experience.
7.1.3. AI as engagement
Designers use different styles of thinking across design stages and individuals. The empirical study is
limited to concurrent design outcomes derived via divergent thinking, a style of thinking that allows
many new ideas of being generated (Colzato et al., 2012). Divergent thinking is dominant in the early
stages of design process where designers aim for opening creative potentials (Runco, 2010). Another
component of human creativity remains to be investigated, i.e. convergent thinking (Colzato et al.,
2012) which involves select design direction and focus on generating one possible solution to
a particular design problem.
Besides, we have not investigated subjective responses from designers because this research
emphasizes to explore the theoretical ground for AI-augmented design supports more than to design
or to evaluate a specific AI-augmented design tool. The novel findings of subjective feedbacks are
likely to be discovered in a variety of design and evaluation of design supports with AI across
different design situations, while it is not the aim of this work. A study alone may fail to provide
holistic understanding of designers’ subjective responses due to multi–variant interaction factors
(Lubert, 2005) and the complex nature of design (Buchanan, 1992).
7.2. Future work

7.2.1. Enrich design information
Design activity requires a variety of information, particularly engineering design and product design
that emphasize design for particular users (Chiu & Shu, 2007; Howard et al., 2011), make decisions
(Ullman, 2002, 2001) and enhance design performance (Goldschmidt & Smolkov, 2006). Future
work could explore emergent AI models that leverage rich information to support sense-making in
inspiration, problem (re)framing to widen design scope, and decision- making to trigger design
actions. The information may come from a variety types of temporal, multi-source, multimodal or
multimedia data. Particularly, the information for decision-making may come from users which an
interactive AI system can learn and respond to.
7.2.2. Support design empathy

As previously mentioned, emotion experience tools can play roles in framing, aligning, paradoxing,
abstracting, and directing. Researchers can investigate novel AI in the design of interactions to augment
the five roles of emotional experience. The novel AI may locate outside the scope of AI generative
methods such as generative adversarial learning (Chai et al., 2018; Zhang et al., 2018; Zhu et al., 2016) and
convolution network (Gatys et al., 2016; Reed et al., 2016) mainly discussed in the framework. The future
of AI, as advocated by AI researchers e.g., Feifei Li, could focus on cooperation and understandings
between AI systems and human cognition. Reinforce learning, as previously discussed in Section 6.1.
could be explored. The other emergent AI models such as the recurrent neural network that can learn
sequence information (e.g., video) could also support collaborative creation with AI, for example,
iterative construction (Gregor et al., 2015). Also, researchers could investigate design methods and
tools to enhance perception and learning of AI suggested information for emotional purposes.
7.2.3. Design practice and subjective response

Design to increase engagement with AI can involve a vast amount of practice. The three fundamental
aspects of engagement are summarized in (Jacques, 1995): providing examples of engaging interac-
tions, suggesting methodologies for evaluating their impacts, and offering practical hints support
learning goals and motivation. These aspects involve human creative tasks, and there may never exist
an optimal HCI solution to provide computational supports (Lubert, 2005). Additionally, the design
is related to various areas and has many different meanings (Buchanan, 1992), which could lead to
different goals and contexts. Future research could investigate design practice and evaluation of the
three potentials of AI-augmented design support in particular design areas, scenarios, and designers
with specific experience, thinking style, and personality. Also, comparisons between the AI-
augmented design framework with traditional approaches can be examined.
7.2.4. Property
AI-augmented design activities include many contributors: designers of the main design task, AI model
designer and architect, engineers that implement AI model, communities that hold data resource used
to train AI model, workers that collect, clean and annotate data, and organizations that develop and
maintain AI computation framework. These stakeholders collectively contribute and yield design
outcomes. The division of design value could raise economic and ethical debates. Future work could
discuss ownership, interests of design outcomes and legal liability among these stakeholders.
In summary, many researchers studied the design process and Artificial Intelligence. However,
our study is distinguished by looking back roles of new AI advances and applications in early design
activity that involves conceptual and high-order intellectual human works. We propose a framework
to suggest potentials of AI in sense-making, problem (re)framing, and decision making during the
early stages of design activity. The framework focuses on what and how AI empowers designers’
cognitive ability rather than design actions in sketching and modeling.
Acknowledgments
We would like to thank all participants attending the experiments and several researchers of Zhejiang University
providing suggestions on transcription and annotation of recordings of individual’s design process. The researchers
include Jia-nan Lou (experiment instructor), and three design experts of Danni Shen, Min Zou and Qifei Wu.
Funding
This work was conducted at Institute of Industry Design, College of Computer Science and Technology in Zhejiang
University, and was supported by Zhejiang Province Public Welfare Technology Research Program [LGF18F020005]
and National Natural Science Foundation of China [31900768].
Notes
We include 10 minutes of preparation when summarizing the duration of design phases. Although designers do not
generate any outcome in practice with design materials and observation of design examples, their working styles and
traits could be exposed in this period.
Background
This work studies design cognition models, design research views, Artificial Intelligence (AI) based design theory and
empirical evidence in design activity. The empirical study involves experienced designers in different areas of
industrial, mechanical, product and art. We take a non-romantic, user-oriented view of design process to discover
opportunity of AI in design activity.
HCI Editorial Record

First received on 20 Jun 2019. Revisions received on 19 Feb 2020. Final manuscript received on 19 Feb 2020.
Notes on contributors
Jing Liao ([email protected], https://www.researchgate.net/profile/Jing Liao7) studies computational design, machine
learning and neural networks at the College of Computer Science and Technology of Zhejiang University, China
30 J. LIAO ET AL.
Preben Hansen ([email protected], https://www.researchgate.net/profile/Preben Hansen) studies creativity, innovation

and information retrieval at the Department of Computer and System Sciences of Stockholm University, Sweden.
Chunlei Chai ([email protected], https://person.zju.edu.cn/en/id) studies design cognition, innovation design and
human engineering at the Institute of Industry Design of Zhejiang University, China. He is the corresponding author
of this article.
ORCID
Jing Liao http://orcid.org/0000-0002-2948-5043
References
Alhadreti, O., & Mayhew, P. (2018). Rethinking thinking aloud: A comparison of three think-aloud protocols.
Proceedings of the CHI 2018 conference on human factors in computing systems (Vol. 44, pp. 1–12). ACM.
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P.N., &
Inkpen, K. (2019). Guidelines for human-AI interaction. In Proceedings of the CHI 2019 conference on human
factors in computing systems (pp. 1–13). ACM.
Ball, L. J., & Christensen, B. T. (2009). Analogical reasoning and mental simulation in design: Two strategies linked to
uncertainty resolution. Design Studies, 30(2), 169–186. https://doi.org/10.1016/j.destud.2008.12.005
Bayazit, N. (1993). Designing: Design knowledge: Design research: Related sciences. In de Vries M.J., Cross N., Grant
D.P. (Eds.), Design methodology and relationships with science (pp. 121–136). Springer.
Bernal, M., Haymaker, J. R., & Eastman, C. (2015). On the role of computational support for designers in action.
Design Studies, 41(Part B), 163–182. https://doi.org/10.1016/j.destud.2015.08.001
Bonnardel, N., & Marme`che, E. (2005, October). Towards supporting evocation processes in creative design:
A cognitive approach. International Journal of Human-Computer Studies, 63(4–5), 422–435. https://doi.org/10.
1016/j.ijhcs.2005.04.006
Buchanan, R. (1992). Wicked problems in design thinking. Design Issues, 8(2), 5–21. https://doi.org/10.2307/1511637
Burgess, A. (2018). Ai prototyping. In The executive guide to artificial intelligence: How to identify and implement
applications for ai in your organization (pp. 117–127). Springer International Publishing.
Buxton, B. (2007). A sketch of the process. In B. Buxton (Ed.), Sketching user experiences (pp. 77–84). Morgan
Kaufmann.
Cai, C. J., Jongejan, J., & Holbrook, J. (2019). The effects of example-based explanations in a machine learning
interface. Proceedings of the IUI 2019 international conference on intelligent user interfaces (pp. 258–262). ACM.
Campbell, M. I., Cagan, J., & Kotovsky, K. (1998). A-design: Theory and implementation of an adaptive, agent-based
method of conceptual design. In Gero J.S., Sudweeks F. (Eds.), Artificial intelligence in design’98 (pp. 579–598).
Springer.
Chai, C., Liao, J., Zou, N., & Sun, L. (2018). A one-to-many conditional generative adversarial network framework for
multiple image-to-image translations. Multimedia Tools and Applications, 77(17), 22339–22366. https://doi.org/10.
1007/s11042-018-5968-7
Chan, J., Fu, K., Schunn, C., Cagan, J., Wood, K., & Kotovsky, K. (2011). On the benefits and pitfalls of analogies for
innovative design: Ideation performance based on analogical distance, commonness, and modality of examples.
Journal of Mechanical Design, 133 (8), 081004: 1-11. https://doi.org/10.1115/1.4004396
Chandrasegaran, S. K., Ramani, K., Sriram, R. D., Horva ́ th, I., Bernard, A., Harik, R. F., & Gao, W. (2013). The
evolution, challenges, and future of knowledge representation in product design systems. Computer-aided Design,
45(2), 204–228. https://doi.org/10.1016/j.cad.2012.08.006
Chen, L., Wang, P., Dong, H., Shi, F., Han, J., Guo, Y., Childs, P. R. N., Xiao, J., & Wu, C. (2019). An artificial
intelligence based data-driven approach for design ideation. Journal of Visual Communication and Image
Representation, 61(May 2019), 10–22. https://doi.org/10.1016/j.jvcir.2019.02.009
Chiu, I., & Shu, L. (2007). Using language as related stimuli for concept generation. AI EDAM, 21(2), 103–121. https://
doi.org/10.1017/S0890060407070175
Chopade, P., Khan, S., Stoeffler, K., Edward, D., Rosen, Y., & Von Davier, A. (2018). Framework for effective
teamwork assessment in collaborative learning and problem solving. In 19th International Conference on
Artificial Intelligence in Education (AIED), LAS, the festival of learning, ARL workshop “assessment and intervention
during team tutoring” AIED-2018 (Vol. 2153, pp. 48–59).
Christensen, B. T., & Schunn, C. D. (2007). The relationship of analogical distance to analogical function and
preinventive structure: The case of engineering design. Memory & Cognition, 35(1), 29–38. https://doi.org/10.
3758/BF03195939
Colzato, L. S., Szapora, A., & Hommel, B. (2012). Meditate to create: The impact of focused-attention and
open-monitoring training on convergent and divergent thinking. Frontiers in Psychology, 3(116), 1-5. https://doi.
org/10.3389/fpsyg.2012.00116
Council, D. (2006). Double diamond design process. Design Council.
Crilly, N. (2015). Fixation and creativity in concept development: The attitudes and practices of expert designers.
Design Studies, 38(May 2015), 54–91. https://doi.org/10.1016/j.destud.2015.01.002
Cross, N. (2001). Design cognition: Results from protocol and other empirical studies of design activity. In Charles M.
Eastman, W. Michael McCracken, Wendy C. Newstetter. (Eds.), Design knowing and learning: Cognition in design
education (pp. 79–103). Elsevier.
Dalsgaard, P. (2017). Instruments of inquiry: Understanding the nature and role of tools in design. International
Journal of Design, 11(1), 21-33. https://pdfs.semanticscholar.org/5b94/177d0e9109d61764e6d4b48d1eb28232bd31.
pdf.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS
Quarterly, 13(3), 319-340. https://doi.org/10.2307/249008
Davis, N. M., Hsiao, C.-P., Singh, K. Y., & Magerko, B. (2016). Co-creative drawing agent with object recognition.
Twelfth artificial intelligence and interactive digital entertainment conference.
Dix, A., & Gongora, L. (2011). Externalisation and design. In Procedings of the second conference on creativity and
innovation in design (pp. 31–42). https://doi.org/10.1186/2191-0855-1-31
Dorst, K., & Cross, N. (2001). Creativity in the design process: Co-evolution of problem–solution. Design Studies, 22
(5), 425–437. https://doi.org/10.1016/S0142-694X(01)00009-6
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017). UX design innovation: Challenges for working with
machine learning as a design material. Proceedings of the CHI 2017 conference on human factors in computing
systems (pp. 278–288).
Färber, B. (2016). Communication and communication problems between autonomous vehicles and human drivers. In
Maurer M., Gerdes J., Lenz B., Winner H. (Eds.), Autonomous driving (pp. 125–144). Springer.
Fallman, D. (2003). Design-oriented human-computer interaction. Proceedings of the CHI 2003 conference on human
factors in computing systems (pp. 225–232).
Fan, J. E., Dinculescu, M., & Ha, D. (2019). Collab draw: An environment for collaborative sketching with an artificial
agent. In Proceedings of the C&C 2019 conference on creativity and cognition (pp. 556–561).
Finke, R. A., Ward, T. B., & Smith, S. M. (1992). Creative cognition: Theory, research, and applications. American
Journal of Psychiatry, 151(5), 777. https://doi.org/10.1176/ajp.151.5.777.
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016,June). Image style transfer using convolutional neural networks. 2016
IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2414–2423). IEEE.
Gentner, D., & Markman, A. B. (1997). Structure mapping in analogy and similarity. American Psychologist, 52(1), 45.
https://doi.org/10.1037/0003-066X.52.1.45
Gero, J. S. (1987). Prototypes: A new schema for knowledge-based design. University of Sydney, Architectural
Computing Unit.
Gero, J. S. (1990). Design prototypes: A knowledge representation schema for design. AI Magazine, 11(4), 26. https://
pdfs.semanticscholar.org/e383/c30e67ae9b5813988c938ac7c6113570f2ad.pdf.
Gero, J. S. (1996). Creativity, emergence and evolution in design. Knowledge-Based Systems, 9(7), 435–448. https://doi.
org/10.1016/S0950-7051(96)01054-4
Gero, J. S. (2006). Understanding situated design computing and constructive memory: Newton, mach, einstein and
quantum mechanics. In Workshop of the European group for intelligent computing in engineering (pp. 285–297).
Gero, J. S., & Kannengiesser, U. (2004). The situated function–behaviour–structure framework. Design Studies, 25(4),
373–391. https://doi.org/10.1016/j.destud.2003.10.010
Gero, J. S., & Kannengiesser, U. (2006). A framework for situated design optimization. In Jos P. van Leeuwen and
Harry J.P. Timmermans (Eds.), Innovations in design & decision support systems in architecture and urban planning
(pp. 309–324). Springer.
Gero, J. S., & Maher, M. L. (1993). Modeling creativity and knowledge-based creative design. Psychology Press.
Gillies, M., Fiebrink, R., Tanaka, A., Garcia, J., Bevilacqua, F., & Heloir, A., Heloir, A., Nunnari, F., Mackay, W.,
Amershi, S., Lee, B. and D’Alessandro, N. (2016). Human-centred machine learning. Proceedings of the CHI 2016
conference extended abstracts on human factors in computing systems (pp. 3558–3565).
Goel, V., & Pirolli, P. (1992). The structure of design problem spaces. Cognitive Science, 16(3), 395–429. https://doi.
org/10.1207/s15516709cog1603_3
Goldschmidt, G., & Smolkov, M. (2006). Variances in the impact of visual stimuli on design problem solving
performance. Design Studies, 27(5), 549–569. https://doi.org/10.1016/j.destud.2006.01.002
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015). Draw: A recurrent neural network for
image generation. Proceedings of the 32nd International Conference on International Conference on machine
Learning-Volume 37 (pp. 1462–1471).
Haritaipan, L. (2019). Towards the creation of creativity tools for real-practice: A review of 112 design tools in the
market. The Design Journal, 22(4), 1–11. https://doi.org/10.1080/14606925.2019.1613800
32 J. LIAO ET AL.
Hatchuel, A., Le Masson, P., & Weil, B. (2011). Teaching innovative design reasoning: How concept–knowledge theory
can help overcome fixation effects. AI EDAM, 25(1), 77–92. https://doi.org/10.1017/S089006041000048X
Hatchuel, A., & Weil, B. (2003). A new approach of innovative design: An introduction to ck theory. DS 31:
Proceedings of ICED 03, the 14th International Conference on Engineering Design, Stockholm
Hay, L., Duffy, A. H., McTeague, C., Pidgeon, L. M., Vuletic, T., & Grealy, M. (2017). A systematic review of protocol
studies on conceptual design cognition: Design as search and exploration. Design Science, 3(e10), 1-36. https://doi.
org/10.1017/dsj.2017.11
Hey, J., Linsey, J., Agogino, A. M., & Wood, K. L. (2008). Analogies and metaphors in creative design. International
Journal of Engineering Education, 24(2), 283. https://pdfs.semanticscholar.org/e594/f88075e5e301b3c733cb0733e393
f86ef9fd.pdf
Howard, T. J., Culley, S., & Dekoninck, E. A. (2011). Reuse of ideas and concepts for creative stimuli in engineering
design. Journal of Engineering Design, 22(8), 565–581. https://doi.org/10.1080/09544821003598573
Howard, T. J., Culley, S. J., & Dekoninck, E. (2008). Describing the creative design process by the integration of
engineering design and cognitive psychology literature. Design Studies, 29(2), 160–180. https://doi.org/10.1016/j.
destud.2008.01.001
Jacques, R. (1995). Engagement as a design concept for multimedia. Canadian Journal of Educational Communication,
24(1), 49–59. https://doi.org/10.21432/T2VG77
Jourdenais, R., Ota, M., Stauffer, S., Boyson, B., & Doughty, C. (1995). Does textual enhancement promote noticing?
A think-aloud protocol analysis. In R Schmidt (Eds.), Attention and awareness in foreign language learning (pp.
183–216).
Kim, J. (2011). Modeling cognitive and affective processes of designers in the early stages of design: Mental categorization
of information processing [Unpublished doctoral dissertation].
Kim, J., & Ryu, H. (2014). A design thinking rationality framework: Framing and solving design problems in early
concept generation. Human–Computer Interaction, 29(5–6), 516–553. https://doi.org/10.1080/07370024.2014.
896706
Köppen, E., & Meinel, C. (2015). Empathy via design thinking: Creation of sense and knowledge. In Plattner H.,
Meinel C., Leifer L. (Eds.), Design thinking research (pp. 15–28). Springer.
Koskinen, I., Zimmerman, J., Binder, T., Redstrom, J., & Wensveen, S. (2013). Design research through practice: From
the lab, field, and showroom. IEEE Transactions on Professional Communication, 56(3), 262–263. https://doi.org/10.
1109/TPC.2013.2274109
Kouprie, M., & Visser, F. S. (2009). A framework for empathy in design: Stepping into and out of the user’s life.
Journal of Engineering Design, 20(5), 437–448. https://doi.org/10.1080/09544820902875033
Kuniavsky, M., Churchill, E., & Steenson, M. W. (2017). The 2017 AAAI spring symposium series technical reports:
Designing the user experience of machine learning systems (Technical Report SS-17-04). Palo Alto.
Landay, J. A., & Myers, B. A. (2001). Sketching interfaces: Toward more human interface design. Computer, 34(3),
56–64. https://doi.org/10.1109/2.910894
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Bourdev, L., Girshick, R., Hays, J., Perona, P.,
Ramanan, D., Zitnick, C. L., & Dollár, P. (2014). Microsoft COCO: Common objects in context. In D. Fleet,
T. Pajdla, B. Schiele, & T. Tuytelaars (Eds.), European conference on computer vision (pp. 740–755). Springer
International Publishing.
Liu, Y., & Lim, S. (2011). Using ontology for design information and knowledge management: A critical review. In
A. Bernard (Ed.), Global product development (pp. 427–433). Springer Berlin Heidelberg.
Löwgren, J., & Stolterman, E. (2004). Thoughtful interaction design: A design perspective on information technology. Mit
Press.
Loukissas, Y. (2012). Co-designers: Cultures of computer simulation in architecture. Routledge.
Lubert, T. (2005). How can computers be partners in the creative process. International Journal of Human-computer
Studies, 63(4), 365–369. https://doi.org/10.1016/j.ijhcs.2005.04.002
Lucero, A. (2012). Framing, aligning, paradoxing, abstracting, and directing: How design mood boards work.
Proceedings of the DIS 2012 conference on designing interactive systems (pp. 438–447). Newcastle Upon Tyne,
United Kingdom. New York: ACM.
Luhmann, N. (2018). Trust and power. John Wiley & Sons.
Maher, M. L. (1990). Process models for design synthesis. AI Magazine, 11(4), 49. https://doi.org/10.1609/aimag.v11i4.
856
Manzini, E. (2009). New design knowledge. Design Studies, 30(1), 4–12. https://doi.org/10.1016/j.destud.2008.10.001
McDonagh, D., & Storer, I. (2004). Mood boards as a design catalyst and resource: Researching an under-researched
area. The Design Journal, 7(3), 16–31. https://doi.org/10.2752/146069204789338424
McDonagh, D., & Thomas, J. (2010). Rethinking design thinking: Empathy supporting innovation. Australasian
Medical Journal, 3(8), 458–464. https://doi.org/10.4066/AMJ.2010.391
McDonald, K. (n.d.). A return to machine learning. Retrieved February 15, 2020, from https://medium.com/@kcimc/
a-return-to-machine-learning-2de3728558eb
Mehrabian, A. (1996). Pleasure-arousal-dominance: A general framework for describing and measuring individual
differences in temperament. Current Psychology, 14(4), 261–292. https://doi.org/10.1007/BF02686918
Moreno, D. P., Blessing, L. T., Yang, M. C., Herna ́ ndez, A. A., & Wood, K. L. (2016). Overcoming design fixation:
Design by analogy studies and nonintuitive findings. AI EDAM, 30(2), 185–199. https://doi.org/10.1017/
S0890060416000068
Mostow, J. (1985). Toward better models of the design process. AI Magazine, 6(1), 44. https://doi.org/10.1609/aimag.
v6i1.468
Mothersill, P., & Bove, V. M. (2018). An ontology of computational tools for design activities. In Proceedings of
DRS2018 International Conference (pp. 1261-1277). Limerick, Ireland, Design Research Society. http://www.pip
mothersill.com/files/drs2018mothersillontology-of-computational-tools.pdf
Oh, C., Song, J., Choi, J., Kim, S., Lee, S., & Suh, B. (2018). I lead, you help but only with enough details:
Understanding user experience of co-creation with artificial intelligence. In Proceedings of the CHI 2018 conference
on human factors in computing systems (pp. 1–13).
Park, J. (2011). Developing a knowledge management system for storing and using the design knowledge acquired in
the process of a user-centered design of the next generation information appliances. Design Studies, 32(5), 482–513.
https://doi.org/10.1016/j.destud.2011.05.001
Park, W., Park, D., Ahn, B., Kang, S., Kim, H., Kim, R., & Na, J. (2019). Interactive ai for linguistic education built on
vr environment using user generated contents. In 2019 21st International Conference on advanced communication
technology (ICACT) (pp. 385–389). https://doi.org/10.1177/1753193418811565
Pedgley, O. (2007). Capturing and analysing own design activity. Design Studies, 28(5), 463–483. https://doi.org/10.
1016/j.destud.2007.02.004
Pourmohamadi, M., Gero, J. S., & Saunders, R. (2011). Cad software as customisation tools: Using FBS protocol coding
scheme to understand the behaviour of mass customisers. In Proceedings of the 16th International Conference on
computer aided architectural design research in asia (pp. 399–408). CUMINCAD.
Reed, S., Akata, Z., Mohan, S., Tenka, S., Schiele, B., & Lee, H. (2016). Learning what and where to draw. In
Proceedings of the 30th International Conference on neural information processing systems (pp. 217–225).
Reich, Y. (1991). Design knowledge acquisition: Task analysis and a partial implementation. Knowledge Acquisition, 3
(3), 237–254. https://doi.org/10.1016/1042-8143(91)90006-9
Rittel, H. W., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169.
https://doi.org/10.1007/BF01405730
Runco, M. A. (2010). Divergent thinking, creativity, and ideation. In J. Kaufman & R. Sternberg (Eds.), The Cambridge
handbook of creativity (pp. 413-446).
Russell, J. A. (1989). Measures of emotion. In Robert Plutchik, Henry Kellerman (Eds.), The measurement of emotions
(pp. 83–111). Elsevier.
Salustri, F. (n.d.). Design concept. Retrieved February 15, 2020, from https://deseng.ryerson.ca/dokuwiki/design:
designconcept
Schmitt, P. (n.d.). The chair project: Four classics. Retrieved February 15, 2020, from https://philippschmitt.com/work/
chair
Schön, D. A. (1987). Educating the reflective practitioner. San Francisco: Josey-Bass Publishers
Schon, D. A., & DeSanctis, V. (1986). The reflective practitioner: How professionals think in action. Taylor & Francis.
Shah, J. J., Smith, S. M., Vargas-Hernandez, N., Gerkens, D. R., & Wulan, M. (2003, 9). Empirical studies of design
ideation: Alignment of design experiments with lab experiments. ASME 2003 International design engineering
technical conferences and computers and information in engineering conference (Vol. 3b: 15th International
Conference on Design Theory and Methodology (pp. 847–856). American Society of Mechanical Engineers.
Sherkat, E., Nourashrafeddin, S., Milios, E. E., & Minghim, R. (2019). Interactive document clustering revisited:
A visual analytics approach. Proceedings of the IUI 2019 international conference on intelligent user interfaces.
Simon, H. A. (1996). The sciences of the artificial. MIT press.
Spector, P. E., & Fox, S. (2005). The stressor-emotion model of counterproductive work behavior. In S. Fox & P. E.
Spector (Eds.), Counterproductive work behavior: Investigations of actors and targets (pp. 151-174). American
Psychological Association. https://doi.org/10.1037/10893-007
Stumpf, S. C., & McDonnell, J. T. (2002). Talking about team framing: Using argumentation to analyse and support
experiential learning in early design episodes. Design Studies, 23(1), 5–23. https://doi.org/10.1016/S0142-694X(01)
00020-5
Tsenn, J., Atilola, O., McAdams, D. A., & Linsey, J. S. (2014). The effects of time and incubation on design concept
generation. Design Studies, 35(5), 500–526. https://doi.org/10.1016/j.destud.2014.02.003
Ullman, D. G. (2001). Robust decision-making for engineering design. Journal of Engineering Design, 12(1), 3–13.
https://doi.org/10.1080/09544820010031580
Ullman, D. G. (2002). Toward the ideal mechanical engineering design support system. Research in Engineering
Design, 13(2), 55–64. https://doi.org/10.1007/s00163-001-0007-4
Visser, W. (2009). Design: One, but in different forms. Design Studies, 30(3), 187–223. https://doi.org/10.1016/j.destud.
2008.11.004
34 J. LIAO ET AL.
Wang, Q., Ming, Y., Jin, Z., Shen, Q., Liu, D., Smith, M. J., Veeramachaneni, K. Qu, H. (2019). Atmseer: Increasing
transparency and controllability in automated machine learning. Proceedings of the CHI 2019 conference on human
factors in computing systems (pp. 681: 1–681:12). ACM.
Ware, C. (2010). Visual thinking: For design. Elsevier.
Wierzbicka, A. (1986). Human emotions: Universal or culture-specific? American Anthropologist, 88(3), 584–594.
https://doi.org/10.1525/aa.1986.88.issue-3
Wright, P., & McCarthy, J. (2008). Empathy and experience in hci. In Proceedings of the CHI 2008 conference on
human factors in computing systems (pp. 637–646).
Yang, Q. (2018). Machine learning as a UX design material: How can we imagine beyond automation, recommenders,
and reminders? In 2018 AAAI spring symposium series.
Zeisel, J. (1984). Inquiry by design: Tools for environment-behaviour research (No. 5). CUP archive.
Zhang, Z., Xie, Y., & Yang, L. (2018). Photographic text-to-image synthesis with a hierarchically-nested adversarial
network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 6199–6208).
Zhu, J. Y., Krähenbühl, P., Shechtman, E., & Efros, A. A. (2016). Generative visual manipulation on the natural image
manifold. In B. Leibe, J. Matas, N. Sebe, & M. Welling (Eds.), Computer vision – ECCV 2016 (pp. 597–613).
Springer International Publishing.
Zimmerman, J., & Forlizzi, J. (2014). Research through design in HCI. In Olson J., Kellogg W. (Eds.), Ways of knowing
in hci (pp. 167–189). Springer.
Zimmerman, J., Forlizzi, J., & Evenson, S. (2007). Research through design as a method for interaction design research
in HCI. Proceedings of the CHI 2007 conference on human factors in computing systems (pp. 493–502).
Appendix A. Literature scope

The literature investigation started from significant journals including Human-Computer Interaction, International
Journal of Human-Computer Studies, ACM Transactions on Computer- Human Interaction, Design Studies,
International Journal of Design, and conferences including International Conference on Human Factors in
Computing Systems, Designing Interactive System, International Conference on Intelligent User Interfaces and
Association for the Advancement of Artificial Intelligence. Besides, we identified studies with its relatedness to design
thinking, design cognition, Artificial Intelligence, and computer-aided design support. Finally, related studies were
examined according to references and the most related studies recommend by Google Scholar.
Appendix B. Experiment brief

Thanks for attending this experiment. You will be invited to the task of designing computer desks. You could observe
several design examples, and use the provided tools to create your design. While designing, kindly recommend you
speak what you think of, draw sketches, and explain with text. The more innovative, practical, and feasible designs, the
better.

Liao 2020

Uploaded by

Copyright:

Available Formats

Liao 2020

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Liao 2020

Uploaded by

Copyright:

Available Formats

Human–Computer Interaction

ISSN: 0737-0024 (Print) 1532-7051 (Online) Journal homepage: https://www.tandfonline.com/loi/hhci20

A framework of artificial intelligence augmented

Jing Liao, Preben Hansen & Chunlei Chai

To link to this article: https://doi.org/10.1080/07370024.2020.1733576

Published online: 30 Apr 2020.

Submit your article to this journal

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

A framework of artificial intelligence augmented design support

ABSTRACT ARTICLE HISTORY

2. Gaps in AI-augmented design

2.1. Background and progress

2.2. In-improving design problem

2.3. Structured design process model

Figure 3. An interactive view of design optimization (Gero, 2006).

3. A framework for design ideation with AI

Figure 4. The framework of AI-augmented design ideation supports.

materials, architecture, etc.

3.1. The origin of inspirations

3.1.1. Elicit psychological representations by concept structure

3.1.2. Empathy thinking by scenario descriptions

3.1.3. Stick to design and tackles of suffering

3.2. Design ideation supports with AI

3.2.1. Representation creation: provide inspiration

Category Approach Limitation

Category Method AADS HCI Aspects

example, a cluster of mathematical representations of entity neighborhood computed by AI could

Category Descriptions in Design Language Descriptions in AI Language

3.2.2. Empathy trigger: widen scope

3.2.3. Engagement: call for design action

4. An empirical study of design rationality

4.1. Aim and method

4.2. Participants and task

Participant Background Experience Ideas Concepts Concepts/Ideas (%)

Figure 9. Participant specifications and design ideation performance.

Code Description Example

4.4. Materials and notations

(1) a sudden whimsy < G >;

● “What else is there on the computer desk itself?” (F01)

4.5.3. Self- instruction

4.5.4. Describe previous example/usage/idea

4.5.5. Divide into cases

● “Regarding age/profession/usage/ergonomics … ” (F02, M02, M06, F11, M08)

4.5.6. Sufferings in design ideation

● “I don’t know what I’m thinking about” (M01)

5. Design principles with AI

5.1. Start from scratch

5.3. Decomposition and integration

6.1. Visible AI for trust and learnability

6.2. Controllable AI for interoperability and reliability

6.3. Adaptive AI for variant backgrounds, attitudes, and expectations

7. Conclusions, limitations and future work

7.1.1. AI as representation creation

7.1.2. AI as empathy trigger

7.2. Future work

7.2.2. Support design empathy

7.2.3. Design practice and subjective response

HCI Editorial Record