See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/332500101

AI for design: Virtual design assistant

Article in CIRP Annals · April 2019

DOI: 10.1016/j.cirp.2019.03.024

CITATIONS READS

21 1,388

6 authors, including:

Sang-Gook Kim Haluk Akay

Massachusetts Institute of Technology Massachusetts Institute of Technology
167 PUBLICATIONS 6,083 CITATIONS 21 PUBLICATIONS 96 CITATIONS

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Solar-powered water splitting with plasmon-assisted nanostructures View project

Airflow Energy Harvesting using Miniaturized Turbine and Piezoelectric Transduction View project

All content following this page was uploaded by Haluk Akay on 13 November 2020.

The user has requested enhancement of the downloaded file.

 CIRP Annals - Manufacturing Technology 68 (2019) 141–144

Contents lists available at ScienceDirect

CIRP Annals - Manufacturing Technology

jou rnal homep age : ht t p: // ees .e lse vi er . com /ci r p/ def a ult . asp

AI for design: Virtual design assistant

Sang-Gook Kim (1)a,*, Sang Min Yoon b, Maria Yang a, Jungwoo Choi b, Haluk Akay a, Edward
Burnell a
a
Massachusetts Institute of Technology, Cambridge, MA, USA
b
Kookmin University, Seoul, Republic of Korea

A R T I C L E I N F O A B S T R A C T

Engineering faces many wicked problems: irreducibly interdisciplinary with multiple competing
Keywords: objectives, and of such large scale and complexity that will require processes to deeply rely on human
Design method insights and power of computation. The resurgence of machine learning offers the possibility for new
Machine learning forms of human/computer collaboration where each fuels hybrid intelligence in complementary ways. A
Hybrid intelligence concept of virtual design assistant (VDA) is developed as a platform to bring the hybrid intelligence in
solving complex design challenges. A deep learning-based abstraction process is developed to provide
VDA a function to extract structured functional requirements from fragmental design specifications and
customer needs.
© 2019 Published by Elsevier Ltd on behalf of CIRP.

1. Introduction significant failures. Many believe that only human experts can
conceptualize and orchestrate large and complex projects at the up-
Recently developed techniques in machine learning and other stream of designing systems. There are two challenging issues in the
artificial intelligence (AI) tools have brought breakthroughs from current practice of heuristic systems design. First, it takes tens of
processing of images, speech and audio, and language translation to years for humans to become area experts through the accumulation
driving of autonomous cars. Since the 1990s, CIRP members have of experience in both successes and failures. Secondly, human
intensively investigated the use of conventional machining learning experts also fail sometimes, especially at the critical time. The
for tool monitoring, process inspection and sensor fusion [1–3], questions one might ask at this stage are “How could we teach junior
conceptual design [4], CAD [5], process planning [6], process engineers, architects and scientists to design complex systems
monitoring and parameter optimization [7], and production control successfully without spending years of effort on the job training?
[8] among others. Conventional machine learning techniques used Could we assist human experts to minimize the probability of failure
for the studies of manufacturing and design, however, were limited by leveraging recent developments in AI and big data?”
in their ability to process huge amounts of raw data due to the We have looked into the use of AI in the up-stream of design
shallow neural network architecture and much slower computing where all the key concepts are determined, and experience-based
hardware available decades ago and thereby research interest on human insight is necessary. Could machine intelligence help this
them has fizzled out. From 1990 to 2000, 27 papers were published in early stage of designing beyond routine design toward the
CIRP Annals Vol. 1 while only 5 were published after 2001. generation of good and novel designs? Our proposed solution is
The resurgence of machine learning (i.e. deep learning) and AI is the use of the hybrid intelligence which is an approach in which
the result of the high-powered Graphic Processing Units readily human intelligence co-evolves with machine intelligence. In this
available at very low cost, deep architecture with massively paper, we describe a deep learning-based design assistant as a first
increased layers and neurons, and massive amounts of data available step to hybrid intelligence that combines human intelligence
on line to train the network. As this new technology spreads to which grows through experience and machine intelligence which
reshape the world and society, bringing the power of computing and can learn from past successes and failures.
AI to all fields of study and disciplines, it is urgently necessary to
revisit the use of AI (deep learning, narrowly speaking) for the 2. Hybrid intelligence for systems design
advancement of design and manufacturing technologies.
The design of real-world systems (engineering, architecture, Many designers, engineers, and researchers emphasize the
software, industrial, financial and social systems) is often a importance of early stage system design, calling it “system
tumultuous endeavor fraught with great triumphs and, at times, architecting.” System architecting is different from “system
engineering,” which focuses more on modeling-based system
analysis such as single variable optimization and multi-variable
* Corresponding author. trade-offs [9]. Great system architects, however, sometimes fail
E-mail address: [email protected] (S.-G. Kim). also, especially at critical times. As the scale of a system goes up or

https://doi.org/10.1016/j.cirp.2019.03.024
0007-8506/© 2019 Published by Elsevier Ltd on behalf of CIRP.
142 S.-G. Kim et al. / CIRP Annals - Manufacturing Technology 68 (2019) 141–144

system heterogeneity increases, complexity can become unman- in the retrospectively generated FR-DP tree structures. For each
ageable by heuristics alone. Incorrect understanding of a problem discipline, we will be able to collect huge amount of data for
in the early stages of design significantly increases the likelihood of training of VDA. In 1978, Nam Suh at MIT introduced a principle-
system failure which has been mostly served only by the experts’ based design approach to manufacturing systems [12]. AD has
intuition, heuristics and experiential perceptions. Engineering provided a way of design thinking that the ad hoc nature of
relies on precise tools, computation, and scientific methods while engineering design could be better structured with the concept of
heuristic approach is a pragmatic guideline from lessons learned domains and the principles to guide decision making in mapping
from past successes and failures, taken to a level of abstraction. The between the domains [13]. While the merits of using AD theory
former alone (without extensive experience) has little success for have been evidenced in academic research papers, we have seen
solving big complex systems (engineering, architecture, software, limited use of AD by industrial practitioners. Many experienced
industrial, financial and social systems); the latter has been the designers agree that AD helps them to cultivate insightful thinking.
guideline for most practitioners in real world cases. In Hybrid But many still find it difficult to apply AD principles to design
Intelligence, our concern is not with the intelligence of artificial practice since using AD effectively also requires designer’s insights
systems, or with constraining human designers, but with the and experience in AD. Benefits of machine learning will help
effectiveness of their collaboration. Instead of finding humans designers effectively use AD for making early design decisions.
replaceable by computational systems such as machine intelli- In the AD theory, this principle has been formally described based
gence, we see humans and computers as working together within on the concept of “design domains” and “mapping.” It is the key for
an ecosystem where each must bring their strengths to bear. designers to learn the what and how relationship between design
Preliminary work in enhancing the tacit knowledge of human has domains. Functional requirements (FRs) are derived from customers’
included geometric programming to support co-evolutionary needs and multiple solution concepts (by the choice of design
design among team members through a software package GPKit parameters, DPs) are conceived in the “functional space,” as shown in
created by one of the authors [10]. A simple machine learning- Fig. 1. AD defines “design” as a mapping between functional domain
based design example developed by another author is “Sketch- and physical domain, which is in the “Functional Space” in Fig.1, while
Helper” [11] which provides stroke guidance by training a stroke most practitioners perform stepwise iterative design activities along
guidance neural network that learns the mapping between the the time-domain, which is in the “Process Space.”
step-wise stroke relations to predict the user’s next stroke. We
propose a framework which will become a tool for broad
investigations of this likely convergence and co-evolution of
man machine intelligence in the field of design and manufacturing.

3. Virtual design assistant (VDA): a platform toward hybrid

intelligence

A concept of “virtual design assistant” (VDA) is developed, like Siri

or Google Assistant which uses voice queries and a natural language
interface to understand the intent of a user, answer questions and
perform proper actions. VDA should offer more than simple answers
and actions of the virtual assistant apps, in order to have human
Fig. 1. Illustration of the use of Axiomatic Design AD to reverse-map past design
designers reach proper design solutions through the collaboration. successes and failures in terms of functional requirements and design parameters,
Expected advices and recommendations include the following. which will then form hierarchical trees between the functional and physical domain
What would a designer do first when facing a new problem to via design.
solve? Could they ask machines as they ask the experts when they
don’t know where to start? Identifying the functional require- VDA should be able to oversee both Functional Space and Process
ments of a completely new system (or a problem) which has never Space while most designers tend to think at the Process Space. VDA
existed is even more complex and difficult since there are no data helps designers check the design matrix at each level of mapping
or knowledge available from prior cases and trials. Answering from what to how, following the first design axiom (Independence
these questions addresses the following major challenges and Axiom), in order not to have their FRs coupled by the chosen DPs. The
functions that could be provided by VDA: second design axiom, Information Axiom, concerns the complexity
of a design solution, which can be applied when there are two or
- Finding abstraction process for system architecting in terms of more non-coupled designs competing against each other (Fig. 1).
finding and structuring correct functional requirements. AD can be also used to reverse-map the past design successes
- Integrating the abstraction process with a generalization design and failures into trees of functional requirements and design
principle such as Axiomatic Design (AD) to map and decompose parameters as shown in the Functional Space in Figs. 1 and 2. The
to physical parameters of design. This will also train the junior reverse-mapped, decomposed and hierarchical past design data
designers to have structured thinking. can be generated in volume and be used to train the reinforced
- Developing a unified framework for systems architecture across neural network of VDA.
disciplines and scenarios.

VDA needs to understand designer’s intent written in natural

language on the design needs and questions. VDA also needs to have
design principles to assess the design decisions at the earliest possible
stage and to provide assistive suggestions. And the VDA should be able
to learn from past successes and failures though deep learning,
providing heuristic expertise collected in each discipline to the designer.
The key for a successful machine learning will be how to train
VDA with existing design cases and data, and on which principle
the DVA recommends/assists the designer. VDA uses a principle-
based design framework namely, “an Axiomatic Design (AD),”
which provides a set of principles that determine good design Fig. 2. Machine intelligence assisted FR-DP mapping and decomposition process,
practices. VDA can use AD to learn from past successes and failures which provides rewards to make the design decisions conform the AD principles.
 S.-G. Kim et al. / CIRP Annals - Manufacturing Technology 68 (2019) 141–144 143

The VDA will provide interactive advice to a human designer to combined loss function consisting of the reconstruction loss and a
reach good design decisions at each stages of design: clustering assignment hardening loss function.

- Suggest where to begin when a human designer struggles with 4.3. Recursive keyword-based cluster decomposition
unstructured, fragmental customer requirements.
- Help establish solution neutral functional requirements (FR) To understand the meaning of the subgroups which are divided
from customers’ needs. using deep neural network-based clustering, we use a seed
- Help map FRs to design parameters (DP). Decomposition keyword-based subset decomposition algorithm. The seed key-
continues through a top-down, zig-zag process (Fig. 2). words are provided by users (interactively) and each group has
- Analyze a design matrix at the early stage of design to verify that analyzed the relationship between seed words and its words in the
the design satisfies the independence axiom and the information subgroup. When k seed keywords are provided by the user, the
axiom. seed word feature vectors are intentionally represented as much as
- Suggest revising DPs to avoid coupled designs which violate the orthogonal to the rest of the vectors to ensure functional
axioms. independence.

There are the three core functions of VDA we need to develop to 4.4. Experiment1
achieve the above: 1) understanding of the user needs and extract
functional requirements, 2) providing suggestions on the goodness We validated the proposed hierarchical clustering algorithm of
of design decisions based on AD, 3) accumulating big data from the VDA by showing how to effectively abstract the highly scattered
past design successes and failures. This paper focuses on the first data in text into subgroups (Fig. 3a) and to extract the meaningful
step of VDA which is to understands the user needs and intension features (key words or center words) of each subgroup. We used
and extract functional requirements in the correct AD syntax Boston Airbnb review comments dataset (https://www.kaggle.
machine can vectorize them. com/airbnb/boston) which is simply chosen among public natural
language datasets available. The proposed algorithm is imple-
mented using Pytorch, an open source deep learning platform, and
4. Extracting functional requirements MATLAB. The dataset was first tokenized as a collection of words
and then filtered with the predefined stop. Secondly, we extracted
Abstraction and generalization have played an important role in 29,380 words in 300-dimension vector using word2vec. We
the minds of scientists to solve complex problems [14]. Abstraction grouped the words into 15 clusters to separate out reviews with
in AI is a technique that reduces the complexity of a problem by foreign languages. By using the loss function, which is combined of
filtering the irrelevant properties while preserving all important reconstruction loss and clustering loss, the deep neural network-
and necessary details to solve a given problem. Instead of the rule- based clustering provides clear separation with different lan-
based technique, a data-driven hierarchical system representation guages. After removing language reviews, the second run deep
and abstraction framework through deep learning have resulted learning based clustering shows very distinct subgroups (Fig. 3b).
much improved and successful AI tools for natural language
processing, text mining and even novel writing in recent years.
Extracting functional requirements from a string of words, often-
fragmented and imperfect user specifications, can be accomplished
by recognizing the designer’s intent contextually and abstracting of
them by hiding low level details as well as decomposing the subset
data using recursive clustering algorithm. For effective key feature
abstraction and decomposition, we first applied existing deep neural
network-based systematic taxonomy of clustering methods to
abstract the text data [15]. Hierarchical clustering algorithm is
composed of three steps: (1) preprocessing to remove the
Fig. 3. Deep neural network-based clustering from scattered feature vectors, a)
meaningless words and convert the words to high-dimensional
filtered English words using t-SNE domain, b) deep learning-based clustering result
vectorized features; (2) feature abstraction using deep learning- (the color represents the number of subgroup), c) seed word-based clustering using
based clustering; (3) recursive seed keyword-based subgroup t-SNE domain.
decomposition (with the input from the human designer).
We intentionally added 5 seed words into filtered feature
4.1. Preprocessing vectors to accelerate the clustering; location, value, communica-
tion, host, and cleanness. Then the DL clusters 15,414 words in the
Natural languages are composed of various characters, symbols, filterer data into 5 subgroups as shown in Fig.3c). 81.15% of the
and numbers. First, VDA reads the documents and tokenizes them words are projected into the “location” keyword, indicating that
as the collection of words. It filters predefined “stop” words from most Airbnb reviews are very closely related to “location.”
the tokenized collection of words. To represent the words as
unique entities, each word is vectorized using word2vec, which is a 4.5. Experiment2
two-layered neural network trained to reconstruct the linguistic
contexts of words [16]. The same method was applied to a set of unstructured design
requirements in manufacturing. From a fully decomposed FR tree
4.2. Clustering of a Chemical Mechanical Polishing machine designed and built at
MIT [17], 146 design requirements were collected and put into a
From a large set of vectorized words (we used 300 dimensions bag of words. Doc2vec, an extension of the word2vec for vectorizing
for vectorization), we abstract the feature vectors by utilizing the phrases rather than words, was used to vectorize each requirement
deep neural network-based systematic taxonomy to cluster the into a 784-dimensional feature vector [18]. Autoencoder-based
data. The autoencoder-based deep neural network is used to deep clustering was applied to recursively generate a binary tree
reduce the dimension of the feature vector, and then reconstruct structure but resulted in unsuccessful clustering and could not
the data from the reduced representation of the original input data. extract higher level FRs either. A key learning from the two
Autoencoder is trained to use standard mean square-based loss experiments is that AI abstraction tools for natural language such
function, and the features are voted into a few subgroups by using as word2vec or doc2vec to vectorize FRs in design is not effective in
144 S.-G. Kim et al. / CIRP Annals - Manufacturing Technology 68 (2019) 141–144

extracting functional requirements from natural language expres- design principles for good design decisions, like Axiomatic Design
sion of design requirements. This finding suggested a new way of (AD), and to provide assistive design suggestions for designers to
vectorizing FRs in the design domain as described below. navigate between functional and physical domains. A key major
challenge in developing a platform for hybrid intelligence is
4.6. Function embedding vs. word embedding identified as the extraction of functional requirements from
unstructured and, sometimes, ill-defined user specifications. We
The existing word embedding tools are for clustering words in found the existing word embedding tools of deep learning do not
documents based on the proximity of word vectors and frequency of extract functional requirements directly from the user specifica-
them. But to extract a functional requirement from a short string of tions. A syntax for FRs has been applied to translate (encode) user
words cannot be achieved by simple frequency counting of word needs and specifications to a collection of functional requirements
vectors in proximity. For this challenge, we created a concept of (FRs) that can be vectorized and understood by the machine
“function embedding” instead of “word embedding.” “Function learning tools to create structure and hierarchy. Once confirmed
embedding” is to vectorize a sting of words which describes a with the design with interactive communication, the vectorized
function wanted. Once vectorized, functions with similar context FRs will be able to utilize the rich set of AD principles, theorems
occupy close spatial positions in n-dimensional vector space, then and corollaries to assess the designer’s decision and provide
the cosine of the angle between such vectors could be used to adequate advices.
determine the orthogonality, redundancy and functional coupling of Leveraging the Deep Learning’s recent progress, we expect to
them. find a way to transform the art of design to a scientifically solvable
Function embedding can be explained with a case below to problem, by integrating human and machine intelligence effec-
show how FRs can be extracted, vectorized and used thereafter. tively. The VDA will have a significant influence on the designing
Consider a case of a building façade design with an ambiguous across the wide spectrum of engineering systems including the
designer’s need such as, “I want my building to have easy access.” future advanced manufacturing systems, such as cyber physical
Firstly, a string of user specified requirements needs to be encoded production systems (CPPS) where design and system architecture
following a syntax of FRs: “action + object + attributes + hereditary still are based on traditional engineering methods facing increased
information (from higher level FRs).” Then the designer’s system complexity [19]. VDA as a deep learning platform and the
statement can be rephrased such as, “FR1: Provide access to my wealth of manufacturing technology accumulated in the CIRP over
building.” This encoding can be done by training a shallow network the years will enable exemplary cases of human/computer
with the FR syntax and many FR examples. collaboration toward smart and intelligent product design and
Using the Axiomatic Design framework, a corresponding design manufacturing.
parameter DP1 can be chosen by the designer or suggested by VDA
as “DP1: a kind of door.” From the solution neutral status of design,
Acknowledgement
we are not supposed to rush to choose a type of door (slide, hung
etc.) as DP1. FR1 needs to be decomposed further as shown in Fig. 4
This work was supported by SUTD IDC Flagship Project Funding
(a). Per VDA’s suggestion, a designer can make lower level FRs in
and MIT/SenseTime Alliance on Artificial Intelligence.
free form text, which can also be encoded into FRs.

References

[1] Dornfeld D, De Vries M (1990) Neural Network Sensor Fusion for Tool Condi-
tion Monitoring. CIRP Annals 39(1):101.
[2] Noori-Khajavi A, Komanduri R (1993) On Multi-Sensor Approach to Drill Wear
Monitoring. CIRP Annals 42(1):71.
[3] Kumara S, Ham I (1990) Use of Associative Memory and Self-Organization in
Conceptual Design. CIRP Annals 39(1):117.
[4] Krause F, Fisher A, Gross N, Barhak J (2003) Reconstruction of Freeform Objects
with Arbitrary Topology Using Neural Networks and Subdivision. CIRP Annals
Fig. 4. FRs of a building entrance design case (a) and t-SNE visualization of building 52(1):125.
entrance FR vectors (b). [5] Teti R, Langella A, D’Addona D (1999) An Intelligent Computation Approach to
Process Planning in Multiple-Step Cold Forging. CIRP Annals 48(1):175.
[6] Monosotori L, Prohaska J (1993) A Step Towards Intelligent Manufacturing:
Using doc2vec, each FRs were vectorized, and then t-SNE Modelling and Monitoring of Manufacturing Processes Through Artificial
visualization tool was used to compare the vectors in two Neural Networks. CIRP Annals 42(1):485.
dimensions (Fig.4(b)). The parent FR is located at a central position [7] Choi GH, Lee KD, Chang N, Kim SG (1994) Optimization of the Process
Parameters of Injection Molding with Neural Network. CIRP Annals 43(1):449.
compared to the sub FRs in this example, which shows the positive [8] Scholz-Reiter B, Muller S, Wiendhal HP (2000) Throughput Time Control in
feasibility that functions can be vectorized with the existing word Production Systems Supported by Neural Networks. CIRP Annals 49(1):331.
embedding tools if the FRs are encoded following the specific syntax. [9] Rechtin E (1991) Systems Architecting: Creating & Building Complex Systems,
Prentice Hall, New Jersey.
Once the functions are encoded into vectors, a metric for the [10] https://gpkit.readthedocs.io/en/latest/.
degree to which child FRs are related to their parent can be [11] Choi J, Cho H, Song J, Yoon SM (2019) SketchHelper: Real-Time Stroke Guidance
established. Similarly, child FRs that may not belong in this branch for Freehand Sketch Retrieval. IEEE Transactions on Multimedia. http://dx.doi.
org/10.1109/TMM.2019.2892301. in press.
can be identified easily. With further development, the virtual [12] Suh NP, Kim S, Bell AC, et al (1978) Optimization of Manufacturing Systems
design assistant will apply the hereditary nature of FRs along the through Axiomatics. CIRP Annals 31:383–388.
same branch in order to structure unstructured and fragmental [13] Suh NP (1990) The Principles of Design, Oxford University Press.
[14] Dreyfus T (1991) in Tall D, (Ed.) Advanced Mathematical Thinking Processes,
requirements into a structured one. Also, vectorized FRs can be Advanced Mathematical Thinking, Springer.
mathematically analyzed to find whether the FRs are coupled or [15] Aljalbout E, Golkov V, Siddiqui Y, Cremers D (2018) Clustering with Deep
not and, if coupled, how serious the coupling is. Learning: Taxonomy and New MethodsCoRR, Vol. abs/1801.07648 .
[16] Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient Estimation of Word
Representations in Vector SpacearVix:1301,3781 .
5. Conclusion [17] Melvin JW, Suh NP (2002) Axiomatic Design of a Chemical Mechanical Polishing
(CMP), MIT Ph.D. Thesis, MIT.
A concept of “virtual design assistant” (VDA) is developed to [18] Le Q, Mikolov T (2014) Proceedings of the 31st international conference on
machine learning. PMLR 32(2):1188–1196.
have three core functions to understand designer’s natural [19] Stark R, Kinda S, Neumeyer S (2017) Innovations in Digital Modelling for Next
language in the syntax of functional requirements, to implement Generation Manufacturing System Design. CIRP Annals 66(1):169.

View publication stats