CIRP Annals - Manufacturing Technology 73 (2024) 9 12

Contents lists available at ScienceDirect

CIRP Annals - Manufacturing Technology

journal homepage: https://www.editorialmanager.com/CIRP/default.aspx

An LLM-based approach for enabling seamless Human-Robot

collaboration in assembly
Christos Gkournelos, Christos Konstantinou, Sotiris Makris (2)*
Laboratory for Manufacturing Systems and Automation, Department of Mechanical Engineering and Aeronautics, University of Patras, Patras, Greece

A R T I C L E I N F O A B S T R A C T

Article history: The complexity in the collaboration between humans and robots in smart manufacturing remains a signifi-
Available online 22 May 2024 cant challenge. This paper introduces an LLM-based manufacturing execution system enhancing Human-
Robot Collaboration (HRC) in smart manufacturing. By leveraging Large Language Models (LLMs), the system
Keywords: provides a natural language interface for operators, integrates with Digital Twins for real-time data, and
Large language models
employs behavior-based control for robots. This integration facilitates intuitive interactions and rapid system
Human robot collaboration
Manufacturing systems
programming, addressing communication complexities in HRC. The effectiveness of this approach is validated
through two HRC assembly case studies, demonstrating significant improvements in collaboration and effi-
ciency.
© 2024 The Author(s). Published by Elsevier Ltd on behalf of CIRP. This is an open access article under the CC
BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)

1. Introduction assistant solution, having an agent-like behavior, by formulating

plans and utilizing external tools [9], lowering the barrier of a collab-
In recent advancements in manufacturing [1], human robot col- oration between robots and humans.
laboration (HRC) has been employed significantly as a means to The tendency of LLMs to perpetuate biases, coupled with their
increase shop floor flexibility. The need for producing a large variety vulnerability to producing hallucinatory [10] or erroneous content
of customizable products increases the required resilience of the [11], presents a significant challenge. It is crucial to strategically
production system, which in turn requires further advances in flexi- deploy self-supervised deep LMs in a manner that not only over-
ble collaborative manufacturing systems [2]. Human centered comes these technical hurdles but also enhances their utility in prac-
manufacturing emphasizes the collaboration of artificial intelligence tical engineering applications.
(AI) agents with human workers while presenting social values. This paper presents a novel approach for efficient and intuitive
Despite the recent advances in recent years, there are still limitations coordination and interaction of human and robotic resources in
in task planning, intuitive interaction, and programming [3]. assembly tasks. Leveraging the extensive semantic knowledge of
Data architecture in HRC-based production systems often remains LLMs, the system is adept at identifying practical tasks for collabora-
highly complex [4], necessitating employing advanced frameworks tive assembly between humans and robots. This approach enables
to handle and process large volumes of data. Moreover, making these easy programming for HRC assembly systems and enhances non-
systems user-friendly for non-specialists is still a significant chal- expert user interactions.
lenge [5]. It’s essential to develop systems with clear, intuitive inter- The system enables production engineers to effortlessly orches-
faces and straightforward interactions to facilitate easy engagement. trate the process schedule and allocate tasks to the resources using
Striking the right balance between managing complex data and text commands. While a query command like “create a schedule for
ensuring user accessibility is crucial in creating effective HRC systems the final assembly of the e-motor with one robot and one worker”,
that seamlessly combine technical sophistication with approachable may elicit a coherent narrative response by an LLM, it lacks practical
interfaces. applicability in the context of assembly execution. This is because
Progress on AI [6], and especially on generative AI, has shown LLMs, in their standard form, are not inherently attuned to the
promising results, creating novel interactions approaches, with sim- manufacturing world. Robots follow precise commands, while
pler scenarios [7]. Studies have shown that such AI models can effi- human workers need clear, concise instructions.
ciently adapt to complicated data structures transformed into their This paper details a method for implementing an LLM applica-
vectorized form [8], and by using their intrinsic knowledge, be able tion to extract relevant information for manufacturing, manage
to respond to queries only by exploiting the nativeness of the natural the task execution, and enable natural language interaction with
language. This advantage of the generative AI, and especially Large human workers and engineers. This solution bridges the gap
Language Models (LLMs), can be interpreted as a middle stage between the narrative capabilities of LLMs and the practical
requirements of a physical, collaborative manufacturing environ-
* Corresponding author. ments and focuses on the creation of a “manufacturing reasoning”
E-mail address: [email protected] (S. Makris). by an LLM.

https://doi.org/10.1016/j.cirp.2024.04.002
0007-8506/© 2024 The Author(s). Published by Elsevier Ltd on behalf of CIRP. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/)
10 C. Gkournelos et al. / CIRP Annals - Manufacturing Technology 73 (2024) 9 12

The paper is organized as follows. The Section 2 presents the sys- instrumental in minimizing hallucination issues. By defining a
tematic methodology and the implementation details of the system. sequence of reasoning steps, this method narrows the response
In Section 3, the application of the proposed system in two industrial space, thereby focusing on targeted, relevant outputs.
case studies is presented. Section 4 presents the evaluation of the sys- The Interaction Agent is primarily engaged in handling the conver-
tem while in Section 5 the conclusions and future developments are sation with the users. It employs a ’plan-and-execute’ strategy and is
reported. equipped with a conversational memory buffer, enhancing its ability
to manage and sustain user interactions. To meet the natural interac-
2. LLM-based execution system for HRC tion requirement, a series of tests were conducted to select the most
suitable LLM for this agent. GPT-3.5 emerged as the preferred choice,
The proposed LLM-based manufacturing execution system is attributed to its high accuracy in generating responses and optimal
designed to integrate and enhance HRC in smart manufacturing envi- response time. This selected model has been fine-tuned using CoT,
ronments. The system is characterized by several key features: enabling it to effectively discern whether an input should be redir-
ected to the Manufacturing Agent or addressed as a general query.
1. System knowledge: Access to manufacturing proprietary knowl- The Manufacturing Agent addresses questions related to the HRC
edge bases and data sources. assembly deployed in the system. A custom LLM, HRC-GPT, based on
2. Online information: Integration with the Digital Twin (DT) of the the gpt-4 0613 model [13], was specifically trained for this purpose.
system allowing the access of dynamically updated data. This fea- This agent is responsible for generating comprehensive production
ture is mandatory when there is a need to obtain and use the run- schedules or modifying existing tasks in the running assembly sched-
time state of the system. ule. HRC-GPT possesses intrinsic knowledge of the production sys-
3. Scalability: Modular architecture for allowing easy reconfigura- tem, a capability enhanced through the integration of manufacturing
tion based on application requirements. The training of the AI data via vector augmentation and prompting techniques, as outlined
components is performed independently to easily add new capa- in Section 2.2. Endowed with "manufacturing reasoning," HRC-GPT is
bilities without compromising the performance of the existing adept at creating accurate assembly schedules and interfacing with
ones. the respective modules that manage resource control. This function-
4. Explainability: A transparent process that can be monitored by ality underscores its pivotal role in the HRC process, enabling precise
humans allowing a level of control in the observations and deci- and efficient coordination of manufacturing activities.
sions.
2.2. Modules
The framework of the proposed system is illustrated in Fig. 1. The
proposed LLM-based approach for HRC is based on an extendable set The methodology proposed in this paper includes two main mod-
of modules, each serving a distinct function within the HRC environ- ules that serve as bridges to the physical world: the Human Interac-
ment. There are two primary categories of modules: a) Agents and b) tion Module and the Robot Behavior Planning module.
Modules. The Agents are neural modules including LLMs, fine-tuned The Human Interaction module is designed to facilitate natural
LMs, or any other AI enabled component with the objective to classify communication between human operators and the system. It is
an action based on inputs. Modules are symbolic tools that imple- equipped with Natural Language Processing (NLP) capabilities,
ment action functions called by Agents. These modules are used for including both speech-to-text and text-to-speech functionalities.
connecting external APIs, such as databases, robot controllers, etc. This module plays a dual role in the LLM-based system: firstly, as
an input interface, enabling human operators to interact seamlessly
with the system through natural language; and secondly, as an out-
put interface, effectively communicating system-generated informa-
tion back to the human users. Its design focuses on ensuring that
interactions are intuitive and user-friendly, thereby enhancing the
overall efficiency and effectiveness of the HRC process.
In managing robotic resources, the system adheres to the Behav-
ior Trees (BTs) methodology [13]. This approach is particularly advan-
tageous across various applications due to its structured manner of
representing and controlling autonomous robots. The Robot Behavior
Planning module plays a critical role in this process. It is tasked with
receiving the BTs generated by the Manufacturing Agent and subse-
quently relaying them for execution. This methodology empowers
the LLM system to produce precise outputs that are not only theoreti-
cally sound but also practically executable in the physical world.
Through this integration, the LLM system transcends traditional
boundaries, effectively bridging the gap between high-level planning
and real-world implementation.

Fig. 1. The proposed LLM-based approach for HRC execution system. 2.3. Manufacturing data and training

The proposed system consists of one formating Agent with NLP The training of the LLM for the specific HRC knowledge integra-
capabilities, two language model (LM) Agents (Interaction Agent tion into our assembly system employs a comprehensive approach,
and Manufacturing Agent), and two action Modules (Robot Behavior utilizing vectorized representations of diverse data sources. As
Planning, Human Interaction). depicted in Fig. 2, the input manufacturing data is organized in sev-
eral structured files. This data encompasses several critical compo-
2.1. Agents nents: process planning details, comprehensive resource
information, geometric layout data, and a collection of primitive
Within the proposed LLM-based execution system for HRC, agents behaviors designed for robotic execution.
are integral neural modules tasked with system observation and Resource data provides an extensive inventory of tools and equip-
interaction. The system incorporates two specialized agents: the ment, outlining their constraints and operational limits. These data
"Interaction Agent" and the "Manufacturing Agent." Both agents are are considered crucial for facilitating improved resource allocation
fine-tuned utilizing the chain-of-thought (CoT) [12] method, which is aiding the planning of task scheduling. Geometric layout data,
 C. Gkournelos et al. / CIRP Annals - Manufacturing Technology 73 (2024) 9 12 11

Table 1
Number of tasks in the industrial use cases.

Number of Tasks Ttotal Trobot Thuman Tcollab

E-motor 16 15 14 1
Air compressor 14 14 14 6

3.1. Task planning and system programming

The initial application of the LLM-based execution system in our

case study involves transforming natural language inputs from pro-
duction engineers into a detailed production schedule. This process is
driven by specific Key Performance Indicators (KPIs). The primary
objective of the system is to generate a schedule that not only
adheres to the semantic rules embedded in the query but is also exe-
cutable in the physical manufacturing environment. Fig. 3 shows an
interaction example on a user query. The system employs a ’chain of
thoughts’ approach, enabling it to grasp the underlying meaning of
the input query comprehensively. Consequently, the system’s output
is not just a theoretical plan but a practical, executable schedule that
can be directly implemented in the physical manufacturing cell.

Fig. 2. The overall system architecture.

extracted from CAD and Digital Twin sources, precisely map the posi-
tions and orientations of all components within the manufacturing
cell, ensuring spatial accuracy in planning. The process plan data
delineates the sequence of assembly and disassembly tasks, while
the primitive behaviors represent a suite of fundamental actions exe-
cutable by robots. Fig. 3. HRC schedule generation from a simple language instruction.
To effectively incorporate this data into the LLM, it undergoes
transformation into a vectorized format using an embedding model. 3.2. Dynamic reconfiguration
Each data segment is processed to generate a semantic vector. This
vectorization is performed using the text-embedding-ada-002 model The second use case for the LLM-based execution system involves
[13], which produces a 1536-dimensional output matrix for each its role in facilitating dynamic interaction with the human operator
data segment. This matrix is used as a foundation for the creation of during execution phases. Central to this interaction is an innovative
the HRC-GPT. The custom trained model employs a cosine similarity interface, named CollabAI, which is deployed on Augmented Reality
metric [14] to retrieve relevant data segments, which are conceptual- (AR) glasses and serves as a critical communication link between the
ized as neighboring vectors in the vector space. The system then gen- operator and the system (Fig. 4). This AR interface is designed to sup-
erates a prompt that combines this retrieved data with the internal port both voice and text modalities. The LLM system provides real-
memory summary. This method in collation with the CoT prompt time guidance and feedback. This level of interaction enhances the
engineering enables the generation of contextually relevant and pre- operator’s situational awareness and decision-making capabilities
cise responses for the HRC system. and ensures that the manufacturing process remains flexible and
responsive to real-time changes.
3. Case study

To validate and assess the effectiveness of the proposed LLM-

based manufacturing execution system, a comprehensive testbed
environment was established, featuring two distinct manufacturing
scenarios. The first scenario, derived from the automotive industry,
involves the intricate assembly of an electric motor and its inverter.
The second scenario originates from the machinery industry, focusing
on the assembly of an industrial air compressor unit. These cases
were specifically chosen for their relevance in the HRC domain, par-
ticularly due to the flexibility they offer in task assignments. Such
flexibility is crucial in evaluating the LLM’s capabilities in schedule
generation and task allocation. Fig. 4. Augmented reality interface of the LLM-system.
Both assembly cases involve a UR10 collaborative robot and
a human operator. Additionally, a variety of tools are provided to 4. Experimental evaluation
facilitate the flexible operation of the robot. These tools include a
pneumatic suction gripper, two mechanical grippers, and four screw- To assess the performance of HRC-GPT, a dataset was created and
drivers two electric and two pneumatic. The tasks designed for this tailored to the evaluation of natural language-based task planning in
environment range from heavy picking and placing operations, suit- assembly cases. Derived from the environments and actions detailed
able for the robot, to screwing tasks. Meanwhile, there are also pick- in two industrial cases, the dataset is divided into three categories
ing and placing tasks that require collaboration of human operator covering different task complexities and Key Performance Indicators
with the robot. A significant proportion of the assembly tasks are (KPIs)
designed to be versatile, capable of being performed by either the Simple dataset comprises data pertaining only to the suitability of
robot or the human operator. In the Table 1, the number of robot, resources for each task. Utilizing this dataset, the system’s perfor-
human, and collaborative tasks are depicted. In this work two use mance is gauged based on Resource Utilization as the sole KPI for pro-
cases of the system are investigated. cess schedule selection.
12 C. Gkournelos et al. / CIRP Annals - Manufacturing Technology 73 (2024) 9 12

Intermediate dataset is enriched with additional information on capabilities. The proposed solution showcased an end to end (from
each task’s process time by the resources. It allows for the analysis of programming to execution) application of an LLM system in an actual
two KPIs: Resource Utilization and Assembly Cycle Time. HRC assembly station. The deployment and evaluation of this
Complex dataset: Incorporating all the information related to the approach using real industrial data have highlighted the significant
assembly process, this is the complete dataset of each case. It enables potential of this system in practical scenarios.
the generation of schedules considering up to five criteria: Resource Future work will go deeper into refining training methodologies
Utilization, Cycle Time, Non-adding Value Activities Time, Ergonom- and prompt engineering strategies to improve the system’s perfor-
ics, and Safety. mance in complex reasoning involving multiple parameters.
The experimental methodology was structured around these three Additionally, another goal is a dataset creation using diverse
dataset categories. Each use case was associated with a group of data- manufacturing tasks, serving as a benchmark to assess and compare
sets, and an additional combined case was created by merging data methodologies, further advancing in the field.
from both the electric motor and the air compressor assembly cases.
A diverse set of input queries were crafted for the experiments. Declaration of competing interest
Initially, one query was designed for each dataset case and type. Sub-
sequently, the general GPT-4 model was utilized to generate multiple The authors declare that they have no known competing financial
variations of the initial query, validating performance across different interests or personal relationships that could have appeared to influ-
phrasings. For the combined case, the queries from both the electric ence the work reported in this paper.
motor and air compressor assembly were employed. Starting from 30
distinct input queries (5 for simple, 10 for intermediate, and 15 for
CRediT authorship contribution statement
complex), our augmentation technique expanded this to a set of
1000 input queries, with 500 for each case (50 for simple, 100 for
Christos Gkournelos: Conceptualization, Formal analysis, Soft-
intermediate, and 350 for complex).
ware, Writing original draft, Writing review & editing. Christos
The output evaluation involved comparing the HRC-GPT’s
Konstantinou: Software, Validation, Writing original draft, Writing
responses to the outputs of the AI-task planning tool [15], which
review & editing. Sotiris Makris: Conceptualization, Funding
served as a ground truth. For each input query, the task planning tool
acquisition, Project administration, Writing review & editing.
was configured accordingly. For the evaluation three metrics were
employed:
Acknowledgments
 Format Accuracy (FA): This metric evaluates the HRC-GPT’s
responsiveness to deliver a structurally correct output. This research has been supported by the “CONVERGING Social
 Executability (EX): This assesses the system’s ability to deliver industrial collaborative environments integrating AI, Big Data and
functionally correct schedule, ready for execution. "Correct" here Robotics for smart manufacturing (Grant Agreement: 101058521)”.
refers to the accurate assignment of resources to tasks they can
perform. Supplementary materials
 Goal Accuracy (GA): This measures the correlation between the
expected output schedule and the generated one, particularly in Supplementary material associated with this article can be found
terms of KPI compliance. in the online version at doi:10.1016/j.cirp.2024.04.002.

The results of the experiments are presented in Table 2. Starting References

from the natural language, up to the generated schedule, the system
demonstrated consistent structural correctness, evidenced by high FA [1] Chryssolouris G (2006) Manufacturing Systems: Theory and Practice, Springer.
across all datasets. EX indicated effective functional schedule genera- [2] Makris S (2021) Cooperating Robots for Flexible Manufacturing, Springer Interna-
tional Publishing.
tion, though with a slight decline in more complex scenarios. This indi- [3] Kru€ ger J, Wang L, Verl A, Bauernhansl T, Carpanzano E, Makris S, Fleischer J, Rein-
cates that LLM based system are not governed by causality but are the hart G, Franke G, Pellegrinelli S (2017) Innovative Control of Assembly Systems
perfect candidate to correlate vast amounts of data, even unseen ones. and Lines. CIRP Annals 66:707–730.
[4] Wang L, Liu S, Liu H, Wang XV (2020) Overview of Human-Robot Collaboration in
The most notable challenge was observed in GA, particularly in more Manufacturing. In: Proceedings of 5th International Conference on the Industry 4.0
intricate tasks, reflecting difficulties in aligning with multiple KPIs. Model for Advanced Manufacturing, 15–58.
Notably, in approximately 70% of cases, the desired output was [5] Wang L, Gao R, Vancza J, Kru € ger J, Wang XV, Makris S, Chryssolouris G (2019)
Symbiotic Human-Robot Collaborative Assembly. CIRP Annals 68:701–726.
achieved with just one additional user input, highlighting the system’s [6] Chryssolouris G, Alexopoulos K, Arkouli Z (2023) A Perspective on Artificial Intelli-
responsiveness and potential for user-guided optimization. Overall, gence in Manufacturing, Springer.
HRC-GPT shows strong structural accuracy, but its functional correct- [7] Zhang Z, Chai W, Wang J (2023) Mani-GPT: A Generative Model for Interactive
Robotic Manipulation. Procedia Computer Science 226:149–156.
ness and multi-KPI optimization, especially in complex environments,
[8] Asai A, Min S, Zhong Z, Chen D (2023) Retrieval-Based Language Models and
present opportunities for further development and refinement. Applications. In: Proceedings of the 61st Annual Meeting of the Association for
Computational Linguistics, Tutorial Abstracts, , 41–46.
Table 2 [9] Ruan J, Chen Y, Zhang B, Xu Z, Bao T, Du G, Shi S, Mao H, Zeng X, Zhao R (2023)
Evaluation of HRC-GPT. TPTU: Task Planning and Tool Usage of Large Language Model-based AI Agents.
NeurIPS-2023 Workshop on Foundation Models for Decision Making, arXiv preprint
Electric Motor Air Compressor Combined arXiv:2308.03427..
[10] Han Y, Liu C, Wang P (2023) A Comprehensive Survey on Vector Database: Stor-
Metrics FA EX GA FA EX GA FA EX GA age and Retrieval Technique. Challenge . arXiv preprint arXiv:2310.11703.
[11] Bender EM, Koller A (2020) Climbing Towards NLU: On Meaning, Form, and
Simple 1 1 0.9 1 0.9 0.9 1 1 1
Understanding In The Age of Data. In: Proceedings of the 58th Annual Meeting of
Intermediate 1 0.8 0.9 1 0.9 0.6 1 0.8 0.8
the Association for Computational Linguistics, 5185–5198.
Complex 1 1 0.8 1 0.8 0.6 1 0.8 1 [12] Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, Chi E, Le Q, Zhou D (2022)
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advan-
ces in Neural Information Processing Systems 35:24824–24837.
5. Conclusions
[13] Cao Y, Lee G (2022) Behavior-Tree Embeddings for Robot Task-Level Knowledge.
2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS),
This research establishes a robust framework using an LLM-based 12074–12080.
system as an orchestrator in manufacturing environments. The [14] Han J, Kamber M, Pei J (2012) Getting to Know Your Data. Data Mining : 39–82.
[15] Evangelou G, Dimitropoulos N, Michalos G, Makris S (2021) An approach for task
primary objective of this paper was to elucidate the concept of and action planning in Human Robot Collaborative cells using AI. Procedia CIRP
“manufacturing reasoning” and its necessity in leveraging LLM 97:476–481.