A Review on The Evolution of Human-computer Interaction

Using Integrated Generative AI in Operating System

Author: Himanshu Singh, Srijana Gautam, Ritika Sinha

1.Abstract: technology, people began to input code

The evolution of human-computer interaction commands to the computer by using the
(HCI) has been marked by a continuous strive for keyboard, and the computer returned the
user-friendliness and accessibility. This review processing results to the display. This was very
paper analyzes two research papers that propose difficult as the user had to remember codes
advancements in Human-Computer Interaction and only programmers and researchers could
using artificial intelligence (AI). The first paper work on these systems. Later, GUI was
explores the technical hurdles that must be introduced. The graphical user interface (GUI)
overcome to realize this vision, including data replaced the command line interface, making
management, model trustworthiness, security, computers easier to use with a mouse and
and ethical considerations. The second paper keyboard. The Xerox Alto was the first
outlines a visionary approach where a network of graphical user interface system, but the Apple
generative AI models handles user requests, Macintosh popularized GUIs in 1984. Since
eliminating the need for traditional applications then, GUIs have been adapted for touch
and complex commands. Through a comparative screens on smartphones, leading to the rise of
analysis, this review paper sheds light on the mobile computing. Natural User Interface
promising potential of AI-powered HCI while (NUI) aims for interaction similar to human-to-
acknowledging the complexities that need to be human communication. It does not require
addressed for its successful implementation. external devices like keyboard or mouse to
interact with the computer system. It uses
2.Introduction: natural human abilities like gestures, voice,
Human- Computer Interaction (HCI) is a study and eye gaze to interact with computers,
focussing in the design of the computer reducing reliance on external devices.
technology and, in particular, the interaction Within the contemporary technological
between humans (the users) and computers. landscape, artificial intelligence (AI) emerges
While initially concerned with computers, HCI has as a potent force with the capability to
since expanded to cover almost all forms of significantly augment and redefine current
information technology design. The evolution of operating systems and user interfaces. The
Human- Interaction interface has undergone burgeoning capacities of large generative
multiple transformations from Command Line models (LGMs), encompassing large language
Interface (CLI) to Graphical User Interface (GUI) models (LLMs) and diffusion models (DMs),
and Natural User Interface (NUI). offer a compelling glimpse into a future where
In CLI, with the development of computer our interactions with technology transcend the
techno- confines of traditional boundaries.
This revolution with the inclusion of Artificial 3.Related Work:
Intelligence and Machine
requiring additional trainingLearning in human-
data. Despite their an alternative
Several studiesmeansshowcaseof interaction
the use offor users
gestures
computer interaction will explore
impressive achievements, LLMs face limitations. the who mayinteraction.
have difficulty usingdesigned
traditional input
for user Ahmed a NUI for
transformative potentialwith
Notably, they struggle of AIlarge
in revolutionizing
contexts and devices like keyboards and mice.
manipulating 3D animations through gestures
user interfaces. We will delve
exhibit a tendency to hallucinate (generate into false
the
capabilities of LGMs, specifically Itlikecanplay,alsopause,
improve anduser rotation. Similarly,
experience by
information). This has spurredLLMs and DMs,
research on exoskeleton robots controlled by gestures have
and elucidate their potential applications in enabling more intuitive and immersive
augmentation techniques. Current been explored by researchers at Cartagena
fostering a more intuitive andonnatural interactions, especially in applications like
augmentation methods focus allowinghuman-
LLMs
computer interaction paradigm. virtual reality (VR) or augmented reality (AR). is
University of Technology. Speech recognition
to interact with external tools through APIs another powerful NUI approach. Tmall genie
(e.g., calling
LLMs a calculator
with their for calculations).
vast data processing This and 4.2.
andVoice Commands:
car machine systems exemplify how voice
work proposes
reasoning a paradigm
capabilities can actshiftasbeyond
the core static
for commands
Robust speech canrecognition
be used to execute various
technology allows
tools. We envision a future where
intelligent interfaces. They can understand and generative functionalities. NUIs can also
users to interact with the system using natural leverage the
models can
generate communicate
natural language,naturally
allowingwith themusersto external commands,
language environment. Automobile
eliminating the needanti-for
and each other,
orchestrate potentially
different systemrendering
componentstraditional
and collision
manual input. systems by Tesla and Toyota utilize
programming
create intuitiveobsolete for the average user. This
user experiences. cameras to detect hazards and trigger safety
vision opens up exciting possibilities. Recent Voice
measures. commands
Sunjin's enhance
S3D system accessibility
employs hand for
DMs further enhance
advancements the potential
in text-to-image by generating
generation pave users with mobility impairments or those who
tracking technology to create realistic,
personalized interfaces. These interfaces can prefer hands-free
the way for personalized, runtime-generated interactive 3D interaction.
visuals. Santos et al.
adapt
GUIs totailored
individualtouserspecific
preferences userandtasks.
even
demonstrate
They can improve eye-tracking based users
efficiency by enabling NUIs,
their mood, leading
Furthermore, deep tolearning
highly customized
models are and
enabling
to perform device control
tasks throughand
quickly gazeefficiently
direction.
user-centric
acquiring designs.
multimodal capabilities beyond These verbal
through examples highlight the diverse
instructions.
images,
This newencompassing audio, video,
paradigm of AI-powered and 3D. goes
interfaces applications of NUIs and their potential to
beyond individual 4.3. Eye Tracking:
4.Proposed Work: systems. By using natural transform human-computer interaction across
language, it enables communication between various
Eye fields. technology
tracking As NUI research progresses,
enables users we to
By comparing
systems that anddon'treviewing
share the thesame
relatedstructure,
papers, can expect even more natural
control the interface and interact with content human
this paper
fostering proposes
a more a novel
unified operating
and efficient system
interaction behaviors
using to be commands.
gaze-based harnessed for intuitive and
design that integrates Natural User Interfaces
landscape. engaging computer interaction.
(NUI) with Artificial Intelligence (AI) to create a This modality enhances accessibility for users
more intuitive, personalized, and natural Recently,
with the focus
mobility has shifted
impairments or towards
conditions that
human-computer interaction experience. The utilizing transformers
prevent them from as generative
using LLMs.input
traditional
potential of AI to automate and study the devices.
These models excel at predicting the next
patterns with Machine Learning can study the Ittoken sequenceefficiency
can improve based on by context, leading
allowing userstoto
user gestures, expressions, preferences and widespreadinterfaces
navigate popularity due more to impressive
quickly and
provide customised and user-friendly interfaces results like.
accurately, especially in scenarios where
to revolutionize the system interfaces. precise selection is Transformer's
required.
LLMs leverage the ability to
4.1. Gesture Recognition: handle massive Assistant:
4.4.AI-powered parameter sizes, enabling
Gesture recognition utilizes cameras to interpret exceptional performance on diverse
An intelligent assistant powered by large
human gestures as commands for various tasks downstream
language tasks, (LLMs)
models even without additional
acts as a central
such as navigating through interfaces, training (zero-shot setting). Furthermore,
interface component, providing contextual
controlling applications, or manipulating objects prompting techniques
awareness, and in-context
natural language learning
processing, and
2.1. Interaction Design Paradigm [1]
in 3D environments. further enhance their capabilities without
proactive assistance.
This modality enhances accessibility by Contextual awareness allows the assistant to
providing
understand user context, such as their current Users can interact with the system using
task or environment, and adapt its responses familiar gestures or voice commands,
accordingly. reducing cognitive load and making the
interface more user-friendly.
Natural language processing enables the
assistant to comprehend user requests and 5.4.Personalized Interaction:
instructions delivered through voice or text,
The system learns user preferences over time,
enhancing the naturalness of interaction.
tailoring interface elements and assistant
Proactive assistance uses AI to anticipate user behavior to individual needs.
needs and offer suggestions, tutorials, or
For example, the assistant can recommend
automate routine tasks, thereby increasing
personalized content, adjust interface settings
efficiency and improving user experience.
based on user preferences, or provide
5.Benefits: targeted assistance based on past
interactions.
The module proposes many benefits by
integrating AI with the NUI in Human User Overall, implementing these modalities in the
Interface. They are as follows- system interface revolutionizes user
interaction, making it more accessible,
5.1. Enhanced Accessibility:
efficient, and personalized, thereby enhancing
NUI features like voice commands and gesture the overall user experience.
recognition make the interface more accessible
6.Challenges:
for users with disabilities, including those with
mobility impairments or visual impairments. The enhanced model comes with many
challenges which we need to overcome. Here
Blind users, for example, can interact with the
are few mentioned.
system using voice commands or gestures,
providing them with greater independence and 6.1.NUI Accuracy and Robustness:
access to digital resources.
Ensuring accurate and reliable interpretation
5.2. Increased Efficiency: of NUI inputs such as gestures and voice
commands across diverse environments poses
AI-powered assistance streamlines workflows
a significant challenge.
by automating tasks and providing context-
aware suggestions, reducing the time and effort Variations in lighting conditions, background
required to perform various activities. clutter, and individual differences in gestures
or accents can affect the accuracy of NUI
For example, the assistant can automatically
recognition.
schedule appointments, set reminders, or
suggest relevant documents based on the user's Addressing this challenge requires robust
context and preferences. algorithms capable of handling such variations
and extensive testing across different
5.3. Improved User Experience:
scenarios to ensure reliable performance.
The natural interaction paradigms of NUI,
6.2.AI Explainability and Trust:
coupled with AI's understanding of user intent,
create a more intuitive and enjoyable user As AI-powered assistants become more
experience. integral to user interactions, building trust by
providing transparency into their decision-
making processes becomes crucial. 7.Future Work:
8.Conclusion:
Users may be hesitant to rely on AI systems if 7.1.Research 9.References:and Development on Advanced
they
This cannot
paper understand
reviews thehow they arrive
evolution of at their NUI
Human- Techniques:
1) https://miro.medium.com/v2/
recommendations
Computer Interaction or responses.
(HCI) and proposes a future Continued resize:fit:1100/format:webp/
research and development are
where AI is integrated into operating systems. 1*sG8BGWXfpng_Gr1Lolr8mA.png
Developing explainable AI (XAI) methods that needed to advance NUI techniques, including
This future
can HCI would
elucidate the use natural user
reasoning interfaces
behind the multi-modal fusion for enhanced accuracy
(NUI) like actions
assistant's gestures
can and
help voice commands
build user trust andto and2) https://miro.medium.com/v2/
robustness.
provide a more
confidence intuitive and personalized user
in the system. resize:fit:1100/format:webp/
experience. The paper acknowledges challenges Integrating multiple modalities such as
1*Ek7FLdFieAmsYiL_8Y8z9Q.jpeg
6.3.Security and Privacy
like NUI accuracy and userConcerns:
privacy but concludes gesture recognition, voice commands, and
that the benefits
Integrating NUI andoutweigh the drawbacks.
AI technologies HCI eye tracking can improve overall interaction
raises
has constantly
concerns about aimed
securityfor risks
user-friendliness.
associated with This reliability and user experience.
3) https://ieeexplore.ieee.org/document/
review
data explores
collection andhow AI can revolutionize HCI. 7.2.Developing
processing. 9382581/ Explainable AI (XAI) Methods:
The future involves AI-powered operating
NUI data, including voice recordings or biometric Further work is required to develop XAI
systems that understand natural language and 4) https://ieeexplore.ieee.org/document/
data from eye tracking, must be handled methods that provide transparent insights
use gestures or voice commands. These systems 10431539/
securely to prevent unauthorized access or into the AI assistant's decision-making
would be more intuitive and personalized. The
misuse. process.
paper examines two research papers. One
explores challenges
Additionally, like data
AI-powered management
personalization and This
raises involves
5) Zheng developing
Xiangwei, algorithms Zhang
Liu Xiaofeng, and
security,concerns
privacy while theregarding
other envisions a network
the collection andof techniques
Yuang, Cui that Lizhen
can explain
and Yu complex
Xiaomei, AI"A
AI models
use of user handling
data user requests.
to tailor Large language
interactions and models portable HCI system-oriented
in a comprehensible manner to end-EEG
models (LLMs) are seen as the core for these users,
recommendations. feature extraction
increasing and acceptance
trust and channel selection
of AI
intelligent interfaces. They can understand and systems. for emotion recognition", International
Implementing robust security measures, such as Journal of Intelligent Systems, vol. 36,
generate natural language, allowing them to
encryption, access controls, and anonymization 7.3.Implementing no. 1, pp. 152-176,Robust2020.
Security Measures:
create a user experience tailored to individual
techniques, is essential to protect user privacy Show in Context Google Scholar
needs. The paper also explores Natural User Ongoing efforts are needed to implement and
within the NUI-AI framework.
Interfaces (NUI) like gesture recognition and enhance robust security
6) Hochheiser Harry andmeasures
Rupa toS protect
Valdez,
voice commands. These NUIs can improve user"Human-Computer data privacy within the NUI-AI
Interaction Ethics
accessibility and make interactions more natural. framework. and Biomedical Informatics", Yearbook
While challenges like NUI accuracy and user
This of Medical Informatics, vol. 29, no. 1, pp.
includes adopting encryption techniques,
privacy exist, the paper argues that the benefits 93-98, 2020.
implementing secure data handling practices,
of a more natural and personalized user Show in Context Google Scholar
and adhering to privacy regulations to
experience outweigh them. The future of HCI
safeguard
7) Tomažuser information.
ýegovnik, Kristina Stojmenova,
seems to be one where AI plays a major role.
In Igor
summary,Tartalja, Igor Tartalja
addressing the and Jaka
identified
Sodnik, "Evaluation of different interface
challenges and pursuing future research and
designs for human-machine interaction
development efforts are essential to realizing
in vehicles", Multimedia Tools and
the Applications,
full potential of NUI-AI
pp. 1-28, 2020. integrated
operating Show systems. By overcoming
in Context these
CrossRef Google
challengesScholarand fostering continued
innovation, we can create a more natural,
Human Computer Interaction with integrated AI
intuitive, and secure computing experience
[2]
for users.
8) Seojeong Lee and Margareta Lützhöft,
"Human-Machine Interaction", SEA
TECHNOLOGY, vol. 61, no. 5, pp. 18-20,
2020.Show in Context Google Scholar

9) Haldar Shefali, Sonali R Mishra, Kim

Yoojung, Hartzler Andrea, Ari H Pollack and
Pratt Wanda, "Use and impact of an online
community for hospital patients", Journal of
the American Medical Informatics
Association, vol. 27, no. 4, pp. 549-557,
2020.Show in Context Google Scholar

10) A. Vaswani, N. Shazeer, N. Parmar, J.

Uszkoreit, L. Jones, A. N. Gomez, et al.,
"Attention is all you need", Advances in
neural information processing systems, vol.
30, 2017. Show in
Context CrossRef Google Scholar

11) J. Devlin, M.-W. Chang, K. Lee and K.

Toutanova, Bert: Pre-training of deep
bidirectional transformers for language
understanding, 2018.Show in
Context Google Scholar

12) K. Clark, M.-T. Luong, Q. V. Le and C. D.

Manning, Electra: Pre-training text encoders
as discriminators rather than generators,
2020.Show in Context Google Scholar

13) C. Campagnano, S. Conia and R.

Navigli, "SRL4E - Semantic Role Labeling
for Emotions: A unified evaluation
framework", Proceedings of the 60th Annual
Meeting of the Association for
Computational Linguistics (Volume 1: Long
Papers), pp. 4586-4601, 2022. Show in
Context CrossRef Google Scholar