B.M.S.

COLLEGE OF ENGINEERING
(An Autonomous Institute, Affiliated to VTU, Belagavi)

DEPARTMENT OF MACHINE LEARNING

(UG Program: B.E. in Artificial Intelligence and Machine Learning)

Course: Capstone Project Phase - 1

Course Code:24AM7PWCP1
AI Assistant for Linux

Review - 2

Presented By,
Rahul Satish Vadhyar 1BM21AI099 Dr. Sandeep Varma N
Abdul Amaan 1BM21AI004 Associate Professor
Shreya Shastry 1BM21AI119 Department of Machine Learning
Shivanshi Singh B.M.S. College of Engineering
1BM21AI117

Semester & Section: 7A,7B

Date:13/01/2025
 Agenda
• Abstract
• Introduction
• Existing systems
• Requirement Analysis
• Problem statement
• Literature survey
• Objectives
• Scope
• Proposed Architecture
• Comparison between assistants
• Interaction with KDE community
• Conclusion
• References
 Abstract
This project highlights the urgent need for advanced, privacy-respecting AI tools in the Linux
ecosystem—an area where such solutions are lagging compared to other platforms. Today’s
Linux users deserve AI-powered features like screen analysis, file search, dictation, text-to-
speech, and secure terminal interaction, all while maintaining complete control over their
data.
Our vision is to create an AI application that processes everything locally, ensuring user data
stays private and secure. This approach aligns perfectly with Linux’s open-source principles,
giving users the flexibility, security, and transparency they expect.
In this paper, we explore the challenges, motivations, and opportunities of building such a
tool. By bridging the gap between privacy and functionality, we aim to empower Linux users
with cutting-edge AI capabilities while staying true to the values that make Linux unique.
 Introduction

● Linux is an open-source, Unix-like operating system kernel that serves as the foundation for various operating systems, offering
flexibility, security, and community-driven development.
● A desktop environment is a graphical interface layer that sits atop an operating system, providing tools, features, and visual
elements like windows, panels, and icons to interact with the system more easily.
● KDE is an open-source software community that develops KDE Plasma, a flexible and powerful graphical desktop environment for
Linux and other operating systems. KDE focuses on customization, user control, and a modern interface.
● LLM stands for Large Language Model, which is an advanced type of artificial intelligence model designed to understand and
generate human-like text. Examples include GPT, Claude and similar AI-based natural language processing systems.
● Our project aims to be a personal, private, and secure virtual assistant for KDE Plasma users.
● It is an attempt at replicating the virtual assistant features found on other platforms(i.e Siri on MacOS and Copilot on Windows).
Introduction
Existing systems
Windows Copilot
• OS: Windows 11
• Based on GPT-4o
• Features:
• Web integration
• image generation
• Limitations:
• No OS level integration
• Requires user to be online
Existing systems
Nyarch Assistant
• OS: Nyarch Linux
• Multiple models
• Features:
• Terminal integration
• Choice of models
• Limitations:
• Not available for general linux users
• Most models need internet
connection
Existing systems
Apple Intelligence(Siri)
• OS: MacOS
• Apple Foundational Model
• Features:
• OS integration
• Access to user files
• Limitations:
• Proprietary model
• Some features need internet
connectivity
 Requirement Analysis
To build a private and secure AI assistant for KDE Plasma, the following requirements are identified:
Functional Requirements

1. Chat LLM functionality.

2. Voice command(Audio speech recognition).
3. Integration with file search.
4. Text-to-speech and dictation features.
5. LLM interaction with the terminal.
6. Screenshot understanding capabilities.
Non-Functional Requirements
1. Performance
a. Low resource consumption to ensure smooth operation on low-end systems.
b. Fast response times for real-time interaction.
2. Usability
a. Intuitive interface design tailored for KDE Plasma.
b. Accessibility features for diverse user needs.
Hardware Requirements
● Minimum 8 GB RAM and quad-core processor.
● Nvidia GeForce 1650 TI or higher, optional but highly recommended

Software Requirements
● Linux distribution with KDE Plasma as the desktop environment.
● Dependencies: Ollama Server, QT6 and its various libraries, ONNX, Curl , Kirigami,
SDL3 (pre release currently)
 Problem Statement

Development of an AI assistant with LLM

integration for Linux systems running
KDE Plasma.
 Literature Survey

Publications by Year Publication Types

Total Publications Surveyed: 35

Virtual Assistants

Papers Speech Recognition NLP Techniques Context Awareness Personalization Privacy and
Security
Linux Based Virtual Basic Naive Bayes Low None Low
Assistance in C
(July 2021)
Python Based AI PyAudio for voice Standard NLP Medium High Medium
Assistance for input pipelines
Computer
(26 April 2024)
Speech Assistant:

Paper Scope Speech NLP Techniques Task automation Privacy and

Interpretation Security

Alexa, Siri,Cortana: Examines popular Standard SR Rule-based NLP Home control, basic Medium
An Introduction to voice assistants like automation
Siri, Alexa, Cortana,
voice assistance etc
(12 Jan 2021)

The Voice Enabled Desktop application Speech Recognition Flexible NLP, Time, date, web Dependency on third
Library supports natural queries, API support part APIs
Personal Assistant for voice assistants language
for Pc using Python using Python
(April 2021)
LLM:

Paper Scope Key Feature Evaluations Year

Intelligent Virtual Task decomposition and Calibration for Success rate on public 2023
Assistants with LLM- automation using LLMs hallucination, action datasets
based Process prediction, iterative
Automation refinement
From Bard to Gemini: Comparison of Google’s Standardized evaluation Performance, usability, 2024
Investigative chatbot evolution protocols, robust and cultural impact
Exploration of Google’s sampling, ethical analysis
Conversational AI monitoring
 Objectives
● Create an AI-driven assistant to facilitate seamless and intuitive interaction with
KDE Plasma.
● Enable the assistant to process inputs from diverse sources, including user prompts,
files, and screenshots, to provide contextually relevant assistance.
● Integrate functionality for executing terminal commands and predefined system
actions to enhance productivity and system control.
● Incorporate dictation and speech output capabilities to support accessibility and
ease of use.
 Scope
● Create a fully functional application that functions as an AI assistant
● Integrate and test various third party libraries for various features such as text-to-
speech, audio-speech-recognition and LLM server
● Create a UI using native toolkit for KDE Plasma(Qt and Kirigami).
● Develop integrations with the rest of the operating system.
● Does not include training a model. Pretrained models will be used.
● Assistant will be designed with the assumption that the user is running KDE Plasma
as desktop environment.
 Proposed Architecture
1. Central Orchestrator: The system's core manages
components like file search, screenshots, terminal
commands, and system actions.

2. Interactive Features: Includes modules for speech

recognition, text-to-speech, and a user interface for
seamless interaction.

3. Backend Support: Powered by the Ollama server, with

a service handler managing APIs and background tasks.
 Proposed Architecture
● Input Modes: Users choose text-based (Chat) or audio-based (ASR -
Automatic Speech Recognition).

● Orchestrator: Processes input, determines appropriate responses, and

directs the flow.

● Output Modes:
Text responses displayed via the interface.
Audio responses generated using TTS (Text-to-Speech) for
playback.
Adapts to user preferences while ensuring a clear and efficient
experience.
 Comparison between assistants
Feature
AI Assistant for KDE
(ours)
Copilot on Windows NyArch Assistant Apple Intelligence(Siri)

Local Processing

Customization

File Search/Indexing

Screen Analysis

Terminal Integration

System Actions

Web integration

Image generation

Privacy

Open Source
Interaction with KDE Community
Interaction with KDE Community
Interaction with KDE Community

<- Problem

<- Resolution
in 2 days
<- Solution
 Conclusion
• We have successfully completed the foundational stages of our project, including the
problem statement, motivation, and scope of work. This groundwork highlights the vision
of creating a secure, private, and locally running virtual assistant tailored for KDE Plasma
users.
• We identified the gap in the Linux ecosystem for a robust, privacy-focused virtual assistant
that combines advanced productivity tools with streamlined workflows.
• These completed milestones set the stage for developing a transformative tool that
prioritizes user needs, promotes KDE Plasma’s viability as an intuitive desktop
environment, and redefines what virtual assistants can achieve in the Linux ecosystem.
 References
• [1] "Apple Intelligence Foundation Language Models," *arXiv*, Jul. 2024. [Online]. Available: https://arxiv.org/pdf/2407.21075. [Accessed: Dec. 15, 2024].
• [2] D. Harley, A. Lee, and P.-M. Bureau, "Perception, Security and Worms in the Apple," *ESET*, Jan. 2010. [Online]. Available:
https://macviruscom.wordpress.com/wp-content/uploads/2010/01/eicar-apple-security. [Accessed: Dec. 15, 2024].
• [3] J. Stratton, "An Introduction to Microsoft Copilot," in *Proc. of the 979-8-8688-0447-2_2*, Charlestown, RI, USA, 2024. [Online]. Available:
https://link.springer.com/chapter/10.1007/979-8-8688-0447-2_2. [Accessed: Dec. 15, 2024].
• [4] "Microsoft Copilot in Digital Communication," *EMWA Journal*, [Online]. Available: https://journal.emwa.org/clinical-trial-transparency-and-disclosure/
digital-communication/article/16074/2-digital-communication.pdf. [Accessed: Dec. 15, 2024].
• [5] S. Bisser, "Introduction to the Microsoft AI conversational platform," in *[Book Title]*, Springer, 2024. [Online]. Available:
https://link.springer.com/chapter/10.1007/978-1-4842-6837-7_1. [Accessed: Dec. 15, 2024].
• [6] V. Këpuska and G. Bohouta, "Next Generation of Virtual Personal Assistance," *IEEE Access*, vol. 6, pp. 8200-8209, 2018. [Online]. Available:
https://ieeexplore.ieee.org/abstract/document/8301638/citations#citations. [Accessed: Dec. 15, 2024].
• [7] D. Arıkan, "Learning with Intelligent Personal Assistance," *Int. J. Hum. Sci.*, vol. X, no. X, pp. XX-XX, 2024. [Online]. Available: https://www.j-
humansciences.com/ojs/index.php/IJHS/article/view. [Accessed: Dec. 15, 2024].
• [8] K. Doppler, C. Wijting, T. Henttonen, and K. Valkealahti, "An Artificial Intelligence Based Virtual Assistance using Conversational Agent," *Int. J. Comput.
Sci. Softw. Eng.*, vol. X, no. X, pp. XX-XX, 2024. [Online]. Available: https://www.scirp.org/journal/paperinformation?paperid=1. [Accessed: Dec. 15, 2024].
• [9] Mayo Clinic Libraries, "Alexa, Siri, Cortana: An Introduction to Voice Assistance," *Int. J. Inf. Manage.*, vol. 38, no. 1, pp. 90-93, 2018. [Online]. Available:
https://www.tandfonline.com/doi/full/10.1080/02763869.2018.1404391. [Accessed: Dec. 15, 2024].
• [10] Riya and R. Sharma, "Linux Based Virtual Assistance in C," *Int. J. Adv. Res. Innov. Technol.*, vol. 6, no. 1, pp. 1-5, 2024. [Online]. Available:
https://www.ijariit.com/manuscripts/v6i1/V6I1-1170.pdf. [Accessed: Dec. 15, 2024].
Suggestions / Questions
Thank you !