Jatin PDF
Jatin PDF
VIRTUAL MOUSE)
CLASS XII
Project Report
Submitted by:
Jatin Kaushal
Board Roll No -___________________
Guided By
Mr. Vikas Tiwari
CERTIFICATE
This is to certify that Jatin Kaushal of Class XII , student of
Allenhouse Public School , Rooma , Kanpur has successfully
completed their Artificial Intelligence Capstone Project on the topic
“ AI Virtual Mouse” under the guidance of “Mr. Vikas Tiwari”
during the year 2024-25.
I am satisfied with their initiative and efforts for the completion of
project file as a part of curriculum of CBSE Class XII Examination.
SIGNATURE SIGNATURE
(PRINCIPAL) (SUBJECT TEACHER)
SIGNATURE
(EXAMINER)
ACKNOWLEDGEMENT
Date: __________
AI PROJECT LOGBOOK
Phase Task Planned Planne Plan Actu Actual Actual Who is Remarks
start date d end ned al end duratio responsible
date dura start date n
tion date (hours,
(hou minutes
rs, )
minu
tes)
Preparing 15-07-24 21-07-24 20- 15-07-24 21-07-24 20-30 hrs All Team
for the 30 Members Collabora
project Coursework
readings hrs tive work
Rate 3/3
yourselves
UnderstandinIdentify users 29-07-24 31-07-24 1 hr 29-07-24 30-07-24 1 hr All Team Collabora
g the users Members tive work
6
Interview 06-08-24 06-08-24 1 hr 06-08-24 06-08-24 1 hr All Team
with user
Members
(2)
Interview 07-08-24 07-08-24 1hr 07-08-24 07-08-24 1hr All Team
with Members
user(3)
Interview 08-08-24 08-08-24 1hr 08-08-24 08-08-24 1hr All Team
with Members
user(4)
Complete 08-08-24 09-08-24 2hr 08-08-24 09-08-24 2 hr All Team Collabora
section 4 Members tive work
of the
Project
Logbook
Rate 3/3
yourselves
Team 09-08-24 10-08-24 2-3 09-08-24 10-08-24 3 hr All Team Collabora
Brainstorming meeting hr Members tive work
to
generate
Testing Invite 02-11-24 02-11-24 2hr 02-11-24 02-11-24 2hr All Team
Creating users to Members
the video test your
prototype
Conduct 03-11-24 5hr 03-11-24 04-11-24 5hr All Team
04-11-24 Members
testing
with users
Complete 05-11-24 05-11-24 4hr 05-11-24 05-11-24 4hr All Team
section 9 Members
of the
Project
Logbook
Rate 3/3
yourselves
Team 08-11-24 08-11-24 2hr 08-11-24 08-11-24 3hr All Team
meeting to Members
discuss
video
creation
8
Write 09-11-24 09-11-24 2hr 09-11-24 09-11-24 2hr All Team
your Members
script
Film 09-11-24 09-11-24 4hr 09-11-24 09-11-24 3hr All Team Collabora
your Members tive work
video
2.3Communications plan
9
Who wasn’t able to attend: None
Purpose of meeting: Research on topic, discussion of ideas and division of roles
Items discussed:
1.Define the purpose and objective of the AI virtual mouse.
2.Determine specific use cases for individual with disabilities.
Things to do (what, by whom, by when):
1.Assign roles, oversee the entire project, set the timelines and track progress.(Sanya)
2.Investigate and evaluate eye tracking technologies at AI algorithms .(Sukhdeep)
2.4.1 Team meeting (Meeting 2)
Date of meeting:22 July 24
Who attended: Sanya,Nanshi,Sukhdeep,Jatin,Arnav and Manali
Who wasn’t able to attend: None
Purpose of meeting: To discuss solutions for the problem
Items discussed: An Computer Vision powered image model responsible
for eye tracking.
Things to do: Learn and research on best AI powered framework that
would help us make our model
2.4.2 Team Meeting(Meeting 3)
Date of meeting: 12 August 24
Who attended: Sanya,Nanshi,Sukhdeep,Jatin,Arnav and Manali
Who wasn’t able to attend: None
Purpose of meeting: To design the solution
Items discussed: Organizing and listing all the requirements required for
construction and designing of model and decided to opt for image based
recognition approach.
Things to do: Start with data preparation and collection
11
3. PROBLEM DEFINITION
3.1 List important local issues faced by your school or community ?
➢ The important local issue faced by our society is for disabled people to use
technology and the people who are unable to use their hands.
12
4.THE USERS
4.1 Who are the users and how they affected by the problem?
➢ The users are individuals with mobility impairments who cannot use
traditional input devices like a mouse or keyboard. They are affected by
limited access to technology, making it difficult to communicate, work, or
perform daily tasks independently.
4.2 What have you actually observed about the users and how the problem
affects them?
➢ We have observed that users with mobility impairments face challenges in
interacting with technology due to their inability to use traditional input
devices. This limits their independence, reduces opportunities for education
and employment, and hinders their ability to fully utilize digital tools.
4.3 Record your interview questions here as well as responses from users.
Interviewer: Can you explain how eye tracking works in your AI Virtual Mouse
Project?
➢ User: It uses computer vision (OpenCV, MediaPipe) to detect and track eye
Movements, translating gaze positions into screen coordinates to
control the Mouse pointer.
Interviewer: What libraries or tools did you use to implement this AI Virtual
Mouse?
➢ User: I used OpenCV for image processing, MediaPipe for facial landmarks,
Pynput for stimulating mouse actions, and NumPy for mathematical
calculations.
Interviewer: What challenges did you face in implementing gaze tracking and
how did you overcome it?
➢ User: Lightning conditions and head movements were challenging. I improved
accuracy by refining facial landmark detection and using techniques to stabilize
gaze-to screen mapping.
Interviewer: How does it handle click events?
➢ User: Eye blinks are used for click detection.
13
4.4 Empathy Map
SAYS THINKS
It’s amazing that I can control my Will it be tiring for my eyes
computer just with my eyes. to use this for long? How
It’s a bit challenging to control can I improve my accuracy
precise actions like dragging and and speed with practice?
dropping.
I wish it could be more accurate in How would it handle rapid
low light conditions. movements or sudden
changes in gaze?
DOES FEELS
Does the system to move the cursor Feels like they have more
using eye gaze. control especially with limited
Calibrates the system regularly for mobility.
improved accuracy. Interested in the technology but
May take breaks to avoid eye strain concerned about its
or fatigue. effectiveness in real life use.
Eager to try an innovative way
of interacting with technology.
4.5 What are the usual steps that users currently take related to the
problem and where are the difficulties?
Voice Control: Often imprecise and struggles in noisy environments.
External Switches: Expensive, require setup, and can be physically
challenging.
Specialized Hardware: High cost and limited availability make it inaccessible
for many users.
General Difficulties: Limited accuracy, affordability, and accessibility.
14
5.BRAINSTORMING
5.1 Ideas
AI Idea #1 Use AI to accurately track eye movements and translate them into precise
cursor control, enabling hands-free interaction.
AI-driven speech recognition to assist users in controlling devices through
AI Idea #2
voice commands.
Machine learning to interpret specific head or facial gestures as input for device
AI Idea #3
interaction.
AI to adapt the gaze tracking system to individual user needs, enhancing
AI Idea #4 accuracy and comfort.
AI-powered guidance and tutorials for users to learn how to use assistive
AI Idea #5 technologies effectively.
High value to users, easy to create High value to users, hard to create
➢ Speech-to-Text ➢ Gaze Tracking
Low value to users, easy to create Low value to users, hard to create
➢ Accessibility Tutorials ➢ Gesture Recognition
Easy Hard
EASE OF DEVELOPMENT
5.6 Based on the priority grid, which AI solution is the best fit for your
users and for your team to create and implement?
➢ Our approach is to create a gaze tracking model that we have trained using custom
computer vision techniques to track eye movements. By leveraging MediaPipe and
Python, we built the entire system from scratch, enabling hands-free interaction
with technology. This approach allows individuals with mobility impairments to
control their devices more efficiently, improving accessibility. We developed and
trained the model to accurately track and translate gaze into cursor movement,
ensuring a seamless user experience.
15
6.DESIGN
6.1 What are the steps that users will now do using your AI solution to address the
problem?
➢ Calibration: Users will first calibrate the system by focusing on specific
points on the screen to align the gaze tracker with their eye movements.
➢ Eye Tracking: Once calibrated, the system will continuously track the user’s
eye movements in real-time.
➢ Cursor Control: Users will move the cursor on the screen by looking at
different areas, with the system translating gaze direction into precise cursor
movement.
➢ Interaction: Users can interact with their device by gazing at buttons, icons, or
text, allowing for hands-free clicking and navigation.
16
7. DATA
17
8. PROTOTYPE
18
9. TESTING
9.2 List your observations of your users as they tested your solution.
➢ Upon testing, we found that the model performed as expected, with an
accuracy of 95%. The accuracy and F1 score for our model were 91% and
0.8, respectively, demonstrating strong performance in gaze tracking and
cursor control.
9.4 Refining the prototype: Based on user testing, what needs to be acted on
now so that the prototype can be used?
➢ User testing made us realize that background color is affecting model’s
decision so we have decided to place the material on white background
while using the model for image classification.
9.5 What improvements can be made later?
➢ Upon testing we found that the model was not fully predictive about some
plastic material which are similar to look at. Upon discussing it with the
team we have decided to update our training set with more diverse images
especially images taken from usage by real life people.
19
10.TEAM COLLABORATION
10.1 How did you actively work with others in your team and with stakeholders?
➢ We collaborated with our stakeholders by investigating the troubles they face
on a daily basis and understanding them to help us make a project that aims
to solve their troubles with full efficiency. Their inputs guided us throughout
our journey.
➢ We as co-members created a group chat to stay in touch and communicate
with each other efficiently at any time of the day comfortably. We made sure
to have open ears for each others’ suggestions and did not let any member's
efforts be unrecognized.
➢ We organized online meetings amongst the group members for
brainstorming ideas.
➢ Sessions with mentors and stakeholders to test the efficiency of the
prototype.
20
11.INDIVIDUAL LEARNING REFLECTIONS
21
12. VIDEO LINK
22
Appendix
Recommended Assessment Rubric (for Teachers)
LOGBOOK AND VIDEO CONTENT
Problem A local problem which has not A local problem which has not been A local problem is
definition been fully solved before is fully solved before is described. described
explained in detail with
supporting research.
The Users Understanding of the user group is Understanding of the user group The user group is described
evidenced by completion of all of is evidenced by completion of but it is unclear how they
the steps in Section 4 The Users most of the steps in Section 4 are affected by the problem.
and thorough investigation. The Users.
Brainstorming A brainstorming session was A brainstorming session was A brainstorming session was
conducted using creative and critical conducted using creative and conducted. A solution was
thinking. A compelling solution critical thinking. A solution was selected.
was selected with supporting selected with supporting
arguments from Section 5 arguments in Section 5
Brainstorming. Brainstorming.
The use of AI is a good fit for the The use of AI is a good fit for The use of AI is a good fit for
solution. The new user experience the solution and there is some the solution.
is clearly documented showing how documentation about how it
users will be better served than they meets the needs of users.
are today.
Data Relevant data to train the AI Relevant data to train the AI Relevant data to train the AI
model have been identified as model have been identified as model have been identified
well as how the data will be well as how the data will be as well as how the data will
sourced or collected. There is sourced or collected. There is be sourced or collected.
evidence that the dataset is evidence that the dataset is
balanced, and that safety and balanced.
privacy have been considered.
A prototype for the solution has
Prototype A prototype for the solution has A concept for a prototype
been created and successfully been created and trained. shows how the AI model
trained to meet users’ will work
requirements.
Testing A prototype has been tested with a A prototype has been tested A concept for a prototype
fair representation of users and all with users and improvements shows how it will be tested.
tasks in Section 9 Testing have have been identified to meet
been completed. user requirements.
Effective team collaboration and Team collaboration among peers
Team There is some evidence of team
collaboration communication among peers and and stakeholders is clearly interactions among peers and
stakeholders is clearly documented documented in Section 10 Team stakeholders.
in Section 10 Team collaboration. collaboration.
Individual Each team member presents a Each team presents an Some team members
learning reflective and insightful account of account of their learning present an account of their
their learning during the project. during the project. learning during the project.
23
VIDEO PRESENTATION
Points Given
3 – excellent
2 – very good
Criteria 1 – satisfactory
Accurate The video presents accurate science and technology and uses
language appropriate language.
Sound and
The video demonstrates good sound and image quality.
image quality
Total points
24