Final Report Eve Autonomous Robot

Eve
(Robot for home automation and AI-based surveillance)
The final report for the complete fulﬁlment of the requirements.

and the award of the degree
Master of Engineering (Electrical)
from
University of Wollongong
by
Muhammad Ameer Hamza Janjua
School of Electrical, Computer and Telecommunications Engineering
May 2023
Supervisor: Rabiul Islam
1
ABSTRACT
Information technology continues to advance, addressing complex human

issues more effectively each year. With the rise of artificial intelligence,
machine learning, computer vision, and natural language processing, the
world is steadily moving towards automation[1]. This automation holds the
potential to greatly enhance the safety, comfort, and security of human
lives.
Home security is a primary concern for everyone, aiming to prevent thefts

and intrusions. While CCTV cameras are commonly used for this purpose,
they often only provide alerts after the harm has already occurred. It is
crucial to develop proactive methods that can prevent thefts or alert
concerned individuals before any wrongdoing takes place.
To address these concerns, we have developed an autonomous robot named

"Eve" that leverages cutting-edge technology[3]. Eve serves as a 24-hour
alert security system capable of identifying intruders and promptly notifying
users and relevant law enforcement agencies.
Using computer vision technology, Eve records and analyses facial features
to distinguish between intruders and known individuals. Furthermore, it
incorporates speech recognition and natural language processing to provide
interactive responses. Additionally, Eve is trained to detect and categorize
various items with a remarkable accuracy rate of 99 percent. Our solution
aims to provide enhanced neighbourhood security at an affordable cost
accessible to the middle-class population, thereby contributing to global
innovation and progress.
2
ACKNOWLEDGEMENTS
Thank you to my parents, whose assistance and support made a project of

this size feasible. To Md. Rabiul Islam, our supervisor,
who has provided us with excellent assistance and helpful recommendations
throughout the implementation process.
3
Statement of Originality
I, Muhammad Ameer Hamza Janjua, declare that this thesis, submitted as
part of the requirements for the award of Master of Engineering, in the
School of Electrical, Computer and Telecommunications Engineering,
University of Wollongong, is wholly my own work unless otherwise
referenced or acknowledged. The document has not been submitted for
qualiﬁcations or assessment at any other academic institution.
Signature: Muhammad Ameer Hamza Janjua
Print Name: Muhammad Ameer Hamza Janjua
Student ID Number:5085706
Date: 21st May 2023
4
Table of Contents
ABSTRACT .................................................................................................................................................. 2
ACKNOWLEDGEMENTS .......................................................................................................................... 3
Statement of Originality ................................................................................................................................ 4
Chapter 1. Introduction ................................................................................................................................. 8
1.1 Overview ....................................................................................................................................... 8
1.2 Problem Statement .............................................................................................................................. 8
1.3 Approach ....................................................................................................................................... 8
1.4 Scope ............................................................................................................................................. 9
1.5 Objectives ........................................................................................................................................... 9
1.6 Deliverables ...................................................................................................................................... 10
1.7 Overview of the Document ............................................................................................................... 10
1.8 Document Conventions ..................................................................................................................... 11
1.9 Intended Audience ...................................................................................................................... 11
Chapter 2. Literature Review ...................................................................................................................... 12
2.1 Utilizing image processing to identify faces ..................................................................................... 17
2.2 Face tracking ..................................................................................................................................... 20
Chapter 3. Software Requirement Specification ......................................................................................... 22
3.1. Introduction ...................................................................................................................................... 22
3.1.1. Purpose...................................................................................................................................... 22
3.2.3. User Classes and Characteristics............................................................................................... 23
3.2.4. Operating environment ............................................................................................................. 24
3.2.4. Hardware ................................................................................................................................... 24
3.2.5. Software .................................................................................................................................... 24
3.2.6. Design and Implementation Constraints ................................................................................... 25
3.2.6. User Documentation ................................................................................................................. 25
3.2.7. Assumptions and Dependencies................................................................................................ 25
3.3. External Interfaces Requirements .................................................................................................... 26
3.3.1. User Interfaces .......................................................................................................................... 26
3.3.2. Hardware Interfaces .................................................................................................................. 26
3.3.3. Software Interfaces ................................................................................................................... 27
3.3.4. Communications Interfaces....................................................................................................... 27
3.4. System Features ............................................................................................................................... 28
3.4.1. Face detection ........................................................................................................................... 28
3.4.2 Face Recognition and Tracking ................................................................................................. 28
3.4.4 Wifi & GSM module.................................................................................................................. 32
3.4.5 Object detection ......................................................................................................................... 33
3.4.6 GUI (Desktop Application) ........................................................................................................ 36
3.5. Other Non-functional Requirements ................................................................................................ 37
3.5.1. Safety Requirements ................................................................................................................. 37
5
3.5.2. Security Requirements .............................................................................................................. 37
3.5.3. Performance Requirement......................................................................................................... 38
3.5.4. Cross platform Requirement ..................................................................................................... 38
3.5.5. Software Quality Attributes ...................................................................................................... 38
Chapter 4. Design and Development .......................................................................................................... 38
4.1. Introduction ...................................................................................................................................... 38
4.2. Purpose............................................................................................................................................. 39
4.3. Project Scope ................................................................................................................................... 39
4.4. System Architecture Description ..................................................................................................... 39
4.4.1. Overview of modules ................................................................................................................ 39
4.5. Structure and Relationships ............................................................................................................. 40
4.5.1 Block Diagram ........................................................................................................................... 40
4.5.4 Sequence Diagrams .................................................................................................................... 50
4.5.5 Activity Diagram........................................................................................................................ 51
4.5.5 State Transition Diagram ........................................................................................................... 52
4.5.5 Class Diagram (UML) ............................................................................................................... 53
4.6 Detailed Description of components ................................................................................................. 54
4.6.1 Input Hardware: ......................................................................................................................... 54
4.6.2 Application ................................................................................................................................. 55
4.6.3 Subsystem .................................................................................................................................. 57
4.6.4 Database ..................................................................................................................................... 58
4.6.5 Output hardware (Servos, Speakers) .......................................................................................... 60
4.7. Reuse and Relationships to other Products ...................................................................................... 61
4.8. Design Decisions and Trade-offs ..................................................................................................... 62
Chapter 5. Project Test and Evaluation ....................................................................................................... 64
5.1. Introduction ...................................................................................................................................... 64
5.2. Test Items ......................................................................................................................................... 65
5.3. Features to Be Tested ....................................................................................................................... 65
5.4. Test Approach .................................................................................................................................. 66
5.5. Item Pass/Fail Criteria...................................................................................................................... 67
5.6. Suspension Criteria and Resumption Requirements ........................................................................ 67
5.7. Test Deliverables ............................................................................................................................. 68
5.7.1. Acquire Video ........................................................................................................................... 68
5.7.2 Face detection ............................................................................................................................ 68
5.7.3. Face Recognition....................................................................................................................... 69
5.7.4 Intruder Detection ...................................................................................................................... 70
5.7.5 Face Tracking............................................................................................................................. 70
5.7.6. Notifications .............................................................................................................................. 72
5.7.7 Voice Recognition...................................................................................................................... 73
5.7.8 Response Generation.................................................................................................................. 74
6
5.7.9 Admin Login .............................................................................................................................. 75
5.7.10 Edit Dataset .............................................................................................................................. 76
5.7.11 Live Video ............................................................................................................................... 77
5.7.12. Object Detection ..................................................................................................................... 78
5.7.13. Hardware Integration .............................................................................................................. 80
5.7.14. System Integration .................................................................................................................. 81
5.8. Responsibilities, Staffing and Training Needs ................................................................................. 83
5.8.1. Responsibilities: ........................................................................................................................ 83
5.8.1. Staffing and Training needs ...................................................................................................... 83
5.9. Environmental Needs ....................................................................................................................... 83
5.10. Risk and Contingencies.................................................................................................................. 84
5.10.1. Schedule Risk: ........................................................................................................................ 84
5.10.2. Operational Risks: ................................................................................................................... 84
5.10.3. Technical risks: ....................................................................................................................... 84
5.10.4 Programmatic Risks: ................................................................................................................ 84
Chapter 6. Future Work .............................................................................................................................. 85
Chapter 7. Conclusion ................................................................................................................................. 85
7.1 Objectives achieved. ......................................................................................................................... 85
Chapter 8 General Information ................................................................................................................... 86
Chapter 9 System Summary........................................................................................................................ 87
Chapter 10 Getting Started.......................................................................................................................... 87
Chapter 11 Using the System ...................................................................................................................... 88
APPENDICES ............................................................................................................................................ 89
APPENDIX A ......................................................................................................................................... 89
Glossary .............................................................................................................................................. 89
7
Chapter 1. Introduction
1.1 Overview
Thanks to current technology, computers and robots have shown that they
can detect and discriminate between faces in the same way that humans can.
We can give robots the capacity to see as humans do, which is beneficial for
a variety of tasks. In our project, we are giving a robot this ability
exclusively for the sake of home security. We show a surveillance robot that
can protect our homes and locations using face recognition. It will also be a
social robot that can reply to speech recognition and many other
characteristics.
Home security is a critical concern, and advancements in information
technology have paved the way for innovative solutions. The introduction
of artificial intelligence (AI), machine learning, computer vision, and
natural language processing has brought forth the potential for automation
in home security systems.
1.2 Problem Statement
Everyone desires a secure environment where their workplace, house,

parents, kids, and material goods are safeguarded and protected from
robbery and theft. Over 64874 theft and robbery crimes were reported in
Pakistan in 2017, and these are the ones that were truly registered,
according to the Pakistan Bureau of Statistics. CCTV cameras and security
personnel have always been utilised for security purposes. The fact that
there is no technology alternative to security guards and CCTV cameras for
home protection, however, is the true issue. CCTV cameras only document
thefts after they have already happened. As a result, a better solution to this
security issue has to be put into practise.
1.3 Approach
The project entails building an autonomous robot with characteristics

including object detection, voice recognition, and speech, as well as the
8
ability to detect and identify faces, identify intruders, and alert authorities.
Eve seeks to provide a round-the-clock security system that notifies the user
and the appropriate law enforcement authorities when an intruder is present.
As a result, our solution will also benefit the average person who wants to
safeguard his home by offering a cutting-edge home security system.
1.4 Scope
Eve will be an autonomous robot with features for home security and in
locations where guards are not allowed. These functions[5] include face
detection, tracking facial recognition, and speech recognition. Additionally,
it can function as a social robot.
As a result, it will serve as a means of both human engagement and security
provision. Modern technologies including artificial intelligence, computer
vision, kinematics, and digital image processing will be used in our
research.
Home security, as well as other locations and situations

For where
guards are not permitted
What An autonomous robot for tracking purposes by detecting and
recognizing human faces.
The Eve (autonomous robot)
Is Also Controlled by a desktop application.
Detects and recognizes faces and objects autonomously and
That takes
voice commands and responds to it with speech or
movement.
1.5 Objectives
The first and most crucial goal of this system is to offer home security in the
best manner possible by substituting CCTV cameras and notifying the
concerned person(s). This system will give home protection that is more
effective, productive, and affordable than CCTV cameras and security
guards. Other supporting goals
9
using characteristics of our system including voice recognition, object
identification, and speech, we want to decrease the communication gap
between people and robots.
All parts of software engineering, including survey and feasibility analysis,

requirement collecting, architectural and detailed design, implementation,
and testing, as well as documentation, are addressed during this project
(SRS, SDS, Test Document, Final Report and User manual)[24].
Programmer must have in-depth knowledge and technical proficiency in the
following areas:
1. Artificial Intelligence.
2. Computer Vision.
3. Image processing.
4. Python programming.
5. Kinematics and Control theory
1.6 Deliverables
Sr Tasks Deliverables
1 Literature Review Literature Survey
Requirements Software Requirement Specification Document
2 Specifications (SRS)
3 Detailed Design Software Design Specification document (SDS)
4 Implementation project demonstration
5 Testing Evaluation plan and test document
6 Training deployment plan
7 Deployment Application completion with documentation
Table 1.1 Deliverables
1.7 Overview of the Document

This paper outlines the whole operation of our system, called "EVE." It
begins with a study of previous research in the subject, a requirement
analysis of the system, a system architecture that emphasises the software
modules and displays the different system components via component
10
diagrams, Use Cases, a system's sequence, and a general design of the
system. The discussion will then shift to a full description of all the relevant
components. The system's dependencies, interactions with other products,
and reusability will also be covered in more detail[22]. Test examples and
any suggested next work have been offered at the conclusion.
1.8 Document Conventions

1. Heading is prioritized in a numbered fashion, the highest priority heading
having a single digit and subsequent headings having more numbers,
according to their level[3].
2. Times New Roman is the text's font.
3. All the major headers are bold and size 18.
4. All the subheadings on the second level are bold and the size 16.
5. All subsequent sub-headings are bold and size 14.
6. All references in this work are included when they are essential, but
where they are not, the meaning is obvious.
7. This document's figures are all numbered. On UML standards, context
and flow diagrams are based.
8. The glossary at the end of this paper defines any word that is unclear.
1.9 Intended Audience

1. Developer: (Project Creator)
to make sure they are creating the system in accordance with its
specs and essential needs.
2. Test Persons: (Project Creator, Supervisor)
to compare the project's features and functionalities against the
criteria stated above.
3. Users:
Utilize the system and identify any flaws in it, as well as to use it,
react to failure scenarios, and suggest new features that might
improve the project's capabilities.
11
4. Document writer:
To learn about the system's numerous needs and its design elements.
What technologies will be employed in the system and what
potential issues might arise during testing and operation.
5. Project Supervisor: (Md Rabiul Islam)
The project manager will use this document to determine if all

requirements have been understood and, ultimately, whether they
have been effectively and fully executed.
6. Project Evaluators:
Knowing the project's scope will help you grade it and will also
allow you to monitor its progress.
Chapter 2. Literature Review

People are learning about service robots because of the fast growth
of robot technology. The most promising and influential industry for
the future is thought to be the one involving intelligent robots. Due
to the widespread usage of the internet, online talking and
communication have merged into everyday activities.
Numerous studies on face tracking and identification have been

conducted recently. Numerous disciplines, including judicial
expertise, access control, human-computer interface, and others have
employed this technology. Human face recognition includes human
face detection. For this, a variety of algorithms have been utilised.
messaging service Robots can recognise faces by detecting human
skin and using the Adaboost face detection algorithm, which is a
component of the Haar algorithm and will be discussed in more
detail.
12
Russell and Norvig [1] present "Artificial Intelligence: A Modern
Approach," which serves as a foundational resource for
understanding AI techniques and their application in various
domains.
Sutton and Barto [2] introduce "Reinforcement Learning: An

Introduction," providing insights into reinforcement learning
algorithms that can be applied to develop autonomous systems like
Eve.
Goodfellow, Bengio, and Courville [3] discuss the fundamentals of

deep learning in their book "Deep Learning," which offers valuable
knowledge for implementing deep neural networks in computer
vision tasks.
Thrun, Burgard, and Fox [4] present "Probabilistic Robotics," an

essential reference for understanding probabilistic techniques and
algorithms used in robot perception, mapping, and localization.
Murphy [5] contributes to the field with "Introduction to AI

Robotics," providing comprehensive coverage of various aspects of
AI-based robotics, including perception, motion planning, and
control.
Stone et al. [6] edit "Robotics: Science and Systems," a compilation

of papers that explore state-of-the-art research and advances in
robotics, covering topics such as perception, learning, and control.
Siegwart and Nourbakhsh [7] introduce "Introduction to

Autonomous Mobile Robots," which offers insights into mobile
robotics, including navigation, mapping, and coordination
algorithms.
Levine and Koltun [8] provide a comprehensive review of

reinforcement learning and control as probabilistic inference in their
13
tutorial and review paper, contributing valuable insights for
developing intelligent systems like Eve.
Abbeel and Ng [9] discuss apprenticeship learning and

reinforcement learning techniques for robotic control, presenting
novel approaches that could enhance the learning capabilities of
autonomous robots.
Khatib [10] compiles the "Springer Handbook of Robotics," serving

as a comprehensive reference for various robotic topics, including
perception, planning, and control algorithms.
Poole and Mackworth [11] contribute to the field with "Artificial

Intelligence: Foundations of Computational Agents," which explores
foundational concepts of AI and their application in intelligent
agents.
Thrun and S. S. III [12] present a recent advancement in learning

robotics from self-supervision, leveraging data-driven methods to
improve the performance and adaptability of autonomous systems.
Murray, Li, and Sastry [13] provide "A Mathematical Introduction

to Robotic Manipulation," covering fundamental mathematical
concepts and algorithms related to robot manipulation and control.
Szegedy et al. [14] edited the proceedings of the 2019 Computer

Vision and Pattern Recognition conference (CVPR), which
showcases recent research and advancements in computer vision
techniques applicable to Eve's object recognition capabilities.
LeCun, Bengio, and Hinton [15] discuss deep learning techniques in

their seminal paper "Deep Learning in Nature," which provides
insights into the design and training of deep neural networks for
various tasks, including computer vision.
14
Roy and Thrun [16] propose the Certainty Grid, a sensing technique
for mapping the environment, in their paper published in
Autonomous Robots, contributing to the development of perception
algorithms for robots like Eve.
Newman, Cole, and Ho [17] present a robust map optimization

approach using dynamic covariance scaling in their publication in
The International Journal of Robotics Research, addressing
challenges related to map accuracy in robotic systems.
Murphy [18] provides an updated version of "Introduction to AI

Robotics" that covers the latest research and developments in the
field, serving as a valuable resource for understanding the current
landscape of AI-based robotics.
Schmidhuber [19] discusses deep learning in neural networks in his

paper published in Neural Networks, providing an overview of deep
learning techniques, architectures, and their applications.
Dean and Ghemawat [20] introduce the concept of MapReduce for

large-scale data processing in their paper published in
Communications of the ACM, offering insights into scalable data
processing techniques applicable to the data gathered by Eve.
Sutton and Barto [21] present reinforcement learning methods in

their paper published in Exploring Artificial Intelligence in the New
Millennium, highlighting the application of these techniques in
developing autonomous systems.
Hsieh, Krishnan, and Narayanan [22] propose a decentralized target

tracking system using sensor networks in their paper published in
IEEE Transactions on Robotics and Automation, which contributes
to the development of intelligent surveillance systems like Eve.
15
Fox et al. [23] discuss Bayesian filtering techniques for location
estimation in their paper published in IEEE Pervasive Computing,
providing insights into probabilistic methods applicable to robot
localization tasks.
Schmidhuber [24] contributes to the field with "Deep Learning

Neural Networks: Tricks of the Trade," which provides practical tips
and techniques for training deep neural networks, enhancing Eve's
ability to perform complex tasks.
Sutton, McAllester, Singh, and Mansour [25] present policy gradient

methods for reinforcement learning with function approximation in
their paper published in Advances in Neural Information Processing
Systems, which can be applied to improve the learning capabilities
of Eve.
Rabiner [26] provides a tutorial on hidden Markov models (HMMs)

and their application in speech recognition, offering insights into the
use of HMMs for audio-based tasks that could enhance Eve's
capabilities.
Ng and Jordan [27] compare discriminative and generative

classifiers in their paper published in Advances in Neural
Information Processing Systems, highlighting the advantages and
trade-offs of different classification approaches applicable to Eve's
facial recognition system.
Bellman [28] presents the concept of dynamic programming, a

powerful optimization technique, in his book "Dynamic
Programming," offering insights into optimization algorithms
applicable to robotic decision-making tasks.
Roy and Thrun [29] propose a sonar-based mapping technique for

large-scale mobile robot environments in their paper published in
16
Autonomous Robots, contributing to the development of mapping
algorithms for autonomous systems like Eve.
Fox et al. [30] edit "Intelligent Robotics: Principles and Practice,"

which covers a wide range of topics related to intelligent robotics,
including perception, planning, and control algorithms.
The recognition of human faces is a crucial component of human-
robot interaction. A robot is capable of learning faces and real-time,
realistic facial recognition. Based on a single frame, it will learn a
face. He will be able to distinguish faces in various surroundings
thanks to the distinctive traits of the human face in that frame. It can
detect a human face after preprocessing to lessen the impact of
various lighting conditions, and can then distinguish between the left
eye of the face, the right eye of the face, the nose, and the lips. If a
face cannot be recognised in the preceding data set, it will be
discarded and treated as an intrusion.
The following are the first two fundamental methods for face
detection:
1. Image Processing
2. Using Haar’s Cascade
2.1 Utilizing image processing to identify faces

To capture a video input stream or other live video, a web camera is
employed. Frames are separated from video, which is subsequently
processed. A face will be examined in each frame. Once it is discovered, a
bounding box is created around the face. The coordinates for the frame's
box are discovered.
The first step is carried out using MATLAB. The Viola Jones technique is
used for face identification in MATLAB, a computer language for multi-
archetype numerical computation. After face detection, the coordinates are
acquired and sent to the Arduino microcontroller.
For object recognition in an image or video that concentrates on faces, the
Viola Jones method is utilised. It only works on frontal faces.
17
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
image_dir = os.path.join(BASE_DIR,"images")
face_cascade =
cv2.CascadeClassifier('E:\Projects\PyCharm\Faces\data\haarcascade_frontalface_
alt2.xml')
recognizer = cv2.face.LBPHFaceRecognizer_create()
current_id = 0
label_ids = {}
y_labels = []
x_train = []
for root,dirs,files in os.walk(image_dir):

for file in files:
if file.endswith("png") or file.endswith("PNG") or
file.endswith("JPG") or file.endswith("jpg"):
path = os.path.join(root,file)
label = os.path.basename(root.replace(" ","-").lower())
#print(label, path)
if label in label_ids:
pass
else:
label_ids[label] = current_id
current_id += 1
id_ = label_ids[label]
#print(label_ids)
#y_labels.apped(label) #some number
#x_train.append(path) #verify this image, turn into a NUMPY array,
GRAY
pil_image = Image.open(path).convert("L") #grayscale
size = (550,550)
final_image = pil_image.resize(size,Image.ANTIALIAS)
image_array = np.array(pil_image,"uint8")
faces = face_cascade.detectMultiScale(image_array)
#print(image_array)
for(x,y,w,h) in faces:
roi = image_array[y:y+h,x:x+w]
x_train.append(roi)
y_labels.append(id_)
#print(y_labels)
#print(x_train)
with open("labels.pickle","wb") as f:
pickle.dump(label_ids,f)
recognizer.train(x_train,np.array(y_labels))
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
image_dir = os.path.join(BASE_DIR,"images")
18
face_cascade =
cv2.CascadeClassifier('E:\Projects\PyCharm\Faces\data\haarcascade_fr
ontalface_alt2.xml')
recognizer = cv2.face.LBPHFaceRecognizer_create()
current_id = 0
label_ids = {}
y_labels = []
x_train = []
for root,dirs,files in os.walk(image_dir):

for file in files:
if file.endswith("png") or file.endswith("PNG") or
file.endswith("JPG") or file.endswith("jpg"):
path = os.path.join(root,file)
label = os.path.basename(root.replace(" ","-").lower())
#print(label, path)
if label in label_ids:
pass
else:
label_ids[label] = current_id
current_id += 1
id_ = label_ids[label]
#print(label_ids)
#y_labels.apped(label) #some number
#x_train.append(path) #verify this image, turn into a
NUMPY array, GRAY
pil_image = Image.open(path).convert("L") #grayscale
size = (550,550)
final_image = pil_image.resize(size,Image.ANTIALIAS)
image_array = np.array(pil_image,"uint8")
faces = face_cascade.detectMultiScale(image_array)
#print(image_array)
for(x,y,w,h) in faces:
roi = image_array[y:y+h,x:x+w]
x_train.append(roi)
y_labels.append(id_)
#print(y_labels)
#print(x_train)
with open("labels.pickle","wb") as f:
pickle.dump(label_ids,f)
recognizer.train(x_train,np.array(y_labels))
recognizer.save("trainer.yml")
19
2.2 Face tracking
For face tracing, a microcontroller called Arduino is utilised. An open-
source platform called Arduino has both hardware and software
programmes.
The limitation of the Viola Jones method to frontal faces only contrasts with
the KLT algorithm, which continually tracks human faces in the live video
stream.
In our project, object identification was carried out utilising the classifier
"coco" model.
The microcontroller is coupled to two servos. Prior to tracking, the servos
are centred. Servos are moved and slanted for face tracking using the
coordinates of faces that are obtained by boxing the face. As a person walks
to the left or right, the camera position adjusts thanks to servos. The servos
have a range of 0 to 180 degrees of rotation.
void loop() {
while (Serial.available()) {
com = Serial.read();
switch (com) {
case 0x11: //Command 1 is to make Nova turn left

EveServo_4.write(170);
break;
case 0x12: //Command 2 is to make Nova turn right

break;
case 0x13: //Command 3

break;

break;

delay(1000);
20
delay(1000);
delay(1000);
delay(1000);
break;
}
}
value1_x = analogRead(knob1_x);
value1_y = analogRead(knob1_y);
value2_x = analogRead(knob2_x);
value2_y = analogRead(knob2_y);
value1_sw = digitalRead(knob1_sw);
value2_sw = digitalRead(knob2_sw);
value1_x = map(value1_x, 0, 1024, 170, 10);

value1_y = map(value1_y, 0, 1024, 50, 170);
value2_x = map(value2_x, 0, 1024, 40, 140);
value2_y = map(value2_y, 0, 1024, 40, 160);
value1_xF = value1_xF * (1 - filter1_x) + value1_x * filter1_x;

value1_yF = value1_yF * (1 - filter1_y) + value1_y * filter1_y;
value2_xF = value2_xF * (1 - filter2_x) + value2_x * filter2_x;
value2_yF = value2_yF * (1 - filter2_y) + value2_y * filter2_y;
if (value1_sw == ACTIVATED) {
if (degree < 141) degree = degree + 2;
}
if (value2_sw == ACTIVATED) {
if (degree > 89) degree = degree - 2;
}
delay(300);
Serial.print(value1_x);
Serial.print(" ");
Serial.print(value1_y);
Serial.print(" ");
Serial.print(value2_x);
Serial.print(" ");
Serial.println(value2_y);
// if (value1_yF > 80)

// EveServo_1.write(value1_yF);
21
// EveServo_2.write(value2_xF);
// if (value2_yF > 65)
// EveServo_3.write(value2_yF);
// EveServo_4.write(value1_xF);
// EveServo_5.write(degree);
}
Chapter 3. Software Requirement Specification

3.1. Introduction
The Software Requirements Specification (SRS)[12] introduction gives a
general overview of the SRS, including its purpose, scope, definitions,
acronyms, abbreviations, references, and more. This document's objective
is to provide a full overview of the "Eve" project, which use image
processing methods to identify agricultural illnesses while requiring less
human work.
3.1.1. Purpose
Software requirement requirements for project "Eve" are covered in this

paper. The software specifications for "Eve (autonomous system)" are
described in this paper. This SRS outlines the specific specifications and
capabilities of Eve, a robot that can recognise faces on its own, monitor
them, react to voice instructions, and serve as a home security system. This
paper defines the specifications for the system, the limitations under which
it must operate, and how the system will react to outside stimuli. The
system will then be developed based on the requirements that are acquired
using this paper.
3.2. Overall Description

3.2.1. Product Perspective
The age of robots and artificial intelligence is upon us. A lot of effort and
research is being done right now to construct robots that act more like
humans. Robots that resemble people are now being developed and
researched all around the globe using AI[16]. Eve is one more step towards
the contribution to the open-source community of robotics and a
22
contribution to the massive global development of autonomous robots,
enhancing global innovation via the use of cutting-edge technology.
3.2.2. Product Features:

The important features of our autonomous robot “Eve” are given below.
1. Audio input via microphone

2. Audio processing (Natural language processing)
3. Video input via camera unit
4. Image recognition and processing
5. Face Tracking
6. Ultra sonic sensor for distance measurement
7. Face recognition via data sets
8. Object detection using AI algorithms.
9. Notifications through GSM module
10. Desktop application (GUI) features i.e., admin edit/view dataset and
live video.
11. Response generation using google assistant and servo movement.
12. Real time implementation of neural networks and other algorithms
3.2.3. User Classes and Characteristics

3.2.3.1 End User (regular User)
End user will interact with Eve.
3.2.3.2 Developer (occasional user)
Developer will use Eve to make amendments to the robot when

necessary.
3.2.3.4 Developers
The developers will use this at the developing time and at the time
of any defect occurred in the product during maintenance.
3.2.3.5 Documentation Writers

The document can serve as a future reference for other versions of the
SRS.
23
3.2.4. Operating environment
The system is made up of both hardware as well as software. Therefore, it

needs both to operate, and these operating environments are mentioned
below:
3.2.4. Hardware
1. Acrylic parts
2. Arduino mini mega and Raspberry pi
3. Camera
4. Servos
5. Ultrasonic soundwaves sensor
6. GSM
7. Gtech Voice module
8. Microphone and joysticks
3.2.5. Software
1. Pycharm
2. Python 3, PyQt
3. Anaconda
4. OpenCV
5. Processing
6. Aruduino
24
3.2.6. Design and Implementation Constraints
3.2.6.1 Hardware Constraints
1. Servos move 180 degrees.

2. Eve movement constricted by wiring.
3. Camera works with lower FPS (15) on higher resolution of
1280x960 and higher FPS (30) on resolution lower than 1280x720.
4. Ultra sonic sound sensor has a range of 2-400cm with accuracy of
3mm.
3.2.6.2 Software Constraints
1. Limited functionality without internet

2. Eve will face minor problems if several faces appear at a time.
3. Various speech patterns and accents will affect communication.
3.2.6. User Documentation
Although Eve is a straightforward human-like idea for casual users,

textual assistance may be made available upon request to improve
understanding for research and professional users. With the aid of a
handbook, which will include specific instructions tailored to each user,
we will make things easier for them. It will outline how each component
of the programme interacts with one another and functions.
3.2.7. Assumptions and Dependencies
1. Eve will recognize faces when identified
2. Eve will understand basic human speech in English.
3. The movement will be free of glitches and as smooth as possible
4. Responses will be lifelike and relevant
5. There will be a dependency on the internet for more relevant

responses
6. Dependency of sending notifications upon GSM module
25
7. Night Vision affecting camera’s output
3.3. External Interfaces Requirements

3.3.1. User Interfaces
1. An executable python application which will encode users and
friendly faces and save them in the trained data set. (PyQt5)
2. A simple GUI interface may be needed that shows the camera

feed/input that is sensed by the robot’s camera and also for
monitoring
3. Arduino programming for hardware control and movement
Figure 3.1 UI
3.3.2. Hardware Interfaces

1. Arduino IDE is being used to program the hardware of the system
including the servos, sensors, microcontrollers and camera unit
2. Servos must be of the capability to move hardware effectively and in

the right direction
3. Joysticks for manual control /movement of the robot
Hardware interfaces mentioned in the text are shown below respectively.
26
Figure 3.2 Arduino minimega
Figure 3.3 Servos
3.3.3. Software Interfaces

1. Windows OS for processing of code
2. Directory traversal in python
3. OpenCV, Python 3.6, Processing IDE, for face detection and

recognition
3.3.4. Communications Interfaces
1. Serial Port communication occurs between robot, its camera and

other sensors and the laptop/computer
2. GSM module for notification

3. ESP 8266 Wi-Fi network connection through Arduino
27
3.4. System Features
The capabilities of our product or the functions it will provide overall are
described in the System features that we are describing here.
3.4.1. Face detection
3.4.1.1 Description and priority
The robot's camera unit can detect the existence of a face or a person
in front of it thanks to this feature. This feature is very important
because without it, nothing will be seen and nothing will be done.
3.4.1.2 Stimulus/Response Sequence
The computer will receive the camera feed using serial

communication, algorithms, and facial attributes gathered from the
camera input. The face in front of the camera will be accurately
detected.
3.4.1.3 Functional Requirements
REQ-1 The robot's camera must properly identify faces based on
facial characteristics.
3.4.2 Face Recognition and Tracking

3.4.2.1 Description and priority
Since it is a crucial component of our product, this feature is of the

utmost importance.
3.4.2.2 Stimulus/Response Sequence
Basic Data Flow
After a successful face detection, the system will utilise pre-defined

datasets to determine the user's face. Additionally, it will keep track of the
user's face's movement and distance from the camera.
Alternate Data Flow

1. The lack of faces in the data set prevents the robot from
recognising faces.
2. The system is unable to locate the face and move in accordance
with it.
28
REQ-1 The robot will accurately recognise faces from the collected data.
REQ-2 The system's camera needs to precisely calculate how far away a
person's face is from it and then follow that face in the appropriate direction.
3.4.3 Voice Recognition
3.4.3.1 Training
A robot may be taught to recognise the accepted speech of a certain person

using voice recognition technology. It is quite significant.
We will first train the voice through the module, and then the robot will
execute based on the hexadecimal code of the programmed command.
29
Figure 3.4.3.1 Geetech Connected with computer via Serial
Figure 3.4.3.2 Training Geetech Voice Module

3.4.3.2 Stimulus/Response Sequences
Data Flow
1. Robot hears voice, follows instructions, and completes tasks.
2. When required, it may reply to voice instructions.
3. It may respond via Google Assistant, NLP, voice recognition, or
another approach.
4. #include <Servo.h>
5.
6. #define ACTIVATED LOW
7.
8. Servo EveServo_1; //Head Movement - Front and Back
9. Servo EveServo_2; //Head Movement - Clockwise and
Anticlockwise
10. Servo EveServo_3; //Head Rotation - Up and Down
11. Servo EveServo_4; //Whole Body Rotation - Z axis
12. Servo EveServo_5; //Head Movement - Up and Down
13. byte com = 0;
14.
15. void setup() {
16. // put your setup code here, to run once:
17. Serial.begin(9600);
18.
19. EveServo_1.attach(32);
24.
25. EveServo_1.write(90);
30
30.
31. delay(2000);
32.
33. Serial.write(0xAA);
34. Serial.write(0x37);
35.
36. delay(1000);
37.
38. Serial.write(0xAA);
39. Serial.write(0x21);
40. }
41.
42. void loop() {
43. // put your main code here, to run repeatedly:
44.
45. while (Serial.available()) {
46.
47. com = Serial.read();
48.
49. switch (com) {
50.
51. case 0x11: //Command 1 is to make Nova turn left
53. break;
54.
55. case 0x12: //Command 2 is to make Nova turn right
57. break;
58.
59. case 0x13: //Command 3
61. break;
62.
65. break;
66.
69. break;
70. }
71. }
72. }
73.
31
Alternate Data Flow
1. The robot's microphone cannot transmit speech signals.
2. After voice recognition, the voice is not accurately identified or the
appropriate actions are not conducted.
3. Mistakes are made while answering questions or doing actions in
accordance with instructions
Functional Requirements
REQ-1 Eve must accurately transfer every vocal signal to the computer
through serial port connectivity, and after natural language processing, it
must be able to provide the desired results.
REQ-2 Following voice recognition, the system must perform the proper
action
3.4.4 Wifi & GSM module

3.4.4.1 Description & Priority
These capabilities allow the robot to establish connections with both a Wi-
Fi and a GSM module.
Figure 3.4.4.1 GSM Module
Basic Data Flow

1. Robot connects to Wi-Fi successfully once the network connection
is set up for it.
2. Robot notifies the user using the GSM module and the sim card in it.
32
Figure 3.4.4.2 GSM Message test
Alternate Data Flow
GSM not having signals and unable to send text messages
REQ-1 The Arduino IDE has to be used to setup the system's network
connection
REQ-2 When a face that is not recognised is detected, the system must
send a notification through the GSM module.
3.4.5 Object detection
3.4.5.1 Description and Priority

The robot can categorise and name various items by using object detection
33
Figure 3.4.5.2 Object Detection
Basic Data Flow

The object detection algorithm, for example, may categorise objects. For
this test scenario, a chair and a bottle are identified and categorised
def run_inference_for_single_image(image, graph):
with graph.as_default():
with tf.Session() as sess:
# Get handles to input and output tensors
ops = tf.get_default_graph().get_operations()
all_tensor_names = {output.name for op in ops for output in
op.outputs}
tensor_dict = {}
for key in [
'num_detections', 'detection_boxes', 'detection_scores',
'detection_classes', 'detection_masks'
]:
tensor_name = key + ':0'
if tensor_name in all_tensor_names:
tensor_dict[key] = tf.get_default_graph().get_tensor_by_name(
tensor_name)
if 'detection_masks' in tensor_dict:
# The following processing is only for single image
detection_boxes = tf.squeeze(tensor_dict['detection_boxes'],
[0])
34
detection_masks = tf.squeeze(tensor_dict['detection_masks'],
[0])
# Reframe is required to translate mask from box coordinates to
image coordinates and fit the image size.
real_num_detection = tf.cast(tensor_dict['num_detections'][0],
tf.int32)
detection_boxes = tf.slice(detection_boxes, [0, 0],
[real_num_detection, -1])
detection_masks = tf.slice(detection_masks, [0, 0, 0],
[real_num_detection, -1, -1])
detection_masks_reframed =
utils_ops.reframe_box_masks_to_image_masks(
detection_masks, detection_boxes, image.shape[0],
image.shape[1])
detection_masks_reframed = tf.cast(
tf.greater(detection_masks_reframed, 0.5), tf.uint8)
# Follow the convention by adding back the batch dimension
tensor_dict['detection_masks'] = tf.expand_dims(
detection_masks_reframed, 0)
image_tensor =
tf.get_default_graph().get_tensor_by_name('image_tensor:0')
# Run inference
output_dict = sess.run(tensor_dict,
feed_dict={image_tensor:
np.expand_dims(image, 0)})
# all outputs are float32 numpy arrays, so convert types as

appropriate
output_dict['num_detections'] =
int(output_dict['num_detections'][0])
output_dict['detection_classes'] = output_dict[
'detection_classes'][0].astype(np.uint8)
output_dict['detection_boxes'] = output_dict['detection_boxes'][0]
output_dict['detection_scores'] =
output_dict['detection_scores'][0]
if 'detection_masks' in output_dict:
output_dict['detection_masks'] =
output_dict['detection_masks'][0]
return output_dict
# In[6]:
for image_path in TEST_IMAGE_PATHS:

image = Image.open(image_path)
# the array based representation of the image will be used later in
order to prepare the
# result image with boxes and labels on it.
35
image_np = load_image_into_numpy_array(image)
# Expand dimensions since the model expects images to have shape: [1,
None, None, 3]
image_np_expanded = np.expand_dims(image_np, axis=0)
# Actual detection.
output_dict = run_inference_for_single_image(image_np,
detection_graph)
# Visualization of the results of a detection.
vis_util.visualize_boxes_and_labels_on_image_array(
image_np,
output_dict['detection_boxes'],
output_dict['detection_classes'],
output_dict['detection_scores'],
category_index,
instance_masks=output_dict.get('detection_masks'),
use_normalized_coordinates=True,
line_thickness=8)
plt.figure(figsize=IMAGE_SIZE)
plt.imshow(image_np)
Alternate Data Flow

Improperly identified object
The REQ-1 Object detection algorithm must function and be able to

categorise various items.
REQ-2 Datasets for various faces, objects accessible for accurate
categorisation of various people.
3.4.6 GUI (Desktop Application)

3.4.6.1 Description and Priority
Logging in allows the administrator to access the dataset, change it, encode
and decode faces, and watch live video.
36
Figure 3.4.2 GUI
Basic Data Flow
1. Administrator can successfully login

2. Administrator may access and change the data set.
3. Administrator can decode and encode faces
Alternate Data Flow
Incorrect username/password prevents admin from logging in and carrying
out further activities.
REQ-1 Provide the proper login and password.
REQ-2 Camera must be operational for admin to watch live footage.
REQ-3 Dataset needs to be accessible.
3.5. Other Non-functional Requirements
3.5.1. Safety Requirements

1. The response time of this application cannot exceed 4 seconds.
2. In the event of data loss, the system will make a backup of the data
and restore it as needed.
3.5.2. Security Requirements

1. The system must only let authorised users to carry out administrator
tasks.
2. Images will be encoded by the system and saved in a data collection.
37
3. The system will operate as a learning base agent by gathering
information from the surroundings.
3.5.3. Performance Requirement

The application must need 400 MB of memory and 100 MB of disc space to
operate.
3.5.4. Cross platform Requirement

The system will be compatible with Microsoft Windows 10 x64 and Ubuntu
3.5.5. Software Quality Attributes

Usability
After setup, voice commands will operate the machine.
Reliability
Application must successfully retrieve information every time Information

retrieval must be 90% successful. That means 90% of information retrieval
efforts must be successful.
Availability
The system must be accessible whenever the user needs it. Up to 5% of

availability failures are acceptable. One minute maximum of system
downtime per day is permitted. Database access must be continuous.
Chapter 4. Design and Development

4.1. Introduction
Security is a serious concern in Pakistan, whether it pertains to a house,
business, or any other location under observation. Traditional CCTV
cameras and other gadgets are expensive and have several drawbacks. As
a result, we are developing a home security robot that uses cutting-edge
technologies like artificial intelligence, computer vision, and image
processing to address these security challenges with the best possible
cost, performance, autonomy, and effectiveness. This article intends to
38
provide sufficient information regarding the creation of an autonomous
robot.
4.2. Purpose
A system must have a strong architecture since it will serve as the project's
fundamental building block and serve as a basic blueprint for its design
and growth. As a result, the design, functionality, and architecture of EVE
are described in this software design specification (SDS) document that we
prepared. Along with the architecture, the paper also includes several
design diagrams and explanations for them, as well as information on the
key parts, building blocks, and system modules that will aid developers in
developing and organising the system appropriately.
4.3. Project Scope

Eve will be an autonomous robot with features for home security and in
locations where guards are not allowed. These characteristics include face
detection, tracking, facial recognition, voice recognition, object
identification, and other functions. Additionally, it can function as a social
robot.
4.4. System Architecture Description

A strong architecture is crucial for a system since it serves as the project's
major building block and serves as a basic blueprint for its design and
growth. As a result, by providing an overview of the system modules, their
structure and linkages, and user interface, the numerous diagrams provided
in this part will explain the comprehensive system design and its
architecture.
4.4.1. Overview of modules

Eve has the following modules:
1. Face detection module

2. Face recognition module
3. Face tracking module
4. Voice recognition module
5. GUI
39
Figure 4.1 System Diagram (Created on Visio)
4.5. Structure and Relationships
This section covers "Eve's" numerous technological specifics. This section

also discusses the connections between the components of hardware and
software. A system's architecture, user interfaces, high-level and low-level
details, as well as its thorough system design, are all fully explained and
shown in the numerous diagrams provided below in this part.
4.5.1 Block Diagram

This graphic shows how the system's hardware and software components
interact and how their core functions relate to one another. Typically, lines
link these blocks, and it is understood that these lines indicate the
connections between the blocks.
40
Figure 4.2 Block Diagram (Created on Visio)
As seen in the picture above, our system's fundamental operation begins

when a human enters the field of view of the robot's camera, which triggers
camera input to the system. Then, our robot or system notices that a face is
in front of the camera. Successful face detection triggers face identification
and tracking through serial port connection, following which the Arduino
instructs the servos to move, producing the output response. Voice
instructions supplied by authorised users may also trigger voice recognition.
The administrator may use a desktop programme to see the camera input and
provide voice instructions.
4.5.2 Dataflow diagram (level 1)
41
Figure 4.3 Dataflow Diagram (created on Visio)
4.5.3 Use case Diagram

Use case diagram shows the ways in which user or actor interact with the
system. There can be more than one actor in the system which can use the
system and there can be more than one way to use a system leading to many
use cases.
42
Figure 4.4 Use Case Diagram created on Lucid Charts
4.5.3.1 Actors
1. Admin
2. Robot
3. Dataset (Secondary actor)
4.5.3.2 Use Cases
Use cases for system as an actor:

1. Face detection
2. Face Recognition
3. Face Tracking
43
4. Voice Detection
5. Voice Recognition
6. Response Generation
7. Hardware Movement
8. Sends notification.
9. Measures Distance
4.6.3.3 Use cases for admin as an actor
1. Login
2. Views Camera input/Video feed
3. Gives Voice Commands
4. Gets Notifications
5. View dataset
6. Change dataset.
4.5.3.4 Description of Use cases
Use case Face Recognition

Using this feature the robot will be able to recognize the
Use case
detected
description face correctly
Normal flow (i)Robot will extract an image of face from camera input.
(ii)User After that, it will match the image of person with

dataset
of previously stored images in the dataset.
(iii)Face recognized successfully
Alternative Face is not recognized, and intruder is detected.
flow
Face is properly detected because the robot will not be
Precondition
able to recognize anything without prior detection
Post Face recognition occurs successfully
condition
Includes Dataset
Table 4.1 Use Case 1
44
Use Case Face detection
Actors Robot
Use case
This feature allows the robot to detect a human face when
description
it comes in robot’s camera view
Normal flow (i)Person comes in front of robot’s camera.
(ii)Camera gives sensor input to the system.
(iii)After getting input, robot detects the face
Alternative Face is not detected properly due to night vision or wrong

flow algorithm
Or changed features of person
Face not detected due to camera lens issues which may
Exception
include
dirt and night vision.
There must be a person in front of camera of robot so that
Precondition
it can
detect it
Post condition Face is successfully detected
Includes N/A
Extends N/A
Use case Face Tracking

Actors Robot
Use case This feature will allow robot to track a face which is not
Description Recognized
Person’s face is not recognized
Robot identifies this face as an intruder

Normal flow
Robot sends alerts/notifications to admin and sends
instructions to servos(hardware movement) to move
in the direction of face track it
Robot is unable to move its hardware in the right

Alternative
direction and
flow face is not tracked
Precondition Face is not recognized
45
Post condition Face is tracked in the right direction
Includes N/A
Extends N/A
Use case Voice detection

Actors Robot
Use case With the help of this use case, robot will be able to detect
description a sound near it.
Normal flow Robot detects a voice through its sensors
Alternative Robot cannot detect that some voice commands are given
flow to it or voice
There must be a voice of a known person which must
Precondition
be
generated
Post Voice is detected successfully
condition
Includes N/A
Extends N/A
Use case Voice Recognition

Actors Robot
robot/system will recognize voice of known users or

Use case
authorized person and it will understand voice commands
description
and respond to them also
(i)voice is detected by microphone.

(ii)voice recognition module recognizes voice
Normal flow
(iii)response is generated in response to voice commands
if the voice is of a known person to Robot
Alternative
Robot can’t recognize voice of unknown persons
flow
Precondition Voice is detected and
Post condition Voice is recognized and response is generated
Includes Geetech hexadecimal dictionary for voice commands
Extends N/A
46
Use case Response Generation
Actors Robot
Description of the Robot will generate response according to the given
use case voice commands
(i)Admin/known user gives commands
(ii) Robot receives instructions via serial from the
Normal flow Arduino.
(iii)Arduino processes the information and makes the
actuators move
Alternative
Actuator movement is not correct
Data Flow
There must be some commands/instructions given
Precondition
Post condition servos/hardware move in the right direction
Includes Actuators
Excludes Chatbot, Google Assistant
Use case Send notifications

Actors Robot
Use case This use case will allow robot to send notifications to
description admin in case of an intruder or unknown person
(i)Face of unknown person/intruder is seen.

(ii) Robot activates its GSM module which sends
Normal flow
notifications to the admin
(iii) In the form of SMS alert
Alternative Notifications were unable to send, GSM module was not
flow activated
Precondition Intruder/unknown person is seen
Post condition SMS alerts are sent through GSM module
Includes GSM module/SMS alert

Extends N/A
Use case Measure Distance

Actors Robot
Using this use case, the robot can measure the distance
Use case
between itself and person’s face and this is mainly used
description
for face tracking
(i)Person comes in front of robot’s camera (ii) Robot
Normal flow
measure distance using ultrasonic sound sensor
Alternative Sensor measures wrong distance ,face tracking doesn’t
flow occur properly
47
Precondition person must be in the region near to robot
Post condition correct face tracking and distance measurement

Includes Ultrasonic sound sensor
Extends N/A
Use case Login

Actors Admin
Use case User can see camera view, change, view dataset, through
description desktop application by using this use case
(i)User enters his username in the required field to log in
to application.
Normal flow
(ii) User the types of his password and presses the login
button
Alternative If incorrect username or password is entered show an

flow error message and login will not be successful
Username and password of user must be already

Precondition
registered
Post condition The user successfully logs in
Includes Authentication
Extends N/A
Use case View camera input

Actors Admin
Use case
This use case provides Admin to view schedule
description
Admin successfully logs in Admin views the robot’s
Normal flow
activity
Alternative Admin is not able to see camera view due to some

flow technical faults in camera or night vision
Precondition Admin is logged in

Post condition Camera view is displayed
Includes Video
Extends live video
Use case Gets notifications

Actors Admin
Use case Admin will be able to receive alerts and notifications
description using this use case
Normal flow Notifications are received by the admin
48
Alternative
Unable to receive notifications
flow
Precondition Admin’s number must be registered in database
Post condition Notifications received through SMS alert
Includes N/A
Extends N/A
Use case Gives voice commands

Actors Admin
Use case Using this use case, the Admin can give voice
description commands to the system also in case of an intruder
Normal flow Admin gives voice commands

Alternative
Could not register command
flow
Admin gives voice commands which must be

Precondition
recognized by the voice recognition module
Serial port communication through Arduino occurs and

Post condition
servos move for response generation
Includes N/A
Extends N/A
Use case Changes dataset (Training images)

Actors Admin
Admin can change the images folder in the data set

Use case through which the system recognizes faces and can add
description more images data set so that the system/robot can
recognize more faces
Admin logs in Admin adds or removes images in data

Normal flow
set and saves data set
Alternative
Admin only views data set and makes no changes
flow
Precondition the desktop application must be logged in by the admin
New images are added in data set or removed. Data set

Post condition
is altered
Includes N/A
Extends N/A
49
4.5.4 Sequence Diagrams
4.5.4.1 Voice Recognition sequence diagram
Figure 4.5 Voice Recognition Sequence Diagram created on Lucidcharts
Figure 4.5 Voice Recognition Sequence Diagram created on Lucidcharts
50
4.5.4.3 Desktop application Sequence Diagram
Figure 4.7 Desktop application Sequence Diagram created on Lucidcharts
4.5.5 Activity Diagram
Figure 4.8 Activity Diagram created on Lucidcharts
51
4.5.5 State Transition Diagram
In this section, state transition of application is shown how it changes to
another state.
Figure 4.9 State Transition Diagram created on Lucidcharts
52
4.5.5 Class Diagram (UML)
Figure 4.10 UML Diagram created on Lucidcharts
Class Description
This is the main class.it will execute first that video it
will extract images and call for face detection method
Main
and acquire video from that video it will extract
images and call for face detection method
In this class input from camera is taken in form images
Face detection
and then it will detect human faces
After receiving voice orders from the administrator,

Voice the voice recognition class will call right away. It will
recognition coincide Voice from data set, followed by servo
movement or a response
Administrators may access desktop apps, It can add or

Admin delete photos from a dataset and see real-time camera
footage. You may also transmit voice instructions.
53
Servos will be able to be moved by hardware class.
The call will come after voice recognition or facial
recognition training. It may monitor faces by moving
Hardware
Eve's hardware (servos) or respond to vocal
instructions by moving servos; for example, when told
to gaze up, it will move servos to look up.
This class will be used to add or delete photos,

recognise voices, or voice recognition or to add or
Dataset
remove images or authenticate or recognise images
from datasets.
If a face matches a picture in the dataset after it is

Face identified by this class, it will proceed to the next step.
recognition face tracking or once again identifying the offered
fresh photos
Table 4.14 Description of Classes
4.6 Detailed Description of components
4.6.1 Input Hardware:

These include:
1. Camera
2. Sensors
3. Microphone, Arduino
Identification Name: Input Hardware

Location: presentation layer of system architecture
Type Hardware components
This is the component that is visible, and the user
interacts with directly.
All input, voice or image is given using this
component.
The requirements that this component fulfills are
Camera operates at lower FPS (15) at higher
resolutions of 1280x960 and higher FPS (30) at
Purpose
lower resolutions of 1280x720.
The ultrasonic sound sensor has a 2-400 cm range,
and a 3mm precision Serial Port communication
occurs between robot, its camera and other sensors
and the laptop/computer.
ESP 8266 Wi-Fi network connection through
Arduino Camera feed will go to computer through
54
serial communication and through the applied
algorithms and facial features acquired by camera
input.it will be successfully detected that there is a
face in front of camera The robot’s camera shall
correctly detect faces according to facial feature
Subordinates Camera, Audio microphone, Ultrasonic sensor
This component 3.1 Input HW interacts with
component 3.4 Database and 3.2 ‘application’
whenever user interacts with the Hardware, the
Dependencies
voice clip and video are stored in database for
processing and are made visible in the application
when required.
Interfaces pins and ports to microprocessors.
Power supply provided to Arduino used to power up
Resources
this component.
Processing ATMEL Processors on microprocessors
Data Audio/video stream.
Table 4.15 Input Hardware
4.6.2 Application
Identific
ation Name: Application UI
Location: presentation layer of system architecture
Type UI components
If needed the user will interact with this component. The

Purpose input
Stream provided by 3.1 input hardware will be displayed
here
along with the faces recognized and command identified.
1. An executable python application which will encode
users and
friendly faces and save them in the trained dataset.
2. A simple GUI interface may be needed that shows the
camera
feed/input that is sensed by the robot’s camera and for
monitoring
3. OpenCV, Python 3.6, Processing IDE, for face
detection and
recognition etc.
Function This component will only be visible when the raspberry pi
55
will
be connected to a screen,
Subordin
ates This component has 2 subordinates, one is responsible for
displaying feed other is responsible for recognizing new
faces
and Commands
Depende This component 3.2 interacts with 3.3 ‘subsystem’, 3.1
ncies ‘Input
Hardware’ and 3.1 ‘Input Hardware’ and 3.4 ‘Databases’
This component is dependent upon ‘subsystem database
and
subsystem whereas in component is dependent upon this
component.
Interface This component will provide interface to component 3.3
s DB and
3.4subsystem when the raspberry pi will be connected to
and
external screen using HDMI
Resource Screen for display, storage device for logs in raspberry pi,
s HDMI
cable, power supply, python code embedded in the
raspberry to
run the application
56
Table 4.16 Application
4.6.3 Subsystem
This component has 7 sub-systems:
1. face detection
2. face recognition
3. face tracking
4. voice detection
5. voice recognition
6. voice commands
7. actuators
Table 4- 17 Subsystem
Identific
ation Name: Subsystem management
Location: Processing layer of system architecture.
Type Processing component
Input that comes in from the input devices component and
data
Sets are extracted from the database and processing is done
Purpose here.
1. It system’s camera shall correctly identify the distance of
person’s face from it and track it in the right direction
2
. The robot shall correctly recognize faces from the dataset.
3 The system shall correctly take actions after voice
. recognition
4 The system shall transmit all the voice signals correctly
. to
computer through serial port communication and after
Natural
language processing it should be able to deliver output
correctly.
5 The system’s network connection must be configured
. through
the arduino
57
4.6.4 Database
Table 4- 18 Database
Identific
ation Name: Database
Location: Application layer of system architecture
Type Data storage component
The function of this component is to use the input data
Function stream
of Audio and video, take assistance from the internet and
provided data set, first detect faces and audio faces are
recognized using the ratio between different points on the
face
and audio is recognized using the frequency matching. It
then
keeps track of the face and generates appropriate response
for
command. This component is also responsible for
classifying
new commands and faces and updating database. Learning
process is also handled here.
Subordi
nates This component has 8 subordinates face detection, face
Identification, face tracking, voice detection, voice
recognition,
voice commands, input stream and decision.
Depende This component is dependent upon 3.1 ‘Input HW’ and 3.4
ncies DB
Whereas 3.2 application and 3.5 ‘output HW’ are
dependent on
it.
Interfac Saves and takes data from database and provides content
es for
actuators and application reside in Arduino minimega and
raspberry pi
Resourc
es Hardware: all input hardware
SW: database, python, and processing (java) code,
Raspian.
58
Data All data from input stream, database, and internet.
Purpose To store data of different component and users

Following functional requirements are fulfilled by
this component
1. Application shall not fail in retrieving information. the
success rate of information
2. Retrieval must be 90%. That is 90% attempts of
retrieving
information must succeed.
Function This component stores all the data fed into it at

development and all new data stored as the system
runs. Tables are kept and transactions are made in
this component
It interacts with (3.2)’application component’ and (3.3)
‘Subsystem component’
Subordi
nates This component has two subordinates
1. Modify database
2. Send data to other components
Depende
ncies This sub system is dependent on (3.3) “Sub System
Component” and vice versa whereas (3.2)
application is dependent on this component
(3.1) Input Hardware is also dependent on this component
Interfac
es SQL database server where all data will be saved
This component is provided interface by (3.3) “Sub
System” for
data transaction
Resourc
es Hardware: micro-SD card on raspberry pi
Software: SQL server and python code
Processi Transaction performed to retrieve data from database to be
ng used
by application and sub system
59
4.6.5 Output hardware (Servos, Speakers)
Table 4- 19 Output Hardware
Identific
ation Name: Output hardware
Location: Presentation layer
Type Hardware component

Purpose This is the component that is visible and the user
interacts with directly. Output whether by voice or
movement is generated by this component
The requirements that this component fulfills are:
1
. Servos must be of the capability to move hardware
effectively and in the right direction
2
. The movement will be free of glitches and as
smooth as possible
Responses will be lifelike and relevant
This component only shows output in form of movement

Function or
sound via audio output unit
Subordi
nates This component has 3 subordinates
1
. Servos
2
. Audio output unit
3
. Acrylic body parts
Depende
ncies No component depends on this component whereas this
component is dependent upon (3.4) Database and (3.3) Sub
system component
Interfac USB ports and pins used to connect to Arduino/raspberry
es pi
60
Resourc Hardware: Power supply provided to Arduino used to run
es this
component
Processi
ng N/A
Data Audio stream, electrical pulses
4.7. Reuse and Relationships to other Products
Our system, the Eve autonomous robot, is a wholly original creation that
neither extends nor has developed from any other application at any level or
from any previously existing system.
1) Our autonomous robot has the potential to develop into a larger, more
complicated system with greater capabilities.
2) Developers may repurpose a variety of hardware utilised in our project
and link it to other hardware systems to create a new, improved system.
3) Our system's components may also be used to build a robot that uses a
laser pointer to automatically find targets. (Can also be fitted with weapons
of a light and medium calibre)
4) By developing AI algorithms that enable our system to automatically
detect and target ground combat units, sea-based combat units, and air-
based combat units with a high-power laser, we can also reuse the robotic
hardware and software components to develop our system into a larger
surveillance system.
5) In order to further enhance our project's autonomy, we may also add
potential machine learning applications.
6) As a result, by continuously expanding the system's services, its
usefulness may be improved.
61
4.8. Design Decisions and Trade-offs
The solution to a recurrent architectural issue is a design pattern. When

creating a system, an architectural design is essential because.
1) A poorly designed system will use more resources.
2) The effectiveness and reaction time of the final product will be quite
poor, which will negatively impact the target user's experience.
3) Ineffective designs make testing and maintenance tasks difficult.
In our project "Eve," we are using MVC (model-view-controller).
An interactive programme is divided into three components using the MVC

paradigm.
1. Model: consists of the fundamental data and functions
2. view: shows the user the information (more than one view may be
defined)
3. Controller: manages the user's input
By separating the many system components, the MVC approach is utilised

to achieve the goal of hiding the user from the system's complexity.
The system is divided into modules through decoupling, which also ensures
effective code reuse.
MVC will be applied in our project as indicated in the figure below:
62
Figure 4.11 MVC Design Pattern
Model-view-controller is what we utilise since it can be applied to many

views, making it particularly suited to our project as we have two views.
1) Desktop app view: As a user or administrator, you may see system

robot/through it actions
2) Servos (parts of hardware movement)
The following graphic illustrates how the controller gets input from sensors,
which might take the form of camera images or spoken instructions. The
controller also reacts to user input and interacts with the objects in the data
model. The input is received by the controller, which may check it before
passing it on to the model. In our scenario, Arduino serves as a controller
and communicates input to the model through the serial port.
After that, the model takes the controller's input and utilises the dataset's
image data to match the picture with the dataset before doing subsequent
operations, such as face detection, face recognition, and voice recognition.
The model then changes the view, which may be any information output
type.
It is conceivable to have many perspectives on the same data, and our
approach likewise employs multiple perspectives.
Therefore, in the event of face tracking, our model refreshes the view by
sending movement orders through Arduino to servos that oversee moving in
the proper directions to track speech or to provide an accurate response in
the case of voice command response generation. The model changes the
second view, which is of the desktop application, by supplying data from
the desktop app.
MVC therefore aids our project by decoupling various system components
and offering support for many views.
63
Chapter 5. Project Test and Evaluation
5.1. Introduction
The acceptable methods, processes, and techniques used to design tests are
described in this section on test plans.
After it has been built, software is tested to verify its quality. Project testing
enhances reliability and efficiency.
This test plan document outlines the proper tactics, procedures, and
management techniques used to organise, carry out, and manage testing of
"Eve." This test plan document will make sure that our system complies
with the criteria for functional, on-functional, and customer needs.
In order to verify that our system complies with both its functional and non-
functional criteria, we will manually test it. As part of this approach, the
tester will manually go through the test cases. The tester examines the
system in the role of an end user and finds any flaws or problems. Unit
testing and integration testing will be used, and each unit will first be tested
independently before being merged with other units. Black box testing is
performed for each item, and acceptance testing is performed for units that
are merged.
The scope, strategy, methodology, and testing technique for EVE are all
included in this paper. The test items' pass/fail criteria are also specified.
Each test case details who will conduct the test, the prerequisites needed to
carry out each test case, the object to be tested, the input, the anticipated
output or results, and, when relevant, the procedural procedures.
64
5.2. Test Items
The following modules, features, and functions of our systems have been
selected for testing in accordance with the specifications stated in our
Software Specifications Document for EVE:
1) Face recognition
2) Face identification
3) Obtain sensor readings and outputs using sensors and hardware
components.
4) Voice response and recognition
5) Desktop Software
6) Dataset
7) Notifications
8) Tracking and measuring distance
9) Detection of objects
The following list outlines the order in which these test items will be
administered:
1. Create the test cases first.

2. Running tests based on the software's generated test cases.
3. Report any flaws found in the test cases that were run.
4. A test report is created when the processes are finished.
5. Manage or incorporate adjustments at a later point in the project's
development.
5.3. Features to Be Tested
The next features are examined:

1) The system must be able to capture real-time video of the for later
processing.
2) The robot's camera must accurately identify faces based on facial
characteristics.
65
3) The robots must accurately classify faces in the data collection as either
known or strange faces, such as intruders.
4) The robot/system must be able to gauge distance and, if a face is

unrecognizably present, track it.
5) The system's capability to detect a particular person's voice and respond
appropriately.
6) The system's capability to accurately move its hardware components,
such as servos, in the case of face tracking and response generation
following speech recognition.
7) The system's capability to alert the administrator in the event of
intruders or an unknown individual
8) The administrator will have access to the desktop application.
9) Through the desktop application, the administrator will be able to
examine the robots' behaviour and alter the dataset (encode/decode faces).
5.4. Test Approach
The system, Eve, requires the development and integration of several

modules since it is a complex system. Additionally, it has both hardware
and software components, making it a complicated system for which
relying just on one testing method is insufficient.
Due to the complexity of our project, several testing methodologies will

be used for each system module and capability.
Unit testing utilising White Box and Black Box testing is part of the plan.
To effectively integrate the system, integration testing is done.
Following the acceptance test plan, a user-acceptance test will be

conducted at the conclusion.
1) Unit testing: Unit testing is carried out at the source or code level
to check for programming language-specific issues including incorrect
syntax and logical flaws as well as to test individual functions or code
66
modules. The purpose of the unit test cases is to validate the programmes'
accuracy.
Developers are responsible for unit testing.
2) Integration test:
In this testing, we'll check that each of the previously examined
components operates correctly when used in tandem.
Following individual testing of each unit, each unit will undergo a module
test. Black box testing is carried out after modular integration.
3) System Testing
System testing will ultimately verify that each module is functional both
on its own and when coupled. The effectiveness of the whole system will
thereafter only be determined by the program's results.
5.5. Item Pass/Fail Criteria
Section 7 of the paper, titled "Test Deliverables," contains information on the

test cases and their specifics. The following criteria will determine whether a
test item is deemed to have passed or failed.
1. The test case satisfies the prerequisites.

2. Different inputs are carried out in the manner described.
3. The test is considered successful if the outcome is as expected from the
output. So, the output is "Pass”.
4. The system is considered to have failed if it fails to function properly or if
it fails to meet the output requirements.
5.6. Suspension Criteria and Resumption Requirements
Unexpected events may always happen in the form of significant defects,

which may cause difficulties and limitations throughout the testing process.
Some test cases may be blocked by these serious flaws because of their
interdependence and potential to limit testing.
67
5.7. Test Deliverables
5.7.1. Acquire Video
Test case number TC 01
Name of testcase Acquire Video
Video is acquired from the camera mounted on

Test items/features the robot. This video will be fed into the system for
further processing.
Testing technique Component testing, Black box testing

Valid Input:
1) Camera’s footage feeding into this module.
Input Specifications (Camera feed will go through serial port
communication over Arduino)
2) Arduino, servos attached to camera.
1)Camera not properly configured with Arduino
Invalid Inputs 2) Camera cap is not removed.
3) Low or no light.
Open the video capture mode using OpenCV
Steps
function
1)Video is acquired through the camera with
Valid Output resolution of 1280x9602 2) Video is captured
Specifications correctly and continuously. 3)Video acquired through
the camera and the OpenCV function.
1) Video unavailable due to improper camera
Invalid Output
configuration
Specifications
2) Video not visible due to dim or no light.
Equipment Hardware: Camera Software: Python3.6, OpenCV 3
Dependencies Not dependent on any test case
Table 5.1 Testcase 1
5.7.2 Face detection

Test case name Face detection
The robot’s camera shall correctly detect faces

Test items/features
according to facial feature extraction
Testing technique Black Box testing, component testing

1) Face of person at distance of 400cm 2) Video
Input Specifications
frame by frame
1) Face angle greater than 50 degrees on either side.
Invalid Input:
2) Face covered with mask
3) Person out of camera’s range
68
1. Camera feed will go to computer through serial
communication for processing.
2. Haar’s cascades will extract the facial features
Steps and facial co-ordinates will be identified in the
video stream/frame
3. Target will be locked depicting that face in front of
the camera is present
1)Extracted facial features through Haar’s cascade 2)

Output
Locking of face coordinates 3)Rectangle formation
Specifications
around face coordinates
Invalid Output: 1)Face not detected

Hardware: camera, Arduino Software: OpenCV,
Equipment NumPy, python, face detection algorithm, Processing
IDE
Inter case
Dependent on video acquiring
Dependencies
Table 5.2 Testcase 2
5.7.3. Face Recognition

Test case name Face Recognition
This test case will test the feature / ability of the
Test items/features
robot to
recognize the faces encoded to its dataset
successfully
1) Video input 2) Face of person detected at distance
of 3) Encoded Dataset for face matching 4 meters
1) Familiar face not encoded in dataset 2) Identical

Invalid Input:
faces it is matched
After face is detected by an algorithm, with the
Steps
dataset and it is recognized if it is a familiar face.
1)Recognized face (matched NumPy array of face
with dataset
2) If detected face is matched with dataset/database
Output face (stored as NumPy array) is recognized
Specifications successfully.
1) Identical faces do not distinguish
2) Person not recognized due to improper lights and
shadows.
Equipment Hardware: camera, Arduino, raspberry pi
69
Software: OpenCV, python IDE, NumPy, AI face
recognition algorithms
Inter case
Dependent on face detection
Dependencies
Figure 5.2 Face recognition
5.7.4 Intruder Detection

Test case name Intruder Detection
Test items/features Robot shall distinguish between intruders and similar
faces and take actions in case of an intruder.
1) Video frames containing face input 2) Detected

face 3) Encoded dataset
1) Error in dataset or limitations of dataset 2) Dataset
Invalid input: not trained properly. 3) Person is not intruder but is
not encoded in dataset
The detected face is matched with the encoded

Steps dataset/database for classification of the face as either
intruder or non-intruder/similar face to the robot
1) Unmatched input of face with encoded dataset i.e

Output
intruder detected. 2) SMS notification send to admin
Specifications
via GSM module
Invalid output: Intruder recognized as friendly face
equipment Hardware: camera, Arduino
Software: OpenCV, python IDE, NumPy, AI face
software
recognition algorithm
Inter case
Dependent on test case 2
Dependencies
Figure 5.3 Intruder detection
5.7.5 Face Tracking
Test case
number TC 05
Test case name Face Tracking

Test The robot’s sensors will track the face of the
items/features person if
its distance is within the range of the robot’s
distance
70
measurement sensor
Testing
technique Black Box testing, component testing
Input
Specifications Valid Input:
1 Camera Video
) frames
2
) Detected face within 400 cm distance
3 Correct Ultrasonic sound waves sensor
) readings
Invalid Input:
1
) Person is out of camera’s range i.e. 4m
2 Incorrect distance measured by ultrasonic
) soundwaves
sensor
3
) Incorrect, jittery servo movement.
1) Face is detected and not recognized and is
Steps classified as
an intruder.
2 Ultrasonic sound waves sensor measures
) distance of
the intruder’s face. If the face is within the
range of the
sensor then movement directions are given to
the servos
via Arduino which then track the face of
intruder (locked
target)
Output Valid output:
Specifications
1)Servo movements to track intruder’s face
Servo.write(up,down,left,write);
Invalid output:
Tracking due to incorrect
discontinued sensor
readings and range
limitation
71
Environmental Hardware: servos, ultrasonic sound wave’s sensor.
needs Software: python
Inter case Dependent on test case 3,4

Dependencies
Table 5.5 Test Case 5
5.7.6. Notifications
Test case
number TC 06
Test case name Notifications
Test
items/features Notifications will be to the admin in case of
intruders/unfamiliar person
Testing
Input
1)Video frames, intruder’s face
2) GSM module
3) Admin information through database
Invalid input:
1)Incorrect admin information
2) GSM package unavailable

If intruder is detected, notifications are sent to the
Steps admin
through GSM module of the robot in the form of
SMS
alerts.
1)Notification in the form of SMS alerts send to
Specifications admin
Invalid Output
2) Message/notification not sent or sent to wrong
number
Environmental Hardware: GSM module

72
needs
Software: Database, python
Inter case Dependent on test case 5
Dependencies
5.7.7 Voice Recognition

Test case
number TC 07
Test case name Voice Recognition
Test The robot shall be able recognize voice of the

items/features specific
person and take actions accordingly.
Testing
Input
1) Voice commands to robot
2) Gtech hexadecimal dictionary for voice
commands,
Invalid Input:
1) Voice commands in language other than English

Steps 1)Voice is detected by microphone
2)Voice recognition module detects voice
3)Voice commands are processed through
Natural
language processing and output is generated
Specifications 1) Servos movement (e.g. if turn right command is
given serial port communication via Arduino will
occur
giving one of the servo to turn right)
2)After natural language processing ,Voice is
recognized and respond is generated in the from
servo
movement in 5 directions accordingly
Invalid output:
73
No response is generated
Hardware: geetech hexadecimal dictionary for

Environmental voice
needs commands, arduino, microphone
Software: NLTK toolkit, natural language
processing
Dependent on voice and voice commands, not on

Inter case any of
Dependencies above mentioned test cases
5.7.8 Response Generation

Test case
number TC 08
Test case name Response generation (servos and Google assistant)
Test The robot will be able to recognize voice commands

items/features and
generate response accordingly
Input
Specifications Valid input :
1) Voice commands to Google assistant built in the
robot
(e.g. Today’s weather)
2) Geetech hexadecimal dictionary for voice
commands,
arduino, microphone, speakers
74
Invalid input:
Voice input in language other than English
1)Voice commands are given by admin and

Steps recognized
by the robot
2 Servos move, if movement (tracking etc. or
) response
to commands) is required.
3 Robot’s Google assistant responds in the response
) of
respective voice commands
Output Valid Output:
Specifications 1 Google assistant of the robot gives correct
) answers.
2
) Servos move in the right direction
Invalid Output:
1
) Commands not understood
2
) No response generated
Environmental Hardware: geetech hexadecimal dictionary for voice
needs commands, arduino, microphone, speakers
Inter case Dependent on voice commands

Dependencies
5.7.9 Admin Login

Test case
number TC 9
Test case name Admin login (GUI application)
Test This test case evaluates the functionality of system

items/features for
successfully logging in of the admin on desktop
application.
Testing
75
Input
Specifications Valid input:
1) Correct User name and password given by admin
2) login button pressed

3
) Internet is available
Invalid output:
1
) Incorrect user login / password entered
2
) Incorrect database
1) username is enteredin the required field to log in to
Steps the
application.
2)Password is entered by the user
3
) Login button is pressed by the user
Output Valid Output:
1
Specifications ) Admin id and password are verified.
2)Admin is logged in and able to access the Desktop
application
Invalid Output:
1 Username not recognized due to limitations of
) database
Environmental Software: desktop application (GUI)

needs
Inter case No
Dependencies
Table 5. 9 TestCase 9
5.7.10 Edit Dataset

Test case
number TC 10
Test case name Edit dataset (GUI application)
Test This test case tests the feature that Admin to change
items/features the
images folder in the data set through which the
system
recognizes faces and an add more images data set so
that
76
the system/robot can recognize more faces
Testing
Input
1)Images of faces to be added in dataset are given
as
input in python desktop application
Invalid
input: Image file format incompatible
Steps 1) Admin is logged in
2)Admin clicks “Add User’’ to add new faces to
dataset(Encoding)
3)Admin clicks “ Delete ” button to decode any face
from data set
4) Admin can view on “Users” box see images in
dataset
5) Admin can see recognized faces in “recognized
face”
box.

Specification
s 1) New image is added to dataset
2)Editing, viewing of dataset is performed
successfully
by the admin
Invalid Output:
Invalid format image not added in dataset
Environment
al Software: desktop application (GUI)
needs
Hardware: laptop containing GUI
Table 5- 10 TestCase 10
5.7.11 Live Video

Test case
number TC 11
77
Test case name Live Video (GUI desktop)
Test
items/features This test case examines the ability of the desktop
application (GUI) to show live video of the robot
Input
Specifications Input values:
1) Admin is logged in
2) Wifi module is connected

3) Robot and GUI are connected
4) Camera feed is available
Steps 1)Admin logs in

2)Admin clicks the “Start” on the desktop
application
(GUI) to view live video
Live video is show on the GUI about the activities

Output of the
Specifications
robot
Invalid output:
Video not available due to incorrect connections
Environmental Software : desktop application (GUI),
needs Hardware : wifi module, micro SD card on
raspberry pi
Inter case Live Video (GUI desktop)

Dependencies
Table 5- 11 TestCase 11
5.7.12. Object Detection

Test case
number TC 12
Test case name Object Detection

Test This test case examines the ability of the robot to
items/features detect
and classify different objects and label them
78
Testing
Input
1
) Different Objects in front of camera.
Invalid Input:
1 Object should be within range of the camera i.e
) 4m
2
) Brightness should be present
Output Output values:

Specifications 1
) Objects are classified by the object detection
algorithm
E.g. Chair and bottle are detected and classified
for
this test case.
Invalid output:
1 Object recognized incorrectly. (chair is
) recognized
as bottle)
2
) Not happened yet but rarely possible.
Environmental Software : Keras, tensor flow library, python
needs
interpreter
Hardware : camera, sensors
Inter case Dependent on test case 1

Dependencies
Table 5.12 Test case 12
79
5.7.13. Hardware Integration
Test case
number TC 013
Test case name Hardware Integration
Test
items/features This test case tests that the different hardware
components of the system are working properly
after
hardware integration
Input
Specifications 1) Camera with resolution 1280x960
2
) Arduino mini mega,
3
) Servos attached to camera
4
) Ultrasonic Sound waves sensor
5) GSM module
6
) Gtech Voice module
7
) Joysticks and Microphone
Testing
technique Black box testing
1
Output ) Camera acquiring video every second
Specifications 2
) Serial port communication with arduino
3
) Servos showing movements with and without
joysticks
4) GSM module working
5
) Gtech module and microphone detecting voice
Hardware: All hardware modules mentioned in

Environmental input
needs specification
80
Software : OpenCV or Webcam to test camera,
Serial
port communication to check servos
Inter case No, some hardware modules are interdependent
Dependencies
Table 5- 13 Test case 13
Figure 5.7 Hardware Integration
5.7.14. System Integration

Test case
number TC 14
Test case name System(Integration) Test

Test
items/features the overall functionality of “Eve”, is checked
here, which includes testing of all the modules
altogether
Testing
Input
1
) Face in front of robot’s camera
2 Camera acquiring video
81
)
3 Encoded dataset, correctly mounted GSM
) module,
GUI application
4 Ultrasonic soundwaves sensor, Servos, Arduino
) ,serial
port communication
Invalid Input:
1 Face behind camera or away from camera’s
) range
and angle
2 Dataset limitations , GSM module not
) configured
properly,
Output Valid output
Specifications 1) detected face
2) recognized face and intruder detected
3
) Tracked face
4) notifications send to admin
5) objects are detected
6) Admin is logged in and can view dataset and can
see
live video.
Invalid output:
1) face not detected, familiar person is recognized
as
intruder,
2) face is not tracked due to incorrect sensor
readings
Table 5- 14 Test Case 14
82
Figure 5- 8 System Integration
5.8. Responsibilities, Staffing and Training Needs

5.8.1. Responsibilities:
The developer shall take part in the testing procedure of the project.
The Testing Procedures and testing methods are being carried out
simultaneously.
by all the members of the project i.e. Eve.
5.8.1. Staffing and Training needs

System testing is important since it guarantees the project's quality.
Without proper understanding of the different testing techniques, including

unit testing, black box testing, white box testing, and integration testing,
testing cannot be done effectively. For the project's testing, a basic
understanding of testing methodologies is required.
Therefore, the developers must have sufficient understanding of these

approaches to critically assess the system in terms of testing methodologies.
5.9. Environmental Needs
Hardware:
1) Camera with resolution 1280x960
2) Arduino mini mega,
3) Servos attached to camera.
4) Ultrasonic Sound waves sensor
5) GSM module
6) GTECH Voice module
7) Joysticks and Microphone
Software:
1) Python 3.7
2) PyCharm
83
3) Anaconda for NLP
4) Java IDE
5) OpenCV
6) Coco for object recognition
7) Kera’s, TensorFlow
8) PyQt 5 for GUI
9) Processing IDE
10) MATLAB
5.10. Risk and Contingencies
There are several risks that might be connected to our endeavour.
5.10.1. Schedule Risk:
Situations that expose a project to defects and interfere with its timely
completion might happen while it is being developed.
The effects of the events might result in the project falling behind schedule,
thus in order to finish it on time, we will need to increase the number of
hours each day that the project is worked on.
5.10.2. Operational Risks:
Our project's operational hazards will be eliminated by:
1) Setting up a timeline with deadlines

2) Including regular deadlines in the objectives to achieve the project's goals
3) Keeping lines of communication open throughout the team.
5.10.3. Technical risks:
By maintaining the once-defined criteria, technical risks that could arise due
to non-conformance with the requirements will be minimised.
5.10.4 Programmatic Risks:
84
In order to keep within the project's parameters and reduce programmatic
risks, the project's scope will be restricted.
Chapter 6. Future Work

This project comprises of many software and hardware components and
may be utilised as a foundation for comprehension and the addition of
elements to increase its size and complexity. Our project may have many
features added to its hardware and software, and its scope can be greatly
expanded. According to us, the following features may be introduced to our
project soon:
1. Building a robot with a laser pointer that can recognise objects on its own
(Can also be fitted with weapons of a light and medium calibre)
2. Include a Gyro and accelerometer module for having better spatial
awareness.
3. Potential uses of Eve with machine learning to learn and adapt to it’s
user’s use cases.
Chapter 7. Conclusion
Modern society increasingly uses automation to replace outdated practises.

Particularly in information technology, automation is occurring. Robots
equipped with advanced technologies like artificial intelligence, machine
learning, and computer vision will rule the planet in the future.
7.1 Objectives achieved.

Consequently, we have developed an autonomous robot employing these
technologies that is effective, high performing, optimised, affordable, and
99% accurate in its output. And these outcomes represent the tasks carried
out by it, such as object detection, speech recognition, and face recognition.
Since Eve is always on duty, it may provide greater home security than
guards and CCTV cameras, which need monthly fees. In the event that it
spots a suspect, it will notify the police, creating a reliable security system.
85
Chapter 8 General Information
System Overview:
Several recognisable faces are intended to be captured by Eve, an
autonomous home security robot, utilising its camera unit from a variety of
angles in order to establish a dataset before installation. Each familiar
individual is given their own dataset along with name.
The desktop application for Eve is used during this installation step.
Following the installation phase, Eve operates by continually filming its
surroundings and identifying each person's face. If a person enters the area
who is not listed in Eve's databases, it will recognise the individual as an
intruder and notify the user or administrator through text message or sound
output via its speaker that a stranger has entered the area. Along with this,
Eve also functions as a home assistant, which means it is fully capable of
carrying out simple daily activities and having conversations, much like
Google Assistant and Amazon Alexa. Despite having a desktop programme
that can display the live stream of the input video and the faces it detects,
Eve is a self-contained device that can function without an interface.
Organization of the manual:
1. General Information, System Summary, Getting Started, and Using the
System make up the five parts of the user's handbook.
2. The system and the intended use are described in broad terms in the
General Information section.
3. A broad overview of the system is provided in section two of the
document. The overview describes how the system's necessary hardware
and software are used.
4. System settings, user access levels, and response in the event of any
potential issues.
5. The Getting Started section describes how to instal and configure the
system. the first time. The system settings are briefly presented in this
section.
6. The section Using the System offers a thorough explanation of system
operations
86
Chapter 9 System Summary
A broad description of the system is provided in the System Summary
section.
System Configuration:
Eve is a self-sufficient machine that has all the parts it needs to work. A
GSM module, a Raspberry Pi, an Arduino Mini, a set of pre-assembled
servos, a camera, a microphone, a speaker, and a port are all included with
Eve. It also has a set of servos. The desktop programme must be executed
on a computer running Windows 7 or later and a functional smartphone in
order to receive the SMS delivered by Eve as an alarm. For Eve's helper
component to function, an internet connection is most important.
User Access Levels:
The admin controls of the desktop programme will be available to the

homeowner or person in charge and anybody who has access to the
specified login credentials, while Eve Robot will be available within the
home and accessible to all occupants.
Contingencies:
The database will not be impacted, and user datasets will be secure in the
event of any problems or system breakdowns.
Chapter 10 Getting Started

How to instal and configure the system for the first time is covered in the
Getting Started section. The system's menu is also briefly described in this
section.
Installation:
Eve System:
1. Remove the camera cover and clean the camera.
2. All wiring connections must be inspected for loose connections

and fixed.
3. Each servo must be angled at 90 degrees.
87
4. Raspberry pie ought to have internet connectivity.
5. Raspberry pie needs a functional and reliable power source.
Desktop Application:
1. The computer must be compatible with Windows 7 or later.
2. The application needs to have access to storage.
System Interface:
The user will be able to log in on their own by inputting their username and
password.
Display Stream and Datasets
The user may browse the live stream, submit their information, and access
previously saved datasets after successfully authenticating into the
programme.
Notification SMS
This function will alert the user through SMS if there is an intruder around
Eve.
Chapter 11 Using the System

The characteristics and operations of the system are described in this section.
Surveillance Mode:
1. The user will launch Eve, which will record its surroundings and search
for both known and unknown faces.
2. Upon seeing an unfamiliar person:
2.1. The user will get notification through SMS.
Login:
The user may log into the desktop programme themselves using this option.
It asks for:
1)Account name
2) Key term
Saving and Accessing Datasets:
88
1. The user will place the person in front of the camera unit of the system
and take many pictures from different angles to add the new person
using the programme.
2. Using the desktop programme, the user may examine, change, and
remove previously used datasets.
Notifications:
Notifications will appear on the smartphone via SMS on a smartphone

through a push notification which will appear when an intruder is detected.
APPENDICES
APPENDIX A
Glossary
1. APP: Application
2. GUI: Graphical User Interface
3. DB: Database
4. SDS: Software Design Specification
5. UML: The Unified Modelling Language (UML) is a general-
purpose modelling language which is designed to provide a standard way to
visualize the design of a system.
7. UI: User Interface
8. WBS: The project management Work Breakdown Structure
References
[1] S. Russell and P. Norvig, Artificial Intelligence: A Modern

Approach. Upper Saddle River, NJ: Prentice Hall, 2016.
[2] R. S. Sutton and A. G. Barto, Reinforcement Learning: An

Introduction. Cambridge, MA: MIT Press, 2018.
[3] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning.

Cambridge, MA: MIT Press, 2016.
[4] S. Thrun, W. Burgard, and D. Fox, Probabilistic Robotics.

Cambridge, MA: MIT Press, 2005.
89
[5] R. Murphy, Introduction to AI Robotics. Cambridge, MA: MIT
Press, 2019.
[6] P. Stone et al., Eds., Robotics: Science and Systems. Cambridge,

MA: MIT Press, 2018.
[7] R. Siegwart and I. R. Nourbakhsh, Introduction to Autonomous

Mobile Robots. Cambridge, MA: MIT Press, 2011.
[8] S. Levine and V. Koltun, Reinforcement Learning and Control as

Probabilistic Inference: Tutorial and Review. Morgan & Claypool,
2018.
[9] P. Abbeel and A. Y. Ng, Apprenticeship Learning and

Reinforcement Learning with Application to Robotic Control. Now
Publishers Inc., 2004.
[10] O. Khatib, Springer Handbook of Robotics. Berlin, Germany:

Springer, 2016.
[11] D. L. Poole and A. K. Mackworth, Artificial Intelligence:

Foundations of Computational Agents. Cambridge, UK: Cambridge
University Press, 2010.
[12] S. L. Thrun and H. W. S. III, Learning Robotics from Self-

supervision. MIT Press, 2021.
[13] R. M. Murray, Z. Li, and S. S. Sastry, A Mathematical Introduction

to Robotic Manipulation. Boca Raton, FL: CRC Press, 1994.
[14] C. Szegedy et al., Eds., CVPR 2019: Computer Vision and Pattern
Recognition. Piscataway, NJ: IEEE Press, 2019.
[15] Y. LeCun, Y. Bengio, and G. Hinton, Deep Learning in Nature.

Nature, vol. 521, no. 7553, pp. 436–444, 2015.
[16] N. Roy and S. Thrun, The Certainty Grid: A Technique for Sensing
the World. Autonomous Robots, vol. 11, no. 3, pp. 305–316, 2001.
[17] P. Newman, D. Cole, and K. Ho, Robust Map Optimization Using

Dynamic Covariance Scaling. The International Journal of Robotics
Research, vol. 26, no. 3, pp. 335–364, 2007.
[18] R. R. Murphy, Introduction to AI Robotics, 2nd ed. Cambridge,

MA: MIT Press, 2020.
[19] J. Schmidhuber, Deep Learning in Neural Networks: An Overview.

Neural Networks, vol. 61, pp. 85–117, 2015.
90
[20] J. Dean and S. Ghemawat, MapReduce: Simplified Data
Processing on Large Clusters. Communications of the ACM, vol. 51,
no. 1, pp. 107–113, 2008.
[21] R. Sutton et A. Barto, Reinforcement Learning: Introduction and

Recent Advances. In J. W. Shavlik et al., Eds., Exploring Artificial
Intelligence in the New Millennium, pp. 65–72. Morgan Kaufmann,
2002.
[22] M. A. Hsieh, S. Krishnan, and V. K. Narayanan, Decentralized

Target Tracking Using Sensor Networks. IEEE Transactions on
Robotics and Automation, vol. 21, no. 3, pp. 208–222, 2005.
[23] D. Fox et al., Bayesian Filtering for Location Estimation. IEEE

Pervasive Computing, vol. 2, no. 3, pp. 24–32, 2003.
[24] J. Schmidhuber, Deep Learning Neural Networks: Tricks of the

Trade. In G. Montavon et al., Eds., Neural Networks: Tricks of the
Trade, pp. 639–660. Springer, 2012.
[25] R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour,

Policy Gradient Methods for Reinforcement Learning with Function
Approximation. In M. S. Kearns et al., Eds., Advances in Neural
Information Processing Systems 12, pp. 1057–1063. MIT Press, 2000.
[26] L. R. Rabiner, A Tutorial on Hidden Markov Models and Selected

Applications in Speech Recognition. Proceedings of the IEEE, vol. 77,
no. 2, pp. 257–286, 1989.
[27] A. Y. Ng and M. I. Jordan, On Discriminative vs. Generative

Classifiers: A Comparison of Logistic Regression and Naive Bayes.
Advances in Neural Information Processing Systems, vol. 14, pp. 841–
848, 2001.
[28] R. Bellman, Dynamic Programming. Princeton, NJ: Princeton

University Press, 1957.
[29] N. Roy and S. Thrun, Sonar-Based Mapping of Large-Scale

Mobile Robot Environments. Autonomous Robots, vol. 8, no. 3, pp.
295–308, 2000.
[31] M. S. Fox et al., Intelligent Robotics: Principles and Practice.

Menlo Park, CA: AAAI Press, 1998.


91


92

Final Report Eve Autonomous Robot

Uploaded by

Copyright:

Available Formats

Final Report Eve Autonomous Robot

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Final Report Eve Autonomous Robot

Uploaded by

Copyright:

Available Formats

Eve

(Robot for home automation and AI-based surveillance)

The final report for the complete fulﬁlment of the requirements.

Master of Engineering (Electrical)

Muhammad Ameer Hamza Janjua

School of Electrical, Computer and Telecommunications Engineering

Supervisor: Rabiul Islam

Information technology continues to advance, addressing complex human

Home security is a primary concern for everyone, aiming to prevent thefts

To address these concerns, we have developed an autonomous robot named

Thank you to my parents, whose assistance and support made a project of

1.2 Problem Statement

Everyone desires a secure environment where their workplace, house,

The project entails building an autonomous robot with characteristics

Home security, as well as other locations and situations

All parts of software engineering, including survey and feasibility analysis,

Table 1.1 Deliverables

1.7 Overview of the Document

1.8 Document Conventions

1.9 Intended Audience

2. Test Persons: (Project Creator, Supervisor)

to compare the project's features and functionalities against the

criteria stated above.

5. Project Supervisor: (Md Rabiul Islam)

The project manager will use this document to determine if all

Chapter 2. Literature Review

Numerous studies on face tracking and identification have been

Sutton and Barto [2] introduce "Reinforcement Learning: An

Goodfellow, Bengio, and Courville [3] discuss the fundamentals of

Thrun, Burgard, and Fox [4] present "Probabilistic Robotics," an

Murphy [5] contributes to the field with "Introduction to AI

Stone et al. [6] edit "Robotics: Science and Systems," a compilation

Siegwart and Nourbakhsh [7] introduce "Introduction to

Levine and Koltun [8] provide a comprehensive review of

Abbeel and Ng [9] discuss apprenticeship learning and

Khatib [10] compiles the "Springer Handbook of Robotics," serving

Poole and Mackworth [11] contribute to the field with "Artificial

Thrun and S. S. III [12] present a recent advancement in learning

Murray, Li, and Sastry [13] provide "A Mathematical Introduction

Szegedy et al. [14] edited the proceedings of the 2019 Computer

LeCun, Bengio, and Hinton [15] discuss deep learning techniques in

Newman, Cole, and Ho [17] present a robust map optimization

Murphy [18] provides an updated version of "Introduction to AI

Schmidhuber [19] discusses deep learning in neural networks in his

Dean and Ghemawat [20] introduce the concept of MapReduce for

Sutton and Barto [21] present reinforcement learning methods in

Hsieh, Krishnan, and Narayanan [22] propose a decentralized target

Schmidhuber [24] contributes to the field with "Deep Learning

Sutton, McAllester, Singh, and Mansour [25] present policy gradient

Rabiner [26] provides a tutorial on hidden Markov models (HMMs)

Ng and Jordan [27] compare discriminative and generative

Bellman [28] presents the concept of dynamic programming, a

Roy and Thrun [29] propose a sonar-based mapping technique for

Fox et al. [30] edit "Intelligent Robotics: Principles and Practice,"

2.1 Utilizing image processing to identify faces

for root,dirs,files in os.walk(image_dir):

for root,dirs,files in os.walk(image_dir):

case 0x11: //Command 1 is to make Nova turn left

case 0x12: //Command 2 is to make Nova turn right