SMART INTERACTIVE MIRROR WITH VOICE ENABLED CONTROL FOR

HOME AUTOMATION USING GOOGLE ASSISTANT

Hariharan P.S [1], Arunkumar A.M [2], Jagadeesh Chandra C.M [3], Darshan M.M [4] ,Saravana Kumar [5]
[1][2][3][4]
Final Year Students Department Of CSE , DSATM
[5]
Associate Professor Department Of CSE , DSATM
and Alpay Kasal at Lit Studios [7] is a touch and signal useful mirror.
The fundamental technique for connecting with the mirror is by
Abstract— Internet of Things, Smart Home, and Smart Cities
have become intriguing issues as of late. The improvement of these
zones becomes spirally because of the rise of cutting edge shrewd contacting the screen. Five understudies at Chalmers University in
gadgets. Savvy mirrors are another expansion to the brilliant home Sweden interacting with the mirror is by touching the screen. Five
family that has been getting a ton of consideration these days by both understudies at Chalmers University in Sweden planned the HUD
business producers and the scholarly community. This paper portrays Mirror [8]. They utilized a two-path mirror to permit the LEDs
how a Raspberry Pi gadget can be utilized to improve such mirrors mounted behind to light up the data. The framework can show data,
with insight and security. The objective is to build up a financially
for example, time, climate, temperature, and a toothbrush clock.
savvy smart mirror framework that functions as an ordinary mirror,
yet in addition have the option to show different sorts of data, for
The savvy reflect portrayed in this paper contrasts from the
example, climate, time and area, recent developments, and clients.
The mirror can give sight and sound administrations while previously mentioned undertakings mostly in that it offers
guaranteeing top of the line security over the whole framework .And exceptional highlights to improve the framework security through
home apparatuses, for example, lights, fans can be controlled through biometric authentica-tion, a voice right hand with included Alexa
the voice directions utilizing Google colleague. abilities and sight and sound capacities. The biometric confirmation
is actualized with multi-variables, facial and voice, to guarantee a
higher security level than that gave by just voice which has been
File Terms shrewd mirror, Raspberry Pi, counterfeit intelli-gence, created in Amazon Alex and Google Home. Additionally, re-
voice acknowledgment, face acknowledgment, Internet of Things, verification is authorized if the client has not been collaborating with
Alexa voice administration the mirror for some time. In this way, it can shield the framework
against a few conventional security dangers, for example, session
I. INTRODUCTION commandeering or man-in-center assault. Also, the voice partner and
the mixed media show actualized in the mirror cause clients to
Brilliant Home has become a rising subject as of late in both appreciate the insight of the savvy home.
scholarly community and industry. With the exception of the well- .
created business Internet of Things (IoT) items, for example, keen The guide of this paper is given as pursues: Section II
entryway locks, savvy fittings, and brilliant indoor regulators, shrewd depicts the review of the framework. Segment III talks about
home has pulled in countless individuals to chip away at self- the subtleties of the framework structure and execution.
persuaded ventures utilizing minimal effort equipment, for example, Segment IV exhibits the exploratory examination, and an end
Raspberry Pi [1]. The known Raspberry Pi based brilliant home is given in Section V.
ventures spread an assortment of themes, from Raspberry Pi fueled
surveillance cameras to robotized pool controllers [2] [3]. Because of II. SYSTEM OVERVIEW
its minimal effort, such undertakings are exceptionally well known in The savvy reflect framework is intended to be an easy to
instruction, where understudies can get occupied with hands-on understand arrangement that offers disentangled and customized
ventures benefits as a mirror. Consequently, the framework was organized to
In this paper, the creators built up a shrewd mirror sys-tem. As give the client an unlimited oversight over all administrations and
another expansion to the shrewd home family, brilliant mirrors capacities through voice communication. Figure 1 delineates a
increased expanding consideration as of late. There have been some schematic diagram of the framework, which can be separated into
fascinating undertakings distributed on the Internet [4]. For example, three unmistakable parts:
Philips HomeLab [5] made an astute individual consideration
Physical structure
framework that executed an Interactive Mirror [6] to give altered
administrations to clients. The framework is equipped for showing Smart mirror software
TV encourages, screen the most recent climate, etc. The mirror has a Cloud services
LCD show joined with a reflected surface and a processor to give the The physical structure of the framework comprises of a single
expected administrations. Another undertaking created by Sam Ewen direction acrylic reflect put over a showcase screen, a Raspberry
Pi gadget, and a mouthpiece. The arrangement permits the
impression of light beams from one surface of the mirror while enabling light from the presentation to go through.
Weather and
Location Weather and location
API

News Headlines AuthenticationFramework RSS News Feeds

Smart Mirror

VoiceAssistant
Request
Native Skill Amazon Webservices

AWS Lambda
Custom Skill Custom Webservices
Host

Response

Processed Response to Custom Skill Request

All the above three figures represent the already existing model.

1st figure represents the embedding working of a smart mirror assemble

The keen mirror programming was created utilizing Python
programming language. It comprises of a few modules which
handle singular elements of the framework, including climate
data, current city, time and date data, news sources and sight and
sound presentation. The mirror shows white characters over a
On the off chance that a face is recognized in a picture
dark foundation for expanded perceivability from the opposite
side of the mirror. outline, the casing is spared. This progression is rehashed
until 20 photos of the client are spared and entitled with a one
The physical structure of the framework comprises of a single of a kind whole number and the picture check (going 1 - 20).
direction acrylic reflect put over a presentation screen, a A one of a kind whole number is produced for every client in
Raspberry Pi gadget, and a mouthpiece. The arrangement permits
the framework. At that point the preparation procedure
the impression of light beams from one surface of the mirror
while enabling light from the showcase to go through. So a client begins. All the spared pictures are stacked into an exhibit,
can see his refection notwithstanding the substance appeared on named faces, and went into a recognizer. The recognizer is
the presentation. The arrangement is at long last encased with a made as an occasion of the OpenCV Local Binary Patterns
wooden casing to give the regular mirror feel to the framework. Histograms class. Neighborhood Binary Patterns (LBP)
depicts a strategy for extricating valuable highlights from face
III. SYSTEM DESIGN AND pictures. Fundamentally, this technique condenses the nearby
IMPLEMENTATION
structure in a picture by contrasting every pixel and its
This area subtleties how the framework is structured and neighbors.
devel-oped. Initial, two key segments of the framework,
//Create a LBH face recognizer object recognizer =
verification system and voice-based direction, are exhibited. cv2.createLBPHFaceRecognizer() //Train images and extract
At that point the whole keen mirror framework is portrayed. face features //Each user has a unique integer Id
Figure 3 shows the structure of the system.in an image. Face recognizer.train(faces, np.array(Id))) //Save extracted features
preparing is the way toward examining a few photos of a to face database recognizer.save
similar individual to become familiar with the highlights
explicit to the person. This procedure makes a database of the
known clients. Face forecast is the way toward distinguishing
a known or pre-enlisted individual in another picture. With the
help of OpenCV, two modules, to be specific client enrollment
and client confirmation, were created in the verification
system. Figure 4 shows the general methodology of face
acknowledgment and enlistment.

User Registration - face detection and training

This module was actualized by taking pictures with a

Raspberry camera. The pictures are passed into an OpenCV
course classifier to distinguish human countenances. A falling
clas-sifier is a model that is pre-prepared with a few hundred
face pictures, called positive models, and negative models -
discretionary pictures of a similar size. An example code bit is
appeared as pursues:
//Import opencv Library
import cv2
//Loading required cascading classifiers
face = cv2.CascadeClassifier(‘frontalface.xml’) //Load image and
convert to grayscale mode
img = cv2.imread(‘image.jpg’)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) //Face
detection in converted image
faces = face.detectMultiScale(gray, 1.3, 5) //Save image
containing faces to disc cv2.imwrite(‘image’+ str(Id))
A. Voice Based Command Wake Word Detection
The framework underpins communication among client and Like other voice-controlled frameworks, this venture actualizes
framework. Clients can control all capacities utilizing voice the utilization of a wake word. A wake word is a word or expression
directions, includ-ing enrollment, confirmation, looking, and marking that is articulated by the client to inform the framework that he is
out. The usage highlights three essential advancements: Google's prepared to give a guidance. The, endless supply of the wake word,
starts to sit tight for guidelines. The keen mirror utilizes the
"Discourse to-Text [12], Google's "Custom Search Engine [13], and
expression "shrewd mirror" as the wake work. When the shrewd
Amazon's "Alexa Voice Service (AVS)" [14].
mirror recognizes the wake word, it tunes in for the accompanying that are promptly accessible to clients of Alexa empowered gadgets.
guidance, for example, login, logout and so on. Maybe the most amazing ability of AVS is that it permits the
Pass phrase Detection formation of custom aptitudes that can be utilized to improve the
extravagance of client experience. The quantity of existing custom
Pass phrase detection refers to accepting and validating a abilities runs into the many thousands
voiceprint from the user. This happens before user enrollment and
user authentication. This task made another Alexa ability called "Shrewd mirror". This
name speaks to what the utilizations state to show that they need to
Registration Module collaborate with the expertise. The expertise is made by visiting
https://developer.amazon.com/alexa-abilities unit and choosing to
The enlistment module works somewhat unique in relation to the make another Alexa aptitude and posting the aims for the new
two cases depicted previously. This module gives the client a brief expertise. Plans are the different reasons/aims a client could have for
and sits tight for a reaction. The module initially requests that the conjuring the expertise. When the goals have been determined, the
client choose and state his username. It at that point tunes in to the clients need to make a cooperation model. A collaboration model
username before mentioning a book translation from the Speech-to- speaks to how clients communicate with the new ability. The model
Text API. The client is provoked to affirm if the content translation is contains different proclamations a client may state to cooperate with
precise after which the framework sits tight for a reaction, either
every one of the goals in the new expertise.
"yes" or "no". Once more, a book translation of the clients reaction is
gotten. On the off chance that it is "yes", the framework continues to
When the model has been fabricated, the Alexa gadget (e.g., the
finish the enrollment. On the off chance that it is "no", the framework
brilliant mirror) associated with the amazon engineer record would
comes back to the initial step to gather username.
have the option to get to the aptitude. An ability named "keen mirror"
for example may contain a plan called "getTime" and an example
Google Custom Search Engine: it is a free stage that empowers
articulation like "disclose to me the time". In the event that a client
designers to perform modified pursuits dependent on Google Search.
says "Alexa, request that shrewd mirror disclose to me the time", the
This administration likewise enables clients to penetrate down to just
Alexa Voice Service would perceive that the client is conjuring the
those genuine site pages that are pertinent. With a functioning
brilliant mirror ability.
Google account, clients can make a free custom web crawler by
visiting the Google custom hunt
Https://cse.google.com/cse to get an API key and custom internet
searcher ID. A Python piece is told underneath to show the best way
to question a custom web crawler.
query = ‘python programming language’; sv =
build(‘customsearch’, ‘v1’,
developerKey = key);
result = sv.cse().list(q = query, cx = cse_id,
searchType = ‘image’, num = 5, fileType =
‘png’,’).execute();

The above code scrap makes an exhibit called "result" which

contains the URLs to five pictures of PNG type that match the
depiction of Python programming language. This undertaking
utilized a custom web search tool to get sight and sound substance
dependent on client voice search inquiries, which is talked about in
detail in the following segment

Amazon Alexa Voice Service (AVS): AVS is a plat-structure that

Amazon has given to permit engineers incorporate "Alexa" into their
tasks. With an AVS empowered item, clients can stream media, get
refreshes on climate, traffic, and news, just as ask general
information inquiries. Alexa (the cerebrum behind the AVS
framework) is activated when a client says the wake word, "Alexa".
When the gadget identifies the wake word, it starts to tune in for a
solicitation. The solicitation is then handed-off to cloud based voice
administration. It is along these lines a necessity to have a working
Internet association. The cloud-based voice administration deciphers
the solicitation and sends an order back to the gadget or for this
situation, the savvy reflect framework.

Extra/custom abilities - Skills are directions that Alexa can

process. They are basically applications however custom-made
explicitly for AVS. Alexa gives a wide scope of custom aptitudes
 This section also contains an indicator showing the status of
the voice assistant. Green means that the voice assistant is
enabled while red means that it is disabled. The voice assistant
is automatically disabled after five minutes of inactivity. Once
this happens, the user needs to be authentication again to
continue using the system.
3) Section C: This segment handles the showcase of
multi-media content (e.g., recordings, pictures, and sound
documents). This segment works with an audience that
always checks the AWS basic line for new demands. For
each new solicitation, the audience gets interactive media
documents coordinating the solicitation utilizing the Google
custom web crawler portrayed previously. Area C in Figure
10 shows a video being played. This is in light of the
accompanying solicitation "Alexa, request that the savvy
Smart Mirror Software Display
reflect play pacific edge."

4) Section D: This segment shows news features pulled

B. Smart Mirror Software from 16 news stations. A news source is shown like
clockwork. The code extricate underneath shows how this
The shrewd mirror programming is created utilizing python
segment was executed.
master gramming language and PyQt 4 [17]. PyQt is a Python
authoritative of the cross-stage GUI toolbox Qt, executed as a //Declare a counter to track progress count = 0;
Python module. The savvy reflect has a presentation that is //Load RSS feed URLs in list
comprised of four unmistakable segments as enclosed Figure 10 l =[‘http://rss.cnn.com/rss/cnn_topstories.rss’,
‘http://www.tmz.com/rss.xml’];
1) Section A: This section displays weather and location //Use feedparser to load feeds
information. The data is obtained from two APIs, namely content = [];
f = feedparser
freegeoip [18] and openweathermap [19]. Below is a code while count < len(l):
snippet showing how to extract data using the APIs. try:
temp = f.parse(l[count])
//Call API to fetch city and region name
if len(temp[‘entries’]) > 0:
url = ‘http://freegeoip.net/json/’ + ip; r = requests.get(url); content.append(temp)
js = r.json(); count = count + 1
except:
//Parse response to obtain city and region response =
count = count + 1
(str(js[‘city’]) + ‘,’ +
pass
js[‘region_name’]);
//Get title/name of first RSS station title = (content[0]‘feed’]
[‘title_detail’]
//Call API to fetch temperature and weather conditions
[‘value’]);
//Data retrieval using developer ID (APPID)
//Get the first feed for the first station feed = (content[0]
r= requests.get(‘http://api.openweathermap.
[‘entries’][0]
org/data/2.5/weather?q = ’ + str(js[‘city’]) + [‘title_detail’][‘value’]);
‘&APPID=b2a86......’)
result = r.json();
//Parse response to get temperature in Kelvin temperature =
(str(int(math.ceil((result https://cse.google.com/cse to obtain an API key and custom
[‘main’][‘temp’]) - 273.15))) + u‘\u00b0’); //Parse response to
get weather conditions condition = result[‘weather’][0][‘description’] search engine ID. A Python snippet is shown below to
demonstrate how to query a custom search engine.
query = ‘python programming language’; sv =
2) Section B: This section shows how time and date infor- build(‘customsearch’, ‘v1’,
mation is retrieved from the computer system. A code snippet developerKey = key);
is as follows: result = sv.cse().list(q = query, cx = cse_id,
searchType = ‘image’, num = 5, fileType = ‘png’,
//Assign current system date and time
now = datetime.datetime.now();
//format value to get current time & date
time = (now.strftime(‘%-H:%M %p’)); https://cse.google.com/cse to obtain an API key and custom
date = (now.strftime(‘%b %d, %Y’)); search engine ID. A Python snippet is shown below to
my_date = date.today();
day = (calendar.day_name[my_date.weekday()]);
demonstrate how to query a custom search engine.

IV. LITERATURE SURVEY S.NO PROJECT

1. Voice Controlled Smart Mirror with Google assistant Allows the user to The system is not
Multifactor Authentication IoT control home designed for
appliances using interaction 6.
at Automation
smart mirror display deeper levels
through voice
2. A Mobile-Programmable Smart Mirror for IoT Allows the users to The system is not
Ambient IoT Environments create and manage capable of 7. Smart Mirro
their profiles based controlling all the interactive d
on what they wish to activities through
display on the smart mobile
interactive mirror
3. Smart Interactive Mirror Display IoT Reduces time needed User would8.not Smart Mirro
in a user’s daily able to access the assistant
routine and provides full features of
a merger of user and smart mirror
technology
4. The Future of Smart Dressing Mirror: An open IoT Solves serious They do not
innovation concept video difficulties to make provide quick
9. Smart Mirro
decision on which access to and vigilanc
clothes to wear or information about
match their clothes the products in the
every time, every store, available
day models and10.sizes Smart Mirro
5. Google Assistant Controlled Home IoT Proposes a cost Accurate results
Automation Google Assistant effective voice are not provided
and Android controlled (Google by the assistant
Assistant) home
automation
V. CONCLUSION  Voice Controlled Smart Mirror with
Multifactor Authentication, Adokiye
Charles Njaka, Na Li, and Lin Li,
Feb 2019

 A Mobile-Programmable Smart Mirror

for Ambient IoT Environments,
Mohammed Ghazal, Tara Al Hadithy,
Yasmina Al Khalil, Dec 2017
 Smart Interactive Mirror Display,
Varsha Singh & Devi Singh, Feb 2018
 The Future of Smart Dressing Mirror:
An open innovation concept video,
Guangyu Gao & Chenchen Bai,
Mar 2016
 Google Assistant Controlled Home
Automation, Manish Prakash Gupta,
May-2018
The system would be able to deliver promising
Output i.e every appliances connected to the
arduino can be controlled through the voice
command and the mirror would be able to deliver
the notifications , texts , to do list and opening of
mobile applications through the voice command.
And even surveillance can be done through the
mirror.

VI. REFERENCES