> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 1

Sixth Sense Technology:

Applications and Comparison
Allan Shivji
Student: BE Computer Engineering
St. John College of Engineering and Technology
Mumbai, India
[email protected]
Neeta Patil
Asst. Professor: Dept. of Computer Engineering
St. John College of Engineering and Technology
Mumbai, India
[email protected]

But, in today’s modern technology world based

Abstract— We grew up interacting with the physical objects environment we are confused to make best decision based on
around us. And there are large number of them that we use these five senses. Today’s technology is a revolutionary way
everyday. These objects are more fun to use than our computing to interface digital information with physical world. Although
devices. Taking about objects, another thing that get’s attached the miniaturization of computing devices allows us to carry
with objects is gestures; how we manipulate these objects & how computers in our pockets, keeping us continually connected to
we use them in everyday life. These gesture are used to interact
the digital world, there is no link between our digital devices
with objects as well as to interact with each other. So using Sixth
Sense Technology we get to know about everyday objects and and our interactions with the physical world. Information is
gestures and how using these objects can leverage (power to confined traditionally on paper or digitally on a screen. Sixth
influence situation) our interaction with digital world. This Sense Technology helps in bridging this gap by giving out the
technology helps us to build bridge between physical world and digital information into tangible world and giving us the
the digital world. It brings part of digital world to physical world, opportunity to interact with this information using out natural
some part of digital information in brought into physical world hand gestures. It can be used in colleges, supermarkets,
which can be manipulated in physical world [1]. Anybody who is schools, map navigator etc. Augmented reality [2] is the
equipped with sixth sense device can use its applications at any concept which makes a bridge between physical world and
place and any time. Working of Sixth sense technology, its
digital world. Augmented reality joins virtual world with the
applications, advantages, future scope and different
implementation approaches are presented in this paper. real world, also the fictitious (not true/real) world can be
experienced. The concept of augmented reality can be
Keywords—Augmented reality, Computer-vision, Fiducially, combined with sixth sense in order to have better gadgets that
Image processing, Visual information. make us machine free. This Sixth Sense device helps in
integrating the information and technologies in our everyday
I.
INTRODUCTION life. Hence, it can make us to have the right information. Sixth
Sense is a wearable gestural interface device that helps us to
Humans, for so many millions of years have got five senses
share digital information with the physical world to use natural
namely, eye, ear, nose, tongue and touch as shown in Fig. 1.
hand gestures. This Sixth Sense device was developed by
which collet information from the happenings around us, that
Pranav Mistry [3], a Ph. D student at Fluid Interfaces Group at
information helps us to make best decision and choose right
the MIT Media Lab. This Sixth Sense prototype has a lot of
choice or action which will in fact benefit us.
applications that demonstrate or shows the usefulness,
viability and flexibility of the system acting as the computer
and your connection to the cloud and all the information
stored on the web. The Sixth Sense recognizes the objects
around you, displaying information automatically and letting
you access it in any way you want and in the simplest way
possible. This technology is based on hand gesture
recognition, image capturing, processing, and manipulation,
etc. Pranav Mistry won the “Invention of the Year 2009”- by
Popular Science. The sixth sense prototype is made using very
common and easily available equipment’s like pocket
projector, a mirror, mobile components, colored caps and a
Fig. 1. Five senses camera. Our natural hand gestures are used to interact with the
digital data. Sixth Sense technology is the science of

978-1-5090-5256-1/16/$31.00 2016
c IEEE 433
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 2

tomorrow with the aim of connecting the digital world with III.
WORKING OF SIXTH SENSE DEVICE
the physical world seamlessly, eliminating hardware devices. The Sixth Sense device is pendant like wearable device [7]. It
consists of camera, mirror, projector and is connected to Smart
II.
ARCHITECTURE OF SIXTH SENSE DEVICE phone. The camera captures the gesture made by the user.
Sixth Sense device is made from a webcam, projector with Then the information is sent to the smart phone for processing.
attached mirror. All of these components are attached to Smart The projector then projects visual images via the mirror on the
phone [4] as shown in Fig. 2. surface. The surface can be anything like the wall, table or
palm of your hand. Hence, the entire world can be a screen
now.When user moves their hands to form different
movements with colored markers on the finger tips, the
camera captures these movements. Recognition of these
gesture is done by computer vision techniques. The markers
act like a tracker.
Basically, camera captures all the gestures one makes with
their hands and the projector helps in projecting the
information on any desired surface. Here the use of mirror is
very important as the projector dangles downwards from the
neck. The color caps are used so that is becomes easy for the
Fig. 2. Architecture of Sixth Sense device software to differentiate between fingers which needs to
perform different operations for the applications. The Sixth
A.
Camera Sense software/code looks for the location of the color
Camera captures the user’s hand gestures [6]. It then sends markers by making use of computer vision techniques. One
the data to the Smart phone. Camera looks and tracks hand can have any number of hand gestures and movements as long
gestures and physical objects using computer-vision based as they are all reasonably identified and differentiated for the
technique. Here camera is the digital eye which connects to system to interpret it, preferably through unique and varied
the outside world in the digital format. fiducially.

B.
Smart Phone IV.
APPLICATIONS

The Smart phone in our pockets get or receives data, text, There are variety of applications implemented using Sixth
voice, etc. anywhere and anyone via the internet. The Smart Sense Technology. Some of them are as follows:
phone here acts as interpreter between all hardware
components camera, projector, etc. Smart phone runs the Sixth A.
Calculator
Sense software and handles all the processing and the You can use the Sixth Sense to project a keypad onto your
connection to the internet. hand as shown in Fig. 3., then we can use that virtual keypad
to calculate values. Not only this, it can be made to project
C.
Projector keyboard as well, hence eliminating use of physical keyboard.
The data that is processed by the software is anticipated on
any surface. The projector ventures the visual data
empowering surfaces and physical items to be utilized as
interfaces. It projects visual information enabling surfaces,
walls and physical objects around us to be used as interfaces.

D.
Mirror
The usage of the mirror is important as the projector
dangles pointing downwards from the neck to the ground
Fig. 3. Make a calculator
surface. Mirror reflects image on surface.
B.
Check Time
E.
Color caps/markers Check time, by drawing circle on wrist, then projector will
Colored markers placed at the tip of the user’s fingers using project watch on wrist as shown in Fig. 4. Hence there is no
simple computer vision techniques. It helps the webcam to more need to wear watches on hand, and also not wasting
track the movement of fingers. The movements and money on branded watches.
arrangements of these fiducials are interpreted into gestures
that act as interaction instructions for the projected application
interfaces.

434 2016 2nd International Conference on Contemporary Computing and Informatics (ic3i)
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 3

the Amazon store or any other website.

Fig. 4. Check time on wrist

Fig. 7. Book rating
C.
Drawing Application
Draw a picture on any desired surface just by moving the H.
Take Pictures
index finger in the air as seen in Fig 1.4.3. Hence, using this Take a picture just by making photo frame or rectangle by
technology we can do drawings without use of pencil or color fingers. The photo will be taken and it will be placed in smart
and paper. And also be able to save our drawings. phone’s memory. A person can resize and look for the photo
whenever needed by projecting it on hand or any surface.
D.
Viewing Maps
We can view maps and also navigate. We can also zoom
into the map just by making gestures like pinch in and out as
seen in Fig. 5. So there will be no need to carry huge paper
maps along with us when we plan for a trip.

Fig. 8. Taking Pictures

I.
Viewing weather information

We can also view the current day’s weather information just
Fig. 5. Viewing Maps
by showing map on paper or any surface to the camera. The
weather information is searched from the internet by the smart
E.
Zooming Features phone.
We can zoom in and out by hand gestures by pinching
fingers of both hands to zoom in and make both hands closer
to each other to zoom just as shown in Fig. 6.

Fig. 9. Weather Information

V.
ADVANTAGES
Fig. 6. Zoom in and out The interface of sixth sense is easy to use and it classifies
digital data into physical world. Hence, the whole world can
F.
Get Product Information
be used as an interface making the whole world your machine
Get information about product in our hand and Amazon [9]. It is easy to carry device, supports multi-touch, easily
gives rating to make a right decision to buy. Just by showing affordable, world and the information are connected to each
the product in front of camera, the camera capture it’s image other, data can be accessed in real world, helps us in making
and then in processing it looks for the information from the right decisions, it acts a computer as well as saves our time
Amazon website. Therefore, we don’t have to open our searching information on the internet.
internet browsers every time to look for the information. Sixth Sense uses hand gestures to interface with digital data.
Information is directly displayed from digital world to
G.
Get Book Information physical world. It is a wearable gesture based device which
Get the information of the book and it’s rating from the can turn any surface an intuitive presentation. It is compact
Amazon book store or any other website. This device would and easy to wear in our neck. This device can be used by
prove a blessing to the book lovers. Just by showing the cover anybody, even the one who don’t know how to use a computer
page of the book to the camera, the camera captures its image mouse or console or keyboard. In today’s modern world it will
and while processing it searches the rating for the book from be easy to catch photographs just by using fingers.

2016 2nd International Conference on Contemporary Computing and Informatics (ic3i) 435
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) < 4

VI.
FUTURE SCOPE hardware -ts are been favorable

Consider a world where Sixth Sense Technology is applied requirement made. approach.
everywhere. In educational field, the hardware components s, on the
used can be minimized. Paper usage and electricity can be handheld
minimized to greater extent. Anybody can make use of a wall device.
or any desired surface to carry out the activities that are
performed on PC. It could be used to monitor agricultural
lands. In medical field it can be used to check the medicines
are genuine or not. People with visual challenge can make use REFERENCES
of this device to read books and recognize objects. Sixth Sense [1] The thrilling potential of Sixth Sense technology by Pranav Mistry
device is more advanced version on computer, and it enables https://www.youtube.com/watch?v=YrtANPtnhyg.
one to compute and use internet on any surface that we can [2] Monika Arora, Basic Principles of Sixth Sense Technology, VSRD-I
find in our surroundings. Implementing this technology in JCSIT, Vol. 2 (8), 2012
[3] Vishmita Yashwant Shetty, Vinayak Bhupendra Rai. (2014, Dec). Sixth
gaming field will let us to play games as if we are playing in Sense Technology. [Online]. 3(12), pp. 1068-1073. Available:
real world. Sixth Sense device can make the world magical https://www.ijsr.net/archive/v3i12/U1VCMTQ1Nzc=.pdf
[10]. [4] https://www.ijsr.net/archive/v3i12/U1VCMTQ1Nzc=.pdf
[5] http://ieeexplore.ieee.org/document/6508367/
[6] https://www.ijarcsse.com/docs/papers/Volume_5/2_February2015/
VII.
DIFFERENT IMPLEMENTATION APPROACHES OF SIXTH V5I2-0180.pdf.
SENSE [7] http://sse.saveetha.com/research/journals/cseitjournals.pdf.
[8] “Sixth Sense Technology and its Applications” -
http://sse.saveetha.com/research/journals/cseitjournals.pdf.
TABLE I. Comparing Approaches [9] http://sallu-sixthsense.blogspot.in/2011/10/sixth-sense-
advantagesand.html
Category Description Disadvantages [10] http://research.ijcaonline.org/volume75/number8/pxc3890514.pdf.
Advantages
A
[11] https://www.scribd.com/doc/47401399/ABSTRACT-for-Sixth-Sense-
Tech-PPT.
[12] Pranav Mistry, Sixth Sense. Fluid Interfaces Group, MIT Media Lab.
http://www.pranavmistry.com/projects/sixthsense/, 2009.
A Hardware Working is It requires [13] http://www.123seminarsonly.com/Seminar-Reports/025/66840620-
used are very much projecting Sixth- Sense-Technology.pdf.
Similar to camera, simple and surface, [14] http://www.billbuxton.com/input14.Gesture.pdf.
[15] http://www.ted.com/talks/pattie_maes_demos_the_sixth_sense.html.
Pranav’s mirror, easy to use. projected
approach projector, image gets
processing affected by
device like light and
smart phone moving
and color background.
markers.

B Recognizes Small in Glasses

hand, sound size, no need designed are
Google and touch. of projecting not favorable;
glasses surface; no it needs
colored further testing.
markers
needed.

C Same Design is Suffers from

hardware simple; bright
Laser needed as in surrounding
using laser is
Keyboard category A, light.
in addition favorable
laser is also than colored
used. markers.

D Same as Lot of Still lot of

screen size; hardware working is
Handheld entirely needed in
and software
Devices depends digital world;
upon improvemen it is least

436 2016 2nd International Conference on Contemporary Computing and Informatics (ic3i)