SIXTH SENSE TECHNOLOGY

A SEMINAR REPORT

Submitted by

Animesh Kumar Tilak (1605340)

in partial fulfillment for the award of the degree

of

BACHELOR OF TECHNOLOGY
IN
COMPUTER SCIENCE & ENGINEERING
At

SCHOOL OF COMPUTER ENGINEERING

KIIT Deemed to be University
BHUBANESWAR
April 2020

i
 KIIT Deemed to be University
School of Computer Engineering
Bhubaneswar, ODISHA 751024

CERTIFICATE

This is to certify that the SEMINAR REPORT entitled SIXTH SENSE TECHNOLOGY, is
a bonafide work carried out by ,in the partial fulfilment of the requirement for the award of
Degree of Bachelor of Engineering (Computer Science & Engineering) at KIIT Deemed to be
University, Bhubaneswar.

Date: 05 / 04 /2020

Dr. Niranjan Kumar Ray

Asst. Professor
School of Computer Science Engg.
(Seminar Supervisor)

ii
 ABSTRACT

Sixth Sense technology is a technology with which a system could be trained to recognize and percept real
world objects and react as desired. Sixth Sense technology bridges the gap between the physical world and the
digital world, bringing intangible, digital information out into the tangible world, and allowing us to interact
with this information via natural hand gestures. Sixth Sense Technology is implemented in “Sixth Sense/WUW
(wear your world)” using gesture recognition, augmented reality, computer vision and radio frequency
identification. It’s just born concept which allows user to connect with the internet seamlessly. Without use of
keyboard, mouse we can see videos access, change, move data simply. But this concept bottle necks lead to
modification of the same by using commands instead of gestures. Sixth Sense technology could be integrated
with voice recognition. Bluetooth device and laser projectors could be used.

The Sixth Sense technology contains a pocket projector, a mirror and a camera contained in a pendant-like,
wearable device. Both the projector the camera and sensors are connected to a coding device (in our case-a
laptop) in the user’s pocket. The projector projects visual information enabling surfaces, walls and physical
objects around us to be used as interfaces; while the camera recognizes and tracks users' hand gestures and
physical objects using computer-vision based techniques. The software program processes the video stream data
captured by the camera and tracks the locations of the colored markers (visual tracking fiducially) at the tips of
the user’s fingers. The movements and arrangements of these fiducially are interpreted into gestures that act as
interaction instructions for the projected application interfaces. The Sixth Sense prototype is used to implement
several applications that have shown the usefulness, viability and flexibility of the system.

iii
 TABLE OF CONTENTS
A. CERTIFICATE..................................................................................................................... ii

B. ABSTRACT......................................................................................................................... iii

1. INTRODUCTION.................................................................................................................. v

2. BRIEF HISTORY ON SELENIUM...................................................................................... vi

3. TESTING TOOLS IN MARKET......................................................................................... vii

4. PERFORMANCE COMPARISION OF SELENIUM AGAINST OTHER TOOLS

4.1 SELENIUM vs. HP QTP/UFT............................................................................... viii
4.2 SELENIUM vs. APPIUM...................................................................................... viii

5. COMPONENTS OF SELENIUM
5.1 SELENIUM INTEGRATED DEVELOPMENT ENVIRONMENT(IDE)........... ix
5.1.1 INSTALLING SELENIUM IDE.............................................................. ix
5.1.2 PROS & CONS OF SELENIUM IDE...................................................... ix
5.2 SELENIUM REMOTE CONTROLLER(RC)......................................................... x
5.2.1 WORKING................................................................................................ x
5.2.2 PROS & CONS OF SELENIUM RC...................................................... xii
5.3 SELENIUM WEBDRIVER................................................................................... xii
5.3.1 WEBDRIVER & THE SELENIUM SERVER....................................... xii
5.3.2 LOCATING WEB ELEMENTS/UI ELEMENTS.................................. xiii
5.3.3 HANDLING POPUP DIALOGS............................................................. xv
5.3.4 PROS & CONS OF SELENIUM WEBDRIVER..................................... xv
5.4 SELENIUM GRID................................................................................................... xv

6. ADVANTAGES OF SELENIUM........................................................................................ xvii

8. CONCLUSION.................................................................................................................... xviii

REFERENCES

iv
 CHAPTER 1 - INTRODUCTION

We use our five natural senses to perceive any information; that information helps us make decisions and
choose the right actions to take. But arguably the most useful information that can help us make the right
decision is not naturally perceivable with our five senses, namely the data, information and knowledge that
mankind has accumulated about everything and which is increasingly all available online. Although the
miniaturization of computing devices allows us to carry computers in our pockets, keeping us continually
connected to the digital world, there is no link between our digital devices and our interactions with the physical
world. Information is confined traditionally on paper or digitally on a screen. Sixth Sense Technology bridges
this gap, bringing intangible, digital information out into the tangible world, and allowing us to interact with this
information via natural hand gestures. ‘Sixth Sense’ frees information from its confines by seamlessly
integrating it with reality, and thus making the entire world your computer. WUW was developed by Pranav
Mistry, a Ph. D student at Fluid Interfaces Group at the MIT Media Lab. The Sixth Sense prototype implements
several applications that demonstrate the usefulness, viability and flexibility of the system acts as the computer
and your connection to the Cloud, all the information stored on the web.

Sixth Sense recognizes the objects around you, displaying information automatically and letting you access it in
any way you want, in the simplest way possible. The device brings us closer to reality and assists us in making
right decisions by providing the relevant information, thereby, making the entire world a computer. The
technology is mainly based on hand gesture recognition, image capturing, processing, and manipulation, etc.
The software of the technology uses the video stream, which is captured by the camera, and also tracks the
location of the tips of the fingers to recognize the gestures. This process is done using some techniques of
computer vision. He invented ‘ Sixth Sense / WUW ( Wear UR World)’ which is a wearable gestural , user
friendly interface which links the physical world around us with digital information and uses hand gestures to
interact with them. This technology is a revolutionary way to interface the physical world with digital
information. Modern technologies include the touch screen techniques which is used widely and it makes ease
of operation and saves utilization time.

This deals with the latest technology called the sixth sense. It’s a wearable interface that augments the physical
world around us with the digital information. It’s just born concept which allows user to connect with the
internet seamlessly. Without use of keyboard, mouse we can see videos access, change, move data simply .But
this concept bottle necks lead to modification of the same by using commands instead of gestures. Speech IC is
used as a database for commands which will be initially trained for storage. It performs the corresponding
commands accessing the operation from the mobile device connected to it and action is projected using a
projector over any surface.

v
 CHAPTER 2 – ORIGIN OF IDEA

This technology is a revolutionary way to interface the physical world with digital information. Modern
technologies include the touch screen techniques which is used widely and it makes ease of operation and saves
utilization time. It’s a hi-tech device seamlessly integrate Analog information with our every day physical
world. The voice is directly performed into operation within fractions of seconds, and the action is projected on
the surface. It’s a portable device and eases the operation which we regularly perform. Basically the sixth sense
technology concept involves the use of hand gestures .the finger tip will contain colored markers and hence
gestures performed will be captured by the camera. Then it’s given to the mobile device for the corresponding
action to be performed. The action is projected on the surface through the projector. Software algorithms and
computer vision technologies will be used to enable the action from the mobile device for the corresponding
gesture captured in the camera. This gesture based technology is used for variety of applications like performing
basic actions, locating points in the map, watching video in news paper, dialing number in hand etc. The slight
modification of this method leads to the use of commands that is analog information into real world. The analog
data is converted into digital and performed as action, as all times the hand gestures cannot be used. This was
how the wearable device is fit to the human body. Here color markers are used in the finger tips. In our
technology we use commands for performing the same operations. Many high technology speech integrated
circuits evolved which makes our operation enhanced with more advanced features. To ensure accurate gesture
recognition and an intuitive interface a number of constraints are applied. A region in the front of the projection
screen is defined as the active zone and the gestures are ignored, if the gestures are performed out of this area.
Gestures are also defined by a set start posture, end posture and dynamic motion between the start and end
postures. Perhaps the use of gestures is most powerful when combined with other input modalities, especially
voice. Allowing combined voice and gestural input has several tangible advantages. The first is purely practical-
ease of expression .Ease corresponds to the efficiency with which commands can be remembered and
expressiveness, size of command vocabulary.

2.1 WHAT IS SIXTH SENSE?

Sixth sense in scientific or non scientific terms defined as Extra Sensory Perception. It involves the reception of
information not gained through any of the five senses. Namely the data, information and knowledge that
mankind has accumulated about everything available online.

Figure 2.1 six senses

2.2 EARLIER SIXTH SENSE PROTOPYPE

Steve Mann is the father of Sixth Sense Technology who made of wearable computer in 1990. He implemented
the neck worn projector with a camera system. Maes’ MIT group, which includes seven graduate students, were
vi
thinking about how a person could be more integrated into the world around them and access information
without having to do something like take out a phone. They initially produced a wristband that would read a
Radio Frequency Identification tag to know, for example, which book a user is holding in a store. They also had
a ring that used infrared to communicate by beacon to supermarket smart shelves to give you information about
products. As we grab a package of macaroni, the ring would glow red or green to tell us if the product was
organic or free of peanut traces — whatever criteria we program into the system. They wanted to make
information more useful to people in real time with minimal effort in a way that doesn’t require any behavior
changes. The wristband was getting close, but we still had to take out our cell phone to look at the information.
That’s when they struck on the idea of accessing information from the internet and projecting it.

Figure 2.2: Earlier prototype

2.3 RECENT PROTOPYPE

The Sixth Sense prototype implements several applications that demonstrate the usefulness, viability and
flexibility of the system acts as the computer and your connection to the Cloud, all the information stored on the
web. The key here is that Sixth Sense recognizes the objects around you, displaying information automatically
and letting you access it in any way you want, in the simplest way possible. The device brings us closer to
reality and assists us in making right decisions by providing the relevant information, thereby, making the entire
world a computer. The technology is mainly based on hand gesture recognition, image capturing, processing,
and manipulation, etc. This process is done using some techniques of computer vision. He invented ‘Sixth
Sense / WUW (Wear UR World)’ which is a wearable gestural , user friendly interface which links the physical
world around us with digital information and uses hand gestures to interact with them.

Figure 3.3 Recent prototypes

vii
 CHAPTER 3 – DETAILED DESCRIPTION

3.1 COMPONENTS
The hardware components are coupled in a pendant like mobile wearable device. The components are:
1. Camera
2. Projector
3. Mirror
4. Mobile component
5. Color markers
6. Microphone

CAMERA
Camera captures an object in view and tracks the user’s hand gestures. It sends the data to smart phone. Camera
recognizes and tracks user's hand gestures and physical objects using computer-vision based techniques. Sixth
Sense system implements a gestural camera that takes photos of the scene the user is looking at by detecting the
‘framing’ gesture. It acts as a digital eye, connecting you to the world of digital information.

PROJECTOR
A tiny LED projector displays data sent from the smart phone on any surface in view-object, wall, or person.
The projector projects visual information enabling surfaces, walls and physical objects around us to be used as
interfaces. The projector projects visual information enabling surfaces, walls and physical objects around us to
be used as interfaces.

MIRROR
The mirror reflects the projection coming out from the projector and thus helps in projecting onto the desired
locations on walls or surfaces. The user manually can change the tilt of the mirror to change the location of the
projection. For example in application where the user wants the projection to go on the ground instead of the
surface in front, he can change the tilt of the mirror to change the projection.

MOBILE
The Sixth Sense system uses a mobile computing device in user’s pocket as the processing device. The software
program enabling all the features of the system runs on this computing device. This device can be a mobile
phone or a small laptop computer. The camera, the projector and the microphone are connected to this device
using wired or wireless connection.

MARKER
Color marker is at the tip of the user’s fingers. Marking the user’s fingers with red, yellow, green, and blue tape
helps the webcam recognize gestures. The camera tracks the movements of the color markers. The movements
and arrangements of these makers are interpreted into gestures that act as interaction instructions for the
projected application interfaces.

MICROPHONE
The microphone is an optional component of the Sixth Sense. It is required when using a paper as a computing
interface. When the user wants to use a sheet of paper as an interactive surface, he or she clips the microphone
to the paper. The microphone attached this way captures the sound signals of user’s touching the paper.

viii
 CHAPTER 4 – WORKING OF SIXTH SENSE TECHNOLOGY
The hardware that makes Sixth Sense work is a pendant like mobile wearable interface. It has a camera, a
mirror and a projector and is connected wirelessly to a Bluetooth smart phone that can slip comfortably into
one’s pocket. The camera recognizes individuals, images, pictures, gestures one makes with their hands.
Information is sent to the Smartphone for processing. The downward-facing projector projects the output image
on to the mirror. Mirror reflects image on to the desired surface. Thus, digital information is freed from its
confines and placed in the physical world. The entire hardware apparatus is encompassed in a pendant-shaped
mobile wearable device. Basically the camera recognizes individuals, images, pictures, gestures one makes with
their hands and the projector assists in projecting any information on whatever type of surface is present in front
of the person. The usage of the mirror is significant as the projector dangles pointing downwards from the neck.
To bring out variations on a much higher plane, in the demo video which was broadcasted to showcase the
prototype to the world, Mistry uses colored caps on his fingers so that it becomes simpler for the software to
differentiate between the fingers, demanding various applications.

The software program analyses the video data caught by the camera and also tracks down the locations of the
colored markers by utilizing single computer vision techniques. One can have any number of hand gestures and
movements as long as they are all reasonably identified and differentiated for the system to interpret it,
preferably through unique and varied fiducially. This is possible only because the ‘Sixth Sense’ device supports
multi-touch and multi-user interaction.

The technology is mainly based on hand gesture recognition, image capturing, processing, and manipulation,
etc. The map application lets the user navigate a map displayed on a nearby surface using hand gestures, similar
to gestures supported by multi-touch based systems, letting the user zoom in, zoom out or pan using intuitive
hand movements. The drawing application lets the user draw on any surface by tracking the fingertip
movements of the user’s index finger.

Figure 4.2: working of the sixth sense

ix
 CHAPTER 5 – RELATED TECHNOLOGIES

Sixth Sense Technology is implemented in “Sixth Sense/WUW (wear your world)” using gesture recognition,
augmented reality, computer vision and radio frequency identification.

5.1 AUGMENTED REALITY

Augmented reality is a term for a live direct or indirect view of a physical real world environment whose
elements are augmented by virtual computer generated imagery. Augmented reality blurs the line between
what's real and what's computer-generated by enhancing what we see, hear, feel and smell. Augmented reality is
one of the newest innovations in the electronics industry. It superimposes graphics, audio and other sense
enhancements from computer screens onto real time environments. Augmented reality goes far beyond the
static graphics technology of television where the graphics imposed do not change with the perspective.
Augmented reality systems superimpose graphics for every perspective and adjust to every movement of the
user's head and eyes.

5.2 COMPUTER VISION

Computer vision is the science and technology of machines that see. It is concerned with the theory behind
artificial systems that extract information from images. An image is a huge array of gray level (brightness)
values of individual pixels. Taken individually, these numbers are almost meaningless, because they contain
very little information about the scene. A robot needs information like "object ahead", "table to the left", or
"person approaching" to perform its tasks. The conversion of this huge amount of low level information into
usable high level information is the subject of computer vision. Earlier algorithms were too computationally
expensive to run in real-time, but also required any type of memory and modeling.

5.3 GESTURE RECOGNITION

Gesture recognition is a topic in computer science and language technology with the goal of interpreting human
gestures via Mathematical algorithms. Gestures can originate from any bodily motion or state but commonly
originate from the face or hand. Current focuses in the field include emotion recognition from the face and hand
gesture recognition. The keyboard and mouse are currently the main interfaces between man and computer.
Humans communicate mainly by vision and sound, therefore, a man-machine interface would be more intuitive
if it made greater use of vision and audio recognition.

5.4 RFID
Radio Frequency Identification is basically an electronic tagging technology that allows the detection, tracking
of tags and consequently the objects that they are affixed to RFID is a technology that uses communication via
radio waves to exchange data between a reader and an electronic tag attached to an object, for the purpose of
identification and tracking. Some tags can be read from several meters away and beyond the line of sight of the
reader.

Figure 5.1:Components of RFID Figure 5.2:Hand gestures

x
 CHAPTER 6 – APPLICATONS & ADVANTAGES

6.1 APPLICATIONS
The basic operations such as enabling clock, inbox, browsing, searching gallery, calendar, seeing contact list etc
are performed regularly in the mobile every time. These operations can be stored as commands in the IC and
then can be accessed on the screen or over any surface using our technology within fractions of seconds.

6.1.1 MAKE A CALL

The sixth sense can project the keypad onto one’s hand which can be used as a virtual screen to make a call.
You can use the Sixth Sense to project a keypad onto your hand, and then use that virtual keypad to make a call.
Calling a number also will not be a great task with the introduction of Sixth Sense Technology. No mobile
device will be required, just type in the number with your palm acting as the virtual keypad. The keys will come
up on the fingers. The fingers of the other hand will then be used to key in the number and call.

6.1.2 CHECK THE TIME

When you draw a circle on your wrist, a virtual watch appears that gives you the correct time. Sixth Sense all
we have to do is draw a circle on our wrist with our index finger to get a virtual watch that gives us the correct
time. The computer tracks the red marker cap or piece of tape, recognizes the gesture, and instructs the projector
to flash the image of a watch onto his wrist.

6.1.3 GET PRODUCT INFORMATION

The sixth sense uses image recognition or a marker technology to recognize the products we pick up and gives
us the information on those products. For example, if you're trying to shop "green" and are looking for paper
towels with the least amount of bleach in them, the system will scan the product you pick up off the shelf and
give you guidance on whether this product is a good choice for you.

6.1.4 TAKE A PICTURE

If you fashion your index finger and thumbs into a square (“framing” gesture), the system snaps a photo at that
time. After taking the desired number of photos, we can project them onto a surface to view them.

6.1.5 CALL UP A MAP

It is helpful to call up the map of our choice and then use our thumbs and index fingers to navigate the map.
The sixth sense also implements map which lets the user display the map on any physical surface and find his
destination and he can use his thumbs and index fingers to navigate the map, for example, to zoom in and out
and do other controls.

6.1.6 FEED INFORMATION ON PEOPLE

It helps to display the relevant information about a person we are looking at. Sixth Sense also is capable of "a
more controversial use”. When you go out and meet someone, projecting relevant information such as what they
do, where they work, and also it could display tags about the person floating on their shirt. It could be handy if
it displayed their facebook relationship status so that you knew not to waste your time.

6.2 ADVANTAGES
Portable, supports multi-touch and multi-user interaction, connectedness between world and information, cost
effective, data access directly from machine in real time, mind map the idea anywhere, assists us in making
right decisions, supports multi touch and multi user interaction, the device serves the purpose of a
computer plus saves time spent on searching information. Sixth Sense also recognizes user’s freehand
gestures (postures) and saves electricity.

xi
 CHAPTER 7 – FUTURE ENHANCEMENTS

Imagine the world where Sixth Sense Technology is applied everywhere. In educational field, the number of
hardware components could be reduced. Usage of papers and electricity could decrease. Students could use any
wall or any surface wherever they are to carry out activities that are done in a PC. Security will be assured for
everyone. It could be helpful in rendering defense services. In medical field, it could be implied to check the
genuinely of drugs. It could be implemented to monitor the agricultural lands. Blind people could be able to
read books and recognize objects. It could be used for the betterment of handicapped people. Sixth sense could
make the world magical.

xii
 CHAPTER 8 - CONCLUSION

Sixth Sense technology recognizes the objects around us, displaying information automatically and letting us to
access it in any way we need. The Sixth Sense prototype implements several applications that demonstrate the
usefulness, viability and flexibility of the system. Allows us to interact with this information through natural
hand gestures. The potential of becoming the ultimate "transparent" user interface for accessing information
about everything around us. Currently the prototype of the device costs around $350 to build. It could change
the way we interact with the real world and truly give everyone complete awareness of the environment around
us. The Sixth Sense prototype implements several applications that demonstrate the usefulness, viability and
flexibility of the system. It will definitely revolutionize the world.

The Sixth Sense software will be open source. As far as this seems to be a little set of items, there will not be
user interfaces or much advanced programs for the users. There will be much harder and secured coding inside
the device to make sure the security of the software. It will be interesting to know the new language for coding
for a sixth sense device

xiii
 REFERENCES

[1] Kirishima, T. Sato, K. Chihara, K.Dept. of Electr. Eng., Nara Nat. Coll. of Technol., Japan Robotics,
“Gesture Spotting and Recognition for Human–Robot Interaction”, IEEE Transactions on Volume: 23,
Issue:2 pp256 – 270., April 2007.

[2] Alon,J.Athitsos, V.Quan, YuanSclarof,” A Unified Framework for Gesture Recognition and
Spatiotemporal Gesture Segmentation”, IEEE transactions on Pattern Analysis and Machine
Intelligence, Volume: 31, Issue:9 pp 1685 - 1699 ., Sept. 2009.

[3] Gomez, A.M. Peinado, A.M. Sanchez, V. Rubio, A.J.dept. eoria de la Senal,” Recognition of coded
speech transmitted over wireless channels Wireless Communications”, IEEE Transactions on Volume:
5, Issue: 9, pp-2555 – 2562., Sept. 2006.

[4] Evans, J.R. Tjoland, W.A. Allred, L.G.Ogden Air Logistics Center, Hill AFB,” Achieving a hands-free
computer interface using voice recognition and speech synthesis”, IEEE Volume: 15, Issue:1, pp 14 16.,
Jan 2000.

[5] Pelaez-Moreno, C. Gallardo-Antolin, A. Diaz-de-Maria,” Recognizing voice over IP: a robust front-end
for speech recognition on the world wide webMultimedia”, IEEE Transactions on Volume: 3, Issue:2,
pp-209 – 218., Jun 2001.

xiv