Available online at www.sciencedirect.

com

Procedia Engineering 41 (2012) 1169 – 1175

International Symposium on Robotics and Intelligent Sensors 2012 (IRIS 2012)

A smart watch with embedded sensors to recognize objects, grasps and

forearm gestures
Elisa Morgantia,*, Leonardo Angelinib, Andrea Adamia, Denis Lalannec, Leandro Lorenzellia,
Elena Mugellinib
a
Fondazione Bruno Kessler, via Sommarive 18, Trento Italy
b
University of Applied Sciences of Western Switzerland, Boulevard de Pérolles 80, Fribourg 1700, Switzerland
c
University of Fribourg, Boulevard de Pérolles 90, Fribourg 1700, Switzerland

Abstract

This article proposes a smart watch for the recognition of gestures with objects. The watch is designed to embed different kinds of sensors
enabling several functionalities: the recognition of tagged objects by means of RFID technology; the recognition of gestures of the
forearm using inertial sensors; the recognition of fingers gestures, hand gestures and grasps by sensing the force exerted by tendons in the
wrist. Although the first two functionalities adopt common solutions already presented in the literature, for the third functionality we
propose a novel approach based on flexible force sensors on the wrist. These sensors are integrated in the belt of the watch and aim to
detect movements of tendons and changes of the shape of the wrist. A feasibility evaluation is presented and discussed. Results show that
force sensors on the wrist are able to retrieve important information about hand and finger movements, although this information can vary
depending on sensor placement. Further improvements for this system are also proposed.

© 2012 The Authors. Published by Elsevier Ltd. Selection and/or peer-review under responsibility of the Centre of
Humanoid Robots and Bio-Sensor (HuRoBs), Faculty of Mechanical Engineering, Universiti Teknologi MARA.
Open access under CC BY-NC-ND license.
Keywords: Smart watch; force sensors; gesture recognition; tangible gestures; grasps.

Nomenclature
EMG Electromyography
I2C Inter-Integrated Circuit: two-wire interface protocol
RFID Radio Frequency Identification

1. Introduction

Gesturing with objects is an interesting approach for the design of Human-Computer Interfaces. Hoven and Mazalek
have recently formalized the interest in tangible gestures in [1]; in this article they stressed the importance of exploring this
new almost undiscovered world.
Back in 1996, Fitzmaurice stated for the first time the importance of using objects to enhance the expressivity of gestures
and his PhD Thesis, Graspable User Interfaces [2], is considered the foundation of Tangible Interaction. In the following
years, many research projects focused on enhancing objects with sensors and computational capabilities, transforming them
in smart objects that become digital interfaces through a plethora of available commands. However, this approach often
implies that the user needs several smart objects, each with specified functionalities, or fewer objects but with several

*
Corresponding author. Tel.: +39-0461-314-442.
E-mail address: [email protected]

1877-7058 © 2012 Published by Elsevier Ltd. Open access under CC BY-NC-ND license.
doi:10.1016/j.proeng.2012.07.297
1170 Elisa Morganti et al. / Procedia Engineering 41 (2012) 1169 – 1175

functions. On one hand, having several objects augmented with sensors is not always possible, especially in mobile
scenarios. On the other hand, for the user is often difficult to remember several functionalities embedded in a single
multifunctional object. To overcome these limitations, we propose the utilization of “dumb” objects, to which either the
designer or the user can associate gestures (commands).
When objects have no intelligence on-board, gestures must be recognized externally. Two main approaches are possible:
an environmental approach, which uses cameras to recognize user’s gestures and objects, and a wearable approach, in which
technologies for gestures recognition are embedded in clothes or accessories worn by the user. Each approach offers some
advantages and some disadvantages. Privacy concerns arise each time cameras are adopted in public environment;
moreover, the problem of occlusion often needs to be addressed for visual gestures recognition. Lighting changes is another
well-known problem for gestures and objects recognition, which is resolved only in part with depth-cameras. On the other
hand, the wearable approach has limited computational capabilities and several constraints for power consumption.
However, nowadays the miniaturization of sensors and microprocessors allows the implementation of devices that often
have sufficient computational capabilities and battery life to operate on-board gesture recognition.
This article proposes a smart watch for the recognition of gestures with objects, focusing the analysis on three main
areas: the recognition of objects, the recognition of gestures of the forearm and the recognition of grasps, finger movements
and wrist rotation. These aspects are treated in Section 2, 3 and 4, respectively. For each section we provide an analysis of
related work and we present the chosen solution for our smart watch. In particular, in Section 4 a novel approach for the
recognition of grasps and finger gestures based on force sensors on the wrist is proposed.
In Fig. 1 a block diagram of the proposed watch are depicted. From an external point of view a large belt encloses the
antenna of the RFID reader. In the inside part of the belt, an array of force sensors allows the detection of tendons
movements. In the upper part, under the display, an ARM9 microcontroller running the Armadeus Linux embedded system
elaborates data coming from sensors. In particular the watch embeds a tri-axial accelerometer, magnetometer and
gyroscope, positioned in the center of the watch; the AD converter for force sensors; the RFID reader; a vibration motor for
haptic feedbacks and a speaker for audio feedbacks. A Bluetooth module allows the transmission of either raw data or
gestures recognized from the embedded microcontroller.

Fig. 1. Block diagram of the smart watch.

2. Object recognition

2.1. Related work

Recognizing objects in the hand is very useful for activity recognition during everyday life but also for gestural
interaction with objects [1]. The most popular approach is based on the RFID technology. An antenna and a RFID reader
can be easily integrated around the wrist and small tags can be attached to the objects that need to be recognized. The first
project that presented a glove equipped with a 125 kHz reader was intended to exploit context information for implicit
Human Computer Interaction [3]. Other projects integrated a 13.56 MHz RFID reader into a bracelet [4] or into a wristband
[5]. The aim of the iBracelet [4] was activity recognition, while in ReachMedia [5] the authors used different objects to
present to the user different menu in an audio interface controlled by free hand gestures. The reading range achieved by
these latter projects is about 10 or 12cm.
An alternative to RFID is integrating a camera in the lower part of the wrist. This concept has been implemented and
tested for high-level activity recognition by Maekawa et al. in [6].
 Elisa Morganti et al. / Procedia Engineering 41 (2012) 1169 – 1175 1171

2.2. Design and feasibility evaluation

Objects can be easily equipped with RFID tags of various dimensions, depending on their shapes. Obviously these tags
should be passive, thus they do not need a battery to communicate with the reader. Integrating an RFID reader in a smart
watch is quite simple, at least at a prototype level, as demonstrated in [4-5]. The biggest difficulty is embedding an antenna
with sufficient gain in the structure of the watch, generally in the belt, in order to recognize objects with a tag that are held
in the hand. RFID readers for passive tags generally work at three standard frequency bands:
x Low Frequency (125 kHz)
x High Frequency (13.56 MHz)
x Ultra High Frequency (433 MHz – 2.4 GHz)
In the literature most projects used LF [3] or HF [4-5] readers and corresponding tags. HF tags generally allow achieving
higher distances with less power, but they suffer of interferences, especially if metallic objects are present. Our attentions
focused on two commercial development modules, the 125 kHz ID Innovations ID-2 reader and the 13.56 MHz SkyeTek
M1-mini module, both with a nominal consumption of about 15 mA at 5 V. Having a lower price and being more robust to
imperfections of the antenna, we chose the ID-2 module for rapid and economic prototyping of a 125 kHz RFID reader with
external antenna. The antenna should have an inductance of 1.08 mH or less. With an inductance of 1.08 mH, the antenna
will match the RFID reader internal capacitance of 1500 pF, giving a theoretical resonance frequency of 125 KHz,
according to “Eq. (1)”. Lower inductance can be tuned increasing the value of the capacitance by adding an external
capacitor.
1 1 (1)
f 1.25*10 5
2S LC 2S 1.08*10 3 *1500 *1012

In order to improve the reading range of the RFID reader in the direction of the forearm, we propose the utilization of a
solenoid antenna, which should be integrated in a large belt of the watch. The inductance of the solenoid is given by the
following formula.
d 2 * n2 (2)
L
18d  40l

where L is inductance in μH, d is coil diameter in inches, l is coil length in inches, and n is number of turns. Inverting “Eq.
(2)” and fixing a length of 10 cm (3.94 inches) and a radius of 4 cm (1.57 inches) we get that 153 turns are necessaries to
obtain an inductance of 1.08 mH (1080 μH),.
Different tags should be tested in order to obtain an appropriate range. Generally bigger tags, like RFID cards, allow
obtaining longer ranges, but obviously are difficult to attach to small objects. Moreover, the orientation of the tag is very
important and should be perpendicular to the axis of the solenoid.
Preliminary tests have been conducted with the ID-2 reader and a commercial rectangular antenna with dimensions of
8,5x12cm. During this first test we have achieved a range of about 4 cm with the card tags, but the reader did not detect the
other smaller tags. Further, we built the aforementioned solenoid antenna and we increased the range of card tags up to 9cm,
and we managed to detect smaller tags. Finally, using another sample of the ID-2 reader, we obtained better results also with
the commercial antenna, showing that the antenna should be fine-tuned for each exemplar in order to maximize the reading
range.

3. Forearm gesture recognition

3.1. Related work

The use of wrist-mounted inertial sensors is a popular approach for the recognition of many gestures of the forearm, like
postures, swipes, rotations and shaking. In [7], the eWatch, a smart watch equipped with a dual-axis accelerometer, has been
used to detect gestures and answer to a simple questionnaire on user feelings along the day. Implemented gestures were
move left, move right and move up-down, to perform the scroll up, scroll down and select actions, respectively. Questions
and possible answers were displayed on the small screen of the watch. The gesture recognition was performed by the ARM7
embedded microcontroller with HMM algorithms. In [8] Raffa et al. presented a similar smart watch (ContextWatch),
equipped with a 3-axis accelerometer, a gyroscope and a Cortex M3 ARM controller in order to perform gesture
recognition. They used a hybrid approach for the gesture recognition process that assigns the segmentation task to the
ContextWatch, while the classification is done on a more powerful mobile device, e.g., a tablet. This approach allows
reducing power consumption required for gesture recognition on the watch. Moreover it reduces wireless traffic in respect to
1172 Elisa Morganti et al. / Procedia Engineering 41 (2012) 1169 – 1175

sensors that send raw data to the device where recognition algorithms are run. The first attempt to benefit from the
specificity of the objects for forearm gestures is depicted in the aforementioned ReachMedia project [5].

3.2. Design and feasibility evaluation

We have chosen to integrate in the watch the most common inertial sensors: an accelerometer, a gyroscope and a
magnetometer, each with three axes. The accelerometer is very useful to recognize swipe and shake gestures and,
eventually, tilting gestures. Rotation of the forearm can be easily detected through the gyroscope. The magnetometer could
be used for tilting gestures and for pointing gestures. This latter gestures is the most difficult to recognize without cameras,
because the orientation of the watch is not sufficient alone to detect the direction where the user is pointing. In fact the
accurate position of the watch and of the pointed object in the 3D space is necessary. The position of the pointed objects can
be fixed in the environment and specified by the system, but the position of the user should be measured through a more
complex system, such as a wireless triangulation or an RFID floor, if cameras must be avoided.
We have configured an ADXL345 accelerometer, a HMC5843 magnetometer and an ITG-3200 gyroscope on the ARM9
microcontroller, using the bus I2C to retrieve data.

4. Hand gestures, fingers gestures and grasps recognition

4.1. Related work

Detecting the posture of the hand from the wrist has been studied first by Rekimoto [9]. His GestureWrist prototype used
electrodes placed in a wrist-band to measure the capacitance of the wrist. The project showed that different postures of the
hand modify the shape of the wrist and consequently the measured capacitance between the top transmitter and electrodes.
Nevertheless results in classifying the “fist” posture against a posture with two extended fingers were not much satisfying
and could significantly vary if the wrist-band position was not stable.
Other important approaches are based on EMG signal analysis: Kim et al. studied the possibility to detect the gesture of
squeezing and left, right and circling wrist movements with sensors placed only on the bottom of the wrist [10]. A broader
range of gestures and grasps can be recognized placing sensors also in the forearm, near the elbow, as shown in [11] and
[12]. Because most part of the muscles is placed in the forearm, only few projects placed EMG sensors only on the wrist.
However the movement of tendons in this zone still arouses the interest of many researchers. In [13] Lim et al. extracted the
information of the wrist shape using an infrared sensor, which detects the amount of light reflected by the wrist during the
different tendon movements, obtaining interesting results for the hold and release gestures. Similarly to grips, finger
movements can be detected by EMG as shown by Saponas at al. in [14].
Another project [15] used a piezoelectric transducer to analyze vibrations induced by finger actions (like rubbing tapping,
etc.) as a sound. A tactile sensors approach can be successfully applied to hand gesture recognitions. In fact, this particular
class of sensors demonstrated to be a powerful tool to measure and detect rich interaction behavior with real world objects.
Over the years tactile sensors have evolved including different transduction methods (e.g., resistive, piezoresistive,
capacitive, optical, ultrasonic, magnetic, piezoelectric, etc.) [16-17]. In this study, we used piezoresistive-based sensors to
detect the force exerted by the tendons in different configurations of the hand and of the wrist.

4.2. Design

Flexible force sensors have been chosen to detect the movements of the tendons. As proof of concept, the sensors have
been purchased from Tekscan [FlexiForce® A201].

Fig. 2: (a) Picture of the wrist strep with four FlexiForce sensors mounted on it; (b) Sensors placement; (right) Strap wore on the wrist and electronics.
 Elisa Morganti et al. / Procedia Engineering 41 (2012) 1169 – 1175 1173

The working principle is based on resistance changes due to the applied force on a piezoresistive ink between two
electrodes on a flexible substrate. The diameter of the active area of the sensors is 1 cm. A piece of silicone has been glued
on the sensible area of the sensor for a better transduction of the force exerted by the tendons.
For the experimental purposes, four sensors have been mounted on a leather strip (Fig. 2 (a) and (c)), granting the
possibility to move sensors when needed, in order to fit different wrists. The position of the sensors has been chosen
according to the forearm anatomy (Fig. 2 (b)): sensor 1 is close to the tendon of the flexor digitorum superficialis tendons
muscle, or next to the palmaris longus tendon, when present; sensor 2 is on the flexor carpi radialis tendon; sensor 3 on the
extensor pollicis brevis and sensor 4 on the extensor digitorum.
The sensors can detect forces up to 1 lb. with a sensitivity that depends upon the polarization circuit. The control
electronics (Fig. 3) consists of a voltage divider with a rail to rail operational amplifier (LTC2201) and variable gain
adjustable through a trimmer (Rg). The output voltage (Vout) is then converted in digital signal using an I2C-based Analog to
Digital Converter (PCF8589) and sent to the ARM9 microcontroller for further elaboration. For the preliminary tests, analog
signals have been acquired using an acquisition board (National Instrument NI PCI-MIO-16E).

Rp R2 (3)
Vout (1  Vcc )
Rs R1  R 2

Fig. 3: Schematic of the conditioning circuit

4.3. Feasibility evaluation

Preliminary tests have been performed in order to check the possibility of discriminating different positions and gestures
by means of force sensors. Several positions of interest have been considered and the results are reported in the Fig. 4-6.
Firstly, free hand gestures, such as flexion and extension, radial deviation and ulnar deviation, pronation and supination
were analyzed, as shown in Fig. 4. The radar representation in Fig. 4 (b) shows that the recognition of the movements is
possible by comparing the signals obtained from the four sensors. The radar graph was obtained using a normalized mean
value.

(a) (b)

Fig. 4: Free hand gestures. (a) Acquired signal from the sensors in the different positions and (b) radar representation of the normalized mean values.

It should be noticed that sensor 3 on the extensor pollicis brevis tendon has great variations while flexing and extending, and
in the radial deviation due to the movement of the thumb. Other relevant contributions are from the flexor carpi radialis
tendon (sensor 2).
Segmenting gestures is an important challenge in the gesture recognition field. The use of objects can help segmenting
gestures, or at least to distinguish them from gesticulation and normal daily activities, but it is not always possible to
recognize objects using RFID tags. Sometimes the tag could be out of range or even not present in the object. Similarly a tag
that is in proximity of the hand could be recognized even if actually the object is not held. Having a system that is able to
detect when an object is held in the hand or not it is very useful to recognize gestures with objects. Thus we tested the four
1174 Elisa Morganti et al. / Procedia Engineering 41 (2012) 1169 – 1175

pressure sensors using the following movements: closing the hand, opening the hand, squeezing a spherical object and
relaxing the hand (Fig. 5). In this case the main variation are related to the activity of the extensor (sensor 4) and flexor
(sensor 1) digitorum tendons in the open-close sequence, while sensor 3 is important in discriminating the squeezing gesture
from relax position.

(b)
(a)

Fig. 5: Segmenting gestures experiment. (a) Acquired signal from the sensors in the different positions and (b) radar representation of the normalized mean
values.

As particular case, we studied the grasp of a pen; we can distinguish between tip grasp and spherical grasp [18] (Fig. 6
(b)). In the graph in Fig. 6, the peak between the positions appears during the transition, while changing the pen grasp.

Fig. 6: Tip grasp and spherical grasp. (a) Acquired signal from the sensors in the different positions and (b) radar representation of the normalized mean
values.

4.4. Discussion

These preliminary tests show that the four force sensors are able to detect wrist and hand postures. However, the actual
shape of the signals strictly depends upon several factors, which introduce a level of incertitude in the measurement. In fact,
it is difficult to place the sensors always in the same place and to reproduce the same position of the hand and of the wrist.
Each imperceptible deviation of the hand position impacts terribly on the response; moreover wrist shapes and muscles
configuration can vary a lot from one person to the other, affecting the recognition procedure. This latter problem could be
avoided by training gestures recognizers only on the user that will use the system, which is an acceptable approach for a
personal device like the smart watch. It should be noticed that the used sensors are quite big compared to the tendons and
they can cover only a few positions around the wrist. Next step would be the design and realization of a distributed array of
smaller sensors [17] that can cover all the area of interest with the aim of reducing the uncertainty due to the placement. At
the same time, with better sampling, the shape of the wrist could be compensated with a pre-calibration. Finally, integration
at the system level should be carefully addressed in order to improve usability and reliability [19].

5. Conclusion

In this work an approach towards the realization of a complete system for forearm and hand gesture recognition with
objects is presented. The key aspect of the system is the integration of different kinds of sensors in a platform with the shape
of an everyday object: a watch.
 Elisa Morganti et al. / Procedia Engineering 41 (2012) 1169 – 1175 1175

Integrating force sensors in the wrist, as shown by the preliminary tests, is very interesting because allows covering most
of the gestures that could not be recognized by inertial sensors. Moreover the possibility to recognize objects introduces an
additional degree of freedom to the gestures possibilities, enhancing the expressivity of the interaction. Combining the three
approaches in a watch, it is possible to obtain a complete platform to recognize forearm gestures, fingers gestures, gestures
with objects and grasps. Several combinations are possible and can vary according to the held object or to the specific
application. For instance, the interaction designer could associate to one object two or three forearm gestures (recognized by
inertial sensors) that are similar to actions done during everyday life with that object. Other objects could change their
functionalities according to how the user is grasping it, thus, in this case, information from force sensors should be used.
Information from different sensors could be either combined, allowing the recognition of complex gestures and enhancing
recognition rates, or alternated, in order to reduce power consumption. For example, force sensors could be used to detect if
an object is grasped and consequently to turn on the RFID reader only when necessary. Similarly, if an object has no
forearm gestures associated, inertial sensors could be turned off.
Different combination of sensors and different recognition algorithms will be tested in order to maximize the
performance of the smart watch. Further, it will be compared to other systems in order to determine the best solution for the
recognition of gestures with objects and of freehand gestures.
Once completely integrated, the system will be able to allow untrained people to interact in their common life with real
objects and customizable gestures in an intuitive and comfortable way.

Acknowledgements

This work has been supported by Hasler Foundation in the framework of “Living in Smart Environments” project.
Special thanks go to Aymen Jlassi for his contribution and to Dr. Ravinder Dahiya for useful discussions.

References
[1] Hoven, E., Mazalek, A., 2011. Grasping gestures: Gesturing with physical artifacts. Artificial Intelligence for Engineering Design, Analysis and
Manufacturing 25, p. 255.
[2] Fitzmaurice, G., 1996. Graspable user interfaces, PhD thesis, University of Toronto
[3] Schmidt, A., Gellersen, H., 2000. “Enabling Implicit Human Computer Interaction: A Wearable RFID-Tag Reader,” 4th International Symposium on
Wearable Computers, p. 193.
[4] Fishkin, K.P., Philipose, M., Rea, A., 2005. “Hands-on RFID: Wireless wearables for detecting use of objects,” Proceedings of the 9th IEEE
International Symposium on Wearable Computers, p. 38.
[5] Feldman, A., Tapia, E.M., Sadi, S., Maes, P., Schmandt, C., 2005. “ReachMedia: On-the-move interaction with everyday objects,” Proceedings of the
th
9 IEEE International Symposium on Wearable Computers, p. 52.
[6] Maekawa, T., Yanagisawa, Y., Kishino, Y., 2010. Object-based activity recognition with heterogeneous sensors on wrist. Pervasive, LNCS 6030, p.
246.
[7] Amft, O., Amstutz, R., Smailagic, A., Siewiorek, D., Tröster, G., 2009. Gesture-controlled user input to complete questionnaires on wrist-worn watches.
Human-Computer Interaction. Novel Interaction Methods and Techniques. LNCS 5611, p. 131–.
[8] Raffa, G., Lee, J., Nachman, L., Song, J., 2010. “Don’t Slow Me Down: Bringing energy efficiency to continuous gesture recognition,” Proceedings of
the 14th International Symposium on Wearable Computers, p. 1.
[9] Rekimoto, J., 2001. “Gesturewrist and gesturepad: Unobtrusive wearable interaction devices,” Proceedings of the 5th International Symposium on
Wearable Computers, p. 21.
[10] Kim, J., Mastnik, S., André, E., 2008.“EMG-based hand gesture recognition for realtime biosignal interfacing,” Proceedings of the 13th International
Conference on Intelligent User Interfaces. p. 30.
[11] Wojtczak, P., Amaral, T.G., Dias, O.P., Wolczowski, A., Kurzynski, M., 2009. Hand movement recognition based on biosignal analysis. Engineering
Applications of Artificial Intelligence 22, p. 608.
[12] Zhang, X., Chen, X., Wang, W., Yang, J., Lantz, V., Wang, K., 2009. “Hand gesture recognition and virtual game control based on 3D accelerometer
and EMG sensors,” Proceedings of the 14th International Conference on Intelligent User Interfaces. p. 401.
[13] Lim, J., Lee, D., Kim, B., Cho, I., 2010. “Recognizing hand gestures using wrist shapes,” International Conference on Computer Electronics, p. 197.
[14] Saponas, T.S., Tan, D.S., Morris, D., Balakrishnan, R., 2008. “Demonstrating the feasibility of using forearm electromyography for muscle-computer
interfaces,” Proceeding of the 26th Annual SIGCHI Conference on Human Factors in Computing Systems. p. 515.
[15] Amento, B., Hill, W., 2002. “The sound of one hand: a wrist-mounted bio-acoustic fingertip gesture interface CHI '02 extended abstracts on Human
factors in computing systems. p. 724.
[16] Dahiya, R. S., Cattin, D., Adami, A., Collini, C., Barboni, L., Valle, M., Lorenzelli, L., Oboe, R., Metta, G., Brunetti, F., 2011. Towards Tactile
Sensing System on Chip for Robotic Applications, IEEE Sensor Journal, 11 (12), p 3216.
[17] Dahiya, R. S., Valle, M., 2012. Robotic Tactile Sensing – Technologies and System, Springer Science + Business Media, Dordrecht. (ISBN:978-94-
007-0578-4)
[18] Saponas, T.S., Tan, D.S., Morris, D., Balakrishnan, R., Turner, J., Landay, J. a., 2009. Enabling always-available input with muscle-computer
interfaces. Proceedings of the 22nd annual ACM symposium on User interface software and technology - UIST ’09 p.167.
[19] Dahiya, R. S., Valle, M., Metta, G., 2008. System Approach: A paradigm for Robotic Tactile Sensing, Proceedings of IEEE- AMC08, The 10th
International Workshop on Advanced Motion Control, 1,pp 110-115.