PoCoMo

Chapter 1
Introduction
Mobile devices are becoming increasingly popular for social and playful interactions.
These offer a powerful computing power needed for high-resolution animation through
embedded graphics processors and always-on Internet connection via 5G cellular
networks, and further advances in mobile projection hardware make it possible to explore
more seamless social interaction between co-locating users with mobile phones.

PoCoMo is a collection of mobile, handheld, projector-camera systems, embedded

in fully functioning mobile phones. The goal of the system is to allow users to interact
with each other and their environment in an ad-hoc, natural way. The phones show
projected characters that line up with the view of the phone’s camera, which is running
software that tracks the characters’ position. Two characters projected in the same space
will quickly respond to each other based on the characteristics of the local environment.

PoCoMo works on commercially available hardware and standard software

libraries to improve the chances of developers to implement it and to provide examples
that support projector-enabled mobile phones. PoCoMo uses a common object tracking
technique to detect other projected characters and to extract visual elements from the
environment to be used in a limited resource environment without significantly reducing
its usability.

The system contributes a novel implementation of new projected interface

techniques where a user’s projected display is split into a number of areas, each can be set
to either private or public. These areas contain tiles where a tile is a space generated by
the user to place their data . A tile assigned by the user in the public area can be seen by
anyone, whereas a tile assigned in private area can only be seen by that user. The system
also introduces augmented reality applications that can be implemented in the system.

Dept. of ISE, SIT, Tumakuru 1 2023-2024

PoCoMo

Chapter 2
Literature Survey

The increasing portability of projection systems since 2003 has enabled researchers to
explore geometrically aware collaborative systems based on orientation. The geometric
issues necessary for adaptive images of multiple projectors into a single display. The
Hotaru system[6] uses static projection to simulate multiple projections. Several
researchers have introduced the flashlight metaphor for interacting with the physical
world, where a portion of the virtual world is revealed by the projection. The presented
techniques for revealing contextual information on the fly also using the flashlight
metaphor [3].

A body of work exploring interaction was presented using personal projection, however
they mostly rely on an external 6-DOF tracking system, and a single- user paradigm [1].
Other researchers have explored playful and game-like possibilities enabled by portable
projection technologies. It describes a handheld projector-camera system projecting a
character that interacts with drawings on a whiteboard. Their system specifically reacts to
edges, either drawn or created by objects it the scene but does not respond other systems
in the same area [1].

Recently, a technique for projected character control via handheld devices. Those were
robot-controlling games using handheld projectors. In this work the camera is mounted on
the robot instead of being embedded in the mobile phone [2]. iLamps provides users with
adaptive projection techniques with a cluster of handheld projectors. iLamps identifies
locations and orientations of multiple projectors and creates a seamless and consistent
image over a projected surface (e.g. planar, spherical etc.) by correcting an overlapping
image given by the projectors [3]. The main purpose of iLamps is different from that of
Hotaru[6] in that it proposes manipulation techniques for overlapping projected displays
of mobile devices.

Dept. of ISE, SIT, Tumakuru 2 2023-2024

PoCoMo
Augmented Surfaces [2] allows users to drag files, such as documents or images, to be
shown on a computer desktop, a table or a wall, by using a laser pointer. Users can easily
share these files with other users and bring them into their own personal computers. The
difference between Augmented Surfaces and Hotaru is that Hotaru[6] provides users with
intuitive methods for supporting their collaborative tasks such as annotating by the user’s
fingers.

Dept. of ISE, SIT, Tumakuru 3 2023-2024

PoCoMo

Chapter 3
Main Concept

3.1 Components of PoCoMo

PoCoMo is a compact mobile projector-camera system, built on a commercially available

cellular smartphone that includes a pico-projector in camera.

Fig 3.1.1 : Components of PoCoMo

Mobile Device
From the following Fig 3.1.1, the mobile device refers as central hub of the PoCoMo
system. It houses the projector, camera, and processing power that enables the system to
function. The mobile device is typically a smartphone or tablet, but it could also be a
dedicated device designed for PoCoMo.

Dept. of ISE, SIT, Tumakuru 3 2023-2024

PoCoMo
Camera
The mobile camera is used to capture the environment around the user. This information
is used to track the user's movements and to project the user's interactions onto the screen.
The camera can also be used to capture images and video that can be shared with other
users.

Pico-Projector
The pico-projector is used to project the user's interactions onto the screen. The
projection can be used to create a shared space where users can interact with each other
and with digital objects. The projector can also be used to create augmented reality
experiences that overlay digital objects onto the real world.

Interaction on the screen

The screen is used to display the user's interactions and the projected content. Users can
interact with the screen using a variety of methods, such as touch, gesture, and voice. The
screen can also be used to display feedback to the user, such as the results of their
interactions.

Relationships between the components

The mobile device, mobile camera, projection, and interaction on the screen are all
interconnected. The mobile device provides the processing power and connectivity that is
needed for the other components to function. The mobile camera provides the input that
is needed for the system to track the user's movements and to project the user's
interactions onto the screen. The projector displays the user's interactions and the
projected content. The interaction on the screen allows users to interact with the system
and with each other as shown in Fig 3.1.1.

Dept. of ISE, SIT, Tumakuru 4 2023-2024

PoCoMo

3.2 Working of PoCoMo

Fig 3.2.1 : Movement Rigs

 Capture User's Movements

The capture of user's movements is primarily accomplished using pose estimation
algorithms. These algorithms are designed to identify and track human body keypoints,
such as joints, in real-time as shown in Fig 3.2.1. Common pose estimation algorithms
include:
OpenPose: OpenPose is a popular open-source pose estimation framework that utilizes
deep learning techniques to achieve accurate and real-time body pose detection.

MediaPipe Hands: MediaPipe Hands is a specialized pose estimation model specifically

designed to track hands and fingers. It offers high accuracy and robustness in various
lighting and background conditions.

BodyPose: BodyPose is another open-source pose estimation framework that employs a

convolutional neural network (CNN) architecture to estimate human body poses. It
provides a balance between accuracy and efficiency, making it suitable for mobile
applications.

 Track User's Movements and Project Interactions

Tracking user's movements and projecting interactions involves combining pose
estimation data with motion prediction algorithms. These algorithms anticipate the user's

Dept. of ISE, SIT, Tumakuru 5 2023-2024

PoCoMo
future movements based on their current trajectory and projected interactions. Common
motion prediction algorithms include:

Kalman Filter: The Kalman filter is a widely used algorithm for tracking and predicting
dynamic objects. It effectively manages noise and uncertainties in the data, making it
suitable for real-time motion tracking.

Hidden Markov Models (HMMs): HMMs are probabilistic models that capture the
underlying patterns in sequential data. They can be used to predict future movements
based on the history of observed poses.

Recurrent Neural Networks (RNNs): RNNs are a class of neural networks well-suited for
handling sequential data like pose sequences. They can learn complex patterns and
relationships in the data, enabling them to predict future movements effectively.

 Render Projected Characters and Objects

Rendering projected characters and objects involves employing 3D rendering algorithms
to create realistic and immersive virtual representations. Common 3D rendering
algorithms include:

Rasterization: Rasterization is a fundamental 3D rendering technique that converts 3D

models into 2D images by projecting them onto a virtual screen. It's widely used due to its
efficiency and simplicity.

Ray Tracing: Ray tracing is a more advanced rendering technique that simulates the
physical behavior of light to create highly realistic images. It produces accurate shadows,
reflections, and refractions, enhancing the visual quality of the virtual environment.

Real-time Rendering Techniques: For real-time performance, PoCoMo may employ

specialized rendering techniques like deferred shading, frustum culling, and level of detail
(LOD) management. These techniques optimize the rendering process to achieve high
frame rates without compromising visual quality.

Dept. of ISE, SIT, Tumakuru 6 2023-2024

PoCoMo

 Allow User Interaction

Enabling user interaction involves employing input recognition algorithms to interpret
gestures, touch, and voice commands. Common input recognition algorithms include:

Gesture Recognition: Gesture recognition algorithms analyze sequences of pose data to

identify specific gestures made by the user. They employ techniques like template
matching, dynamic time warping, and machine learning to classify gestures accurately.

Touch Recognition: Touch recognition algorithms process touch events from the
touchscreen to identify specific interactions, such as taps, swipes, and multi-touch
gestures. They employ algorithms like filtering, segmentation, and pattern matching to
distinguish between intentional and unintended touches.

Voice Recognition: Voice recognition algorithms convert spoken language into text. They
employ techniques like acoustic modeling, language modeling, and deep learning to
achieve accurate speech recognition.

 Enable Real-time Interaction

Enabling real-time interaction involves employing networking protocols to synchronize

user actions and interactions across multiple devices. Common networking protocols for
real-time communication include:

WebSockets: WebSockets provide a persistent bi-directional communication channel

between devices, allowing for continuous data exchange and real-time synchronization.

UDP (User Datagram Protocol): UDP offers low-latency data transmission, making it
suitable for real-time applications where latency is critical. However, it does not guarantee
data delivery or order, requiring additional mechanisms for error handling and
synchronization.

Dept. of ISE, SIT, Tumakuru 7 2023-2024

PoCoMo
MQTT (Message Queue Telemetry Transport): MQTT is a lightweight publish-subscribe
protocol designed for resource-constrained devices, making it suitable for mobile
applications. It provides efficient data transmission while maintaining real-time
communication capabilities.

Dept. of ISE, SIT, Tumakuru 8 2023-2024

PoCoMo

Chapter 4
Applications

Fig 4.1 : Gaming

 Entertainment:
Immersive gaming: PoCoMo can revolutionize gaming by enabling players to physically
interact with virtual characters and environments, creating a more immersive and
engaging gaming experience as shown in Fig 4.1.

Enhanced storytelling: It can bring stories to life by allowing readers to interact with
characters, explore virtual worlds, and participate in the narrative.

Augmented reality experiences: It overlays virtual elements onto the real world, creating
interactive experiences for entertainment purposes, such as virtual scavenger hunts,
interactive museums, and augmented reality games.

 Education:
Interactive learning experiences: PoCoMo can transform traditional classroom settings
into engaging and interactive learning environments. Students can interact with virtual
Dept. of ISE, SIT, Tumakuru 9 2023-2024
PoCoMo
characters, manipulate objects, and explore concepts in a more immersive and memorable
way.

Language learning: Facilitates language learning by providing interactive scenarios where

students can practice conversational skills and engage in role-playing activities with
virtual characters.

 Collaboration:
Remote collaboration: PoCoMo can facilitate real-time collaboration between remote
teams by providing a shared virtual workspace where participants can interact with each
other and with virtual objects.

Distributed meetings: Enhance distributed meetings by allowing participants to interact

with each other using avatars, whiteboard tools, and virtual presentations.

 Marketing:
Interactive product demonstrations: PoCoMo can transform product demonstrations into
interactive experiences where potential customers can interact with virtual products,
explore features, and ask questions in real-time.

Virtual showrooms and exhibitions: It can create immersive virtual showrooms and
exhibitions where customers can explore products, interact with virtual guides, and
receive personalized recommendations.

Dept. of ISE, SIT, Tumakuru 1 2023-2024

PoCoMo

Chapter 5
Limitations

 Limitations related to PoCoMo:

Accuracy and robustness: The accuracy of user tracking and gesture recognition
algorithms can be affected by various factors such as lighting conditions, background
clutter, occlusions, and variations in user movements.

Latency and performance: Real-time interaction requires low latency to prevent

delays between user actions and their corresponding virtual representations. Achieving
consistent low latency across devices and environments remains a challenge.

Scalability: Supporting multiple users in a shared virtual space demands efficient

networking protocols and resource management to handle the increased bandwidth and
computational demands.

Hardware constraints: The processing power and battery life of mobile devices pose
limitations on the complexity of 3D rendering and the overall performance of the
PoCoMo system.

User acceptance and adoption: The widespread adoption of PoCoMo depends on

user acceptance and familiarity with mixed-reality interactions. Addressing user concerns
regarding privacy, safety, and social acceptability is crucial.

Camera placement: To date, few commercially available mobile devices have placed
the camera in line of sight of the projection area. This limitation hinders the development
of AR applications that require precise alignment between the user's real-world
interactions and the projected virtual content.

Dept. of ISE, SIT, Tumakuru 1 2023-2024

PoCoMo

 Future Work related to PoCoMo:

Improved tracking and recognition algorithms:
Developing more robust and accurate algorithms for user tracking, gesture recognition,
and object recognition is essential for enhancing the user experience and expanding the
system's applicability.

Real-time performance optimizations:

Optimizing rendering algorithms, networking protocols, and resource management
techniques is crucial for achieving consistent low latency and real-time responsiveness
across devices and environments.

Scalable networking solutions:

Investigating and implementing scalable networking protocols and resource management
strategies to support a large number of concurrent users in shared virtual spaces.

Addressing camera placement limitations:

Investigating and developing solutions to place the camera in line of sight of the
projection area will enable more precise and immersive AR applications. This could
involve advancements in camera placement mechanisms, camera lens designs, or
software-based calibration techniques.

Dept. of ISE, SIT, Tumakuru 1 2023-2024

PoCoMo

Chapter 6
Conclusion

PoCoMo is a promising mobile projection system that has the potential to revolutionize
the way we interact with the digital world. It combines the power of mobile devices with
the immersiveness of projection technology to create a truly interactive and engaging
experience. However, the system faces several limitations, such as accuracy, latency,
scalability, and hardware constraints. To overcome these challenges, future work should
focus on improving tracking and recognition algorithms, optimizing rendering and
networking, developing scalable solutions, and enhancing hardware capabilities. As these
limitations are addressed, PoCoMo has the potential to have a profound impact on
education, entertainment, collaboration, and marketing.

Despite the limitations that PoCoMo faces today, the system holds immense potential to
transform the way we interact with technology. By combining the portability and
versatility of mobile devices with the immersiveness of projection technology, it offers a
unique platform for creating interactive and engaging experiences that go beyond the
limitations of traditional screens.

Dept. of ISE, SIT, Tumakuru 1 2023-2024

PoCoMo

Chapter 7
References

[1] Shilkrot, Roy, Seth Hunter, and Patricia Maes. "PoCoMo: projected collaboration
using mobile devices." Proceedings of the 13th International Conference on Human
Computer Interaction with Mobile Devices and Services. 2011.

[2] R. T. Chadalavada, H. Andreasson, R. Krug and A. J. Lilienthal, “That's on my mind!

Robot to Human Intention Communication through on-board Projection on Shared Floor
Space,” in European Conference on Mobile Robots (ECMR), pp. 1-6, September, 2015.

[3] T.Machino, S. Iwaki, H. Kawata, Y. Yanagihara, Y. Nanjo and K. Shimokura,

“Remote-collaboration system using mobile robot with camera and projector,” in
Proceedings of IEEE International Conference on Robotics and Automation, pp. 4063-
4068, May, 2006.

[4] Y. Xiao, Z. Zhang, A. Beck, J. Yuan and D. Thalmann, “Human-robot interaction by

understanding upper body gestures,” Presence:Teleoperators and Virtual Environments,
vol. 23, no. 2, pp. 133-154,2014.

[5] Hosoi, K., Dao, V.N., Mori, A., and Sugimoto, M. VisiCon : A Robot Control
Interface for Visualizing Manipulation Using a Handheld Projector Cooperative Robot
Navigation Game via Projection. Human Factors, pp 99-106, 2007.

[6] Sugimoto, M., Miyahara, K., Inoue, H., and Tsunesada, Y. Hotaru: intuitive
manipulation techniques for projected displays of mobile devices. Human-Computer
Interaction-INTERACT,57-68,2005.

Dept. of ISE, SIT, Tumakuru 1 2023-2024

Emerging Mobile App Development Trend References

Dept. of ISE, SIT, Tumakuru 1 2023-2024