PROCESAMIENTO DIGITAL DE

IMAGENES Y VISION ARTIFICIAL

Semana 15 – sesión 1
Introducción a la Realidad Aumentada
Objetivos de la sesión

• Análisis de imágenes
Trend Towards Invisible Interfaces

Trend from room scale to invisible computing

Making Computers Invisible
hide the computer in the real world
Ubiquitous Computing
put the user inside the computer
Virtual Reality
Making Interfaces Invisible

• Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented
interaction with real world environments. In Proceedings of the 8th Annual ACM Symposium on
User interface and Software Technology. UIST '95. ACM, New York, NY, 29-36.
Graphical User Interfaces

Separation between real and digital worlds

WIMP (Windows, Icons, Menus, Pointer) metaphor
Virtual Reality

• Circa 1985
Ubiquitous Computing

Computing and sensing embedded in real world

Particle devices, RFID, motes, arduino, etc
From Reality to Virtual Reality

Ubiquitous Computing Augmented Reality Virtual Reality

Augmented Reality

• 1977 – Star Wars

 AUGMENTED REALITY
•

Defining Characteristics [Azuma 97]

•

• Combines Real and Virtual Images

• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space

•AZUMA, R. T. (1997). A SURVEY OF AUGMENTED REALITY. PRESENCE, 6(4), 355-385.

 AUGMENTED REALITY VS.VIRTUAL REALITY

Augmented Reality
• Virtual Reality
•

• System augments the real • Totally immersive

world scene environment
• User maintains a sense of • Senses are under control of
presence in real world system
• Needs a mechanism to • Need a mechanism to feed
combine virtual/real worlds virtual world to user
• Hard to register real and • Hard to make VR world
virtual interesting
 MIXED REALITY
•

A system that combines

•

real and virtual objects and

information.
 • MILGRAM’S REALITY-VIRTUALITY CONTINUUM
Mixed Reality (MR)

Real Augmented Augmented Virtual

Environment Reality (AR) Virtuality (AV) Environment

Milgram coined the term “Augmented Virtuality” to identify

•

systems which are mostly synthetic with some real world imagery
added such as texture mapping video onto virtual objects.
•P. MILGRAM AND A. F. KISHINO, TAXONOMY OF MIXED REALITY VISUAL DISPLAYS. IEICE TRANSACTIONS
ON INFORMATION AND SYSTEMS, E77-D(12), PP. 1321-1329, 1994.
 AUGMENTED VIRTUALITY
•

VR with windows into the real world

•
AR VS VR
•
 OTHER CONTEXT: THE METAVERSE
•

Neal Stephenson’s “SnowCrash”

•

The Metaverse is the convergence of:

•

1. virtually enhanced physical reality

2. physically persistent virtual space
Metaverse Roadmap
•

• http://metaverseroadmap.org/
 METAVERSE DIMENSIONS
•

Augmentation technologies that layer information onto

•

our perception of the physical environment

Simulation refers to technologies that model reality
•

Intimate technologies are focused inwardly, on the identity

•

and actions of the individual or object

External technologies are focused outwardly, towards the
•

world at large
FOUR KEY METAVERSE COMPONENTS
•
 MIRROR WORLDS
•

Mirror worlds are informationally-

•

enhanced virtual models of the

physical world.
• Google Earth, Street View, etc

• ogle Maps
 •LIFELOGGING
Technologies record and report the intimate states and life
•

histories of objects and users

• Nokia LifeBlog, Nike+, FitBit
 NARRATIVE CLIP
•

Wearable camera
•

• Automatic picture capture - 2 pics/minute

• http://getnarrative.com
 • AR APPLICATION AREAS
• Computer games
• Map-assisted navigation
• Urban planning / Design / construction
• Tourism / Cultural heritage
• Maintenance
• Training
• Battlefield information display
• Robot-assisted surgery
• Virtual tours—real estate, medical facilities, etc.
AUGMENTED REALITY EXAMPLES
•
 AR IN TELECONFERENCING
•

Person works at real desk

•

Remote collaborator represented

•

by picture or video or “talking head”

Objects of discussion; e.g. a patient’s
•

brain image, might also be fused into

visual field

•FROM UNIVERSITY OF WASHINGTON HIT LAB

CNN VIRTUAL VIEW (2008)
•

•HTTPS://YOUTU.BE/V7FQ_ESMJMS
POKEMON GO
•
MICROSOFT HOLOLENSE
•

•HTTPS://YOUTU.BE/XGAKDCEZVWG
 • EXAMPLE: HAUNTED BOOK/AR BOOK

• Camera hidden in lamp object

• AR content seamlessly integrated into real book
• Natural page turning/manipulation interaction

•SCHERRER, C., PILET, J., FUA, P., & LEPETIT, V. (2008, SEPTEMBER). THE HAUNTED BOOK. IN
PROCEEDINGS OF THE 7TH IEEE/ACM INTERNATIONAL SYMPOSIUM ON MIXED AND
AUGMENTED REALITY (PP. 163-164). IEEE COMPUTER SOCIETY.
AR BOOK DEMO
•

•HTTPS://YOUTU.BE/XBSFAMN78TC
 GAMING: ROCK-EM SOCK-EM
•

Shared AR Demo
•

Markerless tracking
•
ROCKEM SOCKEM DEMO
•

•HTTPS://YOUTU.BE/HXTQ1QBMLIW
 SALES AND MARKETING
•

Connect with brands and branded objects

•

Location Based Experiences

•

• Lynx Angels
Web based
•

• Rayban glasses
Mobile
•

• Ford Ka campaign
Print based
•

• Red Bull Magazine

PEPSI AR EXPERIENCE (2014)
•

•HTTPS://YOUTU.BE/GO9RF9GMYPM
ASSEMBLY AND MAINTENANCE
•

© 1996 S. Feiner, B. MacIntyre, &

A. Webster, Columbia University

© 1993 S. Feiner, B. MacIntyre, &

D. Seligmann, Columbia University
 •MAINTENANCE SYSTEMS
Ngrain
•

• http://www.ngrain.com/
• Training authoring tool
• Model based AR tracking
ScopeAR
•

• http://www.scopear.com/
• Remote assistance
• Image based tracking
NGRAIN EXAMPLE
•

•HTTPS://YOUTU.BE/XTHEAOBGZS4
 MEDICAL AR TRIALS
•

Sauer et al. 2000 at Siemens

•

Corporate Research, NJ
Stereo video see through
•

•F. SAUER, ALI KHAMENE, S. VOGT: AN AUGMENTED REALITY NAVIGATION SYSTEM WITH A SINGLE-CAMERA
TRACKER: SYSTEM DESIGN AND NEEDLE BIOPSY PHANTOM TRIAL, MICCAI 2002
MEDICAL DEMO
•

•HTTPS://YOUTU.BE/KFQULGEYH7W
 • INTERACTIVE MUSEUM EXPERIENCES
• BlackMagic
• Virtual America’s Cup
• 410,000 people in six months
• MagicPlanet
• TeManawa science museum
• Virtual Astronomy
• Collaborative AR experience
• AR Volcano
• Interactive AR kiosk
• Scienceworks museum, Melbourne
DIGITAL BINOCULAR STATION
•

•HTTP://WWW.DIGITALBINOCULARSTATION.COM/
DIGITAL BINOCULAR STATION
•

•HTTPS://YOUTU.BE/MQLAKBX0OM4
CITYVIEWAR APPLICATION
•

•HTTPS://YOUTU.BE/FDGRXXJX4SE
 COLLABORATION EXAMPLE: HOLOPORTATION
•

Augmented Reality + 3D capture + high bandwidth

•

http://research.microsoft.com/en-us/projects/holoportation/
•
HOLOPORTATION VIDEO
•

•HTTPS://YOUTU.BE/7D59O6CFAM0
 • HUMAN OPERATING WITH AR

Think of a heads up display on your auto windshield, or on the

•

instrument panel. What could be there to help you navigate?

(Vectors to nearby eating places? Blinking objects we might
•

collide with? Congestion of nearby intersections? Web pages?)

 •SPECIAL DEVICES NEEDED TO
FUSE/REGISTER REAL AND GENERATED
IMAGES
Computer Graphics Problem – Human sees
•

graphics generated from 3D/2D models

Tracking/Pose Problem – Graphics system
•

needs to know how the human is viewing

the 3D environment
Optics Problem – Human sees both real and
•

virtual environments

•FROM UNIVERSITY OF WASHINGTON HIT LAB

 DIFFICULT AUGMENTATION PROBLEM
•

Human very sensitive to misregistration

•

• Some applications OK – e.g. circuit board

inspection
Tough calibration procedures for average user
•
 COMBINING THE REAL AND VIRTUAL WORLDS
•

We need:
•

• Precise models
• Locations and optical properties of the viewer (or
camera) and the display
• Calibration of all devices
• To combine all local coordinate systems centered
on the devices and the objects in the scene in a
global coordinate system
 • COMBINING THE REAL AND VIRTUAL WORLDS (CONT)
Register models of all 3D objects of interest with their
•

counterparts in the scene

Track the objects over time when the user moves and interacts
•

with the scene

 REALISTIC MERGING
REQUIRES
Objects to behave in physically
•

plausible manner
• Occlusion
• Collision detection
• Shadows
 • STRONG VS. WEAK AR
• Weak AR
• Imprecise tracking
• No knowledge of environment
• Limited interactivity
• Handheld AR

• Strong AR
• Very accurate tracking
• Seamless integration into real world
• Natural interaction
• Head mounted AR
 • AR TRACKING EXAMPLES
Optical / vision-based tracking
•

• Ensures portability
• Large number of tracked objects
• Registration and low latency are
crucial for AR systems
Sourceless inertial orientation tracking
•

GPS (or other) position tracking

•

• Generally requires high accuracy

• GPS enables mobile (global) outdoor AR

•HTTPS://YOUTU.BE/TQGAQAFLGG0
 “MOBILE” OUTDOOR AR
•

1997 “Touring Machine”

•

Backpack systems (Feiner)

User wears/carries:
•

• Computer
• HMD
• Inertial tracker
• GPS unit/antenna
• Input device(s)
 •PERFORMANCE ISSUES WITH TRACKING
Accuracy of the registration of the real and virtual image
•

• Noise – position and pose of camera with respect to the

real scene
• Image distortions
Time delays
•

• Update rate for generating the augmenting image

• Can limit registration accuracy – “1 ms = 1mm error”
AR DISPLAY TECHNOLOGIES
•

Monitor Based
•

• Laptops
• Cell phones
• Projectors (more Ubiquitous Computing)
Head Mounted Displays
•

• Video see-through
• Optical see-through
 • MONITOR BASED AUGMENTED REALITY
Simplest available
•

Treat laptop/cell phone as a window through

•

which you can see AR world.

Sunglasses demo
•
• MONITOR BASED AR – SUCCESSFUL
COMMERCIALIZATION
Yellow line in football broadcasts
•

Glowing hockey puck

•

Replace times square

•

billboards with own ads

during broadcast
Baseball cards
•

Ad campaigns
•
OPTICAL SEE-THROUGH HMD
•
VIDEO SEE-THROUGH HMD
•
Cierre