Computer Graphics and Computer Vision

Professor Demetri Terzopoulos

Graphics and Vision: A Unified View

This

major field is about Computers and Images (CG)

Computer Graphics

Computational models images and videos Synthesis Forward problem

Computer Vision

(CV)

Images and videos computational models Analysis Inverse problem

Page 1

History of Computer Graphics and Computer Vision

Two PhD

projects at MIT in the early 1960s

Ivan E. Sutherland, 1963

Sketchpad, a man man-machine graphical communication system

Lawrence G. Roberts, 1963

Machine perception of three three-dimensional solids

CG

and CV have developed as independent fields CG and CV have started to synergize

In recent years, years Exciting

synergies and crosscross-fertilization with other fields

Physics, biology, cognitive science, artificial intelligence, art, .

What is an Image / Video?

Array A

of pixels (one (one or more numbers)

video is a time sequence of images they are formed:

Illumination

How

Objects in the world (static or dynamic) Illumination (light sources) Imaging device (eye, camera)
We want to synthesize
Camera

and analyze them

Page 2

Why? Images and movies are everywhere!

Entertainment: Motion pictures & Games Virtual worlds (especially for movies and games) Industrial design Scientific and medical visualization HumanHuman -computer interaction Fine arts Etc. Robotics Automotive Visual Surveillance / Biometrics Industrial inspection Medical imaging Remote sensing Image and video retreival Etc. For a deeper understanding of the physical world and living systems, including the human brain

Computer Graphics
The

art and science of creating imagery by computer themes

Three main research

Modeling

How do we model (mathematically represent) objects? How do we construct models of specific objects? How do we represent the motions of objects? How do we give animators control of this motion? How do we simulate the real real-world behavior of light? How do we simulate the formation of images?

Animation

Rendering

Page 3

Standard Display Devices

CRT
(Cathode Ray Tube)

LCD
(Liquid Crystal Display)

Plasma

Exotic Display Devices

Immersive

HeadHead -Mounted Holographic

Autostereoscopic

Volumetric

Page 4

Movies
To reality

and beyond !

Movies
Special

effects

Page 5

Movies

Digital Compositing

Page 6

Digital compositing

Digital Compositing

Page 7

Cartoons

Games
Focus

on interactivity

Page 8

Games

Metal Gear Solid 4

Page 9

ComputerComputer -Aided Design

Precision

modeling visualization

Engineering

ComputerComputer -Aided Design

Its not just about visualization Simulation

is also useful

Page 10

Visualization: Scientific

Molecular Biology Environmental Physics

Visualization: Medical

Page 11

Visualization: Architectural

http://www.diamondschmitt.com/

Visualization: Info

Smith and Fiore

Page 12

Graphical User Interfaces

3D User Interfaces

Steven Schkolne Sculpture Art

Page 13

Digital Art
Genetically

evolved
Carl Sims

Digital Art

The Highest-Grossing Film of All Time, ^ 2nd

Jason Salavon

Page 14

Elements of CG
The graphics pipeline

Modeling

Animation

Rendering

Modeling
Representing

objects geometrically on a computer

Point clouds Texture maps Polygon meshes

S f Surface patches t h NURBS

Parametric curves

Page 15

Modeling
Alternative

Voxels

representations

Blobs Subdivision surfaces

Modeling
Altering

geometric models

Affine transform

Ying, Kristjansson, Biermann, Zorin

Page 16

Scanning Shapes

Digital Michaelangelo Project

Plant Modeling
P.

Prusinkiewicz, Univ. of Calgary

Page 17

Plant Modeling

Rendering
Key

Illumination

elements
Camera Geometric Shape & Reflectance

Pixels

Page 18

Camera Model

Rendering
Draw

visible surfaces onto display

Camera (view direction)

Clipping
Image Plane Camera (eye position)

Scan Conversion

Page 19

Reflectance Modeling

Reflectance Modeling

Diana the Huntress,

Henrik Wann Jensen

Page 20

Subsurface Scattering

Translucency and varied levels of light penetration can be created using subsurface scattering effects (nVIDIA)

Rendering
Non Non-photorealistic rendering

Aaron Hertzmann

Page 21

Rendering: Information

Agrawala and Stolte

Mapblast.com

Animation
Keyframe

animation

Keyframing

Page 22

Animation
Motion capture

Animating Golem in LOTR

Page 23

Markerless Motion Capture

Max Plank Institute of Informatics, Germany

Animation
Physical

simulation

Dynamics

R. Fedkiw et al., Stanford U.

Page 24

Atlanta in Motion
J.

Hodgins, et al., Georgia Tech

NBC - Los Angeles

Coverage

of Prof. Petros Faloutsos digital stuntman

Page 25

Falling Sideways

Falling Backward, Rolling Over, Rising, and Balancing in Gravity

Help,

Ive fallen!

and I can get up!

Page 26

Rising From a Supine Position

Virtual Celebrity
Virtual

Celebrity Productions, LLC

Page 27

Facial Anatomy

Muscle Model

Skin Model
Epidermis Dermis

Muscle Layer

RealReal -Time Facial Simulation

Page 28

NeckNeck -HeadHead-Face Animation

Animation
Behavioral

animation
Ethology

Page 29

Evolved Virtual Creatures

Karl

Sims, 1994 algorithms evolution

Genetic

Mimic biological

Animation
Pixar:

Luxo Jr. (1986)

Page 30

Animation
Example:

Geris Game - Pixar

Animation
The Animatrix
Final Flight of the Osiris

Page 31

Reality is *Very* Complex

Virtual Reality
Artificial

life and other natural phenomena

Discuss later

Page 32

Many Open Research Problems

How

to model/render/animate a complex scene?

Computer Vision
Related

fields

Image processing Pattern P recognition ii Visual perception

Image understanding

True IU seems to involve a great deal of human intelligence Automated systems y are still far from human p performance Some good solutions in constrained special cases

(e.g., inspection: IC manufacturing, circuit boards)

Inverse

problems are generally tougher to solve

Page 33

WHAT DO YOU SEE?

AND NOW?

Page 34

Vision Research Themes

Edge

and region extraction segmentation reconstruction: From images to surfaces from X

Image Visual Shape

Contours Shading Stereo Motion

Object Object Event

tracking recognition (including faces)

and activity recognition

Interactive Image Analysis Using Snakes

Page 35

Retinal Angiogram Segmentation

Cartographic Modeling: Rivers

P. Fua. Fast, Accurate and Consistent Modeling of Drainage and Surrounding Terrain. International Journal of Computer Vision, 1998.

Page 36

Cartographic Modeling: Buildings and Roads

Automated Image Retreival

Find pictures

of the

Sydney opera house

in a large database on the web

Page 37

Space Robotics: Mars Rover

SelfSelf -Driving Cars: Intelligent Transportation

Professor

Stefano Soatto

Page 38

Visual Servoing

Tom Drummond, Univ. of Cambridge

ModelModel -Based Heart Motion Tracking

Page 39

3D Reconstruction
From image(s)

to model(s)

Inverting 3D projection Image Image-based modeling

3D model Viewpoint

Image

Cooking with Kurt

Model Model-based scene

reconstruction/animation

Page 40

Human Modeling from Video

M. Dimitrijevic, S. Ilic, and P. Fua CVPR04, S. Ilic, and P. Fua PAMI06

PerformancePerformance -Based Animation

[Pighin, Szeliski,

Salezin, 1999]

Page 41

PerformancePerformance -Based Animation

[Pighin, Szeliski,

Salezin, 1999]

PerformancePerformance -Based Animation

[Pighin, Szeliski,

Salezin, 1999]

Page 42

ALife Modeling Pyramid

Computational model

of animals and humans Cognition Learning Behavior

Perception Biomechanics Physics / Locomotion

An Artificial Fish Model

Body
Innate Characteristics

Action Selection

Brain
Learning

Behavior Routines Sensors

Optimization

Motor Controllers

Perception

Behavior
Biomechanical Model

Motor

Display Model

Page 43

ALife Characters in Interactive Games

These autonomous characters

are alive

Nintendogs (2005) CyberLifes Creatures (1996)

Page 44

ALife for Vision

Deformable organisms for automated medical

image

segmentation
Memory and prior knowledge Perception

Brain

Plan or schedule Interactions with other creatures

Perceptual attention mechanism

Muscle actuation causes shape deformation

S ke

le to

Sensors

Underlying medial based Shape representation

Muscles and limbs

body upper/right lower/left genu

N-2

fornix splenium

rostrum
N-1 N

3 2 1

Active Visual Tracking

Stereo Retinal Images Retinal Backprojection of Color Model

Top View

Oblique View

Page 45

Active Vision System

Color Models

Optical flow minimization

Color histogram
intersection & backprojection

Retinal Image Stream

Further Visual Analysis

Stabilization Module

Foveation Module

Gaze Control

Locomotion Control

Self-Animating Virtual Pedestrians SelfIn a Virtual Train Station (Wei Shao)

Autonomous pedestrian

behavior

Concourses Main Waiting Room

Old Penn Station in NYC

Page 46

Simulation of Pedestrian Activity in the Train Station

Virtual Vision

Simulated

visual sensor networks

Page 47

Persistently Observing a Pedestrian

CS Faculty Conducting CG and CV Research

Stanley Osher (CV - Mathematics) Stefano Soatto (CV - Computer Science) Demetri Terzopoulos (CG & CV - Computer Science) Zhuowen Tu (CV - Neurology) Alan Yuille (CV - Statistics) SongSong -Chun Zhu (CV - Statistics)

Page 48

Courses Offered
CS CS CS

174A Introduction to Computer Graphics 174B Image Image-Based Modeling and Rendering 174C Computer Animation

Plus several graduate courses in graphics and vision:

Artificial Life for Computer Graphics and Vision Machine Vision Humanoid Character Simulation Building an Autonomous Robot Etc.

Thank you!

Page 49