CGLecture 02 Interactive Graphics
CGLecture 02 Interactive Graphics
1/39
What is Computer Graphics? (1/2)
Computer graphics generally means creation, storage and manipulation
of models and images
Such models come from diverse and expanding set of fields including
physical, biological, mathematical, artistic, and conceptual/abstract
structures
Frame from animation by William Latham, shown at
SIGGRAPH 1992. Latham creates his artwork using
rules that govern patterns of natural forms.
2/39
What is Computer Graphics? (2/2)
William Fetter coined term “computer graphics” in
1960 to describe new design methods he was pursuing
at Boeing for cockpit ergonomics
Created a series of widely reproduced images on “pen
plotter” exploring cockpit design, using 3D model of
human body.
“Perhaps the best way to define computer graphics is to find out what it is not. It is not a machine. It is
not a computer, nor a group of computer programs. It is not the know-how of a graphic designer, a
programmer, a writer, a motion picture specialist, or a reproduction specialist.
Computer graphics is all these – a consciously managed and documented technology directed toward
communicating information accurately and descriptively.”
Computer Graphics, by William A. Fetter, 1966
3/39
What is Interactive* Computer Graphics? (1/2)
User controls content, structure, and appearance of objects and their
displayed images via rapid visual feedback
Basic components of an interactive graphics system
input (e.g., mouse, stylus, multi-touch, in-air fingers…)
processing (and storage of the underlying representation/model)
display/output (e.g., screen, paper-based printer,
video recorder…)
First truly interactive graphics system,
Sketchpad, pioneered by Ivan Sutherland 1963 Ph.D. thesis Note CRT monitor, light
pen, and function-key
Sketchpad, A Man-Machine Graphical Communication System panels – the “organ
console” showing bi-
Used TX-2 transistorized “mainframe” manual operation
at MIT Lincoln Lab
* Sometimes called real-time computer graphics, and in certain
contexts, real-time rendering 4/39
What is Interactive Computer Graphics? (2/2)
Almost all key elements of interactive graphics system are expressed in first
paragraph of Sutherland’s 1963 Ph.D. thesis
5/39
What is Batch Computer Graphics?
Today, we still use non-interactive batch mode (aka offline rendering) for final
production-quality video and film (special effects – FX). Rendering a single frame of
The Good Dinosaur (2015, a 24 fps movie) averaged 48 hours on a 30,000-core render
farm!
8/39
Enabling Modern Computer Graphics (3/5)
Input Devices
Mouse, tablet & stylus, multi-touch, force feedback, and other game controllers
(e.g., Wii U), scanner, digital camera (images, computer vision), etc.
Body as interaction device
http://youtu.be/zXghYjh6Gro
9/39
Enabling Modern Computer Graphics (4/5)
Many form factors
Smartphones/laptops/desktops/tablets
Smart watches Android Phones
Apple iPhone Tablets
Head-mounted displays (HMDs)
HTC Vive
Augmented Reality
Virtual Reality
AR vs VR: Different experiences! Microsoft’s first
Surface Apple Watch Android Wear
Brown’s old Cave Microsoft Hololens Vive Oculus Rift Google Cardboard
10/39
Digression: Augmented Reality
Easily accessible AR through smartphones
11/39
Digression: Cave Redesign
Old Cave:
4 1024 x 786 projectors on 8’ x 8’ walls (8-10 pixels per inch)
Too low resolution and brightness for many applications, and got
worse (brightness, contrast deteriorated over time)
New Cave:
69 projectors onto cylindrically
curved screen 8’ radius, floor, ceiling
140 million pixels
Powered by a ~69 gpu cluster
No right angles, up to 40 pixels per
inch (can’t see individual pixels at Brown’s new Cave, the YURT
normal viewing distance)
12/39
Enabling Modern Computer Graphics (5/5)
Software Improvements
Algorithms and data structures
Modeling of materials
Rendering of natural phenomena
“Acceleration data structures” for ray tracing and other renderers
Parallelization
Most operations are embarrassingly parallel: calculating value of one
pixel is often independent of other pixels
Distributed and Cloud computing
Send operations to the cloud, get back results, don’t care how
Rendering even available as internet service!
13/39
Environmental Evolution (1/5)
Character Displays (1960s – now)
Display: text plus alphamosaic pseudo-graphics (ASCII art)
Object and command specification: command-line typing
Control over appearance: coding for text formatting
(.p = paragraph, .i 5 = indent 5)
Application control: single task
14/39
Environmental Evolution (2/5)
Vector (Calligraphic, Line Drawing)
Displays (1963 – 1980s)
Display: line drawings and stroke text; 2D and 3D transformation hardware
Object and command specification: command-line typing, function keys, menus
Control over appearance: pseudo-WYSIWYG
Application control: single or multitasked, distributed computing pioneered at
Brown via mainframe host <-> minicomputer satellite
Term “vector” graphics survives as “scalable vector graphics” SVG library from
Adobe and W3C – shapes as transformable objects rather than just bitmaps
15/39
Environmental Evolution (3/5)
2D bitmap raster displays for PCs and workstations
(1972 at Xerox PARC - now)
Display: windows, icons, legible text, “flat earth” graphics
Above, a classic WIMP
Note: late 60’s saw first use of raster graphics, especially for interface. The technology,
flight simulators
at its core, remains largely
Minimal typing via WIMP GUI (Windows, Icons, Menus, the same today. Below, a
Pointer): point-and-click selection of menu items and modern WIMP interface.
objects, direct manipulation (e.g., drag and drop), “messy
desktop” metaphor
Control over appearance: WYSIWYG (which is really
WYSIAYG, What You See Is All You Get – not pixel-accurate
or controllable)
Application control: multi-tasking, networked client-server
computation and window management (even “X terminals”)
16/39
Environmental Evolution (4/5)
3D graphics workstations (1984 at SGI – now)
could cost up to $1M for high-end!
Display: real-time, pseudo-realistic images of 3D
scenes
Object and command specification: 2D, 3D and N-D
input devices (controlling 3+ degrees of freedom) and
force feedback haptic devices for point-and-click,
widgets, and direct manipulation Graphics workstations such as
Control over appearance: WYSIWYG (still WYSIAYG) these have been replaced with
commodity hardware (CPU + GPU),
Application control: multi-tasking, networked e.g., our MaxBuilts + NVIDIA cards
(client/server) computation and window management
17/39
Environmental Evolution (5/5)
High-end PCs with hot graphics cards (NVIDIA GeForce™, AMD
Radeon™) have supplanted graphics workstations
Such PCs are clustered together over
high speed buses or LANs to provide
“scalable graphics” to drive tiled
PowerWalls, CAVEs, etc.
Also build GPU-clusters as number crunchers,
e.g., protein folding, weather prediction
Now accessible to consumers via
technologies like NVIDIA’s
SLI (Scalable Link Interface) bridge
SLI Image from http://www.overclock.net/t/1606562/official-
nvidia-titan-x-pascal-owners-thread/3410 18/39
Graphics Display Hardware
Vector (calligraphic, stroke, Raster (TV, bitmap, pixmap) used
random-scan) in displays and laser printers
Driven by display commands Driven by array of pixels (no semantics,
lowest form of representation)
(move (x, y), char(“A”) , line(x, y)…)
Note “jaggies” (aliasing errors) due to
Survives as “scalable vector graphics” discrete sampling of continuous primitives
Ideal Vector
Drawing Drawing Outline Filled
19/39
Conceptual Framework for Interactive Graphics
Graphics library/package is intermediary between application and display
hardware (Graphics System)
Application program maps application objects to views (images) of those
objects by calling on graphics library. Application model may contain lots of
non-graphical data (e.g., non-geometric object properties)
User interaction results in modification of image and/or model
This hardware and software framework is 5 decades old but is still useful
Software Hardware
Graphics
System/
Application Graphics GPU
Application
Library
Model / database program
20/39
Graphics Library
Examples: OpenGL™, DirectX™, Windows Presentation Foundation™
(WPF) accessed via XAML, RenderMan™, HTML5 + WebGL™
Primitives (characters, lines, polygons, meshes,…)
Attributes
Color, line style, material properties for 3D
Lights
Transformations
Immediate mode vs. retained mode
immediate mode: no stored representation, package holds only
attribute state, and application must completely draw each frame
retained mode: library compiles and displays from scenegraph
that it maintains, a complex DAG. It is a display-centered extract
of the Application Model
21/39
Application Distinctions: Two Basic Paradigms
Sample-based graphics vs Geometry-based graphics
22/39
Sample-based Graphics (1/3)
Sample-based graphics: Discrete samples
are used to describe visual information
pixels can be created by digitizing images,
using a sample-based “painting” program,
etc.
often some aspect of the physical world is
sampled for visualization, e.g., temperature
across the US
example programs: Adobe Photoshop™,
GIMP™ , Adobe AfterEffects™ (which came
out of CS123/CS224!)
23/39
Sample-based Graphics (2/3)
Pixels are point locations with associated sample values, usually of light
intensities/colors, transparency, and other control information
When we sample an image, we sample the point location along the
continuous signal and we cannot treat the pixels as little circles or squares,
though they may be displayed as such
CRT* beam illumination pattern
light intensity
1 pixel
Visualization of a Can’t visually resolve
mathematical pixel grid LCD display adjacent pixels on CRT
Sample values can also be input numerically (e.g., with numbers from computed dataset)
Once an image is defined as pixel-array, it can be manipulated
Image editing: changes made by user, such as cutting and pasting sections, brush-type tools, and
processing selected areas
Image processing: algorithmic operations that are performed on image (or pre-selected portion of image)
without user intervention. Blurring, sharpening, edge-detection, color balancing, rotating, warping. These
are front-end processes to Computer Vision, Computational Photography
25/39
Sampling an Image
Lets do some sampling of CIT building
3D scene
A color value is measured at every grid point and used to color corresponding
grid square 0 = white, 5 = gray, 10 = black
27/39
What’s the Disadvantage?
WYSIAYG (What You See Is All You Get): No additional
information
no depth information
can’t examine scene from different point of view
at most can play with the individual pixels or groups of
pixels to change colors, enhance contrast, find edges, etc.
But increasingly great success in image-based rendering
to fake 3D scenes and arbitrary camera positions. New
images constructed by interpolation, composition,
warping and other operations.
For a computational and cognitive science perspective,
Take James Tompkin’s Computer Vision (CSCI1430)
“Scene Reconstruction from High Spatio-Angular
Resolution Light Fields” by Kim, Zimmer et al., 2013
28/39
Geometry-Based Graphics (1/2)
Geometry-based graphics (also called scalable
vector graphics or object-oriented graphics):
geometrical model is created, along with various
appearance attributes, and is then sampled for
visualization (rendering, a.k.a image synthesis)
often some aspect of physical world is visually
simulated, or “synthesized”
examples of 2D apps: Adobe Illustrator™ and Corel
CorelDRAW™
examples of 3D apps: Autodesk’s AutoCAD™,
Autodesk’s (formerly Alias|Wavefront’s) Maya™,
Autodesk’s 3D Studio Max™
29/39
Geometry-Based Graphics (2/2)
Geometry-based graphics applications
Store mathematical descriptions, or “models,” of geometric elements (lines,
polygons, polyhedrons, polygonal meshes…) and associated attributes (e.g., color,
material properties).
Geometric elements are primitive shapes, primitives for short.
Images are created via sampling of geometry for viewing, but not stored as part of
model.
Users cannot usually work directly with individual pixels in geometry-based
programs; as user manipulates geometric elements, program resamples and
redisplays elements
Increasingly rendering combines geometry- and sample-based graphics,
both as performance hack and to increase quality of final product
CG animated characters (geometry) on painted or filmed scene images (samples)
30/39
What is Geometric Modeling?
What is a model?
Captures salient features (data, behavior) of object/phenomenon being
modeled
data includes geometry, appearance, attributes…
note similarity to OOP ideas
Modeling allows us to cope with complexity
Our focus: modeling and viewing simple everyday objects
Consider this:
Through 3D computer graphics, we have abstract, easily changeable 3D forms, for the
first time in human history
Has revolutionized working process of many fields – science, engineering, industrial
design, architecture, commerce, entertainment, etc. Profound implications for visual
thinking and visual literacy
“Visual truth” is gone in the Photoshop and FX-saturated world (but consider painting
and photography…) – seeing no longer is believing…(or shouldn’t be!)
31/39
Modeling vs. Rendering
Modeling Rendering
Create models Take “picture” with camera
Apply materials to models
Place models around scene Both can be done with commercial software:
Place lights in scene
Place the camera Autodesk MayaTM ,3D Studio MaxTM, BlenderTM, etc.
Point Light
Spot
Light
Directional Light
Ambient
Light
32/39
Decomposition of a Geometric Model
Divide and Conquer
Hierarchy of geometrical components
Reduction to primitives (e.g., spheres, cubes, etc.)
Simple vs. not-so-simple elements (nail vs. screw)
Head
Shaft
Point
composition decomposition
33/39
Hierarchical (Tree) Diagram of Nail
Object to be modeled is (visually) analyzed, and then decomposed into collections of
primitive shapes.
Tree diagram provides visual method of expressing “composed of” relationships of
model
34/39
Composition of a Geometric Model
Translate
Scale and Translate
Primitives Composition
in their own modeling in world (root)
coordinate system coordinate system
Primitives created in decomposition process must be assembled to
create final object. Done with affine transformations, T, R, S (as in
above example). Order matters – these are not commutative!
35/39
Upcoming Topics
We manipulated primitive shapes with geometric
transformations (translation, rotation, scale). These
transformations are essential for model organization,
process of composing complex objects from simpler
components.
Hierarchical models and geometric transformations are
also essential for animation – create and edit scenegraphs
Once object’s geometry is established, must be viewed on
screen: map from 3D geometry to 2D projections for
viewing, and from 2D to 3D for 2D input devices (e.g., the
mouse or pen/stylus, or touch)
While mapping from 3D to 2D, object (surface) material
properties and lighting effects are used in rendering one’s
constructions.
This rendering process is also called image synthesis
36/39
In Summary
Computer graphics involves both real-time / interactive applications and batch /
offline applications
Both equally important, but different use cases
Photo-realism has really advanced
But it still takes dozens of hours on fastest computers to mimic physics of photons
interacting with physical environments
Hardware evolution from vector to raster graphics
Vector graphics survives as Scalable Vector Graphics, which transforms without artifacts
Geometry-based vs. image-based graphics
Mathematical definition vs. pixel manipulation
Pixels are discrete samples of continuous functions
Causes artifacts (“jaggies”/ “aliases”) to appear (we will study fixes – “anti-aliasing”)
Geometric models typically constructed hierarchically
Scene graph data structure
37/39
What is Graphics Processing Unit (GPU)?
It is a processor optimized for 2D/3D graphics, video, visual computing,
and display.
It is highly parallel, highly multithreaded multiprocessor optimized for
visual computing.
It provide real-time visual interaction with computed objects via
graphics images, and video.
It serves as both a programmable graphics processor and a scalable
parallel computing platform.
Heterogeneous Systems: combine a GPU with a CPU
FIGURE A.2.2
Contemporary PCs with
Intel and AMD CPUs. See
Chapter 6 for an
explanation of the
components and
interconnects in this
figure. Copyright © 2009
Elsevier
The origin
In this example, the origin or (0,0) is in the bottom-left of the screen. It is also possible to have
the origin in the top-left, in which case the rows (y coordinates) are numbered downwards.