Image Processing and Image

Filters in Java
by John F. McGowan, Ph.D.
Desktop Video Expert Center
NASA Ames Research Center

John F. McGowan, Ph.D.

2/16/98 1
E-Mail: [email protected]
 Introduction
n Java is too slow for production image
processing, but may offer advantages for
prototyping and demonstration of image
processing algorithms.
n Algorithm developers and scientists want
a tool that allows them to concentrate on
their work (math, science), not on the
tool (C, Mathematica, Java).

John F. McGowan, Ph.D.

2/16/98 2
E-Mail: [email protected]
 Image Filters in Java
n Standard Java Wrapper for Image
Processing Operations.
n java.awt.image Package
• ImageFilter and RGBImage Filter classes
• user derived filter classes
n In JDK 1.0
n Extended slightly in JDK 1.1
n Extended heavily in JDK 1.2 Beta
John F. McGowan, Ph.D.
2/16/98 3
E-Mail: [email protected]
 Image Filters and NASA
n Astronomical Images
• Planets
• Sun and Stars
n Satellite Images
n Aerial Photographs
n Images from Simulations
• Computational Fluid Dynamics
• Solar Models
John F. McGowan, Ph.D.
2/16/98 4
E-Mail: [email protected]
 Image Filters and NASA
n Edge Detection and Enhancement
n Image Segmentation
n Image Enhancement
n Image Analysis
• e.g. Fourier Transforms
n Computer Vision
n Special Effects

John F. McGowan, Ph.D.

2/16/98 5
E-Mail: [email protected]
 Outline of Talk
n Desktop Video Expert Center
n Algorithm Development
n Image Filters and Image Processing in
Java
• Java Demo
• How Java Images Work
n Conclusions

John F. McGowan, Ph.D.

2/16/98 6
E-Mail: [email protected]
 Desktop Video Expert Center
n Advanced Applications (R and D)
n Installation and Support of
Videoconferencing Systems
n NASA Space Shuttle MBONE (Multicast
Backbone) Broadcasts over Internet
n Evaluations of COTS Desktop Video
Products
n http://zeus.arc.nasa.gov/
John F. McGowan, Ph.D.
2/16/98 7
E-Mail: [email protected]
 DVEC Advanced Applications
n Research and Evaluate Future Desktop
Video and Networked Video
Technologies.
n Develop New Technologies as
Appropriate
n Software and Algorithm Development
n http://zeus.arc.nasa.gov/adv_apps.html

John F. McGowan, Ph.D.

2/16/98 8
E-Mail: [email protected]
 Some Current Projects
n DCTune: Perceptual Optimization of
JPEG Images (with Vision Science and
Technology Group)
n Studies of Wavelet, Fractal, and Other
Leading Edge Video Coding
Technologies
• Smooth full-motion video over Internet?
• Smooth full-motion video over telephone?

John F. McGowan, Ph.D.

2/16/98 9
E-Mail: [email protected]
 DCTune
n Patented NASA algorithm developed by
Andrew B. Watson, Al Ahumada, and
others with the Vision Science and
Technology Group.
n Mathematica Prototype
n Converted to compiled C language
binary executable
n http://vision.arc.nasa.gov/dctune1.1.html

John F. McGowan, Ph.D.

2/16/98 10
E-Mail: [email protected]
 Who am I?
n Converted DCTune from Mathematica
Prototype to portable ANSI C language
version.
n Developed commercial MPEG-1 and
MPEG-2 audio and video playback
(decoder) software written in C for
PC/Windows, Power Macintosh, and
Unix platforms.

John F. McGowan, Ph.D.

2/16/98 11
E-Mail: [email protected]
 Who am I?
n Image,Video, and Audio Compression
Algorithms
n Ph.D. in Physics, University of Illinois at
Urbana-Champaign
• Maximum Likelihood Fitting Methods
• Monte Carlo Simulations
• Pattern Recognition

John F. McGowan, Ph.D.

2/16/98 12
E-Mail: [email protected]
 Developing Algorithms
n Cumbersome process
n Symbolic Manipulation Programs such
as Mathematica, MATLAB, and MAPLE
n C/C++ Prototypes
n Pros and Cons to Both Methods
n Looking for Better Tools to Develop,
Prototype and Demonstrate Algorithms

John F. McGowan, Ph.D.

2/16/98 13
E-Mail: [email protected]
 Traditional Algorithm
Development
n Proof of Concept in Software on
Supercomputer, Mainframe or Unix
Workstation
n Make Presentation or Publish a Paper
n Present Simple Demo
• Grayscale Images
• Very Slow
n Fund Further Development
John F. McGowan, Ph.D.
2/16/98 14
E-Mail: [email protected]
 Traditional Algorithm
Development
n Design and Fabricate VLSI Chip
Implementing the Algorithm
n Develop Better Software and wait for
CPU’s to get Fast Enough
• Optimize the Software for Speed
• Add Color Support
• Support Arbitrary Picture Dimensions

John F. McGowan, Ph.D.

2/16/98 15
E-Mail: [email protected]
 The New World
n CPU Speeds Heading Above 500 MHz
n Algorithms Can Run on Desktop PC
n Very Fast Transition from Prototype to
Product
n Argues for developing in C/C++ in
PC/Windows environment.
n What can Java do?

John F. McGowan, Ph.D.

2/16/98 16
E-Mail: [email protected]
 Mathematica Advantages
n Symbolic manipulation
n Interpreted, Interactive
n High Level Mathematics
• matrix multiplication m = a . b
• special math functions
• much more
n Built-In Graphical Rendering
• Show[graphics]
John F. McGowan, Ph.D.
2/16/98 17
E-Mail: [email protected]
 Mathematica Disadvantages
n Slow
n Proprietary (Wolfram Research)
n Expensive ($2000)
n Requires Much Memory (300 MB)
n Arcane Syntax
• e.g., MapThread[f, {{a,b,c},{ap,bp,cp}}]
n Learning Curve for C or other procedural
language programmers
John F. McGowan, Ph.D.
2/16/98 18
E-Mail: [email protected]
 Other Symbolic Packages
n MATLAB
n MAPLE
n REDUCE
n Probably similar advantages and
disadvantages to Mathematica.

John F. McGowan, Ph.D.

2/16/98 19
E-Mail: [email protected]
 Many Symbolic Language
Prototypes
n Hard to reach mass audience.
• For example, must own Mathematica
n Usually must convert to C, assembler, or
an ASIC for a product.
n Usually must embed C version in
platform OS or API
• Video for Windows codec
• QuickTime Component
John F. McGowan, Ph.D.
2/16/98 20
E-Mail: [email protected]
 C/C++ Advantages
n Fast/Compiled
n Executables are Free
n Unix GNU Compiler is Free
n Mac and PC Commercial Compilers
n De Facto Standard
• Lingua Franca of Programmers
• ANSI Standard

John F. McGowan, Ph.D.

2/16/98 21
E-Mail: [email protected]
 C/C++ Disadvantages
n No Built-In Graphics
• Microsoft Windows (PC)
• Mac Toolbox (MacOS)
• X Windows (Unix)
• Many Graphic File Formats (e.g. PPM)
n No High Level Mathematics
n No symbolic manipulation
n Porting Problems
John F. McGowan, Ph.D.
2/16/98 22
E-Mail: [email protected]
 ANSI C Porting Problems
n Byte-Order Dependent Code
n Gaps in Standard such as definition of
bit shift by a negative number: a >> -1
n Bugs in Compilers. Video Compression
Algorithms Stress Compilers.
n Complex Multimedia Algorithms Don’ t
Always Port Easily

John F. McGowan, Ph.D.

2/16/98 23
E-Mail: [email protected]
 Many C/C++ Prototypes
n MPEG Software n Read/write
Simulation Group sequences of still
n Berkeley MPEG images (PPM, TGA
Tools etc.)
n Wavelet Image n Often Grayscale
Construction Kit Only
n Telenor H.263 Code n Much work to
convert to graphical
applications.

John F. McGowan, Ph.D.

2/16/98 24
E-Mail: [email protected]
 Possible Java Advantages
n Immediate Demonstration on Web
• Administrators and Funding Agencies
• Colleagues
• Potential Customers and Licensors
n Built-In Graphics Rendering
n Image and ImageFilter Classes
n Portable
n Free
John F. McGowan, Ph.D.
2/16/98 25
E-Mail: [email protected]
 Possible Java Disadvantages
n Slow (Interpreted Byte-code)
n No High Level Mathematics
n No symbolic manipulation
n Algorithm Must be Ported from Java to C
to Create Product
n Not Portable
n Not Free

John F. McGowan, Ph.D.

2/16/98 26
E-Mail: [email protected]
 Java Image Processing Demo
n Demonstration of several standard edge
enhancement and edge-detection
algorithms.
n Moderately complex algorithms.
n A Real World Test of Java
n http://zeus.arc.nasa.gov/sword.html

John F. McGowan, Ph.D.

2/16/98 27
E-Mail: [email protected]
 Motivation for Demo
Algorithms
n Study edge enhancement and edge
detection algorithms for possible
incorporation in a compression
algorithm.
n Edge detection solves many problems:
character recognition, flaw detection in
silicon wafers, remote surveillance, and
many other applications.

John F. McGowan, Ph.D.

2/16/98 28
E-Mail: [email protected]
 Why Edge Detection for
Compression?
n Leading lossy video compression
algorithms such as Discrete Cosine
Transform and wavelets throw away or
distort critical details at edges.
n Need to preserve sharp edges and lines
for low bitrate image coding to look
“natural”to a human viewer!

John F. McGowan, Ph.D.

2/16/98 29
E-Mail: [email protected]
 Edges and Block DCT

JPEG Encoder in Paint Shop Pro

John F. McGowan, Ph.D.
2/16/98 30
E-Mail: [email protected]
 Edges and Wavelets

Wavelet Image Construction Kit (Geoff Davis)

John F. McGowan, Ph.D.
2/16/98 31
E-Mail: [email protected]
 Demo Java Filter Applet
n Java Filter Applet (using JDK 1.1.4)
n http://zeus.arc.nasa.gov/sword.html
n The Filter Applet works differently under
different Web browsers and operating
systems. :-(
n Fails completely for a few browsers and
operating system combinations. :-(

John F. McGowan, Ph.D.

2/16/98 32
E-Mail: [email protected]
 Images in Java
n Poorly documented
n java.awt Package
• Image class
n java.awt.image Package
• ImageProducer, ImageConsumer, etc.
n Here is my educated guess how it works

John F. McGowan, Ph.D.

2/16/98 33
E-Mail: [email protected]
 Images in Java
n Image class
n ImageProducer interface
n ImageConsumer interface
n ImageFilter class (implements
ImageConsumer interface)
n FilteredImageSource class (implements
ImageProducer interface)

John F. McGowan, Ph.D.

2/16/98 34
E-Mail: [email protected]
 Java Graphics and Images

Applet::paint(Graphics g) { g.drawImage(image,...) }

image = createImage(FilteredImageSource)

Applet Image FilteredImageSource

(ImageProducer)

Java Virtual Machine decides when to update data?

John F. McGowan, Ph.D.

2/16/98 35
E-Mail: [email protected]
 Java Image Processing
Java
Graphics FilteredImageSource

ImageProducer
Applet.getImage().getSource()
ImageFilter

Java Virtual Machine decides when to update data?

John F. McGowan, Ph.D.

2/16/98 36
E-Mail: [email protected]
 Image Class
n Abstract class represents a displayable
image in a platform-independent way.
n java.awt Package (NOT java.awt.image)
n Don’ t use constructor to create an Image
• Applet.getImage() or createImage(producer)
n Image.getSource() returns the
ImageProducer that produces the image
data.
John F. McGowan, Ph.D.
2/16/98 37
E-Mail: [email protected]
 ImageProducer Interface
n Defines the methods that any class that
produces image data must define to
enable communication with
ImageConsumer classes.
n FilteredImageSource (java.awt.image) is
an ImageProducer
n java.awt.image Package

John F. McGowan, Ph.D.

2/16/98 38
E-Mail: [email protected]
 ImageProducer Interface
n SOURCE of the image data
n not the IMAGE data
n file on hard disk
n URL of image (JPEG or GIF) on network
n a filter that processes an image
• FilteredImageSource

John F. McGowan, Ph.D.

2/16/98 39
E-Mail: [email protected]
 ImageConsumer Interface
n Interface defines the methods necessary
for a class that consumes image data
to communicate with a class that
produces image data, an
ImageProducer.
n ImageFilters in Java implement the
ImageConsumer Interface
n java.awt.image Package

John F. McGowan, Ph.D.

2/16/98 40
E-Mail: [email protected]
 ImageFilter Class
n java.awt.image Package
n Implement the ImageConsumer interface
n Represent a filter that performs an
operation on an image
n Filter method run when the ImageFilter
retrieves new data from an
ImageProducer? (JVM?)

John F. McGowan, Ph.D.

2/16/98 41
E-Mail: [email protected]
 FilteredImageSource Class
n Implements the ImageProducer Interface
n Constructor
FilteredImageSource(ImageProducer
orig, ImageFilter imgf)
n Image.getSource() retrieves the
ImageProducer associated with a Java
Image

John F. McGowan, Ph.D.

2/16/98 42
E-Mail: [email protected]
 FilteredImageSource Class
n Represents an object that takes data
from another ImageProducer and
applies a filter, represented by an
ImageFilter class, and passes the
filtered image on to another java object.
n Is itself an ImageProducer

John F. McGowan, Ph.D.

2/16/98 43
E-Mail: [email protected]
 Image Events?
n ImageProducers send “events”to
ImageConsumers requesting image
data.
n Images, ImageConsumers,
ImageProducers maintain local buffers
with copies of the image.
n Different Java Virtual Machines decide
to send image events at different times.
John F. McGowan, Ph.D.
2/16/98 44
E-Mail: [email protected]
 Image Events?
n Standard does not define or products
incorrectly implement the handshaking
between ImageProducers, Consumers,
and other graphics components.
n Updating of data in different objects
occurs at different times with different
Java Virtual Machines.

John F. McGowan, Ph.D.

2/16/98 45
E-Mail: [email protected]
 Conclusions
n ImageFilter Class is a Great Idea.
n Java GUI easy to code.
n Slow, but slow is acceptable for a
prototype.
n Portability is NOT THERE YET!
• Biggest problem with Java ImageFilters.
• Limits DEMONSTRATION of prototypes.

John F. McGowan, Ph.D.

2/16/98 46
E-Mail: [email protected]
 Suggestions for Sun
n Improve Portability
• standardize image updating?
n More Speed (Interpreted Java)
n Java to Machine Code Compiler
n High Level Math Classes
• Matrices and Matrix Operations
• Special Functions (e.g. Error Function)

John F. McGowan, Ph.D.

2/16/98 47
E-Mail: [email protected]
 Where to Get This Talk
n http://zeus.arc.nasa.gov/sword.html
• The Java Demo
n http://zeus.arc.nasa.gov/jugfeb18.pdf
• Adobe PDF Format Version
n http://zeus.arc.nasa.gov/jugfeb18.ppt
• Microsoft Windows Power Point Version
n http://zeus.arc.nasa.gov/
• Desktop Video Expert Center Web Site
John F. McGowan, Ph.D.
2/16/98 48
E-Mail: [email protected]