COMP 478/6771

Image Processing

Instructor: Dr. Yaser Esmaeili

Email: [email protected]

Department of Computer Science and Software Engineering

Concordia University

Slides modified from materials provided by Drs. Yiming Xiao & Tien Bui
 COMP 478/6771
• Instructor: Yaser Esmaeili
• Email: [email protected]
• Office hour: Mon. and Fri. : 12:30 - 13:00 ET (via ZOOM, by
an appointment via email)
• Classroom: LS 210 SGW
• Lectures: Tues: 2:45 pm ~ 5:30 pm
• Tutorials/Labs/Marker:
Sepehr Ghamari Tutorial [email protected]

Marzieh Adeli Tutorial [email protected]

Shamsabad
Sami Ben Brahim Marker [email protected]
Arash Harirpoush Marker [email protected]
Maryam Valipour Marker [email protected]

✓ LabDDDI-Monday——20:30~22:00 ET - (Room - H967) 2

✓ LabDDDJ-Monday——20:30~22:00 ET - (Room - H903)
Textbook: Digital Image Processing, 4th Edition, R. C.
Gonzalez and R. E. Woods, Prentice Hall.

Optional reference: Digital Image Processing using Matlab,3rd , 2nd

Edition, R. C. Gonzalez, R. E. Woods, S. L. Eddins, Prentice Hall.
3
• Course outline & planning document: key milestones, course
overview, and deadlines
• Moodle page: All course materials, discussion forum,
assignment submission, and course announcements - Please
check it regularly

• Assignments: There will be four assignments.

✓ Assignments must be done individually.
✓ Assignments are due strictly on the due date.
✓ No submissions will be accepted ONE day after the due date,
and in that period, there is a penalty of 50% for each day late.
✓ Assignments are worth 20% of the final mark.

• Tutorials: 12 sessions worth 10% of the final mark.

4
• Project: 30% of the final mark.
✓ Goal: deep understanding of the work through a team working project
and easy (transparent) transfer learning to others
✓ Students may complete the project in a team of 3~4 members
✓ Select an image processing title (following the definition mention in the
class, e.g. segmentation, denoising, captioning, face detection re
various objectives etc, from the publications in the next slide)
✓ MATLAB programming delivery (entire/partially)
✓ Final report should be no more than 10 pages long and contain a brief
paper review, details of the implementation, and all relevant codes
✓ In-class presentation

• Mid-term test: one mid-term test worth 25% of the final mark [Week 8].

• Final exam: 3-hr final exam at the end of term, worth 35% of the final mark.

• Note: A) There will be 120 (= 100+20 (bonus)) as the total mark for the
course! B) Paper’s submission with your own novelty/contributions

• Supporting software: MATLAB, Image Processing toolbox, Signal Processing

5
toolbox, Wavelet toolbox, etc.
 Project Description (1)
Publication h5-index h5-median
1 IEEE/CVF Conference on Computer Vision and Pattern Recognition 422 681

2 European Conference on Computer Vision 238 390

3 IEEE/CVF International Conference on Computer Vision 228 366

4 IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 179 318

5 IEEE Transactions on Image Processing (TIP) 138 199

6 Pattern Recognition 111 160
IEEE/CVF Computer Society Conference on Computer Vision and
7 108 176
Pattern Recognition Workshops (CVPRW)
8 Medical Image Analysis 103 171

h5-index is the h-index for articles published in the last 5 complete years. It is the largest number h such
that h articles published in 2018-2022 have at least h citations each.

h5-median for a publication is the median number of citations for the articles that make up its h5-index.

LN-1 ‫مبانی بینایی کامپیوتر‬

 Project Description (2)
➢ Deadlines:

➢ Grouping + Project Selection: (week 4)

➢ MATLAB Programming of the paper (week 8) [5%]

➢ Presentation and Q&A (Weeks 12/13) [10%]

➢ MATLAB Programming over the alternative data sets / comparing the results
and final report (Week 13) [15%]

LN-1 ‫مبانی بینایی کامپیوتر‬

8
Overview • Definitions
• Motivations
• Applications

Review • Matrices and Vectors

• Probability/Statistics
• Linear Systems

9
 Imaging

Camera
Obscura
concept

10
https://owlcation.com/humanities/Leonardo-da-Vincis-Camera-Obscura
Imaging History of photography

11
 What is a digital image?
• Sensors: CCD (Charge-Coupled Device) & CMOS (Complementary
Metal-Oxide Semiconductor)
✓ Photon -> electron -> voltage

https://www.teledynedalsa.com https://silentpeakphoto.com

12
CCD from a Sony alpha DSLR-
CMOS sensor from Canon 5D Mark II
A300 (10.14 Megapixel)

Bulk processing vs. element-wise processing

13
 What is a digital image?
• Divided into array of light-sensitive cells called photosites or pixels
• Each pixel has a gray-level or brightness (e.g, 0-255)
• For color pixel: needs three channels or bands (RGB)
• Resolution or size: number of rows and columns in the image
1280 x 720 (720p) for HD video
For brain MRI images: ~256x256x120 (~1x1x1 mm3)

www.javatpoint.com/concept-of-pixel 14
Color channels
What Is Digital Image Processing?

• A sample
digital image.
662*640*256.

x
• f(x,y): A two-dimensional function,
where x and y are spatial coordinates,
and the amplitude of f at any pair of
coordinates (x,y) is called the intensity
or gray level of the image at that point.
• x size: 662, y size: 640, gray levels:
256
• Digital image: x, y, and the amplitude
values of f are all finite, discrete
quantities
• A sample
color
digital
image,
800*600*2
4 bits
• Pixel: The elements of a digital image.

Pixels
The Origins of Digital Image Processing

• One of the first applications of digital

images was in the newspaper industry,
when pictures were first sent by
submarine cable between London and
New York.
 From computers, meaningful image
processing tasks appeared.
• Medical imaging
• Remote Earth resource observations
• Astronomy
• High-energy plasmas and electron
microscopy
Examples of Fields tat Use Digital Image
Processing

• Electromagnetic energy spectrum

 What is Digital Image Processing?

• An image is a function of two spatial variables f(x,y), where

(x,y) denote the planar image coordinates, and f(.) is the
amplitude of the image at the given location.
• Digital image processing: manipulate an image as a function
with different algorithms and systems to obtain new images.

x 24
 Digital Image Representation

A pixel in 3x3 neighborhood, at

location (x,y)

Intensity values = {0, 0.5, 1}

25
 Digital Image Representation

A pixel in 3x3 neighborhood, at

location (x,y)

26
 Image Sampling and Quantization

Analogue information Digital information

27
 Image Acquisition

Camera lens

28
Image representation

29
• An MxN digital grayscale image: (for color f(x,y) would be
1 colour component)

• Book Convention: (same as MATLAB, except MATLAB starts

index at 1 instead of 0)

(1,1) y (columns)

x (rows) 30
Digital Image Fundamentals
 The Grayscale and Its Perception
• We call the intensity L = f(x,y) the “gray level” value

• The range of values of Lmin < L < Lmax is called the

grayscale.

• Commonly, we use the range [ 0 , L-1 ] for integer L

that is a power of 2.
Black White

32
 Digital Image Fundamentals

Pixel intensity is 2k
Image size is NxN
 Storage Needs for Images

• Image MxN pixels, 2B gray levels, c color

components
– Size = M x N x B x c
– Example: MxN=1024x1280, B=8, c=3 (24 bit
RGB image)
• Size = 31,457,280 bits (or 3.75 MBytes)

• Need to have (lossy) compression!

34
35
Number of Pixels and Resolution

36
Number of Pixels Resolution

37
Number of Pixels Resolution

38
 Summary
• Definition: An image is a 2-dimensional light intensity
function, f(x,y), where x and y are spatial coordinates,
and f at (x,y) is related to the brightness of the image at
that point.
• Definition: A digital image is the representation of a
continuous image f(x,y) by a 2-D array of discrete
samples.
• The amplitude of each sample is quantized to be
represented by a finite number of bits.
• Definition: Each element of the 2-D array of samples is
called a pixel (Picture Element)

39
 Revisit: What is Digital Image Processing?

Processing images generally results in one of several kinds of outputs:

• Another image: g(x,y) = G (f(x,y)) (low-level processing)
• A set of attributes: edges, segments, objects (mid-level)
• An algorithmic “understanding” (high-level)

Processing Vision

Low-level Mid-level High-level

Denoising Segmentation Image recognition
Contrast enhancement Contours
Image sharpening

40
Denoising Landmark labels Face recognition

Image classification
Denoising Segmentation 41
Take a break!
42
 Why do we process images?
• Facilitate storage and transmission of pictures
• Compression of digital images
• Transmission of images
• Prepare an image for display
• Resizing images
• High-dynamic range (HDR) display
• Enhance and restore images
• Restoring old movies
• Better medical scans
• Information extraction from images
• Reading zip codes, bar codes
• Robotics: e.g., Self-driving cars 43
 General Areas/Techniques of Image
Processing

• Compression
• Recognition/classification
• Denoising
• Enhancement
• Restoration
• Segmentation
• Inpainting
• Super-resolution
• Watermarking
• Others

44
 Areas of Image Processing
Compression

Image compression standard:

- JPEG (1992): Joint Photographic Expert Group: works with color and grayscale
images.
- Compression ratio of lossless methods is not high enough.
- JPEG uses transform coding (DCT = discrete cosine transform)
- JPEG 2000 uses wavelet transforms
45
Image Compression
At 158:1 compression ratio

Courtesy of Minh Do
46
Areas of Image Processing
Recognition and classification

Some important steps:

• Preprocessing (denoise, deblurr, contrast

enhancement,…)
• Segmentation
• Feature extraction (scale space,
multiresolution, invariant features,
occlusion)
• Recognition (feature matching, distance
measure,…)
• Classification (machine learning)

Applications:
medical imaging, pattern recognition, video
processing, document processing,
industrial applications

47
 Recognition-Classification
Feature extraction (e.g. content-based image retrieval)

Images Signatures

Feature
extraction

Similarity Measurement

Feature
Query
extraction
N best matched images

48
 Areas of Image Processing
Denoising

➢ To give an image a softer effects ➔ smoothing or blurring

➢ Removal of small details (aesthetic effects)
➢ Bridging of small gaps in lines or curves
➢ image smoothing using Gaussian mean filter

49
 Areas of Image Processing
Denoising

“Many important images contain noise that is signal

dependent, and obeys a Poisson distribution. A familiar
example is that of radiography. The signal in a
radiograph is determined by photon counting statistics
and is often described as particle-limited, emphasizing
the quantized and non-Gaussian nature of the signal.
Removing noise of this type is a more difficult problem.”

Le Triet et al. J. Math Imaging Vision

50
 Areas of Image Processing
Restoration

Given image g, related to true image f through blur h and noise

Inverse Problem: restore f, given h

and statistics for the noise. Keeping
edges sharp and in the correct location
is a key problem !
51
 Areas of Image Processing
Segmentation

support.zoom.us

Objective: Partition an image  into n sub-regions, such that:

1. image data u(x, y) vary smoothly and/or slowly within each i

2. image u(x, y) varies discontinuously and/or rapidly across

52
the boundary C between different i
 Areas of Image Processing
Inpainting

Real inpainting example

from http://www.topcstudio.com/inpainting.htm

➢ How to estimate some missing or damaged regions in an image?

➢ Inpainting (retouching, repairing): originated from art conservation
or restoration
➢ A manual procedure to restore a damaged artwork (e.g., mediaeval
painting)
53
 Areas of Image Processing
Out-painting

https://www.sciencetimes.com/articles/39764/20220906/openai-
introduces-outpainting-dall-e-expands-borders-picture.htm

➢ DALL-E is one of the most popular text-to-image generators, with

over a million users
➢ DALL-E - stable diffusion model
➢ Extension of the image for resizing or content generation
➢ Extrapolate contents based on partial information at the image 54
boundary
 Related Subjects to Inpainting
• Disocclusion (object removal, image completion)
– Fundamentally the same problem as image inpainting

Image completion (object removal)

Block recovery [Rane03] [Sun05]
55
Automatic Image Inpainting Examples

Courtesy of Beretalmio et al. [Bertalmio2000] 56

Also see http://www.iua.upf.es/~mbertalmio/restoration1.html
 Areas of Image Processing
Super-resolution

Super-resolution (SR) reconstruction is able to produce high-resolution (HR)

image(s) from a set of low-resolution (LR) images. Color video resolution
enhancement using 30 input frames (courtesy of Dr. Milanfar at UCSC)

Your source of training

data are important!

Image: Twitter / @Chicken3gg 57

 Areas of Image Processing
Watermarking
What is watermarking?
Watermarking means protecting the image with a copyright
notice or a logo that denotes the origin of the
image/photo. Such watermark prevents people from
using your images and photos without your consent. If
they use it they must use it with the your mark.

58
What are the applications?

59
 Face Detection (identity, sex, age, etc)

towardsdatascience.com/an-intro-to-deep-
learning-for-face-recognition-aa8dfbbc51fb

www.youtube.com/watch?v=Pc2aJxnmzh 60
0&feature=emb_title
 Digital Document Processing

• Textlines and keywords detection in handwritten documents

• Classification of handwritten documents

User-entered
search keywords:
abonnement
résilier
Scene Text Detection
 Digital photography
squeeze

Content-aware image scaling (https://helpx.adobe.com)

regular HDR

High-dynamic-range (HDR) photography

(http://pristinekk.blogspot.com)
 Automated decisions / robotics

www.graphitech.net

https://becominghuman.ai/computer-vision-applications-in-self-
driving-cars-610561e14118

Product quality control Self-driving cars

64
 Medical Imaging
1953 1972 1973
1895

X-ray
CT MRI
PET
ultrasound

T1w MRI

T2w
Special image characteristics & features
3D image: voxel instead of pixel
4D image: 3D image in time series
 Medical Imaging
Blood vessel enhancement Generate CT from MRI

Image Enhancement Image synthesis

Brain tissue classification

Image segmentation Contrast correction

 Computer Vision

Tesla’s driverless system detecting objects in a foggy scenario via Source

LN-1 ‫مبانی بینایی کامپیوتر‬

Deep Learning for Computer Vision

Examples of Image Captioning

LN-1 ‫مبانی بینایی کامپیوتر‬

Hyperspectral
images

LN-1 ‫مبانی بینایی کامپیوتر‬

 ‫‪Q#1 .‬‬

‫•‬ ‫?‪What is this‬‬

‫‪LN-1‬‬ ‫مبانی بینایی کامپیوتر‬

 ‫‪Q#2 .‬‬

‫•‬ ‫?‪What is this‬‬

‫‪LN-1‬‬ ‫مبانی بینایی کامپیوتر‬

 Distance Metrics
• Distance metrics play a crucial role in image
processing for various tasks, helping to
quantify the similarity or dissimilarity between
images or image features.
• E.g., recognition, retrieval, segmentation, and
quality assessment
Wikipedia:
Hausdorff Distance
• https://towardsdatascience.com/9-
distance-measures-in-data-science-
918109d069fa
• https://medium.com/@eskandar.sahel/expl
oring-common-distance-measures-for-
machine-learning-and-data-science-a-
comparative-analysis-ea0216c93ba3
• https://en.wikipedia.org/wiki/Hausdorff_dist
ance
 Summary

• Introduction of key concepts and definitions

• General areas/techniques in image processing

• Applications in various domains

• Distance Metrics

81