CSE 483: Computer Vision

Image Segmentation

Prof. Mahmoud Khalil

Spring 2022
1
Segmentation Goals
Goal: identify groups of pixels that go together

2
3
 Image Segmentation – Point Detection
This is not a mathematically
This filter gives highest response
proven filter, this is just made
only for my required detection, and
for the context of the current
can give smaller responses
situation... �‫ﻫﻠﻴﻬ‬.
� Matches the
otherwise, so I can just threshold
point that is white surrounded
the result to eliminate them. The
by all black. If it's continuous,
HPs are the threshold and filter size
all weights are fired and so we
get 8 * (-1)*8 = 0. Otherwise 8

4
Image Segmentation – Line Detection

Max response = indeed a line = 6 * 255, so let's apply a threshold on that value.

5
Image Segmentation – Edge Detection

7
 Apply threshold to detect edges. But this thing stays
constant for a while, are these multiple edges?

Easier to detect w/ 2nd order derivative by connecting the two impulses to

get center of edge at their zero-crossing, but scales very badly with noise
Can deal with the problem of noise by
doing average blurring as preprocessing
step before doing edge detection.

Obviously you won't see with your eyes

the effect of blurring, since you didn't
even see the noise in the first place.
This is machine-level stuff.
Edge Detection
(Gradient Operators)
They all come in pairs because we use two
perpendicular directions to detect edges in
them and with knowledge of both we can
detect the edge's direction (one filter may
only detect one direction). Roberts goes 45
and -45, the other two do vertical and horiz
10
 Partial Derivatives of an Image

12
(-1, 1) implies that when you see a transition from 0 to 255
the result is white, which means the beginning of an edge
will be white, and the end will be black... (2nd order)
 triangle means 1st order, superscript
Image Gradient means further order (so a superscript
of 2 means 2nd order, 3 = 3rd, etc...)

13
Designing an Edge Detector

This kernel is of a
smoothing filter

The result of applying convolution with

this filter is elimination of noise (good)
and widening of gradient (semi-bad)...
It's semi-bad cause the gradient is
wider but we can still get the edge with
zero-crossing in second order detection

After this step (1st order

derivative), as mentioned
before we should apply a
threshold so we can get
14
useful information
 In this slide we see that, since
smoothing filter and 1st O.D.
Derivative Theorem of Convolution are linear operations, merging
them gives the same result

This is the 1st O.D. to the

Gaussian (smoothing) kernel

15
 As I was saying, 2nd O.D.'s zero crossing will counteract
the widening of the ramp, and since noise was removed
due to blurring, not much of an issue to actually use 2nd
O.D., an additional bonus is that we can get the direction
of the edge by getting the slope at the zero-crossing. In
Laplacian of Gaussian 1st O.D., the response would've been negative or
positive based on direction (which means the threshold
in that scenario should've been applied on absolute vals)

2nd O.D. to Gaussian

(smoothing) kernel

17
 2D Edge Detection Filters

Source: Steve Seitz 18

Edge by Derivative of Gaussian

16
 Smoothing with a Gaussian equivalent to the amount of smoothing

Source: Kristen Grauman, UT-Austin 19

 Effect of σ on derivatives

Source: Kristen Grauman, UT-Austin 20

Edge Detection Generally speaking, edge detection is a pre-processing step.

• Criteria for optimal edge detection (Canny 86):

• Good detection:
• minimize the probability of false positives (detecting spurious
edges caused by noise),
• false negatives (missing real edges)
- Good localization:
• edges must be detected as close as possible to the true edges.
- Single response constraint:
• minimize the number of local maxima around the true edge (i.e.
detector must return single point for each true edge point)

21
Edge Detection
• Examples:

This one is a result

of small threshold 22

or too much smooth

 CannyBoii doesn't do anything new, this is
what we've been doing for the past 20 slides
or so... Smoothen-->Deriv.-->Calc. Mag. + Dir.
Canny Edge Detector
• Smooth by Gaussian
x2  y2
S  G * I 1 
G  e 2 2

2 
• Compute x and y derivatives
T
  
S   S S   Sx  Sy 
T

 x y 
• Compute gradient magnitude and orientation

S  S x2  S y2

Sy
  tan 1

Sx
23
Canny Edge Operator

S  G * I   G * I
T
 G G 
G  
 x y 

T
 G G 
S   *I *I
 x y 

24
 Now that's what's new about CannyBoii

Non‐maximum suppression for each orientation

After detecting the edge (step 3), he
At q, we have a gets the orientation of the edge at each
maximum if the point (by getting the perpendicular to
the edge at the respective point) and
value is larger applies non-maximum suppression at it,
than those at both which simply means "if not max. value
p and at r. among values, get suppressed you little
piece of !@$%". This basically fixes the
Interpolate to get problem of "too many responses".
these values.

Source: D. Forsyth 26
 And so after you're done doing thinning at that point, to
Edge linking get to the following point simply move in the direction
of the edge, which is the tangent... ‫�ﻤﻞ ﻗﺮا�ﺔ‬
Assume the marked point
is an edge point. Then we
construct the tangent to the
edge curve (which is
normal to the gradient at
that point) and use this to
predict the next points We use it to predict which of
the already-existing points to
(here either r or s). go to next, that is expected to
be part of the thinner edge.

Source: D. Forsyth 27
 Non‐Maximum Suppression Examples

Source: D. Forsyth 28
 The second additional step that CannyBoii did
was this "Hysteresis Thresholding" thing...

Thresholding

Source: D. Forsyth High Threshold (Strong Edges) Small Threshold (Weak Edges) 29
 Hysteresis thresholding
• Use a high threshold
Start
to start edge curves
and a low threshold
Connects
to continue them. strong edges

Source: S. Seitz 30
Hysteresis Thresholding

31
 Hysteresis Thresholding

Source: L. Fei-Fei 32
Canny Edge Detector
Sx

Sy

25
Canny edge detector
0. Pre-processing: Smoothen w/ Gaussian
1. Filter image with x, y derivatives of Gaussian
2. Find magnitude and orientation of gradient
3. Non‐maximum suppression:
• Thin multi‐pixel wide “ridges” down to single pixel width
4. Thresholding and linking (hysteresis):
• Define two thresholds: low and high
• Use the high threshold to start edge curves and the low threshold to continue them

• MATLAB: edge(image, ‘canny’)

33
Effect of  (Gaussian kernel spread/size)

original Canny with Canny with

The choice of  depends on desired behavior

• large  detects large scale edges
• small  detects fine features
34
 Humans implicitly segment stuff in their incredible brains and
then re-draw based on the semantics they conclude. Machines
can't do that, they must analyze first and then get the semantic.

Low‐level Edges vs. Perceived Contours

Source: L. Lazebnik
35
Thresholding

42
Global Thresholding

43
The Role of Illumination

Has no notion of localization.

To it, there are a bunch of
grays, and a bunch of blacks.
Can't tell which is background, 45
which is foreground.
Adaptive Thresholding

Global Thresholding
It is bound to fail. Always.
Because just cutting off at a
certain color, no matter what
color you choose, will take out
part of the object, because it
shares part of the spectrum
with the background.

46
Adaptive Thresholding
Yay more hyper-parameters :) ! The new one is the size of grid cells based on which
you want to do your histogram thresholding. The purpose of the grid is to localize
parts (adapt within the image). It's still failing though, so let's try and fix it up.
Decent background
w.r.t. foreground

Small background
w.r.t. foreground
 Optimal Global Adaptive Thresholding
Histogram plot of overlapping regions p1 and p2, where
would the threshold be? So far we've been analyzing if
regions are separated (in fact that's how we detected
that they are separate regions in the first place...)

We could keep trying to place T at different points on the z-axis,

and see how large the two regions would be if they were cutoff at
that position (so for p1, its right end would be T, and for p2, its left
end would be T). Mathematically, this can be represented by:

So, we want to maximize this function. Let's call the function e(T), where it
is a function in terms of T (not z, because z does not change!). So, simple
maxima and minima tells you get the derivative = 0 and voila.

All good, but in practice, how do we know what region 48

is p1 and what region is p2? ArTiFiCiAL iNTeLLiGeNcE.
Basically analyze a crap ton of similar images manually to do human segmentation
and then guide your algorithm by its results to "guess" better.