Part III Registration and Segmen-

tation
 19 Histogram-based image segmenta-
tion

This tutorial aims to implement some image segmentation methods based

on histograms (thresholding and “k-means” clustering).

The different processes will be applied on the following images:

(a) Cells. (b) Muscle cells, author: Damien

Freyssenet, University Jean Monnet,
Saint-Etienne, France.

19.1 Manual thresholding

The most simple segmentation method is thresholding.

• Visualize the histogram of the grayscale image ’cells’.

• Make the segmentation with a threshold value determined from the

image histogram.

19.2 k-means clustering

Let X = {xi }i∈[1;n],n∈N be a set of observations (the points) in Rd . The k-means
clustering consists in partitioning X in k (k < n) disjoint subsets S̃ such that:
k X
X
S̃ = arg minS={Si }i≤k kxj − µi k2 (19.1)
i=1 xj ∈Si
 Histogram-based image segmentation 218

where µi is the mean value of the elements in Si . The k-means algorithm is iterative.
(1)
From a set of k initial elements {mi }i∈[1;k] (randomly selected), the algorithm
iterates the following (t) steps:

• Each element of X is associated to an element mi according to a distance

criterion (computation of a Voronoi partition):

n o
(t) (t) (t)
Si = x j : xj − m i ≤ xj − mi∗ , ∀i∗ ∈ [1; k] (19.2)

• Computation of the new mean values for each class:

(t+1) 1 X
mi = (t)
xj (19.3)
|Si | x (t)
j ∈Si

(t) (t)
where |Si | is the number of elements of Si .

19.3 Grayscale image, k = 2 in one dimen-

sion
The objective is to binarize image image ’cells’, which is a grayscale image. The set
X is defined by X = {I(p)}, for p being the pixels of the image I.

• Implement the algorithm proposed below (Alg. 6).

• Test this operator on the image ’cells’.

• Test another method of automatic thresholding (defined by Otsu in

[26]).

• Compare the values of the thresholds (manual and automatic).

 19.4 Simulation example, k = 3 in two dimensions 219

Data: Original image A

Data: Stop condition ε
Result: thresholded image
Initialize T0 , for example at 21 (max(A) + min(A));
done ← F alse;
while NOT done do
Segment the image A with the threshold value T ;
it generates two classes G1 (intensities ≥ T ) and G2 (intensities < T );
Compute the mean values, denoted µ1 , µ2 , of the two classes G1 , G2 ,
respectively;
Compute the new threshold value Ti = 12 (µ1 + µ2 );
if |Ti − Ti−1 | < ε then
done ← T rue
end
end
Segment the image with the estimated threshold value.
Algorithm 6: K-means algorithm for automatic threshold computation of
grayscale images.
For use with python, use the module skimage. filter and the function
threshold_otsu .

19.4 Simulation example, k = 3 in two di-

mensions
The objective is to generate a set of 2-D random points (within k = 3 distinct classes)
and to apply the k-means clustering for separating the points and evaluating the
method (the classes are known!).

Write a function for generating a set of n random points around a point

with coordinates (x, y).

We will use the python function randn from the module numpy.random.

• Use this function to generate 3 set of points (in a unique matrix)

around the points (0, 0), (3, 4) and (−5, −3).
 Histogram-based image segmentation 220

• Use the python function sklearn . cluster . KMeans for separating the
points. The result is presented in Fig.19.1.

Verify the utility of the option n_init =10.

Figure 19.1: Resulting clustering of random points.

19.5 Color image segmentation using K-means:

k = 3 in 3D
The k-means clustering is now used for segmenting the color image representing
the muscle cells ’Tv16.png’.

Which points have to be separated? Transform the original image into a

vector of size N × 3 (where N is the number of pixels) which represent the
3 components R, G and B of each image pixel.

The python function numpy.reshape does the same transformation.

19.5 Color image segmentation using K-means: k = 3 in 3D 221

• Visualize the 3-D map (histogram) of all these color intensities.

• Make the clustering of this 3-D map by using the K-means method.

• Visualize the corresponding segmented image.

 Histogram-based image segmentation 222

19.6. Python correction

19.6.1 Manual thresholding

Visual analysis of histogram

When manually choosing a threshold value, one has to analysis the histogram (Fig.
19.2).

import numpy as np
2 import imageio
import matplotlib . pyplot as plt # plots
4 from skimage import filter # otsu thresholding

6 # read image
cells =imageio.imread(’ cells . png’) ;
8
# display histogram
10 fig =plt . figure () ;
plt . hist ( cells . flatten () , 256)
12 fig . show();
fig . savefig ( " histo . pdf" ) ;

Figure 19.2: Original image and its histogram.

(a) Original image. (b) Histogram.

4000

3000

2000

1000

0
0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
 19.5 Color image segmentation using K-means: k = 3 in 3D 223

Segmentation

1 fig =plt . figure () ;
plt . subplot (1,2,1)
3 plt . imshow(cells , plt . cm.gray) ; plt . title ( ’ Original image’) ;
plt . subplot (1,2,2)
5 plt . imshow(cells>80, plt . cm.gray) ; plt . title ( ’Manual segmentation’) ;
fig . savefig ( "manual.pdf") ;

19.6.2 Automatic thresholding

def autothresh (image):

2 """ Automatic threshold method
@param image: image to segment
4 @return : threshold value
"""
6 s = 0.5∗( np.amin(image) + np.amax(image));
done = False ;
8 while ~done:
B = image>=s;
10 sNext = .5∗( np.mean(image[B]) + np.mean(image[~B]));
done = abs( s-sNext) <.5;
12 s = sNext;
return s

The results are displayed using the following code (Fig. 19.3):

1 # Automatic threshold
s_auto= autothresh ( cells ) ;
3
# Otsu thresholding
5 s_otsu = filter . threshold_otsu ( cells ) ;

7 plt . figure () ;
plt . subplot (1,2,1)
9 plt . imshow(cells>s_auto, plt . cm.gray) ; plt . title ( ’Automatic thresholding ’ )
plt . subplot (1,2,2) ;
11 plt . imshow(cells>s_otsu , plt . cm.gray) ; plt . title ( ’Otsu thresholding ’ ) ;
 Histogram-based image segmentation 224

Figure 19.3: Automatic thresholding and thresholding by Otsu. Results are almost
identical because threshold values are 105.3 and 105, respectively.
(a) Automatic threshold. (b) Otsu threshold.

19.6.3 k-means clustering

Different techniques can be found in the scikit documentation. A point cloud will
first be generated, from 3 clustered cloud points. The objective is then to segment
all the points into their original cluster.

Imports

1 import numpy as np
import matplotlib . pyplot as plt
3 import time

5 from sklearn . cluster import KMeans

19.5 Color image segmentation using K-means: k = 3 in 3D 225

Generation of point clouds

1 def generation (n, x, y) :

Y = np.random.randn(n, 2) + np.array ([[ x, y ]]) ;
3 return Y

5 points1=generation (100, 0, 0) ;
points2=generation (100, 3,4) ;
7 points3=generation (100, - 5, - 3) ;

9 pts=np.concatenate (( points1 , points2 , points3 ) ) ;

plt . plot ( pts [:,0], pts [:,1], ’ ro ’ ) ;
11
plt . show();

6
8 6 4 2 0 2 4 6

Figure 19.4: Point cloud.

Histogram-based image segmentation 226

k-means clustering

n=3; # number of clusters

2
# k−means initialization
4 k_means = KMeans(init=’k−means++’, n_clusters=n, n_init =10)
t0 = time . time () ; # computation time
6 k_means. fit ( pts ) ; # kmeans segmentation

8 t_batch = time . time () - t0 ;

10 # retrieve results
k_means_labels = k_means.labels_ ;
12 k_means_cluster_centers = k_means. cluster_centers_ ;

14 # plot
fig = plt . figure ()
16 colors = [ ’#4EACC5’, ’#FF9C34’, ’#4E9A06’]

18 # k−means
# zip agregates values two by two
20 for k, col in zip (range(n) , colors ) :
my_members = k_means_labels == k
22 cluster_center = k_means_cluster_centers[k]

24 # display points
plt . plot ( pts [my_members, 0], pts[my_members, 1], ’w’,
26 markerfacecolor=col , marker=’. ’ )

28 # display centroid
plt . plot ( cluster_center [0], cluster_center [1], ’o’ ,
30 markerfacecolor=col , markeredgecolor=’k’ ,
markersize=6)
32 plt . title ( ’KMeans’)
plt . show()
34 fig . savefig ( "kmeans.pdf") ;
 19.5 Color image segmentation using K-means: k = 3 in 3D 227

KMeans
6

6
8 6 4 2 0 2 4 6

19.6.4 Color image segmentation

Three different colors can be observed in the image. The objective is to separate
the 3 colors with the help of the K-means algorithm. Thus, the segmentation is
performed in the RGB color space, and each pixel is represented by a point in this
3D space. Initialization steps are identical to previous code. The data is converted
from a color image (of size (n, m, 3)) to a vector (of size (n × m, 3)), done by the
reshape function of numpy.

# load color image

2 cells =imageio.imread(’Tv16.png’) ;
[nLines , nCols,channels] = cells . shape
4 # reshape data
data = np.reshape( cells , (nLines∗nCols, channels) ) ;
6 k_means. fit ( data ) ;
# convert result to an image
8 # as we got labels , we expand the dynamic (multiply by 70)
segmentation = 70∗np.reshape(k_means.labels_ , (nLines , nCols) ) ;
10
fig =plt . figure () ;
12 plt . imshow(segmentation, cmap=plt.cm.gray);
imageio.imwrite( "segmentation_kmeans.png", segmentation) ;

3D scatter plot

This is a method to display colors in the RGB cube. This method is really slow,
depending on your GPU.
Histogram-based image segmentation 228

Figure 19.5: Segmentation result.

(a) Original image. (b) Segmented.

1 from mpl_toolkits . mplot3d import Axes3D # 3D scatter plot

# plot
3 colors = [ ’#4EACC5’, ’#FF9C34’, ’#4E9A06’]

5 fig = plt . figure ()
ax = fig . add_subplot (111, projection =’3d’ )
7
# Plot scatter points
9 for k, col in zip (range(n) , colors ) :
my_members = k_means_labels == k
11 cluster_center = k_means_cluster_centers[k]
ax. scatter ( data[my_members, 0], data[my_members, 1],
13 data[my_members, 2], c=col)
ax. scatter ( cluster_center [0], cluster_center [1],
15 cluster_center [2], s=30, c=col )
 20 Segmentation by region growing

This tutorial proposes to program the region growing algorithm.

20.1 Introduction
The region growing segmentation method starts from a seed. The initial region
first contains this seed and then grows according to

• a growth mechanism (in this tutorial, the N8 will be considered)

• an homogeneity rule (predicate function)

The algorithm is simple and barely only needs a predicate function:

Data: I: image
Data: seed: starting point
Data: queue: queue of points to considere
Result: visited: boolean matrix, same size as I
begin
queue.enqueue( seed );
while queue is not empty do
p = queue.dequeue();
foreach neighbor of p do
if not visited(p) and neighbor verifies predicate then
queue.enqueue( neighbor );
visited( neighbor ) = true;
end
end
end
return visited
end
Algorithm 7:
To get the coordinate of the mouse click, use the matplotlib connect utilities.
Define a function def onpick.
 Segmentation by region growing 230

# start by displaying a figure ,

2 # ask for mouse input ( click )
fig = plt . figure () ;
4 ax = fig . add_subplot (211) ;
ax. set_title ( ’ Click on a point ’ )
6 # load image
img = misc. ascent () ;
8 ax.imshow (img, picker = True, cmap = plt . gray () ) ;
# connect click on image to onpick function
10 fig . canvas.mpl_connect ( ’ button_press_event ’ , onpick) ;
plt . show () ;
12 def onpick(event) :
""" connector """

20.2 Region growing implementation

The seed pixel is denoted s.

• Code the predicate function: for an image f and a pixel p, p is in the

same segment as s implies |f (s) − f (p)| ≤ T .

• Code a function that performs region growing, from a starting pixel

(seed).

• Try others predicate functions like:

– pixel p intensity is close to the region mean value, i.e.:

|I(p) − ms | ≤ T

– Threshold value Ti varies depending on the region Ri and the

intensity of the pixel I(p).It can be chosen this way:
σi
Ti = (1 − )·T
mi
 20.2 Region growing implementation 231

20.3. Python correction

20.3.1 imports

1 import queue
from scipy import misc
3 import matplotlib . pyplot as plt
import numpy as np

20.3.2 Predicate
This function defines the agregation condition.

def predicate (image, i , j , seed) :

2 f=image[i, j ];
g=image[seed[0], seed [1]];
4 return abs( f -g)<20

The following code is used to start the region growing from a pixel manually
clicked on an image.

# start of code
2 fig = plt . figure () ;
ax = fig . add_subplot (211) ;
4 ax. set_title ( ’ Click on a point ’ )

6 # load image
img = misc. ascent () ;
8 ax.imshow(img, picker=True,cmap=plt.gray () ) ;

10 fig . canvas.mpl_connect(’button_press_event ’ , onpick)
plt . show();

And here comes the main function for region growing. The result is illustrated
in Fig.20.1.
Segmentation by region growing 232

1 def onpick (event) :

"""
3 this functions gets the event ’ click ’, and starts the region growing algorithm
Notice that img is a global variable
5 """
print ( "x, y: " , event . xdata , event . ydata) ;
7 # original pixel
seed = np.array ([ int (event . ydata) , int (event . xdata) ]) ;
9 q = queue.Queue () ;

11 myfunctions = [ predicate , predicate2 , predicate3 ];

13 for index , f in enumerate(myfunctions):

15 # initializes the queue

q.put (seed) ;
17
# Visited matrix : result of segmentation
19 # this matrix will contain 1 if in the region , −1 if visited but not in the
֒→ region , 0 if not visited
visited = np.zeros (img.shape)
21 #−−−−−−−−−−−−−−−
# Start of algorithm
23 visited [seed [0], seed [1]] = 1;

25 while not q.empty () :

p = q. get () ;
27
for i in range (max (0, p[0] - 1) , min (img.shape [0], p[0] + 2) ) :
29 for j in range (max (0, p[1] - 1) , min (img.shape [1], p[1] + 2) ) :
if not visited [ i , j ]:
31 if f (img, i , j , seed , visited ) :
visited [ i , j ] = 1;
33 q.put (np.array ([ i , j ]) ) ;
else :
35 visited [ i , j ] = - 1;

37 # visited matrix contains the segmentation result

# display results
39 ax = fig . add_subplot (142+index) ;
ax.imshow ( visited == 1) ;
41
imageio.imsave(f . __name__ +’_seg.python.png’, ( visited ==1). astype ( ’ int ’ ) )
43
fig . canvas.draw () ;
45 plt . show();
return ;

Notice that values −1 of the visited matrix avoid testing multiple times the same
pixel. In the predicate function, the visited matrix is used in case of adapting the
20.2 Region growing implementation 233

predicate to the current region. In the next case, the candidate pixel’s graylevel is
compared to the mean gray value of the region. The results are illustrated Fig.20.1.

def predicate2 (image, i , j , seed , visited ) :

2 f = int (image[i , j ]) ;
m = np.mean(image[visited==1]) ;
4 return abs ( f - m) < 20

Another predicate function would be:

def predicate3 (image, i , j , seed , visited ) :

2 f = int (image[i , j ]) ;
m = np.mean(image[visited==1]) ;
4 s = np. std (image[ visited ==1]) ;
return abs ( f - m) < 20 ∗ (1 -s/m)
Segmentation by region growing 234

Figure 20.1: Region growing illustration for pixel (189,136). The segmentation result
highly depend on the order used to populate the queue, on the predicate function
and on the seed pixel.
(a) Original image. (b) Segmented region.

(c) Segmented region. (d) Segmented region.

 21 Hough transform and line detec-
tion

This tutorial introduces the Hough transform. Line detection operators are
implemented.

21.1 Introduction
This tutorial deals with line detection in an image. For a given point of coordinates
(x, y) in R2 , there exists an infinite number of lines going by this point, with
different angles θ. These lines are represented by the following equation:

ρ = x · cos(θ) + y · sin(θ).

Thus, for each point (x, y) (Fig. 21.1a) corresponds a curve parametered by
[θ, ρ], where θ ∈ [0;2π] (Fig. 21.1b). The intersection of these curves represents a
line (in this case, y = x).
Figure 21.1: Representation of the Hough transform.
(a) Different points in R2 . (b) Hough transform of the 3 points.
Y ρ
y=x
(3,3)
•
(2,2)
•
(1,1)
•

X θ

(1, 1)

(2, 2)

(3, 3)

21.2 Algorithm
The (general and simple) method for line detection is then:
 Hough transform and line detection 236

1. Compute contours detections (get a binary image BW).

2. Apply the Hough transform on the contours BW.

3. Detect the maxima of the Hough transform.

4. Get back in the Euclidean space and draw the lines on the image.

Results should look like in Fig. 21.2.

Figure 21.2: Lines detection via Hough transform.

(a) Hough transform and maxima detection. Angles θ (b) Line detection.
are represented in abscissa, pixels ρ are represented in
ordinates. The detection of the absolute maxima of this
images will lead to the lines.

21.3 Hough transform

Code a function that will transform each point of a binary image into a
curve in the Hough space. For each curve, increment each pixel by one in
the Hough space.
 21.4 Maxima detection 237

21.4 Maxima detection

Use or code a function to detect maxima (regional maxima). For each maxi-
mum, keep only one point.

21.5 Display lines

For each maximum, display the corresponding line above the original image.
 Hough transform and line detection 238

21.6. Python correction

This correction makes use of the python modules numpy, opencv, skimage and
matplotlib.

1 import numpy as np
import cv2
3 import matplotlib . pyplot as plt
from skimage. feature import peak_local_max

21.6.1 Contours detection

The contours are detected using the Canny edge detection method. In this code,
the method from OpenCV is employed, see Fig.21.3a.

img = cv2.imread( ’TestPR46.png’) ;

2 plt . figure ()
plt . imshow(img)
4
# perform contours detection
6 edges = cv2.Canny(img,100,200) ;
plt . figure ()
8 plt . imshow(edges)

21.6.2 Hough transform

Notice that OpenCV contains Hough fonctions: HoughLines and HoughLinesP. The
result (sinogram) of the image is presented Fig.21.3b.
21.5 Display lines 239

0
200
400
600
800
1000
1200
1400
1600

0 500 1000 1500

(a) Canny edge detection. (b) Representation of the sinogram and detec-
tion of the maxima in the Hough space.

Figure 21.3: The algorithm of the Hough line detection consists in detecting the
edges, then representing each pixel in the Hough space and finally detecting the
maximal value in the sinogram.

## Hough transform
2 # size of image
X = img.shape [0];
4 Y = img.shape [1];

6 angular_sampling = 0.01; # angles in radians

8 # initialization of matrix H
rho_max = np.hypot(X,Y);
10 rho = np.arange(-rho_max, rho_max, 1) ;
theta = np.arange (0, np.pi , angular_sampling) ;
12 cosTheta = np.cos( theta ) ;
sinTheta = np. sin ( theta ) ;
14 H = np.zeros ([ rho. size , theta . size ]) ;

16 # Hough transform
# loop on all contour pixels
18 for i in range(X):
for j in range(Y) :
20 if (edges[ i , j ] != 0) :
R = i ∗cosTheta + j ∗sinTheta ;
22 R = np.round(R + rho. size /2) . astype ( int ) ;
H[R,range(theta . size ) ] += 1;
24
plt . imshow(H);
 Hough transform and line detection 240

21.6.3 Maxima detection

The function peak_local_max from skimage is used to detect local maxima in the Hough
transform. Matrix H is first smoothed with a Gaussian filter and represented in
Fig.21.3b.

# Maxima detection
2 G = cv2.GaussianBlur(H, (5,5) , 5) ;
maxima = peak_local_max(H, 5, threshold_abs =150, num_peaks=5);
4 plt . figure () ;
plt . imshow(G);
6
# display maxima on Hough transform image G
8 plt . scatter (maxima [:,1], maxima [:,0], c=’r ’ ) ;
plt . show();

21.6.4 Resulting lines

The result is shown in Fig.21.4.

1 # display the results as lines in image

for i_rho , i_theta in maxima:
3 print rho[i_rho ], theta [ i_theta ]
a = np.cos( theta [ i_theta ])
5 b = np. sin ( theta [ i_theta ])
y0 = a∗rho[i_rho]
7 x0 = b∗rho[i_rho]
y1 = int (y0 + 1000∗( -b) )
9 x1 = int (x0 + 1000∗( a) )
y2 = int (y0 - 1000∗( -b) )
11 x2 = int (x0 - 1000∗( a) )

13 cv2. line (img,(x1,y1) ,( x2,y2) ,(0,0,255) ,2)

15 # display in window
cv2.imshow(’hough transform’, img);
17 # write resulting image
cv2.imwrite( ’cv_hough.png’, img);
 21.5 Display lines 241

Figure 21.4: Lines detected with the Hough transform.

21.6.5 OpenCV builtin function

import cv2
2 import numpy as np

4 # read image and convert it to gray

img = cv2.imread(’TestPR46.png’)
6 gray = cv2. cvtColor (img,cv2.COLOR_BGR2GRAY)
edges = cv2.Canny(gray, 100, 200, apertureSize = 3)
8
# threshold value for lines selection :
10 # lower value means more lines
threshold = 150;
12
# perform lines detection
14 lines = cv2.HoughLines(edges, 1, np.pi /180, threshold )

16 # display lines
for rho, theta in lines [0]:
18 print rho, theta
a = np.cos( theta )
20 b = np. sin ( theta )
x0 = a∗rho
22 y0 = b∗rho
x1 = int (x0 + 1000∗( -b) )
24 y1 = int (y0 + 1000∗( a) )
x2 = int (x0 - 1000∗( -b) )
26 y2 = int (y0 - 1000∗( a) )
print x1, y1, x2, y2
28 cv2. line (img,(x1,y1) ,( x2,y2) ,(0,0,255) ,2)
 22 Active contours

This tutorials aims at introducing the active contours (a.k.a. snakes) method
as originally presented in [18]. It is a segmentation method based on the
optimization of a contour that will converge to a specific object.

22.1 Definition
A snake is a parametric curve v(s) with s∈[0; 1[. The energy functional is represented
by:
Z 1 Z 1
Esnake = Eint (v(s))ds + Eext (v(s))ds.
0 0

The internal energy is detailed in Eq.22.1. The first order derivation constrains
the length of the curve, the second order derivation constrains the curvature. The
parameters α and β a priori depend on s, but for simplicity, constant values will be
taken.
 
1
Eint (v(s)) = α(s)|v ′ (s)|2 + β(s)|v ′′ (s)|2 (22.1)
2 | {z } | {z }
length curvature

The snake will be attracted by some edges points from the external energy (the
image to be segmented). The, the energy functional will be in a local minimum: it
is shown that the Euler-Lagrange equation is satisfied:

−αv (2) + βv (4) + ∇Eext = 0

with v (2) and v (4) denoting the 2nd and 4th order derivatives. To solve this equation,
the gradient descent method is employed: the snake is now transformed into a
function of the position s and the time t, with Fext = −∇Eext

∂s
= αv (2) − βv (4) + Fext
∂t
 Active contours 244

Figure 22.1: Illustration of the active contours segmentation method. Two energies
are at stake: internal energies depend only on the snake shape and control points,
external energies are related to the image properties.

22.2 Numerical resolution

The spatial derivatives are approximated with the finite difference method, and the
snake is now composed of n points, i ∈ [1; n]:

x(2) = xi−1 − 2xi + xi+1

x(4) = xi−2 − 4xi−1 + 6xi − 4xi+1 + xi+2
The gradient descent method can be written as, with γ being the time step and
controls the convergence speed:
xt − xt−1
= A · xt + fx (xt−1 , yt−1 )
γ
yt − yt−1
= A · yt + fy (xt−1 , yt−1 )
γ
with
 −2α−6β α+4β −β 0 ··· 0 −β α+4β

 α+4β −2α−6β α+4β −β 0 ··· 0 −β

 −β α+4β −2α−6β α+4β −β 0 ··· 0 
 0 −β α+4β −2α−6β α+4β −β 0 ··· 

A=
 .
.. . .. . .. . .. . .. .. 

 . 

 0 ··· −β α+4β −2α−6β α+4β −β 0 
 0 ··· 0 −β α+4β −2α−6β α+4β −β 
−β 0 ··· 0 −β α+4β −2α−6β α+4β
α+4β −β 0 ··· 0 −β α+4β −2α−6β

The forces fx and fy are the components of Fext . For example for an image I,
if ∗ denotes the convolution and Gσ a gaussian kernel of standard deviation σ:
Fext = −∇(k∇(Gσ ∗ I)k)
22.2 Numerical resolution 245

• Generate a binary image (of size n × n pixels, with values 0 and 1)

containing a disk (of a radius R).

• Generate the initial contour as an ellipse, with the same center as the
disk.

• Generate the pentadiagonal matrix A. The different parameters can

be α = 10−5 and β = 0.05.

• Program the iterations and visualize the results. For example, γ = 200
and 1000 iterations may give an idea of the parameters to use.
 Active contours 246

22.3. Python correction

1 import numpy as np
import matplotlib . pyplot as plt
3 import scipy
import scipy . ndimage. filters
5 from scipy import interpolate
import progressbar

22.3.1 Binary image generation

A disk (Fig.22.2) is generated via the meshgrid function.

# disk : number of points

2 n = 1024;
# disk : radius
4 R = 300;
# construct a binary image of a disk
6 X, Y = np.meshgrid(np.arange(-n /2, n /2,1) , np.arange(-n /2, n /2,1) ) ;
I = X∗∗2+Y∗∗2<=R∗∗2;
8 I = I . astype ( ’ float ’ )

Figure 22.2: Circle.

22.3.2 Initial contour

The choice of the initial contour (Fig.22.3) is crucial in this method. The parameters
used in this example ensure the convergence of the snake.
 22.2 Numerical resolution 247

step =.01;
2 x = n/2 + 400 ∗ np.cos(np.arange (0,2∗ np.pi+step , step ) ) ;
y = n/2 + 200 ∗ np. sin (np.arange (0,2∗ np.pi+step , step ) ) ;

The different parameters are defined by:

1 k =.1;
alpha = .0001;
3 beta = 10;
gamma= 100;
5 iterations = 1000;

Figure 22.3: Circle.

750

700

650

600

550

500

450

400

350

300
100 200 300 400 500 600 700 800 900 1000

22.3.3 Matrix construction

This is maybe the hardest part of this code, with the use of the scipy . sparse . diags
function.

N = x. size ;
2 X = np.array ([ -beta , alpha+4∗beta , - 2∗alpha- 6∗beta , alpha+4∗beta , -beta , -beta ,
֒→ alpha+4∗beta , -beta , alpha+4∗beta ])
A = scipy . sparse . diags (X, np.array ([ - 2, - 1, 0, 1, 2, N-2, N-1, -N+2, -N+1]), shape
֒→ =(N,N)) . toarray () ;
4 AA = np. identity (N)-gamma∗A;
invAA = np. linalg . inv(AA);
 Active contours 248

22.3.4 External forces

The external forces are computed with the following code. Notice that axis 0
correspond to the vertical axis (y) and and the axis 1 corresponds to the horizontal
axis (x).

1 # external forces computation

G = scipy . ndimage. filters . gaussian_gradient_magnitude(I , 30) ;
3 # Notice that horizontal axis x is 1, and vertical axis is 0
Fy = scipy . ndimage.prewitt(G, axis =0) ;
5 Fx = scipy . ndimage.prewitt(G, axis =1) ;

22.3.5 Display results

To enhance the role of the external forces, the arrows showing the force are displayed
( quiver function, see Fig.22.4).

imshow(I ,[])
2 hold on
plot ([ x;x (1) ], [y; y (1) ], ’g’ , ’ linewidth ’ , 3) ;
4 # display arrows for external forces
step =20;
6 subx = 1: step : size ( I ,1) ;
suby = 1: step : size ( I ,2) ;
8 [Xa, Ya] = meshgrid(subx, suby) ;
quiver (Xa, Ya, Fx(subx, suby) , Fy(subx,suby)) ;

Figure 22.4: External forces that will be applied to the snake.

1000

800

600

400

200

0
0 200 400 600 800 1000
 22.2 Numerical resolution 249

22.3.6 Convergence algorithm

1 # interpolation methods to get values of the external forces at the

# coordinates of the snake
3 sx , sy = I . shape;
ix = interpolate . interp2d (np.arange(n) , np.arange(n) , Fx) ;
5 iy = interpolate . interp2d (np.arange(n) , np.arange(n) , Fy) ;
# loop for convergence of the snake
7 bar = progressbar . ProgressBar () ;
for index in bar(range( iterations ) ) :
9 fex = np.array ([ float ( ix (XX,YY)) for XX,YY in zip (x,y) ]) ;
fey = np.array ([ float ( iy (XX,YY)) for XX,YY in zip (x,y) ]) ;
11 # print (np.max(fex) , np.min(fex) )
x = np.matmul(invAA, x+gamma∗fex);
13 y = np.matmul(invAA, y+gamma∗fey);

The results are displayed in Fig.22.5.

Figure 22.5: Result of the snake converging toward the disk, after 1000 iterations
with the proposed parameters.

200

400

600

800

1000
0 200 400 600 800 1000
 23 Watershed

This tutorial aims to study the watershed transform for image segmentation.
In image processing, an image can be considered as a topographic surface.
If we flood this surface from its minima and if we prevent the merging of
the water coming from different sources, we partition the image into two
different sets: the catchment basins separated by the watershed lines.

The different processes will be applied on the following images:

(a) circles (b) gel

23.1 Watershed and distance maps

The objective is to individualize the disks by disconnecting them with the distance
map.

• Calculate the distance map of the image ’circles’.

• Take the complementary of this distance map and visualize its minima.

• Calculate the watershed transform of the inverted distance map.

• Subtract the watershed lines to the original image.

Look at the documentation of scipy . ndimage.morphology for Euclidean or

Chamfer distance transform. Reconstruction operator can be found in
skimage.morphology.
 Watershed 252

23.2 Watershed and image gradients

• Calculate the Sobel gradient of the image ’gel’.

• Visualize the minima of the image gradient.

• Apply the watershed transform on the gradient image.

The watershed transform, applied in a direct way, leads to an over-segmentation.

To overcome this limitation, the watershed operator can be applied on a filtered
image.

• Smooth the original image with a low pass filter. To stay in the math-
ematical morphology field, you can use an alternate morphological
filter (opening followed by closing). A Gaussian filter is also a good
solution.

• Calculate the gradient operator on the filtered image.

• Calculate the corresponding watershed.

23.3 Constrained watershed by markers

In order to indivudualize the image spots, we have to determine the internal and
external markers for the constrained watershed.

• Calculate the gradient (Sobel) of the filtered image.

• Calculate the internal markers (minima of the filtered image) and

external (watershed of the filtered image) as minima of the gradient
image.

• Calculate the corresponding watershed.

• Superimpose the watershed lines of the resulting segmentation to the

original image.
 23.3 Constrained watershed by markers 253

23.4. Python correction

import numpy as np
2 from scipy import misc
import matplotlib . pyplot as plt
4 from scipy import ndimage as ndi
from scipy . ndimage.morphology import distance_transform_edt
6 from scipy . ndimage.morphology import distance_transform_cdt
from skimage import morphology

23.4.1 Watershed and distance map

Regional maxima

The following code is an implementation of the regional maxima (from mathematical

morphology definition). It deals with the case of float images as input.

1 def rmax(I) :
"""
3 Own version of regional maximum
This avoids plateaus problems of peak_local_max
5 I : original image, int values
returns : binary array , with True for the maxima
7 """
I = I . astype ( ’ float ’ ) ;
9 I = I / np.max(I) ∗ 2∗∗31;
I = I . astype ( ’ int32 ’ ) ;
11 h = 1;
rec = morphology.reconstruction ( I , I+h);
13 maxima = I + h - rec ;
return maxima>0

Distance map followed by a watershed

This method is a classical way of performing the separation of some objects by

proximity or influence zones. It is illustrated in Fig.23.1.
 Watershed 254

A = imageio.imread(’ circles . tif ’ ) ;

2 # chamfer distance gives integer distances
dm = distance_transform_edt (A);
4 # regional maxima
local_maxi = rmax(dm);

The local maxima will be used as a marker for the watershed operation, in order
to perform the separation of the grains, see Fig.23.1.

1 # watershed segmentation for separating the circles

markers = ndi . label ( local_maxi , np.ones ((3, 3) ) ) [0]
3 W = morphology.watershed(-dm, markers, watershed_line=True)
# separation of the grains
5 B = A & W==0;
separation = ndi . label (B, np.ones ((3,3) ) ) [0];

Figure 23.1: Steps of the separation of the grains.

(a) Original image. (b) Distance map. (c) Separation of the grains.

23.4.2 Watershed and image gradients

The gradient image amplifies the noise. Thus, the watershed operator directly ap-
plied to the gradient of the image produces an over-segmented image (see Fig.23.2).
23.3 Constrained watershed by markers 255

def sobel_mag(im):
2 """
Returns Sobel gradient magnitude
4 im: image array of type float
returns : magnitude of gradient (L2 norm)
6 """
dx = ndi . sobel (im, axis =1) # horizontal derivative
8 dy = ndi . sobel (im, axis =0) # vertical derivative
mag = np.hypot(dx, dy) # magnitude
10 return mag;

Figure 23.2: Performing the watershed on the gradient image is usually not a good
idea.
(b) Amplitude of the gradient (So-
(a) Original image. bel). (c) Watershed segmentation.

In fact, this method produces as many segments as there are minima in the
gradient image Fig.23.3.

Figure 23.3: Local minima of the gradient image.

Solution: filtering the image

Before evaluating the gradient, the image is filtered. The number of minima is
lower and this leads to a less over-segmented image (Fig.23.4).
 Watershed 256

SE = morphology.disk(2) ;
2 O = morphology.opening(gel, selem=SE);
F = morphology.closing(O, selem=SE).astype ( ’ float ’ ) ;
4 g = sobel_mag(F) . astype ( ’ float ’ ) ;

Figure 23.4: Even if the gradient is performed on the filtered image, there is still a
high over-segmentation.

23.4.3 Watershed constrained by markers

The watershed can be constrained by markers: the markers can provide the correct
number of regions. This method imposes both the background (external markers)
and the objects (internal markers). The results are illustrated in Fig.23.5. The
ultimate erosion of the internal markers is used to deconnect these markers from
the external markers.

local_maxi = rmax(255-F) ;
2 markers = ndi . label ( local_maxi , np.ones ((3, 3) ) ) [0]
W = morphology.watershed(F, markers, watershed_line=True)
4
markers2 = local_maxi | (W==0);
6 M = ndi. label (markers2, np.ones ((3, 3) ) ) [0]
segmentation = morphology.watershed(g, M, watershed_line=True);
8
gel [segmentation==0] = 255;
10 plt . imshow(gel);
plt . show();
12 imageio.imwrite( "segmentation.python.png", gel ) ;
23.3 Constrained watershed by markers 257

Figure 23.5: Watershed segmentation by markers.

(a) Markers of the back-
ground and of the objects. (b) Final segmentation.
 24 Segmentation of follicles

This practical work aims to investigate image segmentation with a direct

application to ovarian follicles. The overall objective is to extract and quan-
tify the granulosa cells and the vascularization of each follicle included in
an ewe’s ovary.

The image to be processed is a 2D histological image of an ewe’s ovary acquired

by optical microscopy in Fig.24.1. The presented image contains one entire follicle
(the white region and its neighborhood) and a part of a second one (right-upper
corner). The follicle is composed of different parts shown in: antrum, granulosa
cells and vascularization. The theca is the ring region around the antrum where
the follicle is vascularized.
Figure 24.1: Different parts of the follicles, to be segmented.
(b) Antrum (dark gray),
(a) Original image with one granulosa cells (light gray)
entire follicle (white region and vascularization (white)
and its neigborhood). of the follicle. (c) Theca.

24.1 Vascularization

• Load and visualize the image.

• Extract the antrum of the follicle.

• Extract the vascularization (inside a ring around the antrum).

 Segmentation of follicles 260

24.2 Granulosa cells

Which kind of processing could be suitable for extracting the granulosa

cells?

24.3 Quantification

Provide some geometrical measurements of the different entities of the

follicle (antrum, vascularization, granulosa cells).
 24.3 Quantification 261

24.4. Python correction

import numpy as np
2 import matplotlib . pyplot as plt
from scipy import misc,ndimage
4 from skimage import morphology

24.4.1 Vascularization
Antrum segmentation

The first step consists in the segmentation of the antrum by thresholding the blue
component. Some post-processes are used to remove artifacts such as holes. This is
illustrated in Fig. 24.2.

A = imageio.imread(’ follicle . png’) ;

2 plt . imshow(A);
plt . show();
4 ## Antrum
# segmentation by mathematical morphology
6 # manual selection of the antrum
B = A [:,:,2];
8 antrum = B > 220;
L = morphology.label(antrum, connectivity =2) ;
10 antrum = L == L [300,300];
antrum = ndimage.morphology. binary_fill_holes (antrum);
12 plt . imshow(antrum);
plt . title ( ’Antrum’)
14 plt . show();

Theca segmentation

The second step provides the segmentation of the theca which is extracted as a
spatial region (corona) adjacent to and outside the antrum. The width of the corona
is selected by the user (expert).
Segmentation of follicles 262

se40 = morphology.disk(40) ;
2 theca = morphology.binary_dilation (antrum, selem = se40) ;
theca = theca - antrum;
4 plt . imshow(theca); plt . title ( ’Theca’)
plt . show()

Vascularization segmentation

The final step extracts the vascularization of the considered follicle. Pixels belonging
to the vascularization are considered to have a low blue component and they should
also be included in the antrum of the follicle.

1 vascularization = B < 140;

vascularization = vascularization ∗ theca ;
3 plt . imshow(vascularization ) ;
plt . title ( ’ vascularization ’ )
5 plt . show();
 24.3 Quantification 263

Results

Figure 24.2: Extraction of the different components of the follicle.

(a) Original image.

(b) Antrum. (c) Theca. (d) Vascularization.

24.4.2 Granulosa cells

First solution

The granulosa cells have a low contrast, so it is difficlut to use thresholding tech-
niques. But we know they are localized between the antrum and the vascularization.
Nevertheless the vascularization is not a closed region outside the antrum. There-
fore, the proposed solution consits in firt trying to close the vascularization region
and taking the corona between this region and the antrum. The result is shown in
Fig. 24.3.
 Segmentation of follicles 264

1 se10 = morphology.disk(10) ;
dil = 1-morphology.binary_closing( vascularization , se10) ;
3 L = morphology.label( dil , connectivity =1) ;
dil = L == L [300,300];
5 granulosa = dil - antrum;
plt . imshow(granulosa);
7 plt . title ( ’ granulosa ’ )
plt . show()

Figure 24.3: Extraction of the granulosa cells of the follicle.

(c) Segmentation of the differ-
(a) Original image. (b) Granulosa cells. ent parts.

Second solution

This first solution is not really robust. The closing of the vascularization region is
not really accurate. A more robust solution consists in using deformable models
to get the corona between the vascularization and the antrum. But this kind of
method is out of the scope of this tutorial.

24.4.3 Quantification

The quantification is easy to process. In addition, we can represent the different

extracted components of the follicle in false colors.
24.3 Quantification 265

result = antrum + 2∗granulosa + 3∗ vascularization ;

2 plt . imshow(result , cmap=’jet’ ) ;
plt . show();
4 plt . imsave(" result . png", result , cmap=’jet’ ) ;

6 follicle = antrum + theca ;

q_vascularization = np.sum( vascularization ) / np.sum( follicle ) ;
8 print ( ’ vascularization : ’ , q_vascularization )
q_granulosa = np.sum(granulosa)/np.sum( follicle ) ;
10 print ( ’Granulosa: ’ , q_granulosa) ;

This quantification gives:

vascularization : 0.0428495233516
2 Granulosa: 0.0913707490403
 25 Image Registration

This tutorial aims to implement the Iterative Closest Point (ICP) method
for image registration. More specifically, we are going to estimate a rigid
transformation (translation + rotation without scaling) between two images.

The different processes will be applied on T1-MR images of the brain in Fig.25.1.

Figure 25.1: Original images.

(a) brain1 (b) brain2

25.1 Transformation estimation

A classical method in image registration first consists in identifying and matching
some characteristic points by pairs. Thereafter, the transformation is estimated
from this list of pairs (displacement vectors).

25.1.1 Preliminaries
Pairs of points are first manually selected.

• Read and visualize the two MR images ’brain1’ and ’brain2’ (moving
and source images).

• Manually select a list of corresponding points.

 Image Registration 268

There is no built-in function for manual selection of pairs of points. You

can use the following points or code your own function, see tutorial 20

# define control points

2 A_points = np.array ([[136, 100], [127, 153], [96, 156], [87,
֒→ 99]]) ;
B_points = np.array ([[144, 99] , [109, 140], [79, 128], [100,
֒→ 74]]) ;

25.1.2 Rigid transformation

With this list of pairs (pi , qi )i , this tutorial proposes to estimate a rigid transfor-
mation between these points. It is composed of a rotation and a translation (and
not scaling). For doing that, we make a Least Squares (LS) optimization, which is
defined as follows. The parameters of the rotation R and the translation t minimize
the following criterion:
X
C(R, t) = ||qi − R.pi − t||2
i

Calculation of the translation :

The optimal translation is characterized by a null derivative of the criterion:
!
∂C X X X
= −2 (qi − R.pi − t)T = 0 ⇔ qi − R. pi = N.t
∂t i i i
P
where N denotes the number of matching pairs. By denoting p̄ = N1 i pi and
P
q̄ = N1 i qi the barycenters of the point sets and by changing the geometrical
referential: p′i = pi − p̄ et qi′ = qi − q̄, the criterion can be written as:
X
C ′ (R) = ||qi′ − R.p′i ||2
i

The estimated rotation R̂ will provide the expected translation:

t̂ = q̄ − R̂.p̄ (25.1)

calculation of the rotation by the SVD method:

We will use the following theorem: Let U.D.V T = K a singular decomposition
of the correlation matrix K = q ′ T .p′ , for which the singular values are sorted in
P
the increasing order. The minimum of the criterion: C(R) = i ||qi′ − R.p′i ||2 is
reached by the matrix :
R̂ = U.S.V T (25.2)
with S = DIAG(1, . . . , 1, det(U ). det(V )).
 25.2 ICP-based registration 269

• Code a function that takes as parameters the pairs of points, and

returns the rigid transformation elements of rotation R̂ and translation
t̂ as defined in Eqs.25.1 and 25.2.

• Apply the transformation to the moving image.

• Visualize the resulting registered image.

• See svd function from scipy . linalg or numpy.linalg.

• See cv2.warpAffine and cv2.transform for transformation
application

25.2 ICP-based registration

When the points are not correctly paired, it is first necessary to reorder them before
estimating the transformation. In this way, the ICP (iterative Closest Points) consists
in an iterative process of three steps: finding the correspondence between points,
estimating the transformation and applying it. The process should converge to the
well registered image.

To simulate the mixing of the points, randomly shuffle them selected on the
first image and perform the registration with the previous method.

See np.random.permutation.

1. From the list of the characteristic points p of the image ’brain1’, find
the nearest neighbors q in the image ’brain2’. Be careful to the order
of the input arguments.

2. Estimate the transformation T by using the LS minimization coded

previously.

3. Apply this transformation to the points p, find again the correspon-

dence between these resulting points T (p) and q and estimate a new
transformation. Repeat the process until convergence.
 Image Registration 270

4. Visualize the resulting registered image.

Look at scipy . spatial . cKDTree for nearest neighbor search.

25.3 Automatic control points detection

The manual selection of points may be fastidious. Two simple automatic methods
for detecting control points are the so-called Harris corners detection or Shi-Tomasi
corner detector.
The function goodFeaturesToTrack returns the corners from the
Shi-Tomasi method.

Replace the manual selection in the two previous parts of this tutorial.

Notice that the automatic points detection does not ensure to give the same
order in the points of the two images. More generally, other salient points detectors
do not give the same number of points, and thus the algorithms have to remove
outliers (non matching points). This case is not taken into account in this tutorial.
 25.3 Automatic control points detection 271

25.4. Python correction

from scipy import misc

2 import matplotlib . pyplot as plt
import numpy as np
4 import cv2
from scipy . spatial import cKDTree

25.4.1 Transformation estimation based on corresponding points

Image visualization

The following code is used to display the images (see Fig.25.2).

1 # Read images and display

A=imageio.imread("brain1 . bmp")
3 B=imageio.imread("brain2 . bmp")
plt . imshow(A,cmap=’gray’);
5 plt . show();
plt . imshow(B,cmap=’gray’);
7 plt . show();

9 # define control points

A_points = np.array ([[136, 100], [127, 153], [96, 156], [87, 99]]) ;
11 B_points = np.array ([[144, 99], [109, 140], [79, 128], [100, 74]]) ;

Figure 25.2: Initial images.

(a) Moving image. (b) Source image. (c) Superimposition.
Image Registration 272

If you want to display and save the fusion of both images, you can use this
function:

1 def superimpose(G1, G2, filename=None):

"""
3 superimpose 2 images, supposing they are grayscale images and of same shape
"""
5 r , c=G1.shape;
S = np.zeros (( r , c ,3) ) ;
7 S [:,:,0] = np.maximum(G1-G2, 0)+G1;
S [:,:,1] = np.maximum(G2-G1, 0)+G2;
9 S [:,:,2] = (G1+G2) / 2;
S = 255 ∗ S / np.max(S);
11 S = S. astype ( ’ uint8 ’ ) ;
plt . imshow(S);
13 plt . show()
if filename!=None:
15 cv2.imwrite(filename , S) ;
return S

Manual selection of corresponding points

With openCV, there is not built-in function to make a manual selection of pairs
of control points. The following code uses a global variable I in order to manage
the display of the points, which are finally stored into the pts variable. First, the
callback function on_mouse is defined to handle mouse event.

pts = [];
2 def on_mouse(event, x, y, flags , param):
"""
4 callback method for detecting click on image
It draws a circle on the global variable image I
6 """
global pts , I ;
8 if event == cv2.EVENT_LBUTTONUP:
pts . append((x, y) ) ;
10 cv2. circle ( I ,( x,y) , 2, (0,0,255) , - 1)

The function cpselect allows the selection of multiple points.

25.3 Automatic control points detection 273

def cpselect () :
2 """
method for manually selecting the control points
4 It waits until ’q’ key is pressed .
"""
6 cv2.namedWindow("image")
cv2.setMouseCallback("image", on_mouse)
8 print ( " press ’q’ when finished" )
# keep looping until the ’q’ key is pressed
10 while True:
# display the image and wait for a keypress
12 cv2.imshow("image", I )
key = cv2.waitKey(1) & 0xFF
14
# if the ’ c ’ key is pressed , break from the loop
16 if key == ord( "q" ) :
break
18
# close all open windows
20 cv2.destroyAllWindows()
return pts ;

Transformation estimation

The rigid transformation is estimated from the corresponding points by the follow-
ing function:
Image Registration 274

1 def rigid_registration ( data1 , data2) :

"""
3 Rigid transformation estimation between n pairs of points
This function returns a rotation R and a translation t
5 data1 : array of size nx2
data2 : array of size nx2
7 returns transformation matrix T of size 2x3
"""
9 data1 = np.array ( data1) ;
data2 = np.array ( data2) ;
11
# computes barycenters , and recenters the points
13 m1 = np.mean(data1,0) ;
m2 = np.mean(data2,0) ;
15 data1_inv_shifted = data1-m1;
data2_inv_shifted = data2-m2;
17
# Evaluates SVD
19 K = np.matmul(np.transpose( data2_inv_shifted ) , data1_inv_shifted ) ;
U,S,V = np. linalg . svd(K);
21
# Computes Rotation
23 S = np.eye (2) ;
S [1,1] = np. linalg . det (U)∗np. linalg . det (V);
25 R = np.matmul(U,S);
R = np.matmul(R, np.transpose (V)) ;
27
# Computes Translation
29 t = m2-np.matmul(R, m1);

31 T = np.zeros ((2,3) ) ;
T [0:2,0:2] = R;
33 T [0:2,2] = t ;
return T;

Then, you can apply this function to the manually selected points.
 25.3 Automatic control points detection 275

# 1st case , rigid registration , with pairs of points in the correct order
2 T = rigid_registration (A_points, B_points) ;

4 # Apply transformation on control points and display the results

data_dest = applyTransform(A_points, T) ;
6 I = B.copy() ;
for pb,pa in zip ( data_dest , B_points) :
8 cv2. circle ( I , totuple (pa) , 1, (255,0,0) , - 1)
cv2. circle ( I , totuple (pb) , 1, (0,0,255) , - 1)
10 plt . imshow(I);
plt . show();
12 # Apply transformation on image
rows, cols = B.shape;
14 dst = cv2.warpAffine(A,T, ( cols , rows)) ;
plt . imshow(dst);
16 superimpose(dst , B, "rigid_manual.png") ;

Figure 25.3: Result of the registration for the manually selected control points.
(a) Without registration. (b) With registration.

The result is good, because the manual selection of the points is good (the points
are given in the correct order for both images).

25.4.2 ICP registration

Random permutation of points

The following code randomly shuffles the points of the first vector. The result is of
course a wrongly registered image, see Fig.25.4.
Image Registration 276

# random permutation of the points

2 p = np.random.permutation(np.arange(4) ) ;
A_points = A_points[p ];
4 T = rigid_registration (A_points, B_points) ;

Figure 25.4: Result of the registration for when the control points are not found in
the same order.
(a) Matching pairs of points. (b) Permulation of the points.

ICP

The previous operations simulates a general non-manual selection of the control

points: there is not reason to finding the points by matching pairs. Thus, a reordering
is necessary. This propositions implies that the number of points is exactly the same
in order to perform the registration process, and that these are matching points
(every point has a matching point in the other image). The ICP method (see code
for icp_transform ) reorders the points via a nearest neighbor rule.
25.3 Automatic control points detection 277

def icp_transform (dataA, dataB) :

2 """
Find a transform between A and B points
4 with an ICP ( Iterative Closest Point ) method.
dataA and dataB are of size nx2, with the same number of points n
6 returns the transformation matrix of shape 2x3
"""
8 data2A = dataA.copy() ;
data2B = np.zeros (data2A.shape) ;
10 T = np.zeros ((2,3) ) ;
T [0:2,0:2] = np.eye (2) ;
12 T [0:2,2] = 0;

14 nb_loops=5;
tree = cKDTree( dataB ) ;
16
for loop in range(nb_loops) :
18 # search for closest points and reorganise array of points accordingly
d, inds = tree . query( data2A);
20
data2B = dataB[inds ,:];
22 # find rigid registration with reordered points
t_loop = rigid_registration (data2A, data2B) ;
24
T = composeTransform(t_loop, T) ;
26 # evaluates transform on control points , to make the iteration
data2A = applyTransform(dataA, T) ;
28
return T;

Automatic extraction of corner points

Generally, the points are automatically detected, and thus, there is no warranty
that they are found in the same order, nor that each pair of point correspond to
matching points (some points –called outliers– need to be eliminated to compute
the correct transformation). In this tutorial, we do not address the problem of
outliers. Please notice that with these parameters and images, by chance, the same
points are detected in the correct order.
Image Registration 278

Figure 25.5: Result of the registration for when the control points are not found in
the same order. The ICP algorithm reorders the points and gives a good result.
(a) Random shuffle of points and
direct rigid transformation estima- (b) ICP registration on the same
tion. points.

1 # Automatic extraction of corner points

# Harris corners detection or Shi and Tomasi
3 # this method do not ensure the correct order in the points (A_points and
# B_points)
5 nb_points = 4; # number of points to detect
corners = cv2.goodFeaturesToTrack(A,nb_points ,0.01,10, blockSize =3,
֒→ useHarrisDetector=False )
7 corners = np. int0 ( corners )
a , b, c = corners . shape;
9 A_points = np.reshape( corners , (a , c) ) ;

11 corners = cv2.goodFeaturesToTrack(B,nb_points ,0.01,10, blockSize =3,

֒→ useHarrisDetector=False )
corners = np. int0 ( corners )
13 a , b, c = corners . shape;
B_points = np.reshape( corners , (a , c) ) ;

The Fig.25.6 displays the results with the automatic detection of corners (in this
case, the method from Shi and Tomasi). Notice that by chance, the points in both
images match. In other cases, one would have to remove outliers.
25.3 Automatic control points detection 279

Figure 25.6: Shi and Tomasi corners detection. By chance, the points correspond
and the ICP method gives a correct result.
(a) (b)