Mutual Information-based Registration

Introduction to mutual information-based registration

Mutual information-based registration is a form of registration which is preferable when two
images need to be aligned but the intensity characteristics are different. When the intensity
characteristics are different it is not possible to use the sum-of-squared-differences energy for
registration. Using the mutual information-based registration a joint histogram is computed.
The joint histogram shows when the intensities of the two image are the same and when they
are not. When two images are aligned the joint histogram will most often consist of a few
histogram bins containing the majority of the intensities, whereas when they are misalligned the
intensities are more spread resulting in more bins. The registration energy is defined as follows:
B B
E ( w )=H F , M with H F , M =− ∑ ∑ pf ,m log ( pf ,m )
f =1 m=1

where pf ,m is the normalized historgram count and can be interpreted as the probability of a
specific intensity combination (f,m). Whereas H F , M is the joint entropy, that tells how
predictable a intensity combination is. The lower the joint entropy the more predictable is the
intensity combinations, thereby the histogram will consist of fewer histogram bins with more
information and a thus more aligned images.

Problems with local minimas for the joint entropy can be found when for example the moving
image only contains background.

To avoid this problem the energy can be calculated as follows:

E ( w )=H F , M − H F − H M ,

where H F and H M are the marginal entropies of the fixed and moving image, respectively. They
are defined as follows:
B B
H F=− ∑ pf log ( p f ) and H M =− ∑ p m log ( p m )
f =1 m=1

In this report rigid image registration will be implemented using mutual information. Two MRI
images are used, where one is rotated around one axis. The joint histograms and mutual
information are computed. Followed by a evaluation of the mutal information when rotating the
image in a range. At last the joint histogram for the rotated moving image and fixed image is
compared with the original one.

Input data and code hints

Python libraries:

import numpy as np
from numpy import linalg as la
import matplotlib.pyplot as plt
import matplotlib as mpl
from mpl_toolkits import mplot3d
#%matplotlib tk
#%matplotlib notebook
%matplotlib inline
plt.ion()
np.set_printoptions( suppress=True )
import nibabel as nib
import scipy
from scipy.ndimage import map_coordinates

Reading the 3D scans for the report

T1_fileName = 'IXI014-HH-1236-T1.nii.gz'
T2_fileName = 'IXI014-HH-1236-T2_rotated.nii.gz'
T1 = nib.load( T1_fileName )
T2 = nib.load( T2_fileName )
T1_data = T1.get_fdata()
T2_data = T2.get_fdata()

Below is code to define a simple interactive viewer class that can be used to visualize 2D cross-
sections of a 3D array along three orthogonal directions. It takes a 3D volume as input and
shows the location a "linked cursor" in all three cross-sections.

The initial location of the cursor is in the middle of the volume in each case. It can be changed by
clicking on one of the cross-sections. The viewer also displays the voxel index v of the cursor.

class Viewer:
def __init__(self, data ):
self.fig, self.ax = plt.subplots()
self.data = data
self.dims = self.data.shape
self.position = np.round( np.array( self.dims ) / 2 ).astype(
int )
self.draw()
self.fig.canvas.mpl_connect( 'button_press_event', self )
self.fig.show()

def __call__(self, event):

print( 'button pressed' )
if event.inaxes is None: return

x, y = round( event.xdata ), round( event.ydata )

#
if ( x > (self.dims[0]-1) ) and ( y <= (self.dims[1]-1) ):
return # lower-right quadrant
 #
if x < self.dims[0]:
self.position[ 0 ] = x
else:
self.position[ 1 ] = x - self.dims[0]

if y < self.dims[1]:
self.position[ 1 ] = y
else:
self.position[ 2 ] = y -self.dims[1]

print( f" voxel index: {self.position}" )

print( f" intensity: {self.data[ self.position[0],
self.position[1], self.position[2] ]}" )

self.draw()

def draw( self ):

#
# Layout on screen is like this:
#
# ^ ^
# Z | Z |
# | |
# -----> ---->
# X Y
# ^
# Y |
# |
# ----->
# X
#
dims = self.dims
position = self.position

xySlice = self.data[ :, :, position[ 2 ] ]

xzSlice = self.data[ :, position[ 1 ], : ]
yzSlice = self.data[ position[ 0 ], :, : ]

kwargs = dict( vmin=self.data.min(), vmax=self.data.max(),

origin='lower',
cmap='gray',
picker=True )

self.ax.clear()

self.ax.imshow( xySlice.T,
extent=( 0, dims[0]-1,
0, dims[1]-1 ),
**kwargs )
 self.ax.imshow( xzSlice.T,
extent=( 0, dims[0]-1,
dims[1], dims[1]+dims[2]-1 ),
**kwargs )
self.ax.imshow( yzSlice.T, extent=( dims[0], dims[0]+dims[1]-
1,
dims[1], dims[1]+dims[2]-1
),
**kwargs )

color = 'g'
self.ax.plot( (0, dims[0]-1), (position[1], position[1]),
color )
self.ax.plot( (0, dims[0]+dims[1]-1), (dims[1]+position[2],
dims[1]+position[2]), color )
self.ax.plot( (position[0], position[0]), (0, dims[1]+dims[2]-
1), color )
self.ax.plot( (dims[0]+position[1], dims[0]+position[1]),
(dims[1]+1, dims[1]+dims[2]-1), color )

self.ax.set( xlim=(1, dims[0]+dims[1]), ylim=(0,

dims[1]+dims[2]) )

self.ax.text( dims[0] + dims[1]/2, dims[1]/2,

f"voxel index: {position}",
horizontalalignment='center',
verticalalignment='center' )

self.ax.axis( False )

self.fig.canvas.draw()

Task 1: Resample the T2-weighted scan to the image grid of the T1-
weighted scan
In the following a resampling of the T2-weighted scan is perfomed. This is done by finding the
voxel-to-voxel transformation between v T 1 and v T 2:

( )
vT 2
1
−1
( )
v
=M T 2 ⋅ M T 1 ⋅ T 1 .
1

At the location v T 2, cubic B-spline interpolation is used to determine the intensity in the T2-
weighted scan, and store it at index v T 1 in the newly created image.

# Defining the affine voxel-to-world matrices for the T1- and T2-
weighted scan
M_T1 = T1.affine
M_T2 = T2.affine
# Defining the inverse of the affine voxel-to-world matrix for the T2-
weighted scan
M_T2_inv = np.linalg.inv(M_T2)

# Defining the T1 coordinate grid in 3D

V1, V2, V3 = np.meshgrid(np.arange(T1_data.shape[0]),
np.arange(T1_data.shape[1]),
np.arange(T1_data.shape[2]),
indexing='ij')

# Stacking the coordinates and adding a row of 1s for homogeneous

coordinates
T1_coords = np.vstack([V1.ravel(), V2.ravel(), V3.ravel(),
np.ones(V1.size)])

# Using the voxel-to-voxel transformation equation

T2_coords = M_T2_inv @ (M_T1 @ T1_coords)

# Remove the homogeneous coordinate

T2_coords = T2_coords[:3]

# Interpolating the T2 data using cubic B-spline interpolation

T2_data_resampled = scipy.ndimage.map_coordinates(T2_data, T2_coords,
order=3).reshape(T1_data.shape)

Below the resampled T2-weighted volume and the T1-weighted and resampled T2-weighted
volume overlaid is visualized. This is done to give an insight into how the T2-weighted image is
rotated. It has been given that the T2-weighted volume is only rotated around a single axis.

# Normalizing data
image_T1 = T1_data / T1_data.max()
image_T2 = T2_data / T2_data.max()
image_moved = T2_data_resampled / T2_data_resampled.max()

#Plots
Viewer( image_moved )
plt.title('T2 in T1 space')

Viewer( image_moved + image_T1 )

plt.title('T2 in T1 space overlayed on T1')

C:\Users\sofie\AppData\Local\Temp\ipykernel_12224\1359666277.py:9:
UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be
shown
self.fig.show()

Text(0.5, 1.0, 'T2 in T1 space overlayed on T1')

The top plot above shows the normalized T2-weighted image in the T1 coordinate grid. The
bottom plot shows the T2-weighted image in the T1 coordinate grid overlayed on the T1-
weighted image. Looking at the bottom plot it becomes apparent that the T2-weighted image is
rotated around the y-axis in the T1-coordinate. The y-axis the one going up through the head.
The top left part of the bottom plot shows the image from above, and it is very clearly rotated in
that image.

Task 2: Compute and visualize the joint histogram

For images of different modalities, such as MRI and CT, registration/alignment by the sum-of-
squared-differences in energy is no longer optimal, as intensity characteristics differ a lot
between methods. An alternative technique is using mutual information (MI). By using this
method, the two images are firstly preprocessed to have only selected descreete intensities.
Afterwards the frequency of finding two mutual intensities in the two images are analysed. This
analysis is done using a joint histogram, therefore we start by determining the joint histogram

( )
h1 , 1 .. . h1 ,B
H= ⋮ .. . ⋮
h B ,1 .. . hB ,B

of the T1-weighted scan and the resampled T2-weighted we found in Task 1, using B=32 bins.
We do this by creating a function that takes two datasets as inputs and uses 32 bins as standard
to compute and plot a 2D histogram. The function then uses this 2D histogram to create a 3D
bar plot.

# Defining a function to make a joint histogram

def create_joint_histogram(T1_data, T2_data, bins=32):
hist, xedges, yedges = np.histogram2d(T1_data.ravel(),
T2_data.ravel(), bins=bins)

xpos, ypos = np.meshgrid(xedges[:-1] + 0.5 * (xedges[1:] -

xedges[:-1]),
yedges[:-1] + 0.5 * (yedges[1:] -
yedges[:-1]))
xpos = xpos.ravel()
ypos = ypos.ravel()
zpos = np.zeros_like(xpos)

bar_thickness = 50 # Defining the thickness of the bars

dx = dy = bar_thickness* np.ones_like(zpos)

dz = hist.ravel()

# Plotting
fig, ax = plt.subplots(subplot_kw=dict(projection='3d'))
ax.bar3d(xpos, ypos, zpos, dx, dy, dz, zsort='average')

ax.set_xlabel(f'T1 Intensity')
ax.set_ylabel(f'T2 Intensity')
ax.set_zlabel('Count')
 plt.show()

We ignore all voxels with an intensity lower than 10, so that the background voxels will not
dominate the image and use the function defined above to plot our 3D bar plot.

# Defining mask to filter out voxels with intensity lower than 10

mask = (T1_data >= 10) & (T2_data_resampled >= 10)

# Normalize
T1_norm = T1_data / T1_data.max()
T2_norm = T2_data_resampled / T2_data_resampled.max()

# Joint histogram
B = 32 # Number of bins

# Filtering out voxels where the intensity of both the T1- and T2-
weighted image is below 10
T1_filtered = T1_data[mask]
T2_filtered = T2_data_resampled[mask]

# Plot 3D histogram
create_joint_histogram(T1_filtered, T2_filtered, B)

The histogram above shows the intensity combinations of the two image histograms depicted as
many entries with small counts, that then add up. Generally, if the two images are well aligned,
most of these intensity combinations will be collected in few of the bins of the histogram plot.
However, if the images are poorly aligned, the histogram will be more spread out over various
bins. For our plot, we see that the two images seem somewhat aligned, but we will now work to
try and improve alignment by use of the mutual information technique.

Task 3: Compute the mutual information between the two images

We now want to determine the mutual information. To do this we write a function that takes
two image volumes defined on the same image grid as input, and returns the mutual
information between the two images:
M I =H F + H M − H F , M

with
B B
H F , M =− ∑ ∑ pf ,m log ( pf ,m ) ,
f =1 m=1

B
H F=− ∑ pf log ( p f ) ,
f =1

and
B
H M =− ∑ p m log ( p m ) .
m =1

The function uses the joint histogram computed as in the previous task (i.e., using B=32 bins
and ignoring all voxels with intensity lower than 10). It computes the mutual information
between the T1-weighted image and the resampled T2-weighted image as well as the mutual
information-based registration (here denoted nMI) in order to avoid pathological solutions.

# Defining function for computing mutual information

def compute_mutual_information(image1, image2, bins=32):
# Filter out voxels with intensity lower than 10
mask = (image1 >= 10) & (image2 >= 10)

# Joint histogram
hist, _, _ = np.histogram2d(image1[mask].ravel(),
image2[mask].ravel(), bins=bins)

# normalize the histogram

p_fm = hist / np.sum(hist)

# Add a small value to avoid log(0) issues

p_fm += 1e-12

# Probabilities of seeing specific intensity combinations

p_f = np.sum(p_fm, axis=1) # Sum over columns to get p_f
p_m = np.sum(p_fm, axis=0) # Sum over rows to get p_m
 H_F = -np.sum(p_f * np.log(p_f)) # marginal entropy of fixed image
H_M = -np.sum(p_m * np.log(p_m)) # marginal entropy of moving
image

# Joint entropy H_F,M

H_FM = -np.sum(p_fm * np.log(p_fm))

# Compute mutual information

MI = H_F + H_M - H_FM # Mutual information
nMI = H_FM - H_F - H_M # MI negative energy function used for
mutual information-based registration

return MI, nMI

# Using the above function to compute the mutual information

MI, _ = compute_mutual_information(T1_data, T2_data_resampled)
print(f'The initial Mutual Information is: {MI}')

The initial Mutual Information is: 0.11674745290694144

Task 4: Evaluate the mutual information across a range of rotation

angles
We now want to identify the angle with approximately the highest mutual information (i.e.,
where the registration is best). To do this we implement a grid search over a range of rotation
angles. We make an educated guess of a suitable range by visually inspecting the images, and
then define a list of candidate angles at intervals of e.g., 5° apart. We loop over all candidate
rotation angles, each time (1) creating a corresponding rotation matrix R ; (2) resampling the T2-
weighted volume accordingly; (3) calculating the mutual information with the function created
above; and (4) storing the mutual information value.

From Task 1, we found that the T2-weighted volume in the T1-coordinates was rotated around
the y-axis. Though, in the following the T2-weighted volume is rotated in the original T2-
coordinates and therefore rotated around the z-axis, thus the rotation angles around the other
axes are clamped to zero.

# Defining functions for each rotational axis

def rotation_matrix_x(theta_x):
"""Rotation matrix around the x-axis."""
return np.array([[1, 0, 0],
[np.cos(theta_x), -np.sin(theta_x),0],
[np.sin(theta_x), np.cos(theta_x),0]])

def rotation_matrix_y(theta_y):
"""Rotation matrix around the y-axis."""
return np.array([[np.cos(theta_y), -np.sin(theta_y), 0],
[0, 1, 0],
[np.sin(theta_y), np.cos(theta_y), 0]])
def rotation_matrix_z(theta_z):
"""Rotation matrix around the z-axis."""
return np.array([[np.cos(theta_z), -np.sin(theta_z), 0],
[np.sin(theta_z), np.cos(theta_z), 0],
[0, 0, 1]])

# Defining function for rotation of the image

def rotate_image(T1_data, T2_data, M_T1, M_T2,R, V1, V2, V3, angle):
angle = np.deg2rad(angle) # angle of rotation
V_T1 = np.array([V1.ravel(), V2.ravel(), V3.ravel(),
np.ones(V1.size)]) # stacking coordinates from the T1-grid and adding
row of 1s for homogeneous coordinates
vox_in_world = M_T1 @ V_T1 # defining the voxels in mm
rotated_world = R @ vox_in_world[:3, :] # Rotating the voxels
rotated_to_vox = np.linalg.inv(M_T2) @ np.vstack((rotated_world,
np.ones((1, V1.size)))) # Using the voxel to voxel equation
T2_rotated = map_coordinates(T2_data, rotated_to_vox[:3],
order=3).reshape(T1_data.shape) # Interpolating the T2 data with
rotated voxels using cubic B-spline interpolation
return T2_rotated

We define a function that maximizes the MI and minimizes nMI, to find the best rotational axis.

def find_best_rotation_angle(T1_data, T2_data, angles,axis='z'):

best_angle = None
best_nMI_angle = None
best_MI = -np.inf # Start with a very low MI
MIs = []
nMIs = []
best_nMI = 1 # Start with a very low nMI
best_resampled_data = None
best_resampled_nMI_data = None

for angle in angles:

# Convert angle to radians
theta = np.deg2rad(angle)

# Define the rotation matrix for the specified axis

if axis == 'z':
R = rotation_matrix_z(theta)
elif axis == 'y':
R = rotation_matrix_y(theta)
elif axis == 'x':
R = rotation_matrix_x(theta)
else:
raise ValueError("Axis must be 'x', 'y', or 'z'")

# Rotate the voxel coordinates

T2_coords_rotated = rotate_image(T1_data, T2_data, M_T1,
M_T2,R, V1, V2, V3, angle)

# Resample T2_data using the rotated coordinates

T2_data_resampled_rotate = T2_coords_rotated

# Compute MI and nMI between T1_data and the rotated T2_data

MI, nMI = compute_mutual_information(T1_data,
T2_data_resampled_rotate)
MIs.append(MI)
nMIs.append(nMI)

# Track the highest mutual information and the corresponding

angle
if MI > best_MI:
best_MI = MI
best_angle = angle
best_resampled_data = T2_data_resampled_rotate

# Track the lowest negative mutual information and the

corresponding angle
if nMI < best_nMI:
best_nMI = nMI
best_nMI_angle = angle
best_resampled_nMI_data = T2_data_resampled_rotate

print(f"Angle: {np.round(angle):<3} degrees, Mutual

Information: {MI:.4f}")

return best_angle, best_MI, best_resampled_data, MIs,

best_nMI_angle, best_nMI, best_resampled_nMI_data, nMIs

# Using the above function to determine the angle resulting in the

highest mutual information
angles = np.arange(-20, 55, 5)
best_angle, best_MI, best_resampled_data, MIs,_, _, _, _ =
find_best_rotation_angle(T1_data, T2_data, angles, axis='z')

print(f'\nBest Angle: {best_angle} degrees with Mutual Information:

{best_MI:.4f}')

Angle: -20 degrees, Mutual Information: 0.0992

Angle: -15 degrees, Mutual Information: 0.1017
Angle: -10 degrees, Mutual Information: 0.1065
Angle: -5 degrees, Mutual Information: 0.1118
Angle: 0 degrees, Mutual Information: 0.1167
Angle: 5 degrees, Mutual Information: 0.1209
Angle: 10 degrees, Mutual Information: 0.1248
Angle: 15 degrees, Mutual Information: 0.1313
Angle: 20 degrees, Mutual Information: 0.1399
Angle: 25 degrees, Mutual Information: 0.1485
Angle: 30 degrees, Mutual Information: 0.1662
Angle: 35 degrees, Mutual Information: 0.2002
Angle: 40 degrees, Mutual Information: 0.2643
Angle: 45 degrees, Mutual Information: 0.3510
Angle: 50 degrees, Mutual Information: 0.2707

Best Angle: 45 degrees with Mutual Information: 0.3510

Based on the mutual information from trying all angles between -20 and 55 in 5 degree
intervals, the best alignment of T1 and T2 is is achieved with 45 degree rotation of T2.

Task 5: Perform automatic registration

We now plot the negative mutual information, i.e. the energy function E ( w )=H F , M − H F − H M
for every 2 degrees in our grid search space, also by using the function find_best_rotation_angle.
We select the one with the lowest energy (i.e., the best angle for registration) and transform the
T2-weighted volume according to this angle. The rotated T2-image overlayed with the T1-image
is visualized below.

angles = np.arange(20, 75, 2)

_, _, _, _, best_nMI_angle, best_nMI, best_resampled_nMI_data, nMIs =
find_best_rotation_angle(T1_data, T2_data, angles, axis='z')

Angle: 20 degrees, Mutual Information: 0.1399

Angle: 25 degrees, Mutual Information: 0.1485
Angle: 30 degrees, Mutual Information: 0.1662
Angle: 35 degrees, Mutual Information: 0.2002
Angle: 40 degrees, Mutual Information: 0.2643
Angle: 45 degrees, Mutual Information: 0.3510
Angle: 50 degrees, Mutual Information: 0.2707
Angle: 55 degrees, Mutual Information: 0.1914
Angle: 60 degrees, Mutual Information: 0.1564
Angle: 65 degrees, Mutual Information: 0.1332
Angle: 70 degrees, Mutual Information: 0.1205

# Plotting
plt.figure()
plt.plot(angles, nMIs, 'o-')
plt.title('E(w) = H_FM - H_F - H_M')
plt.grid(True)
plt.xlabel('Angle')
plt.ylabel('E(w)')
print(f'\nBest Angle: {best_nMI_angle} degrees with Mutual
Information: {best_nMI:.3f}')

Viewer( best_resampled_nMI_data / best_resampled_nMI_data.max() +

T1_data / T1_data.max())
plt.title(f'Best rotated T2 ({best_nMI_angle} degrees) overlayed on
T1')
Best Angle: 45 degrees with Mutual Information: -0.351

C:\Users\jaco8\AppData\Local\Temp\ipykernel_43176\1359666277.py:9:
UserWarning: Matplotlib is currently using
module://matplotlib_inline.backend_inline, which is a non-GUI backend,
so cannot show the figure.
self.fig.show()

Text(0.5, 1.0, 'Best rotated T2 (45 degrees) overlayed on T1')

Above the result of the energy function (negative mutual information) is seen at top. When
calculating the engergy function the best result is when E(w) is the smallest possible. The plot
above confirms the same result as in task 4 where we found that the best rotation angle is 45
degrees. When looking at the bottom plot above showing the rotated T2-imag overlayed the T1
it becomes pparent that the two images almost perfectly aligns when the T2-image is rotated 45
degrees.

Task 6: Compare the joint histogram before and after registration

We now visualize the joint histogram before and after registration.

# Alter the function so we can use subplots

def create_joint_histogram2(T1_data, T2_data, bins=32, ax=None):
hist, xedges, yedges = np.histogram2d(T1_data.ravel(),
T2_data.ravel(), bins=bins)

xpos, ypos = np.meshgrid(xedges[:-1] + 0.5 * (xedges[1:] -

xedges[:-1]),
yedges[:-1] + 0.5 * (yedges[1:] -
yedges[:-1]))
xpos = xpos.ravel()
ypos = ypos.ravel()
zpos = np.zeros_like(xpos)

bar_thickness = 50
dx = dy = bar_thickness * np.ones_like(zpos)
 dz = hist.ravel()

# Plot using the provided axis `ax`

ax.bar3d(xpos, ypos, zpos, dx, dy, dz, zsort='average')

ax.set_xlabel('T1 Intensity')
ax.set_ylabel('T2 Intensity')
ax.set_zlabel('Count')

# Filter out voxels with intensity lower than 10

mask_2 = (T1_data >= 10) & (best_resampled_data >= 10)

# Normalizes data
T2_norm_2 = best_resampled_data / best_resampled_data.max()

# Apply mask
T1_filtered_2 = T1_data[mask_2]
T2_filtered_2 = best_resampled_data[mask_2]

# Create subplots: 1 row, 2 columns

fig, ax = plt.subplots(1, 2, figsize=(9, 5),
subplot_kw=dict(projection='3d'))
# First histogram on the left subplot
create_joint_histogram2(T1_filtered, T2_filtered, B, ax=ax[0])
ax[0].set_title("Original Data")
# Second histogram on the right subplot
create_joint_histogram2(T1_filtered_2, T2_filtered_2, B, ax=ax[1])
ax[1].set_title("45 degrees rotated data")

# Show the subplots

plt.tight_layout()
plt.show()
For the 45 degrees rotation (right), we see that the data's spread is more condensed, meaning
that the two images are more similar after the rotation than before.

Conclusion
We have now seen how using mutual information-based registration can help align two
different modalities for medical imaging. Both computing the mutual information and using the
energy function the results showed that the best rotation angle was 45 degrees. This was
confirmed by looking at the rotated T2-image overlayed the T1 and by the spread of values in the
histogram which showed what was introduced in the introduction.