🏞️

Part5: Image processing

Created @November 6, 2024 8:07 PM
Person Nick Yao Larsen
Start Time @October 30, 2024
Status In progress

import skimage as ski

import scipy as sp
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.cm as cm

from matplotlib.colors import LinearSegmentedColormap, ListedColormap

## upload images
img = ski.io.imread(img_name)

## display image
plt.imshow(img,cmap='gray')
sns.heatmap(kernel,annot=True,cmap='gray',ax=axes[])

1 Introduction: Digital image

1.1 Formation of digital images
1.2 Image classes
1.2.1 Binary image
1.2.2 Grayscale/intensity image
1.2.3 Indexed/labeled image
1.2.4 RGB image
2 Colors
2.1 RGB pixels
2.2 Convert color to grayscale
2.3 Convert grayscale images to color
2.3.1 Custom colormap
2.3.2 Convert to indexed images using threshold
3 Point processing methods
3.1 Gray level enhancement
3.1.1 brightness
3.1.2 Contrast
3.2 Image histograms
3.2.1 histogram stretching
 3.2.2 Thresholding
3.3 Global thresholding
3.3.1 Basic automatic thresholding
3.3.2 Variance minimization: Otsu’ method
3.4 Image arithmetic
4 Neighborhood processing methods
4.1 Correlation
4.2 Problems on the borders and solutions
4.3 Application of convolution
4.3.1 image blur/smoothing kernel/low pass filter
4.3.2 remove noise by rank filters/order-statistics filters
4.3.3 edge detection
5 Morphological operations
5.1 Dilation: g(x, y) = f (x, y) ⊕ SE
5.2 Erosion: g(x, y) = f (x, y) ⊖ SE
5.3 Opening:(f (x, y) ⊖ SE) ⊕ SE
5.4 Closing: (f (x, y) ⊕ SE) ⊖ SE
6 Geometric image transformation
6.1 Affine transformations
6.1.1 scaling/resize/stretch: change size and shape
6.1.2 rotation
6.1.3 cropping: keep only a portion of the original image
6.1.4 shearing: “pulling” a corner of the image
6.2 Projective transformations
6.3 Interpolation
6.3.1 nearest neighbor
6.3.2 bi-linear interpolation
6.3.3 bi-cubic interpolation
6.4 Image registration
6.4.1 unassisted image registration
6.4.2 interactive registration

1 Introduction: Digital image

1.1 Formation of digital images
An images f(x,y) is represented as a 2D array/matrix
discrete
x and y cannot take on any value! They must be within the bounds of the image.
Width = number of pixels in x-direction
Height = number of pixels in y-direction
Size = (width × height)
 ### shape of image
image.shape

### scale of image: change shape of image-array

# downscale will decrease resolution of the image
ski.transform.rescale( img, scale=(m,[n,...]) )

### zoom into region of interest (ROI)

## define ROI
roi_start = (500, 100) # point on the bottom left
roi_height, roi_width = 300, 250
roi = retina[roi_start[0]:roi_start[0]+roi_height,roi_start[1]:roi_start[1]+roi_w

## show the orignal image and ROI

plt.figure(figsize=(10,5))
plt.subplot(1,2,1)
plt.imshow(retina)
# draw a rectangle an the plot
rect = plt.Rectangle(roi_start[::-1],roi_width,roi_height,edgecolor='r',facecolor
plt.gca().add_patch(rect)
plt.title('Original RGB Image with ROI')

plt.subplot(1,2,2)
plt.imshow(roi)
plt.title('Zoomed in ROI')

plt.show()

each dot/cell in an image is called a pixel

each cell has a value, this value is a pixel intensity
the size of an area in a scene is represented by the number of pixels in an image
sample frequency ~ resolution: low resolution leads to data reduction

Quantization
 🏷️ Why are bits interesting?
the human visual system cannot detect more than 256 different gray levels in an image
available
often this quantization results in a representation of 1 Byte (8 bits), since 1 byte
corresponds to the way memory is organized inside a computer (0 and a high charge
quantized to 255)

### Grayscale image quantization of 4 colors

# Step 1: Rescale the intensity values to the range [0, 1]

# image/255

# Step 2: Scale the rescaled values to the desired color range

# now the range is [0,n_colors-1]
n_colors = 4
scaled_values = image/255 * (n_colors-1)

# Step 3: Round the scaled values to represent discrete color levels

# [0,0.5) = 0, [0.5,1.5) = 1, [1.5,2.5) = 2, [2.5,3] = 3, get 4 discrete values
rounded_values = np.round(scaled_values)

# Step 4: Rescale the rounded values to bring them back to the original intensity
# [0,3] -> [0,1] -> [0,255]
image_quantized = rounded_values * 255 / (n_colors-1)

1.2 Image classes

### image classes convertation (grayscale image)

# convert from int to double float

ski.util.img_as_float64(img)

# convert from float to int

ski.util.img_as_int(img)

There are four different image classes or encoding schemes:

1.2.1 Binary image
number is either 0 or 1 indicating black or white
logical arrays
they are often used as image “mask”, (i.e. to isolate selected
segments of an image)
threshold between 0 and 1 can be defined

1.2.2 Grayscale/intensity image

number indicates grayscale intensity
float format: 0 to 1
int format: 0 to 255

### count number of different intensities

np.unique(img).size

1.2.3 Indexed/labeled image grayscale image: [row, col]

pixel number is pointer (i.e. index) to an RGB color map.

permit easy change of colorization schemes
need an accompanying colormap ot a predefined colormaps for
visualization
For pixel operations, you need to convert to grayscale or RGB.

1.2.4 RGB image

number array indicates intensity of [red, blue, green] (0 to 1 OR 0 to RGB image: [row, col, rgb]
255)
three separate arrays are required

2 Colors
Red, green, blue are called Primary Colors
R, G, B were chosen due to the structure of the human eye
R, G, B are used in cameras as they got three sensors

2.1 RGB pixels

 in a color image, each pixel consists of three values: red, green and
blue
color pixel = [Red, Green, Blue]

Typically each color value is represented by an 8-bit value meaning that 256 different shades of each
color can be measured (totally 2563 colors can be represent in an pixel)
the actual representation might be three images - one for each color, but it can also be a three-
dimensional vector for each pixel, hence an image of vectors

### separate the red, green and blue channels of an RBG image

# red_img, green_img and blue_img are still RGB images

red_img = np.zeros_like(img)
red_img[:,:,0] = img[:,:,0]

green_img = np.zeros_like(img)
green_img[:,:,1] = img[:,:,1]

blue_img = np.zeros_like(img)
blue_img[:,:,2] = img[:,:,2]

### Swap color channels

# [R,G,B] = [1(G),2(B),0(R)]
img_new = img[:,:,[1,2,0]] # !!img_new and img share pointer in this method

2.2 Convert color to grayscale

Intensity = WR ∗ R + WG ∗ G + WB ∗ B, WR + WG + WB = 1
​ ​ ​ ​ ​ ​

1
as default, the three colors are equally important, hence WR = WG = WB =
​ ​ ​

3
​

optimal weight vary based on the task

 ### covert from color to grayscale (de-dimension)

# compute luminance of an RGB image

img_color = ski.color.rgb2gray(img)
plt.imshow(img_color,cmap='gray')

2.3 Convert grayscale images to color

2.3.1 Custom colormap
a colormap is a m×3 matrix, with values in double format in the
range [0,1]
is used to map pixel data to the actual color values
https://matplotlib.org/devdocs/users/explain/colors/colormaps.html#colormaps

### Generate a colormap

# create a list of colors

colorslist = ["black", "white", "magenta", "blue", "yellow", "red"]
#OR colorslist = [[0, 0, 0], [1, 0, 0], [0, 1, 0], [0, 0, 1]]

# generate a colormap based on the color list

mycmap = ListedColormap(colorslist,name='myCoMap')

2.3.2 Convert to indexed images using threshold

a method for colorizing grayscale images: Slicing the image into levels
slices the image into N levels using an equal step threshold process

### slice the intensity levels into 6 levels

## Step1: make sure the values range from [0-255]

# normalize the value to [0,1]

img_norm = ( img-np.min(img) )/( np.max(img)-np.min(img) )

 # OR: img_norm = ski.exposure.rescale_intensity(img,out_range='float64')

# rescale to [0,255]
img1 = np.uint8(img_norm*255)

## Step2: slice the intensity levels by digitizing

# number of values we want to get
num_categories = len(colorslist)
# seperate [0,255] into num_categories
# we need num+1 points
bin_edges = np.round( np.linspace(0,255,num_categories+1) )
# digitize
digitized_img = np.digitize(img1, bins=bin_edges[1:-1])

### display image using a costum colormap

plt.imshow(img,cmap=cm.get_cmap(mycmap))

3 Point processing methods

point processing is now defined as an operation which calculates the new value of a pixel in g(x,y) based
on the value of the pixel in the same position in f(x,y)
Simplest type of operation because the pixel value itself is independent on the surrounding pixels

3.1 Gray level enhancement

3.1.1 brightness
 brighter image darker image

the brightness is the intensity

change brightness: g(x, y) = f (x, y) + b
if b > 0, brighter image
if b < 0, darker image
be aware of image saturation (image bleaching)
the pixels cannot exceed its maximum or minimum values

3.1.2 Contrast
contrast describes the level of detail we can see, or the difference between pixel values

change contrast: g(x, y) = a ∗ f (x, y)

if a > 0, contrast increase
if a < 0, contrast decrease
be aware of image saturation
apply gray level transformation in
“sections”
the slope of T (r):
if slope > 1, contrast increases
if slope < 1, contrast decreases
if slope = 1, no change

### automatically adjusting the brightness and contrast of the image

img_adapted = ski.exposure.equalize_adapthist(img)*255

3.2 Image histograms

How can we tell if an image is too dark or too bright?
A histogram is a graphical representation of the frequency of events
the horizontal axis represents the possible numbers in a matrix/picture
the vertical axis (height) represents the count of the numerical value on x-
axis in the matrix

A histogram is a discrete function p(k) = nk

N
​

nk is the number of pixel with the k-th gray level

​

N is the total number of pixels

p(k) gives the probability of k appearing in the image
histograms tell us about the intensity/color distributions of a picture
histograms can act as a sort of fingerprint of a picture
but be aware, two completely different pictures may have the same histogram

### plot image histogram and draw the thresholds

plt.figure(figsize=(8,5))
plt.hist(img.ravel(), bins=256,range=(0,255),density=True,alpha=0.7)

plt.vlines(x=0.1, ymin=0,ymax=50, color='red', linestyle='dashed', linewidth=2, labe

plt.vlines(x=0.4, ymin=0,ymax=50, color='red', linestyle='dashed', linewidth=2, labe
plt.vlines(x=0.7, ymin=0,ymax=50, color='red', linestyle='dashed', linewidth=2, labe

plt.ylim([0,45])

plt.title('Grayscale Image Histogram')

plt.xlabel('Pixel Value')
plt.ylabel('Frequency')

plt.legend()
plt.show()

### define a function to display the image and the histogram

def display_image_and_histogram(img, img_title='Original Image',hist_title='Image Hi

plt.figure(figsize=(12,5))

plt.subplot(1,2,1)
plt.imshow(img, cmap='grey', vmin=0, vmax=255)
 plt.title(img_title)
plt.colorbar()

plt.subplot(1,2,2)
plt.hist(img.flatten(), bins=100, range=(0,255), color='blue')
plt.title(hist_title)
plt.xlabel('Pixel Value')
plt.ylabel('Frequency')

plt.show()

3.2.1 histogram stretching

humans cannot tell the difference between graylevel values too close to each other
so, spread out graylevel values

### define a function of linear stretch

def img_linear_stretch(img, new_min, new_max):

min_intensity = np.min(img)
max_intensity = np.max(img)
new_img = (new_max-new_min)/(max_intensity-min_intensity)*(img-min_intensity)+ne
new_img = np.uint8(new_img)

return new_img

### define a cumulatice stretching

def img_cdf_stretch(img):
# calculate the counts number of values from 0 to 255
counts, values = np.histogram(einstein.flatten(), bins=256, range=(0,255))
# convert intensity value into cumulative probability [0,1]
cdf_normalize = np.cumsum(counts)/sum(counts)
# the values in map is the probability of [0,1] corresponding to [0,255]
map = np.uint8(cdf_normalize * 255)
 # use the intensity value of img as index
return map[img]

3.2.2 Thresholding
segmentation is about image analysis, not image manipulation
the task: information vs. noise; foreground (object) vs. background
use graylevel mapping and the histogram
when two peaks/nodes of a histogram correspond to object and noise, find a threshold value T that
separates the two peaks

result: a binary image where object pixels =1 and noise/background pixels = 0

### Apply manuel threshold

# Set the threshold you choose based on histogram

Threshold = 100

# Threshold
img_threshold = img <= Threshold
#OR: img_threshold = img >= Threshold

### invert binary image (白底和黑底互换）

img_invert = ski.util.invert(img)

# OR display inverted image directly

plt.imshow(~img,cmap='gray')

3.3 Global thresholding

3.3.1 Basic automatic thresholding
automatic (basic for bimodal histogram)
assumption: a separation of the histogram peaks → separation the background and object
approach
1. select an initial estimate for T
2. segment the image using T → image is divided into two groups G1 and G2
​ ​

3. compute the average gray level value m1 and m2 por the pixels in regions G1 and G2
​ ​

4. compute a new threshold: T = 12 (m1 + m2 )

​ ​ ​

5. repeat step 2-4 until the difference in T is smaller than a predefined parameter T0 ​

3.3.2 Variance minimization: Otsu’ method

### Apply Otsu method to choose the threshold

Threshold_otsu = ski.filters.threshold_otsu(img)

img_threshold = img <= Threshold_otsu

#OR: img_threshold = img >= Threshold_otsu
3.4 Image arithmetic
arithmetic: +, -, ×, /
perform simple math on pixel level
apply the operation as point processing
for substraction, you may be interested only in the absolute difference

### Create noise on the image

# this function add noise to the image

# the result image ranges [0,1]
# if the original image ranges [0,255], it'll be converted automatically
# mode: 'salt','pepper','s&p'
img_noise = ski.util.random_noise(img, mode='gaussian')

# imgae with noise

img_noise = (img_noise*255).astype(np.uint8)

apply mask: matric dot multiply （*）

### apply mask on the image

# load the mask

mask = ski.io.imread('b_mask.tif')
# apply
img_mask = blood.astype('float64')*mask
plt.imshow(img_mask,cmap='gray')

overflow / underflow
the result of calculation may be smaller than 0 or larger than 255
solution: use an intermediate image: pixel values are float values (32 bits / 4 bytes), can store almost
any number
1. write computation result into intermediate image
2. rescale intermediate image to values [0,255] and write results into 8 bit image
image−min
N ormalizedimage = ​

max−min
​

4 Neighborhood processing methods

4.1 Correlation
 Correlation vs. Convolution
when apply convolution the filter is flipped both horizontally and vertically before being applied to
the input image
in image processing we use correlation, but (nearly) always call it convolution! We’re silly like that.
when the filter is symmetric: correlation = convolution
Convolution on images

### conduct convolution

# use grayscale image |you can convert RGB image to grayscale by "ski.color.rgb2gray
img_filter = sp.signal.convolve2d(img,kernel)

4.2 Problems on the borders and solutions

the output image is smaller than the input. And the bigger the kernel, the bigger the problems. | 边缘通常不
包含重要信息，这个问题只有在combine images with the same size的时候才需要特别考虑，我们需要尽量
保持形状一致 |

copy the input image after processing: copy the outer border in the input
padding: add a fixed value around the edge: 0, 255 | this process will change histogram |
truncate kernel when at the edge | complex and not well-defined |
mirror padding: copy the outer border in the output

4.3 Application of convolution

 🏷️ Template matching
the filter is called a template or a mask
the template is just an ordinary picture, which is correlated with the input image
the brighter the value in the output, the better the match

4.3.1 image blur/smoothing kernel/low pass filter

the simplest filter: Mean filter (spatial low pass filter)
another option: Gaussian filter
degree of blurring = kernel size
application
remove identity or other details

preprocessing: enhance objects by clurring and thresholding

### Apply smoothing filters

Size = (5,5) # size of the filters

Sigma = 0.5 # parameter for Gaussian filter
 # apply mean filter
img_mean = ski.filters.rank.mean(img,footprint=np.ones(Size))
# apply gaussian filter or add Gaussian noise
img_gaussian = ski.filters.gaussian(img,sigma=Sigma)

4.3.2 remove noise by rank filters/order-statistics filters

very useful to avoid noise of the image
not based on correlation but still neighborhood processing
principle
1. define a mask/kernel size
2. sort all pixel-values within the mask into ascending order
3. select a pixel-value according to the filter type: median, min, max, range,…

example: median filter

good for cleaning salt-and-pepper noise
good at removing noise in binary images
better than the mean filter as blurring is
minimized and edges stay sharp

### Apply median filter to remove noise

Size = (5,5) # size of the filters

img_median = ski.filters.median(img,footprint=np.ones(Size))

4.3.3 edge detection

Why are edges interesting?

edges can represent the information in the image (the objects)
a higher level of abstraction (less data to process)
edges are features independent of illumination, as opposed to color information
 object recognition and detection often use edge feature
machine vision: excellent for measurements of size
What are edges?

if we represent an image by height as opposed to intensity, then edges correspond to places where
we have steep hills
for each point in this image landscape, we have two gradients: x-direction and y-direction.
Edge detection steps
1. noise reduction | because edge detect filters are high-resolution sensitive |
2. edge enhancement: calculate candidates for the edges

Gradient vector: g = [gx , gy ]

​ ​ ​

Magnitude: gm = ​ gx2 + gy2

​ ​

### Calculate the gradient using rank-based method

img_gradient = ski.filters.rank.gradient(img,footprint)

3. edge localization: decide which edge candidates to keep

threshold the gradient magnitude
if the magnitude in g(x, y) > threshold, I(x, y) = 255, else I(x, y) = 0
Algorithms
Sobel conclusion
 vertical and horizontal sobel filter
Canny
Prewitt Operator
Scharr Operator

### Apply edge detect filter

## sobel filter return grayscale image

img_sobel = ski.filter.sobel(img)
# threshold img_sobel to convert it into binary image
threshold = ski.filter.threshol_otsu(img_sobel)
img_sobel_otsu = img_sobel <= threshold

## canny filter return binary image directly

img_canny = ski.filter.canny(img)

### Apply filter to fill holes

img_fill = sp.ndimage.binary_fill_holes(img)

5 Morphological operations
### Generate a structural element (SE)

## generate a cross SE
Radius = 1 # the size of SE will be 2*Radius+1
SE = ski.morphology.disk(Radius)

## display it
sns.heatmap(SE,cmap='gray',cbar=False, annot=True)

2D structure element (kernel)

5.1 Dilation: g(x, y) = f(x, y) ⊕ SE
### Apply dilation
img_dilated = ski.morphology.dilation(img,SE)

∃1 in the result of element multiply in SE overlap ⟹ output = 1, otherwise, output = 0

objects grow and holes are filled

sharp corners are preserved

5.2 Erosion: g(x, y) = f(x, y) ⊖ SE

### Apply erosion
img_eroded = ski.morphology.erosion(img,SE)

∀ results in SE overlap = 1 ⟹ output=1, otherwise, output = 0

objects shrink and noise specks are removed

application example: counting objects
counting these coins is difficult because they touch each other
solution: use thresholding and erosion to separates them

5.3 Opening:(f(x, y) ⊖ SE) ⊕ SE

### Remove small white dots by opening
img_open = ski.morphology.opening(img,SE)

# OR: remve objects smaller than min_size (pixels)

img_move_objects = ski.morphology.remove_small_objects(img, min_size)

first do erosion, and then do dilation using same SE

 isolate objects and remove small objects (better than erosion)
idempotent: repeated operations have no further effects! in other words, you always get the same
results
use large SE that fits into the big objects

5.4 Closing: (f(x, y) ⊕ SE) ⊖ SE

### Remove small black holes by closing
img_close = ski.morphology.closing(img,SE)

# OR: remve holes smaller than min_size (pixels)

img_move_holes = ski.morphology.remove_small_holes(img, min_size)

fill holes but keep original size and shape (better than dilation)
idempotent

6 Geometric image transformation

6.1 Affine transformations
straight lines remain straight
parallel lines remain parallel
rectangles may become parallelograms
need 3 point-pairs to calculate transformation

6.1.1 scaling/resize/stretch: change size and shape

6.1.2 rotation
 usually rotation is defined to rotate around the center point

### Conduct rotation

img_rotated = ski.transform.rotate(img,angle)
# angle: rotation angle in degrees in counter-clockwise direction.

6.1.3 cropping: keep only a portion of the original image

usually the crop area is defined by a rectangle

通过直接索引image array中的部分像素点就能实现

6.1.4 shearing: “pulling” a corner of the image

shift pixels horizontally or vertically with different amount depending on the position

### Conduct shearing

## Generate a shearing matrix

shear_h = 0.3 # horizontal shear angles, clockwise
shear_v = -0.1 # vertical shear angles
shear_matrix = ski.transform.AffineTransform(shear=(shear_h,shear_v))

## apply the transformation matrix

img_shear = ski.transform.warp(img,shear_matrix)

### Conduct translation by pixels

tx = -20 # move 20 pixels to right

 ty = 0
tform = ski.transform.AffineTransform( translation=(tx,ty) )
img_trans = ski.transform.warp(img,tform)

6.2 Projective transformations

straight lines remain straight
parallel lines may converge towards a vanishing point

🏷️ Conduct Project Transformation

Source Coordinate System:
The source coordinate system represents
the original image. Each point in the source coordinate system is defined by its (x, y) coordinates.
Destination Coordinate System:
The destination coordinate system represents
the transformed image. Like the source coordinate system, points in the destination system are
defined by (x, y) coordinates. These coordinates determine where the corresponding points from
the source image will be placed in the transformed image.
Mapping Points:
When you apply the transformation matrix to the source points, it computes the coordinates of
the corresponding points in the destination.

### Apply projective transformation on image

# Define source cordinate (变形后的坐标点)

src = np.array([[0, 0], [0, height], [width, height], [width, 0]])
# Define destination cordinate (要变形的区域)
dst = np.array([[155, 15], [65, 40], [260, 130], [360, 95]])

# Calculate transform matrix

tform = ski.transform.ProjectiveTransform()
tform.estimate(src, dst)
 # Conduct tranformation using the matrix
img_warped = ski.transform.warp(img, tform)

6.3 Interpolation
when doing transformations, it is often not possible to map pixels 1 to 1
hence, spatial transformations usually require some form of interpolation to add possible anti-aliasing
interpolation methods: since computation increases with the number of pixels that are considered,
there is a trade off between quality and computational time

6.3.1 nearest neighbor

the output pixel is assigned the value of the closest pixel in the transformed
image. An input pixel may fail into two or more output pixels

6.3.2 bi-linear interpolation

the output pixel is the weighted average of the transformed pixels in the nearest 2×2 neighborhood
6.3.3 bi-cubic interpolation
the weighted average is taken over a 4×4 neighborhood

6.4 Image registration

Image registration is the alignment of two or more images so they best superimpose. To achieve the
best alignment, it may be necessary to transform the images.
image registration can be quite challenging even when the images are very similar. Frequently, the
images to be aligned are not that similar, perhaps because they have been acquired using different
modalities.
the difficulty in accurately aligning images presents a significant challenge to image registration
algorithms, so the task is often aided by a human intervention or the use of embedded markers for
reference.

6.4.1 unassisted image registration

relies on an optimization technique to maximize the correlation, or other measure of similarity between the
images

6.4.2 interactive registration

some reduction in sharpness is seen in the realigned image as a result of information lost in the distortion process

uses human pattern recognition skills to aid the alignment process, usually by selecting corresponding
reference points in the images
the number of reference pairs required is the same as the number of variables needed to define a
transformation
an affine transformation will require a minimum of 3 reference points
a projective transformation requires 4 variables
more reference points generally improve the alignment

### Create and play the movie of the series of images

%matplotlib tk
# Deactivate pop up => %matplotlib inline
import matplotlib.animation as animation
fig = plt.figure()
images = []
# We can just plot each picture directly without creating the axes, since we need to
for i in range(36):
images.append([plt.imshow(image_stack[:,:,i],cmap='gray', animated=True)])

# Create the animation object, where we input the figure and the images
ani = animation.ArtistAnimation(fig, images, interval=50, repeat=False)
plt.axis('off')
plt.show()

# Save the animation as a GIF or MP4

ani.save('rotation_animation.gif', fps=20) # Save as GIF
# ani.save('rotation_animation.mp4', writer='ffmpeg', fps=20) # Save as MP4
%matplotlib inline