DTU Compute

Introduction to Medical Image

Analysis
Rasmus R. Paulsen
DTU Compute

[email protected]

http://courses.compute.dtu.dk/02512

Plenty of slides adapted from Thomas Moeslunds lectures

DTU Compute

Week 3
Pixelwise operations and colour images

2 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

What can you do after today?

 Compute and apply a linear gray transformation
 Describe and compute the image histogram
 Implement and apply histogram stretching
 Implement and apply gamma transformation
 Implement and apply log and exp mappings
 Describe and use thresholding
 Describe and use automatic thresholding
 Perform conversions between bytes and doubles
 Use addition and subtraction of images
 Explain the benefits of bi-modal histograms
 Identify images where global thresholding can be used for
object extraction

3 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

…and you can even more

 Describe the basic human visual system including rods and cones
 Describe subtractive colors
 Describe additive colors
 Describe the RGB color space
 Describe the normalised RGB color representation
 Describe the use of the Bayer pattern in digital cameras
 Describe the HSI color space
 Convert from an RGB to a grey level value
 Convert from an RGB value to an HSI value
 Describe the use of different color spaces
 Implement and use color thresholding in RGB space
 Implement and use color thresholding in HSI space

4 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Gray value mappings

 Mapping
0 1 2 3 4 5 – To make correspondence
between two sets of values
 Look-up-table
– A table of mappings

5 17 53 75 99 180

5 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

7 DTU Compute, Technical University of Denmark Image Analysis 2020

 DTU Compute

Gray value mappings

1  Mapping
– To make correspondence
out between two sets of values
>0.1  Mapping function
– out = f(in)
0.5  What happens with the
values?
0.1 – Values with difference 0.1
– Output values “spread out”

0 1
0.5
in

8 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Why change gray level values

 When could it be good to change the gray level
values?
– Lack of contrast
– Make the image lighter
– Make the image darker

9 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Point processing
Input Output
1 2 0 1 3
12
2 1 4 2 2
9

1 0 1 0 1

1 2 1 0 2

2 5 3 1 2

• The value of the output pixel is only dependent on the value of one input pixel
• A global operation – changes all pixels

10 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Point processing
 Grey level enhancement
– Process one pixel at a time independent of all other pixels
– For example used to correct Brightness and Contrast
 Known from the television remote control

Too high Too low Too high Too low

Correct brightness brightness contrast contrast

11 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Brightness

 The brightness is the intensity

 Change brightness:
– To each pixel is added the value b
– f(x, y) is the input image
– g(x, y) is the (enhanced) output image
 If b>0 : brighter image
 If b<0 : less bright image

𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 𝑓𝑓 𝑥𝑥, 𝑦𝑦 + 𝑏𝑏

12 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Brightness

13 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Contrast
 The contrast describes the level of details
we can see
 Change contrast
 Each pixel is multiplied by a
– f(x, y) is the input image
– g(x, y) is the (enhanced) output image
 If a>1 => more contrast
 If a<1 => less contrast

𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 𝑎𝑎 ∗ 𝑓𝑓(𝑥𝑥, 𝑦𝑦)

14 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Combining brightness and contrast

 A straight line
 Called a linear transformation
 Here a = 0.7 and b = 20

out

in

𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 𝑎𝑎 ∗ 𝑓𝑓 𝑥𝑥, 𝑦𝑦 + 𝑏𝑏
15 DTU Compute, Technical University of Denmark Image Analysis 2020
DTU Compute

17 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Combining brightness and contrast

 A straight line
 Called a linear transformation
 Here a = 0.7 and b = 20
 What will the result be on the
output image?
out
– More bright (b > 0)
– Less contrast (a < 1)

in

𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 𝑎𝑎 ∗ 𝑓𝑓 𝑥𝑥, 𝑦𝑦 + 𝑏𝑏
19 DTU Compute, Technical University of Denmark Image Analysis 2020
DTU Compute

Histogram Reminder
 A histogram normally contains the same number of “bins” as
the possible pixel values
 A bin stores the number of pixel with that value

20 DTU Compute, Technical University of Denmark Image Analysis 2020

22 DTU Compute, Technical University of Denmark
DTU Compute

Back to the histogram

 The shape of the histogram tells us a lot!

23 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Histogram inspection

Dark image

Bright image

24 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Histogram inspection

Low contrast

High contrast

25 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Histogram stretching

 How do we optimise the image using the histogram?

– Minimum and maximum values?
– Stretch it so new minimum = 0 and new maximum = 255

26 DTU Compute, Technical University of Denmark Image Analysis 2020

28 DTU Compute, Technical University of Denmark
DTU Compute

Histogram stretching
 We want
– Min = 0
– Max = 255
 We have
– Min = 32
– Max = 208

𝑣𝑣𝑚𝑚𝑚𝑚𝑚𝑚,𝑑𝑑 − 𝑣𝑣𝑚𝑚𝑖𝑖𝑖𝑖,𝑑𝑑
𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 𝑓𝑓 𝑥𝑥, 𝑦𝑦 − 𝑣𝑣𝑚𝑚𝑚𝑚𝑚𝑚 + 𝑣𝑣𝑚𝑚𝑚𝑚𝑚𝑚,𝑑𝑑
𝑣𝑣𝑚𝑚𝑚𝑚𝑚𝑚 − 𝑣𝑣𝑚𝑚𝑚𝑚𝑚𝑚

29 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Histogram stretching formula

𝑣𝑣𝑚𝑚𝑚𝑚𝑚𝑚,𝑑𝑑 − 𝑣𝑣𝑚𝑚𝑖𝑖𝑖𝑖,𝑑𝑑
𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 𝑓𝑓 𝑥𝑥, 𝑦𝑦 − 𝑣𝑣𝑚𝑚𝑚𝑚𝑚𝑚 + 𝑣𝑣𝑚𝑚𝑚𝑚𝑚𝑚,𝑑𝑑
𝑣𝑣𝑚𝑚𝑚𝑚𝑚𝑚 − 𝑣𝑣𝑚𝑚𝑚𝑚𝑚𝑚

 Desired min value vmin,d = 0

 Desired max value

vmax,d = 255

 Current min value vmin = 32

 Current max value vmax = 208

30 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Histogram stretching

255
𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 𝑓𝑓 𝑥𝑥, 𝑦𝑦 − 32
176
31 DTU Compute, Technical University of Denmark Image Analysis 2020
DTU Compute

Effect of histogram stretching

208 255
32 0

32 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Histogram stretching – weaknesses

 A single pixel value of 0 or 255 ruins it
 Sometimes you want
– To stretch only the high pixel values
– While “compressing” the low pixel values
– Non-linear mapping

33 DTU Compute, Technical University of Denmark Image Analysis 2020

35 DTU Compute, Technical University of Denmark
DTU Compute

Deep learning and color/gray scale

transformations
 Deep learning needs training data
– Input image
– Ground truth labels or classes http://uc-r.github.io/feedforward_DNN

 When you lack data you can

augment your data
– Create artificial versions
– Adding variation Luc, Pauline, et al. "Predicting deeper into the future of
semantic segmentation." IEEE International Conference on

– Changing gray / color levels in the Computer Vision (ICCV). Vol. 1. 2017.

image
– Point wise operations
https://www.quora.com/What-does-the-term-
semantic-segmentation-mean-in-the-context-
of-Deep-Learning

36 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Other mappings
 Non-linear mappings
 Not always nice to work with byte images
– Better to work with image with values in [0,1]

Conversion to [0,1]
Back to bytes

Byte image
Non-linear transformation Byte image

37 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Working with bytes and doubles

 A byte contains integer values [0,255]
– A byte can not store 127.4232
 A value of type double can contain “all numbers”
 Why not use doubles always?
– One double = 8 bytes in the memory
– Images become very large!
– Many things can be done with bytes

38 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Map pixels to [0,1]

 In Matlab it is easiest to create a new image of type
double
– Itemp = double(I);
– (temp means temporary and is used by many programmers
for variables that quickly are thrown away)
 Conversion to [0,1]

1
𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 𝑓𝑓(𝑥𝑥, 𝑦𝑦)
255
39 DTU Compute, Technical University of Denmark Image Analysis 2020
DTU Compute

Pixels back to bytes

 Input pixels are [0,1]
 We want them to be [0,255]
 Simple linear transformation equal to
– Contrast?
– Brightness?

𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 255 ∗ 𝑓𝑓(𝑥𝑥, 𝑦𝑦)

 Back to bytes
– Ifinal = uint8(Itemp);

40 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Gamma mapping
 Gamma mapping is used in televisions and flat
panels
 Can increase the contrast (dynamics) in more
selected part of the histogram
 Many games have a possibility for a gamma
correction

41 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Gamma curves
 Named after the Greek letter
gamma
 What happens to the dark
areas
– With 0.45?
– With 2.22?

𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 𝑓𝑓(𝑥𝑥, 𝑦𝑦)𝛾𝛾

42 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Perform the gamma mapping

Conversion to [0,1]
Back to bytes

Byte image
Gamma mapping Byte image

43 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Results of gamma mapping

0.45 2.22

44 DTU Compute, Technical University of Denmark Image Analysis 2020

46 DTU Compute, Technical University of Denmark
DTU Compute

Logarithmic mapping

Why?

𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 𝑐𝑐 log (1 + f x, y )

255
𝑐𝑐 =
log(1 + 𝑣𝑣𝑚𝑚𝑚𝑚𝑚𝑚 )

Maps from [0,255] to [0,255]

47 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Logarithmic mapping – when?

 For images with very
bright spots
 Low intensity pixel
values are enhanced

48 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

What do we get out of pixel mappings

 Spreading out or compressing pixel values
– Better for humans to see
– New information – no!

49 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Now for something different

 Until now image processing
– Input image transformed to output image
 Now for something more like image analysis
 Segmentation
– Segment the image into regions
 Background and objects for example

50 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Thresholding
 A threshold T is a value
– Pixels below that value is set to 0 (background)
– Pixels equal or above is set to 1 (object)
 One threshold value for the entire image
– Difficult to choose!

if 𝑓𝑓 𝑥𝑥, 𝑦𝑦 ≤ 𝑇𝑇 then 𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 0

if 𝑓𝑓 𝑥𝑥, 𝑦𝑦 > 𝑇𝑇 then 𝑔𝑔 𝑥𝑥, 𝑦𝑦 = 255

51 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Thresholding

T=110

Background
and bone
have same
value!

T=128 T=140

52 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Thresholding based on the histogram

The bones are visible in the histogram!

But mixed with soft-tissue

T=128 T=140
T=110

53 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Automatic Tresholding

54 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute Otsu, Nobuyuki. "A threshold selection method from gray-level
histograms." IEEE transactions on systems, man, and
cybernetics 9.1 (1979): 62-66.

Automatic Tresholding
Otsu’s method

 Two classes: background and

object
 T divides pixels into object and
background
 Compute pixel value variance in
each class
 Find T that minimises combined
variance
55 DTU Compute, Technical University of Denmark Image Analysis 2020
DTU Compute

Segmentation – histogram shaping

 With a threshold you want

a histogram with two peaks
– Bimodal
 An ideal histogram has well
separated peaks
Background  Obtaining a bi-modal
Object histogram is very important
in the image acquisition

Ideal

56 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Histogram shaping
 It is not possible to “unmix”
using gray level
transformations

Should be Should be
higher lower

57 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

How to obtain good histograms

 With cameras
– Light
– Setup
– Camera
– Lens
– Backlight?

58 DTU Compute, Technical University of Denmark Image Analysis 2020

60 DTU Compute, Technical University of Denmark
DTU Compute

Colour images and colour perception

The Human Eye

https://askabiologist.asu.edu/rods-and-cones

61 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Color sensitivity

https://askabiologist.asu.edu/rods-and-cones
62 DTU Compute, Technical University of Denmark Image Analysis 2020
DTU Compute

Object colors
Subtractive colors

All other colors than red absorbed

63 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Object colors
Additive colors
 Additive colours: Final colour
is made by mixing red,
green, and blue
 RGB = Red, Green, and Blue
 Television, computers, digital
cameras use the “RGB color
space”
 Typically the values of R, G,
and B lie between 0 and 255

64 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

RGB Colours
 When alle three “Lamps” are turned of we get
black
 When all three “lamps” are on what do we get?

(0,255,0)
RGB = (0,0,0) (255,0,0) (0,0,255)

(255,255,0) (0,255,255)

(255,0,255)

RGB = (255,255,255)
65 DTU Compute, Technical University of Denmark Image Analysis 2020
DTU Compute

Color camera with one sensor

http://www.skyandtelescope.com/astronomy-resources/astrophotography-tips/redeeming-
color-planetary-cameras/

66 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

RGB color space

67 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Converting colour to grayscale

v = 0.2989 * R + 0.5870 * G + 0.1140 * B

68 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Normalised RGB colors

69 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Another RGB representation

(r,g,I)

70 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

HSI Color Reprentation

 Hue – the dominant wave length in the perceived
light (the pure color)
 Saturation – the purity of the color
 Intensity – the brightness of the color (sometimes
called the value)

71 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Converting between RGB and HSI

 You have an RGB value
 You want the corresponding HSI value

72 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Why other colorspaces

 Why should we use for example HSI ?

73 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Melanoma segmentation
 An algorithm that can
do pixelwise
classification
– Background / skin
– Melanoma

 Use the colors

74 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Melanoma segmentation – color variation

75 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Color thresholding

76 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Color thresholding

77 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Color variation
 The major variation is in the brightness
– This will spread out the values in RGB space
 The Hue is rather constant
 HSI Space
– HUE and saturation rather stable
– Only variation in intensity / value

78 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Contrast in medical images

 How do we optimise image

acquisition when we want to
look at
– Bones
– Brain structures
– Cancer

79 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Image acquisition - bone

 X-rays
– goes through soft tissue with
little loss
– are attenuated in bone
 CT scanners use X-rays
– Good for imaging bones
 A simple threshold can
often extract the bones
 Areas with only bone and
soft-tissue will have a
bimodal histogram

Attenuation - the gradual loss in intensity

80 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Image acquisition – brain structures

 Magnetic Resonance Imaging
(MRI) is often used
 Much more difficult to
explain!
– Based on very powerful
magnetic fields and radio waves
 Needs water molecules!
 Bone is black!

81 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Image acquisition - cancer

 CT scan
 Liver cancer
– Very difficult to see

82 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

What makes cancer cells special?

83 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Cancer metabolism
 Cancer cells typically have a
high metabolism
– They eat more!

 Some substances are easier

to see on different scanners
– Bone on CT

84 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Using the cancer metabolism

 Something that is to see
+
 Something that is being
eaten by the cancer

 A tracer

85 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Contrast using tracers

 A commonly used tracer is
– 18F-FDG = 18F-fluorodeoxyglucose
 Used in oncological PET
– Oncology : Cancer
– PET: positron emission tomography
 Positron-emitting radioactive
isotope fluorine-18
CT image
 Glucose is a “sugar”

PET-FDG image

86 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

PET

Wikipedia

87 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

PET Image
 Areas with high glucose
intake will be brighter
– Higher intake of
radioactive molecules
 Bimodal histograms in
areas with cancer cells
 Big research topic

Wikipedia
88 DTU Compute, Technical University of Denmark Image Analysis 2020
DTU Compute

High-Resolution PET scanner at

Rigshospitalet

PhD project by Oline Vinter Olesen –

now startup

89 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Combining Images
 CT is good for bone
 PET is good for cancer
 What if I want to see both?
 PET/CT scanner
– Patient scanned in both a CT and a PET
scanner

 Image registration
– Take two or more separate
images
– Combine them using image
registration
– More about that later

90 DTU Compute, Technical University of Denmark Image Analysis 2020

DTU Compute

Thresholds visited
 The tumour became much
more separated from the
background
 Perhaps a simple threshold is
enough now?
 The best solution
– Clever imaging techniques
and
– Intelligent image analysis

91 DTU Compute, Technical University of Denmark Image Analysis 2020

92 DTU Compute, Technical University of Denmark
93 DTU Compute, Technical University of Denmark
96 DTU Compute, Technical University of Denmark
DTU Compute

Next week
 Neighbourhood processing
– Filtering

98 DTU Compute, Technical University of Denmark Image Analysis 2020