Application of Wavelets to

Document Image Processing, Graph

Processing and Video Processing

Dr. S A Angadi
Professor, Dept. of Computer Science and Engineering,
PG Centre,
Visvesaraya Technological University, Belgaum
Contents
• Applications to Document Image Processing
Contd…
• Graphs and their Applications
• Graphs and Wavelets
• Video Processing
• Video Processing and Wavelets
• Summary
 3

Text Region Extraction in Low Resolution Images

of Display Boards using Wavelet Features

• The proposed work is texture based and uses wavelet

features for text region detection and extraction.

• The wavelet energy features are obtained at 2

decompostion levels on every 50x50 pixels block of the
image and potential text blocks are identified using
newly defined discriminant functions.

• Further, the detected text blocks are merged to extract

text regions
• The proposed method is robust and achieves a detection
rate of 97% on a variety of natural scene images.
 4

Text Region Extraction using Wavelet

Features

The proposed method comprises of 4

phases;
• Preprocessing for binarization and dividing image into
50x50 blocks.

• Wavelet Energy features computation on every 50x50

pixels block

• Classification of blocks into text and non text categories

• Merging of text blocks to detect text regions

 5
Text Region Extraction using Wavelet Features
Input Image

S A Angadi M M Kodabagi
Department of CSE, Basaveshwar Engg College

Block diagram of text extraction technique using wavelet features

 6

Text Region Extraction using Wavelet Features

Preprocessing and dividing image into 50x50 blocks

• Initially the input image is preprocessed for binarization.

• Further, the converted image is divided into 50x50 blocks. For each block,
DWT (Discrete Wavelet Transform) is applied to obtain wavelet energy
features at each level j.

• There are several benefits of using larger sized image blocks for extracting
energy features
• One such benefit is, the larger size image blocks cover more details and
hence extracted features give sufficient information for correct classification
of blocks into text and non-text categories

• The other benefits include; robustness and insensitiveness to variation in

size, font and alignment.
 7

Text Region Extraction using Wavelet Features

 8

Text Region Extraction using Wavelet Features

Feature Extraction

• Totally 3 features are extracted from every block at each level j and
are stored into a feature vector Yi (Subscript “i” corresponds to ith
block).

• The feature vector Yi also records block coordinates which

corresponds to minimum and maximum row and column numbers of
the block.

• Feature vectors of all N blocks are combined to form a feature matrix

D.

• D = [ Y1, Y2, Y3…………………… YN]

• Yi = [rmin, rmax, cmin, cmax, fj1, fj2, fj3 j=1,2];
 9

Text Region Extraction using Wavelet Features

Where;
rmin, rmax, cmin, cmax- corresponds to coordinates of ith block in
terms of minimum and maximum row and column numbers.

• And fj1, fj2 , and fj3 are described as in equations below;

• fj1 = (∑ abs (Hj (x, y))) / (M * N)

• fj2 = (∑ abs (Vj (x, y))) / (M * N)
• fj3 = (∑abs ( Dj (x, y))) / (M * N)
Where,
H,V and D corresponds to detail coefficients of
wavelet transform.
M X N represents size of each detail band
fj1, fj2 , fj3 are wavelet energy features of detail bands
at each level j
 10

Text Region Extraction using Wavelet Features

Classification
•The classification phase of the proposed model classifies every block into
two classes’ w1 and w2.

•Where, w1 corresponds to text blocks and w2 corresponds to non-text

blocks category. Newly defined descriminant functions are used in this
phase for classifying the blocks into text and non-text.

•The descriminant functions uses thresholds for corresponding wavelet

energy features fj1, fj2 , and fj3 .

•The discriminant functions are as follows;

 11

Text Region Extraction using Wavelet Features

The classified text blocks are stored into vector B.

 12

Text Region Extraction using Wavelet Features

• B = [Ci, i = 1,N1]

Ci = [rmin, rmax, cmin, cmax ]

Where
• Ci corresponds to coordinates of ih text block.
• rmin, rmax, cmin, cmax corresponds to the
coordinates of ith block in terms of minimum and
maximum row and column numbers

• N1 is Number of text blocks.

• After this phase, the classified text blocks are

subjected to merging process
 13

Text Region Extraction using Wavelet Features

Merging of text blocks to detect text regions

• The merging process combines the potential text blocks Ci
connected in rows and columns, to obtain new text
regions ri, whose coordinates are recorded into vector R.

• R = [ri , i = 1,W]

ri = [rmin, rmax, cmin, cmax ]

• ri corresponds to ith text region

• rmin, rmax, cmin, cmax corresponds to the coordinates of ith

text region.

• W is number of text regions.

 14

Text Region Extraction using Wavelet Features

Results and Analysis

• The proposed methodology has produced good results
for low resolution images containing text of different
size, font, and alignment with varying background
• The approach also detects nonlinear text regions

• The methodology has achieved an overall detection

rate of 97%, and a false reject rate of 3% while
testing on 3240 low resolution display board images
 15

Text Region Extraction using Wavelet Features

• The reason for rejection of some of the text blocks

is due to low values of wavelet energy features of
blocks containing minute part of the text, which is
too weak for acceptance to classify blocks as text
blocks.

• The system is developed in MATLAB and evaluated

with 3240 low resolution images on Intel Core i7
3632-QM Processor @ 2.20GHZ. And the method
reports an average processing time 1.2 seconds.
 16

Text Region Extraction using Wavelet Features

• The performance of the methods has been tested

for localizing English text, Kannada text in
images

• As the proposed texture features do not capture

language dependent information, the methods
can also be extended for text localization from
the images of other languages with little or no
modifications
 17

Text Region Extraction using Wavelet Features

Sample output
 18

Text Region Extraction using Wavelet Features

 19

Text Region Extraction using Wavelet Features

LVQ- Neural Network Based Signature Recognition

System Using Wavelet Features

Signature Test Signature

Image Knowledge Image
Acquisition base Acquisition

Preprocessing
Preprocessing

LVQ-NN
(recognizer)
Feature Feature
Extraction Extraction

Signature
Recognition
 LVQ- Neural Network Based Signature Recognition

System Using Wavelet Features

Image Acquisition:
Preprocessing:

i) Gray Scale Conversion

ii) Binarization
iii) Fitting Boundary Box
LVQ- NN Based Signature Recognition System Using Wavelet Features

Feature Extraction
-Features: 1. maximum horizontal projection position
2. maximum vertical projection position
-Technique: Descrete Wavelet Transform
Ac Hc Vc Dc

-Steps to extract features using wavelet:

i) The whole image is divided into 8 blocks

ii) DWT with the filter called ‘db4’ is applied to each of the 8 blocks
LVQ- NN Based Signature Recognition System Using Wavelet Features
Feature extraction contd…
iii) original image is diagonally divided to get 4 blocks

iv) To each of the blocks, first level DWT is applied

v) For each of the detailed coefficient matrices, features such as

maximum horizontal projection positions and maximum vertical
projection positions are extracted
 LVQ- NN Based Signature Recognition System Using Wavelet Features
Block Vertical
MHP1 position, MVP2 position Block Vertical MHP25 position, MVP26 position
1 Horizontal
MHP3 position, MVP4 position 5 Horizontal MHP27 position, MVP28 position
DiagonalMHP5 position, MVP6 position Diagonal MHP29 position, MVP30 position
Block Vertical
MHP7 position, MVP8 position
2 HorizontalMHP9 position, MVP10 Block Vertical MHP31 position, MVP32 position
Diagonal position 6 Horizontal MHP33 position, MVP34 position
MHP11position,MVP12 Diagonal MHP35 position, MVP36 position
position
Block Vertical MHP13position,MVP14 Block Vertical MHP37 position, MVP38 position
3 Horizontal position 7 Horizontal MHP39 position, MVP40 position
Diagonal MHP15position,MVP16 Diagonal MHP41 position, MVP42 position
position
MHP17position,MVP18 Block Vertical MHP43 position, MVP44 position
position 8 Horizontal MHP45 position, MVP46 position
Block Vertical MHP19position,MVP20 Diagonal MHP47 position, MVP48 position
4 Horizontal position
Diagonal MHP21position,MVP22
Block positionVertical MHP49 position, MVP50 position
List of MHP23position,MVP24
1 Horizontal MHP51 position, MVP52 position
position
Features Block Vertical MHP53 position, MVP54 position
Extracted 2 Horizontal MHP55 position, MVP56 position

Block Vertical MHP57 position, MVP58 position

3 Horizontal MHP59 position, MVP60 position

Block Vertical MHP61 position, MVP62 position

4 Horizontal MHP63 position, MVP64 position
LVQ- NN Based Signature Recognition System Using Wavelet Features

Experimentation
-Experiment 1: 8 blocks and DWT
Accuracy was too low (45% to 80%)

Persons Accuracy(%)
15 84
20 70
25 66
30 55
35 51
40 41
45 47
50 45
LVQ- NN Based Signature Recognition System Using Wavelet Features

- Experiment 2: 8+4 blocks and DWT

Accuracy was improved (60% to 87%)

Persons Accuracy(%)
15 88
20 78
25 75
30 70
35 67
40 65
45 62
50 60
LVQ- NN Based Signature Recognition System Using Wavelet Features

-Experiment 3: Samples of signature of each member is increased (8 to12).

Accuracy is improved as expected from
(74% to 94%)

Persons Accuracy(%)
15 94
20 91
25 88
30 85
35 80
40 77
45 76
50 74
Euclidean Distance Based Offline Signature Recognition System Using Global
and Local Wavelet Features

Training Image Test Image

Acquisition Acquisition

Preprocessing Preprocessing

Feature Feature
Extraction Extraction

Knowledge Base Euclidean

Distance

Signature
Recognised
Training Testing
Phase Phase
 Euclidean Distance Based Offline Signature Recognition System Using Global
and Local Wavelet Features

Image Acquisition:
Preprocessing:
i) Gray Scale Conversion
ii) Binarization
iii) Thinning
iii) Fitting Boundary Box:
Euclidean Distance Based Offline Signature Recognition System Using Global
and Local Wavelet Features

Feature Extraction
-Global Features:
Height
Width
Area
Centroid-x, Centroid-y
maximum horizontal projection position
maximum vertical projection position
-Local Features:
Energy features
E=∑ I(i,j) / MXN
-Technique: Discrete Wavelet Transform
Euclidean Distance Based Offline Signature Recognition System Using Global and
Local Wavelet Features
First: The whole image is divided into 16 blocks

Second: First level DWT with the filter called ‘db4’ is applied to each of
the 16 blocks. DWT decomposes each block into four sub-images

Third: For each of the detailed coefficient matrices, energy features are
extracted using the equation
E=∑ I(i,j) / MXN
 List of Features Extracted
Sl. Feature Sl. Feature Sl. Feature Sl. Feature
N0 N0 N0 N0
1. Height 16. VEF4 31. VEF9 46. VEF14
2 Width 17. DEF4 32. DEF9 47. DEF14
3. Area 18. HEF5 33. HEF10 48. HEF15
4. Centroid-X 19. VEF5 34. VEF10 49. VEF15
5. Centroid-Y 20. DEF5 35. DEF10 50. DEF15
6. HEF1 21. HEF6 36. HEF11 51. HEF16
7. VEF1 22. VEF6 37. VEF11 52. VEF16
8. DEF1 23. DEF6 38. DEF11 53. DEF16
9. HEF2 24. HEF7 39. HEF12 54. HPH
10. VEF2 25. VEF7 40. VEF12 55. VPH
11. DEF2 26. DEF7 41. DEF12 56. HPV
12. HEF3 27. HEF8 42. HEF13 57. VPV
13. VEF3 28. VEF8 43. VEF13 58. HPD
14. DEF3 29. DEF8 44. DEF13 59. VPD
15. HEF4 30. HEF9 45. HEF14
 Euclidean Distance Based Offline Signature Recognition System Using Global and
Local Wavelet Features

Classification/Testing

-The Euclidean distance between points p and q is the length of the line

segment connecting them. In Cartesian coordinates, if p = (p1, p2,..., pn)
and q = (q1, q2,..., qn) are two points in Euclidean n-space, then the distance
from p to q, or from q to p is given by equation

n
d(p,q) d(q, p)  (q1  p1) (q2  p2) ...(qn  pn) 
2 2 2
i i
(q
i1
 p )2

In testing, all the features of test image are extracted. Using the
Euclidean distance we generate matching scores by matching a
test signature features with all the signatures of database. A
signature is matched if this distance lies in a range of subjective
threshold.
Euclidean Distance Based Offline Signature Recognition System Using Global and
Local Wavelet Features

Experiments

Experiment 1 Experiment 2
Graph Representations
• V - number of vertices (nodes), E- number of
edges
• Edge array - size: E
• Adjacency matrix - size: V * V
• Adjacency lists - size: V + E
• Real representations are often more complicated
Graph Representation Properties
• V - number of vertices E - number of edges
matrix list edge array

space V2 V+E E
find edge 1 V E

insert edge 1 1 1
path v to w? V2 V+E E lgV
Graph Processing Problems
• Traversal, shortest paths, longest paths
• Simple and strong connectivity
• Sub graphs, cliques
• Planarity (no edge crossing)
• Isomorphism
• Graph drawing
• Compute some static value of the graph
degree distribution, clustering coefficient
• Identify some subset of edges or vertices
-spanning tree, connected components
• Graph algorithms are difficult to test for correctness
• All of these problems are characterized by high/ large data content
• Hence there is scope for abstracting this information and wavelets are
handy tools for such applications
Graph Processing and Wavelets
• Many interesting scientific and real world problems
involve manipulating/ processing structured data
• Examples include data defined on network-like
structures, data defined on manifolds or irregularly
shaped domains, and data consisting of “point clouds”,
such as collections of feature vectors with associated
labels
• Graphs can be used to represent data of many real
world entities/ processes
• The properties of the graphs can be brought out in
matrices such as adjacency matrix etc.
Graph Processing and Wavelets
• The adjacency or other matrix representations of the weighted graph can
be subjected to wavelet transforms, which helps in abstracting the
information into smaller dimensioned data[David K Hammand et al,
“Wavelets on graphs via spectral graph theory”, Applied and
Computational Harmonic Analysis, vol. 30, no. 2, pp. 129-150, 2011]

• For example, computer networks, transportation (road, rail, airplane)

networks or social networks can all be described by weighted graphs, with
the vertices corresponding to individual computers, cities or people
respectively.

• The graph wavelet transform could be useful for analyzing data defined on
these vertices, where the data is expected to be influenced by the
underlying topology of the network
Graph Processing and Wavelets

• Weighted graphs can also be used to describe the

similarity relationships between “point clouds” of
vectors

• Many approaches for machine learning or pattern

recognition problems involve associating each
data instance with a collection of feature vectors
that hopefully encapsulate sufficient information
about the data point to solve the problem at hand
Graph Processing and Wavelets
• A flexible construction for defining wavelet transforms for data
defined on the vertices of a weighted graph [David K Hammand et
al, “Wavelets on graphs via spectral graph theory”, Applied and
Computational Harmonic Analysis, vol. 30, no. 2, pp. 129-150, 2011]

• For many problems, the local distance relationships between data

points are crucial for successful learning or classification

• These relationships can be encoded in a weighted graph by

considering the data points as vertices and setting the edge weights
equal to a distance metric A m,n = d(vm, vn) for some function d : RN ×
RN → R
Graph Processing and Wavelets
• The spectral graph wavelets in this setting could
find a number of uses for analysis of data
defined on such point clouds
• They may be useful for regularization of noisy or
corrupted data on a point cloud, or could serve
as a building blocks for building a hypothesis
function for learning problems
• This can be used for various graph problems
such as isomorphism detection
• The spectral graph wavelets in this setting could
find a number of uses for analysis of data
defined on such point clouds
• They may be useful for regularization of noisy or
corrupted data on a point cloud, or could serve
as a building blocks for building a hypothesis
function for learning problems
Graph Processing and Wavelets

• This problem is approached by working in the

spectral graph domain, i.e. using the basis
consisting of the eigenfunctions of the graph
Laplacian L
• Given a simple graph G with n vertices, its
Laplacian matrix   is defined as:
Laplacian Matrix of a graph

Labeled graph Degree matrix Adjacency matrix Laplacian matrix

Graph Processing and Wavelets
• This tool from spectral graph theory, provides an analogue of the Fourier transform
for functions on weighted graphs

• In this construction, the wavelet operator at unit scale is given as an operator valued
function Tg = g(L) for a generating kernel g

• Scaling is then defined in the spectral domain, i.e. the operator Tt g at scale t is given
by g(tL)

• Applying this operator to an input signal f gives the wavelet coefficients of f at scale t

• These coefficients are equivalent to inner products of the signal f with the individual
graph wavelets

• These wavelets can be calculated by applying this operator to a delta impulse at a

single vertex, i.e. ψt,m = Tt gδm
Graph Processing and Wavelets
• Haar Wavelet like basis are also available for processing
of graphs
• Wavelets and dendrograms
▫ Dendrograms:
A dendrogram (from Greek dendron "tree"
and gramma "drawing") is a tree diagram frequently used
to illustrate the arrangement of the clusters produced by 
hierarchical clustering. Dendrograms are often used
in computational biology to illustrate the clustering
of genes or samples
▫ Wavelets for processing dendrograms are proposed ( Haar)
Graph Processing and Wavelets
• Wavelets for learning applications using graphs
▫ A machine learning framework for constructing
wavelets on weighted graphs that can sparsely
represent a given class of signals
▫ The lifting scheme as applied to the Haar
wavelets, and is based on the observation that the
update and predict steps of the lifting scheme are
similar to the encode and decode steps of an auto-
encoder
▫
Video Processing
• Video is a rich information source
▫ frames (individual images)
▫ links between frames (cuts, fades, dissolves)
▫ changes in color, shapes, motion of both camera
and objects
▫ acquisition (shot angles, camera motion)
▫ each type of video has its own characteristics
(commercials, news, sports)
Video Structure
• Frame: typically 1/25 or 1/30 seconds
• Shot: sequence of similar frames
▫ elementary video units
▫ a single event
• Clip / Scene: sequence of shots consecutive in
time, space, action
• Episode: consecutive scenes
▫ intro, news, reporter, weather
Overview of Video Processing
Video
Manipulation Video
Display
Video
Video Compression Video
Acquisition Database
Video
Computer Transmission
Graphics
Video Computer
Analysis Vision
Wavelets and Video Processing
• The wavelets are very good in compression of
data
• Sub band coding principle of wavelets have
found use in video compression
• The size of the videos and their storage and
transmission requirements are placing a heavy
demand on computer based processing hence
there is a need for efficient compression
Wavelets and Video Compression
Because of their inherent multi-resolution
nature, wavelet-coding schemes are especially
suitable for applications where scalability and
tolerable degradation are important
Recently the JPEG committee has released its
new image coding standard, JPEG-2000, which
has been based upon DWT.
Wavelets and Video Compression
• Wavelet Video Coding - Classification
• Intraframe coding (e.g. MJPEG)
• 3D wavelet coding without MC
• Hybrid video coding using wavelet-based
texture coding
• In-Band Motion Compensation Prediction
• Motion Compensated Temporal Filtering
• In-Band Motion Compensated Temporal
Filtering
Wavelets and Video Compression
Wavelets and Video Compression
• Motion compensation is key
▫ To achieve good compression performance
▫ To guarantee visual quality – non MC/interframe
coding with same SNR usually looks worse
• Motion-compensated Wavelet video coding
▫ Temporal MC prediction followed by Wavelet
Transform
▫ Wavelet Transform followed by temporal MC
prediction in wavelet domain
▫ 3D Wavelet with MC
Wavelets and Video De-noising
• With the maturity of digital video capturing
devices numerous applications have been
emerging
• These include teleconferencing, remote
surveillance, multimedia services and digital
television etc.
• However, the video signal is almost always
corrupted by noise from capturing devices or
during transmission due to random thermal or
other electronic noises.
Wavelets and Denoising
• It is now well-known that wavelet transforms with
some regularity have strong decorrelation and can
well represent an image with just a few large
coeffcients
• Therefore, it is far more reasonable to assume that
the wavelet coeffcients are independent than the
original spatial domain pixels
• This explains why impressive denoising results have
• been achieved by simply thresholding or shrinking
each wavelet coeffcient independently
Wavelets and Video Denoising
• Spatial Domain Video De-noising
▫ Temporal Only
▫ Spatio Temporal
▫ 2D De-noising for each video frame
• Wavelet based Video De-noising
• sequential 2-D spatial and 1-D temporal denoising, in
which they first do sophisticated wavelet-based image
denoising for each frame and then recursive temporal
averaging
• However, 2-D spatial filtering tends to introduce
artefacts and to remove weak details with noise
Wavelets and Video Denoising

• 2-D spatial wavelet, and recursive filter to

preserve low latency is proposed by researchers
• In such a way we minimize spatial blurring to
preserve the weak spatial details in still Areas

• The key to processing directly in the wavelet

domain is an efficient shift-invariant transform,
allowing spatial motion to be meaningfully
reflected in the wavelet coefficients
Summary
• Wavelet especially DWT is a versatile tool for
image processing/ video processing/ graph
processing applications
• Wavelets can be implemented as filters
• Wavelets can be implemented using the lifting
scheme also
Thank You