Graph Spectral Image Processing Gene Cheung pdf

download

https://ebookmass.com/product/graph-spectral-image-processing-
gene-cheung/

Explore and download more ebooks at ebookmass.com

 We believe these products will be a great fit for you. Click
the link to download now, or visit ebookmass.com
to discover even more!

Digital Image Processing Using MATLAB Rafael C. Gonzalez

https://ebookmass.com/product/digital-image-processing-using-matlab-
rafael-c-gonzalez/

Feature extraction and image processing for computer

vision Fourth Edition Aguado

https://ebookmass.com/product/feature-extraction-and-image-processing-
for-computer-vision-fourth-edition-aguado/

Methods and techniques for fire detection : signal, image

and video processing perspectives 1st Edition Çetin

https://ebookmass.com/product/methods-and-techniques-for-fire-
detection-signal-image-and-video-processing-perspectives-1st-edition-
cetin/

Digital Image Processing 3rd Edition Rafael C. Gonzalez

And Richard E. Woods

https://ebookmass.com/product/digital-image-processing-3rd-edition-
rafael-c-gonzalez-and-richard-e-woods/
Design for Embedded Image Processing on FPGAs, 2nd Edition
Donald G. Bailey

https://ebookmass.com/product/design-for-embedded-image-processing-on-
fpgas-2nd-edition-donald-g-bailey/

Machine Learning Algorithms for Signal and Image

Processing Suman Lata Tripathi

https://ebookmass.com/product/machine-learning-algorithms-for-signal-
and-image-processing-suman-lata-tripathi/

Image Processing for Automated Diagnosis of Cardiac

Diseases Rajeev Kumar Chauhan (Editor)

https://ebookmass.com/product/image-processing-for-automated-
diagnosis-of-cardiac-diseases-rajeev-kumar-chauhan-editor/

Machine Intelligence, Big Data Analytics, and IoT in Image

Processing Ashok Kumar

https://ebookmass.com/product/machine-intelligence-big-data-analytics-
and-iot-in-image-processing-ashok-kumar/

Machine vision inspection systems. Vol.1: image

processing, concepts Malarvel M (Ed.)

https://ebookmass.com/product/machine-vision-inspection-systems-
vol-1-image-processing-concepts-malarvel-m-ed/
Graph Spectral Image Processing
 SCIENCES
Image, Field Director – Laure Blanc-Feraud
Compression, Coding and Protection of Images and Videos,
Subject Head – Christine Guillemot

Graph Spectral Image

Processing

Coordinated by
Gene Cheung
Enrico Magli
First published 2021 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.

Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers,
or in the case of reprographic reproduction in accordance with the terms and licenses issued by the
CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the
undermentioned address:

ISTE Ltd John Wiley & Sons, Inc.

27-37 St George’s Road 111 River Street
London SW19 4EU Hoboken, NJ 07030
UK USA

www.iste.co.uk www.wiley.com

© ISTE Ltd 2021

The rights of Gene Cheung and Enrico Magli to be identified as the author of this work have been
asserted by them in accordance with the Copyright, Designs and Patents Act 1988.

Library of Congress Control Number: 2021932054

British Library Cataloguing-in-Publication Data

A CIP record for this book is available from the British Library
ISBN 978-1-78945-028-6

ERC code:
PE7 Systems and Communication Engineering
PE7_7 Signal processing
 Contents

Introduction to Graph Spectral Image Processing . . . . . . . . . . . . xi

Gene C HEUNG and Enrico M AGLI

Part 1. Fundamentals of Graph Signal Processing . . . . . . . . . . . 1

Chapter 1. Graph Spectral Filtering . . . . . . . . . . . . . . . . . . . . . 3

Yuichi TANAKA
1.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2. Review: filtering of time-domain signals . . . . . . . . . . . . . . . . . . 4
1.3. Filtering of graph signals . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.1. Vertex domain filtering . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.2. Spectral domain filtering . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.3. Relationship between graph spectral filtering and classical
filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4. Edge-preserving smoothing of images as graph spectral filters . . . . . 11
1.4.1. Early works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4.2. Edge-preserving smoothing . . . . . . . . . . . . . . . . . . . . . . 12
1.5. Multiple graph filters: graph filter banks . . . . . . . . . . . . . . . . . . 15
1.5.1. Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.5.2. Perfect reconstruction condition . . . . . . . . . . . . . . . . . . . 17
1.6. Fast computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.6.1. Subdivision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.6.2. Downsampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.6.3. Precomputing GFT . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.6.4. Partial eigendecomposition . . . . . . . . . . . . . . . . . . . . . . 22
1.6.5. Polynomial approximation . . . . . . . . . . . . . . . . . . . . . . 23
1.6.6. Krylov subspace method . . . . . . . . . . . . . . . . . . . . . . . . 26
vi Graph Spectral Image Processing

1.7. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.8. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

Chapter 2. Graph Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Xiaowen D ONG, Dorina T HANOU, Michael R ABBAT and Pascal F ROSSARD
2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.2. Literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.2.1. Statistical models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.2.2. Physically motivated models . . . . . . . . . . . . . . . . . . . . . 35
2.3. Graph learning: a signal representation perspective . . . . . . . . . . . . 36
2.3.1. Models based on signal smoothness . . . . . . . . . . . . . . . . . 38
2.3.2. Models based on spectral filtering of graph signals . . . . . . . . . 43
2.3.3. Models based on causal dependencies on graphs . . . . . . . . . . 48
2.3.4. Connections with the broader literature . . . . . . . . . . . . . . . 50
2.4. Applications of graph learning in image processing . . . . . . . . . . . 52
2.5. Concluding remarks and future directions . . . . . . . . . . . . . . . . . 55
2.6. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Chapter 3. Graph Neural Networks . . . . . . . . . . . . . . . . . . . . . . 63

Giulia F RACASTORO and Diego VALSESIA
3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.2. Spectral graph-convolutional layers . . . . . . . . . . . . . . . . . . . . 64
3.3. Spatial graph-convolutional layers . . . . . . . . . . . . . . . . . . . . . 66
3.4. Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.5. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Part 2. Imaging Applications of Graph Signal Processing . . . . . . . 73

Chapter 4. Graph Spectral Image and Video Compression . . . . . . 75

Hilmi E. E GILMEZ, Yung-Hsuan C HAO and Antonio O RTEGA
4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.1.1. Basics of image and video compression . . . . . . . . . . . . . . . 77
4.1.2. Literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.1.3. Outline of the chapter . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.2. Graph-based models for image and video signals . . . . . . . . . . . . . 79
4.2.1. Graph-based models for residuals of predicted signals . . . . . . . 81
4.2.2. DCT/DSTs as GFTs and their relation to 1D models . . . . . . . . 87
4.2.3. Interpretation of graph weights for predictive transform coding . . 88
4.3. Graph spectral methods for compression . . . . . . . . . . . . . . . . . . 89
4.3.1. GL-GFT design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.3.2. EA-GFT design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.3.3. Empirical evaluation of GL-GFT and EA-GFT . . . . . . . . . . . 97
 Contents vii

4.4. Conclusion and potential future work . . . . . . . . . . . . . . . . . . . 100

4.5. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Chapter 5. Graph Spectral 3D Image Compression . . . . . . . . . . . 105

Thomas M AUGEY, Mira R IZKALLAH, Navid M AHMOUDIAN B IDGOLI, Aline
ROUMY and Christine G UILLEMOT
5.1. Introduction to 3D images . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.1.1. 3D image definition . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5.1.2. Point clouds and meshes . . . . . . . . . . . . . . . . . . . . . . . . 106
5.1.3. Omnidirectional images . . . . . . . . . . . . . . . . . . . . . . . . 107
5.1.4. Light field images . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.1.5. Stereo/multi-view images . . . . . . . . . . . . . . . . . . . . . . . 110
5.2. Graph-based 3D image coding: overview . . . . . . . . . . . . . . . . . 110
5.3. Graph construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.3.1. Geometry-based approaches . . . . . . . . . . . . . . . . . . . . . 117
5.3.2. Joint geometry and color-based approaches . . . . . . . . . . . . . 121
5.3.3. Separable transforms . . . . . . . . . . . . . . . . . . . . . . . . . . 125
5.4. Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5.5. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128

Chapter 6. Graph Spectral Image Restoration . . . . . . . . . . . . . . 133

Jiahao PANG and Jin Z ENG
6.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
6.1.1. A simple image degradation model . . . . . . . . . . . . . . . . . . 133
6.1.2. Restoration with signal priors . . . . . . . . . . . . . . . . . . . . . 135
6.1.3. Restoration via filtering . . . . . . . . . . . . . . . . . . . . . . . . 137
6.1.4. GSP for image restoration . . . . . . . . . . . . . . . . . . . . . . . 140
6.2. Discrete-domain methods . . . . . . . . . . . . . . . . . . . . . . . . . . 141
6.2.1. Non-local graph-based transform for depth image denoising . . . 141
6.2.2. Doubly stochastic graph Laplacian . . . . . . . . . . . . . . . . . . 142
6.2.3. Reweighted graph total variation prior . . . . . . . . . . . . . . . . 145
6.2.4. Left eigenvectors of random walk graph Laplacian . . . . . . . . . 150
6.2.5. Graph-based image filtering . . . . . . . . . . . . . . . . . . . . . . 155
6.3. Continuous-domain methods . . . . . . . . . . . . . . . . . . . . . . . . 155
6.3.1. Continuous-domain analysis of graph Laplacian regularization . . 156
6.3.2. Low-dimensional manifold model for image restoration . . . . . . 163
6.3.3. LDMM as graph Laplacian regularization . . . . . . . . . . . . . . 165
6.4. Learning-based methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
6.4.1. CNN with GLR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
6.4.2. CNN with graph wavelet filter . . . . . . . . . . . . . . . . . . . . 171
6.5. Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
6.6. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
viii Graph Spectral Image Processing

Chapter 7. Graph Spectral Point Cloud Processing . . . . . . . . . . . 181

Wei H U, Siheng C HEN and Dong T IAN
7.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
7.2. Graph and graph-signals in point cloud processing . . . . . . . . . . . . 183
7.3. Graph spectral methodologies for point cloud processing . . . . . . . . 185
7.3.1. Spectral-domain graph filtering for point clouds . . . . . . . . . . 185
7.3.2. Nodal-domain graph filtering for point clouds . . . . . . . . . . . 188
7.3.3. Learning-based graph spectral methods for point clouds . . . . . . 189
7.4. Low-level point cloud processing . . . . . . . . . . . . . . . . . . . . . . 190
7.4.1. Point cloud denoising . . . . . . . . . . . . . . . . . . . . . . . . . 191
7.4.2. Point cloud resampling . . . . . . . . . . . . . . . . . . . . . . . . 193
7.4.3. Datasets and evaluation metrics . . . . . . . . . . . . . . . . . . . . 198
7.5. High-level point cloud understanding . . . . . . . . . . . . . . . . . . . 199
7.5.1. Data auto-encoding for point clouds . . . . . . . . . . . . . . . . . 199
7.5.2. Transformation auto-encoding for point clouds . . . . . . . . . . . 206
7.5.3. Applications of GraphTER in point clouds . . . . . . . . . . . . . 211
7.5.4. Datasets and evaluation metrics . . . . . . . . . . . . . . . . . . . . 211
7.6. Summary and further reading . . . . . . . . . . . . . . . . . . . . . . . . 213
7.7. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

Chapter 8. Graph Spectral Image Segmentation . . . . . . . . . . . . . 221

Michael N G
8.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
8.2. Pixel membership functions . . . . . . . . . . . . . . . . . . . . . . . . . 222
8.2.1. Two-class problems . . . . . . . . . . . . . . . . . . . . . . . . . . 222
8.2.2. Multiple-class problems . . . . . . . . . . . . . . . . . . . . . . . . 226
8.2.3. Multiple images . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
8.3. Matrix properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
8.4. Graph cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
8.4.1. The Mumford–Shah model . . . . . . . . . . . . . . . . . . . . . . 234
8.4.2. Graph cuts minimization . . . . . . . . . . . . . . . . . . . . . . . 235
8.5. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
8.6. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237

Chapter 9. Graph Spectral Image Classification . . . . . . . . . . . . . 241

Minxiang Y E, Vladimir S TANKOVIC, Lina S TANKOVIC and Gene C HEUNG
9.1. Formulation of graph-based classification problems . . . . . . . . . . . 243
9.1.1. Graph spectral classifiers with noiseless labels . . . . . . . . . . . 243
9.1.2. Graph spectral classifiers with noisy labels . . . . . . . . . . . . . 246
9.2. Toward practical graph classifier implementation . . . . . . . . . . . . . 247
9.2.1. Graph construction . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
9.2.2. Experimental setup and analysis . . . . . . . . . . . . . . . . . . . 249
 Contents ix

9.3. Feature learning via deep neural network . . . . . . . . . . . . . . . . . 255

9.3.1. Deep feature learning for graph construction . . . . . . . . . . . . 258
9.3.2. Iterative graph construction . . . . . . . . . . . . . . . . . . . . . . 260
9.3.3. Toward practical implementation of deep feature learning . . . . . 262
9.3.4. Analysis on iterative graph construction for robust classification . 267
9.3.5. Graph spectrum visualization . . . . . . . . . . . . . . . . . . . . . 269
9.3.6. Classification error rate comparison using insufficient training
data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
9.3.7. Classification error rate comparison using sufficient training
data with label noise . . . . . . . . . . . . . . . . . . . . . . . . . . 270
9.4. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
9.5. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272

Chapter 10. Graph Neural Networks for Image Processing . . . . . . 277

Giulia F RACASTORO and Diego VALSESIA
10.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
10.2. Supervised learning problems . . . . . . . . . . . . . . . . . . . . . . . 278
10.2.1. Point cloud classification . . . . . . . . . . . . . . . . . . . . . . . 278
10.2.2. Point cloud segmentation . . . . . . . . . . . . . . . . . . . . . . 281
10.2.3. Image denoising . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
10.3. Generative models for point clouds . . . . . . . . . . . . . . . . . . . . 286
10.3.1. Point cloud generation . . . . . . . . . . . . . . . . . . . . . . . . 286
10.3.2. Shape completion . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
10.4. Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
10.5. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294

List of Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
 Introduction to Graph Spectral
Image Processing

Gene1 C HEUNG1 and Enrico M AGLI2

York University, Toronto, Canada
2
Politecnico di Torino, Turin, Italy

I.1. Introduction

Image processing is a mature research topic. The first specification of Joint

Photographic Experts Group (JPEG), now the predominant image coding standard on
the Internet, was published in 1992. MPEG1, the first digital video compression
standard by ISO, was standardized in 1993. The IEEE International Conference on
Image Processing (ICIP), the flagship image processing conference held annually for
the IEEE Signal Processing Society (SPS), was also started in 1993 and has been in
existence for 27 years, making it older than many image processing researchers now
studying in graduate schools! Given the topic’s maturity, it is a legitimate question to
ask if yet another book on image processing is warranted. As co-editors of this book,
we emphatically answer this question with a resounding “Yes”. We will first discuss
the following recent technological trends, which also serve as motivations for the
creation of this publication.

1) Sensing and Display Technologies: The advent of image sensing technologies,

such as active depth sensors and display technologies like head-mounted displays
(HMD), in the last decade alone, means that the nature of a digital image has
drastically changed. Beyond higher spatial resolution and bit-depth per pixel, a
modern imaging sensor can also acquire scene depth, hyper-spectral properties, etc.
Further, often acquired image data is not represented as a traditional 2D array of pixel
information, but in an alternative form, such as light fields and 3D point clouds. This
means that the processing tools must flexibly adapt to richer and evolving imaging
contents and formats.

Graph Spectral Image Processing,

coordinated by Gene C HEUNG and Enrico M AGLI. © ISTE Ltd 2021
Visit https://ebookmass.com today to explore
a vast collection of ebooks across various
genres, available in popular formats like
PDF, EPUB, and MOBI, fully compatible with
all devices. Enjoy a seamless reading
experience and effortlessly download high-
quality materials in just a few simple steps.
Plus, don’t miss out on exciting offers that
let you access a wealth of knowledge at the
best prices!
xii Graph Spectral Image Processing

2) Graph Signal Processing: In the last eight years, we have also witnessed the
birth of a new signal processing topic – called graph signal processing (GSP) – that
generalizes traditional mathematical tools like transforms and wavelets, to process
signals residing on irregular data kernels described by graphs (Shuman et al. 2013).
Central to GSP is the notion of graph frequencies: orthogonal components, computed
from a graph variation operator like the graph Laplacian matrix, that generalize the
notion of Fourier modes to the graph domain, spanning a graph signal space. Because
of its inherent powerful generality, one can easily adopt or design GSP tools for
different imaging applications, where a node in a graph represents a pixel, and the
graph connectivity is chosen to reflect inter-pixel similarities or correlations. For an
example of the GSP tool being used for image restoration, see Figure I.1 for an
illustration of a graph spectral method called left eigenvectors of the random walk
graph Laplacian (LeRAG) for JPEG image dequantization (Liu et al. 2017). GSP tools
can also easily adapt to the aforementioned modern imaging modalities, such as light
field images and 3D point clouds, that do not reside on regular 2D grids.

Figure I.1. Visual comparison of JPEG dequantization methods for a butterfly at

QF = 5. The corresponding PSNR values are also given. For a color version of this
figure, see www.iste.co.uk/cheung/graph.zip

3) Deep Neural Networks: Without a doubt, the singular seismic paradigm shift
in data science in the last decade is deep learning. Using layers of convolutional
filters, pointwise nonlinearities and pooling functions, deep neural network (DNN)
architectures like convolutional neural networks (CNN) have demonstrated superior
performance in a wide range of imaging tasks from denoising to classification, when
a large volume of labeled data is available for training (Vemulapalli et al. 2016;
Zhang et al. 2017). When labeled training data is scarce, or when the underlying
data kernel is irregular (thus complicating the training of convolutional filters and the
selection of pooling operators), how to best design and construct DNN for a targeted
image application is a challenging problem. Moreover, a CNN purely trained from
labeled data often remains a “black box”, i.e. the learned operators like filtering remain
unexplainable.
 Introduction xiii

Motivated by these technological trends, we have focused this book on the theory
and applications of GSP tools for image processing, covering conventional images
and videos, new modalities like light fields and 3D point clouds, and hybrid
GSP/deep learning approaches. Different from other graph-based image processing
books (Lezoray and Grady 2012), we concentrate on spectral processing techniques
with frequency interpretations such as graph Fourier transforms (GFT) and graph
wavelets, drawing inspiration from the long history of frequency analysis tools in
traditional signal processing. Graph frequency analysis enables the definition of
familiar signal processing notions, such as graph Fourier modes, bandlimitedness,
and signal smoothness, using graph spectral methods that can be designed.

Specifically, the content of this book is structured into two parts:

1) The first part of the book discusses the fundamental GSP theories. Chapter 1,
titled “Graph Spectral Filtering” by Y. Tanaka, reviews the basics of graph filtering
such as graph transforms and wavelets. Chapter 2, titled “Graph Learning” by X.
Dong, D. Thanou, M. Rabbat and P. Frossard, reviews recent techniques to learn
an underlying graph structure given a set of observable data. Chapter 3, titled
“Graph Neural Networks” by G. Fracastoro and D. Valsesia, overviews recent works
generalizing DNN architectures to the graph data domain, where input signals reside
on irregular graph structures.

2) The second part of the book reviews different imaging applications of GSP.
Chapters 4 and 5, titled “Graph Specral Image and Video Compression” by H.E.
Egilmez, Y.-H. Chao and A. Ortega and “Graph Spectral 3D Image Compression”
by T. Maugey, M. Rizkallah, N. M. Bidgoli, A. Roumy and C. Guillemot, focus on the
design and applications of GSP tools for the compression of traditional images/videos
and 3D images, respectively. Chapter 6, titled “Graph Spectral Image Restoration” by
J. Pang and J. Zeng, focuses on the general recovery of corrupted images, e.g. image
denoising and deblurring. As a new imaging modality, Chapter 7, titled “Graph
Spectral Point Cloud Processing” by W. Hu, S. Chen and D. Tian, focuses on the
processing of 3D point clouds for applications, such as low-level restoration and
high-level unsupervised feature learning. Chapters 8 and 9, titled “Graph Spectral
Image Segmentation” by M. Ng and “Graph Spectral Image Classification” by M. Ye,
V. Stankovic, L. Stankovic and G. Cheung, narrow the discussion specifically to
segmentation and classification, respectively, two popular research topics in the
computer vision community. Finally, Chapter 10, titled “Graph Neural Networks for
Image Processing” by G. Fracastoro and D. Valsesia, reviews the growing efforts to
employ recent GNN architectures for conventional imaging tasks such as denoising.

Before we jump into the various chapters, we begin with the basic definitions in
GSP that will be used throughout the book. Specifically, we formally define a graph,
graph spectrum, variation operators and graph signal smoothness priors in the
following sections.
xiv Graph Spectral Image Processing

I.2. Graph definition

A graph G(V, E, W) contains a set V of N nodes and a set E of M edges. While

directed graphs are also possible, in this book we more commonly assume an
undirected graph, where each existing edge (i, j) ∈ E is undirected and contains an
edge weight wi,j ∈ R, which is typically positive. A large positive edge weight wi,j
would mean that samples at nodes i and j are expected to be similar/correlated.

There are many ways to compute appropriate edge weights. Especially common
for images, edge weight wi,j can be computed using a Gaussian kernel, as done in the
bilateral filter (Tomasi and Manduchi 1998):
   
li − lj 22 xi − xj 22
wi,j = exp − exp − [I.1]
σl2 σx2

where li ∈ R2 is the location of pixel i on the 2D image grid, xi ∈ R is the intensity

of pixel i, and σl2 and σx2 are two parameters. Hence, 0 ≤ wi,j ≤ 1. Larger geometric
and/or photometric distances between pixels i and j would mean a smaller weight
wi,j . Edge weights can alternatively be defined based on local pixel patches, features,
etc. (Milanfar 2013b). To a large extent, the appropriate definition of edge weight is
application dependent, as will be discussed in various forthcoming chapters.

A graph signal x on G is a discrete signal of dimension N – one sample xi ∈ R

for each node1 i in V. Assuming that nodes are appropriately labeled from 1 to N , we
can simply treat a graph signal as a vector x ∈ RN .

I.3. Graph spectrum

Denote by W ∈ RN ×N an adjacency matrix, where the (i, j)th entry is

Wi,j = wi,j . Next, denoteby D ∈ RN ×N a diagonal degree matrix, where the
(i, i)th entry is Di,i = j Wi,j . A combinatorial graph Laplacian matrix L is
L = D − W (Shuman et al. 2013). Because L is real and symmetric, one can show,
via the spectral theorem, that it can be eigen-decomposed into:

L = UΛU [I.2]

where Λ is a diagonal matrix containing real eigenvalues λk along the diagonal, and
U is an eigen-matrix composed of orthogonal eigenvectors ui as columns. If all edge

1 If a graph node represents a pixel in an image, each pixel would typically have three color
components: red, green and blue. For simplicity, one can treat each color component separately
as a different graph signal.
 Introduction xv

weights wi,j are restricted to be positive, then graph Laplacian L can be proven to be
positive semi-definite (PSD) (Chung 1997)2, meaning that λk ≥ 0, ∀k and
x Lx ≥ 0, ∀x. Non-negative eigenvalues λk can be interpreted as graph
frequencies, and eigenvectors U can be interpreted as corresponding graph Fourier
modes. Together they define the graph spectrum for graph G.

The set of eigenvectors U for L collectively form the GFT (Shuman et al. 2013),
which can be used to decompose a graph signal x into its frequency components via
α = U x. In fact, one can interpret GFT as a generalization of known discrete
transforms like the Discrete Cosine Transform (DCT) (see Shuman et al. 2013 for
details).
Note that if the multiplicity mk of an eigenvalue λk is larger than 1, then the
set of eigenvectors that span the corresponding eigen-subspace of dimension mk is
non-unique. In this case, it is necessary to specify the graph spectrum as the collection
of eigenvectors U themselves.
If we also consider negative edge weights wi,j that reflect inter-pixel
dissimilarity/anti-correlation, then graph Laplacian L can be indefinite. We will
discuss a few recent works (Su et al. 2017; Cheung et al. 2018) that employ negative
edges in later chapters.

I.4. Graph variation operators

Closely related to the combinatorial graph, Laplacian L, are other variants of
Laplacian operators, each with their own unique spectral properties. A normalized
graph Laplacian Ln = D−1/2 LD−1/2 is a symmetric normalized variant of L. In
contrast, a random walk graph Laplacian Lr = D−1 L is an asymmetric normalized
variant of L. A generalized graph Laplacian Lg = L + diag(D) is a graph Laplacian
with self-loops di,i at nodes i – called the loopy graph Laplacian in Dörfler and
Bullo (2013) – resulting in a general symmetric matrix with non-positive
off-diagonal entries for a positive graph (Biyikoglu et al. 2005).
Eigen-decomposition can also be performed on these operators to acquire a set of
graph frequencies and graph Fourier modes. For example, normalized variants Ln
and Lr (which are similarity transforms of each other) share the same eigenvalues
between 0 and 2. While L and Ln are both symmetric, Ln does not have the constant
vector as an eigenvector. Asymmetric Lr can be symmetrized via left and right
diagonal matrix multiplications (Milanfar 2013a). Different variation operators will
be used throughout the book for different applications.

2 One can prove that a graph G with positive edge weights has PSD graph Laplacian L via the
Gershgorin circle theorem: each Gershgorin disc corresponding to a row in L is located in the
non-negative half-space, and since all eigenvalues reside inside the union of all discs, they are
non-negative.
xvi Graph Spectral Image Processing

I.5. Graph signal smoothness priors

Traditionally, for graph G with positive edge weights, signal x is considered

smooth if each sample xi on node i is similar to samples xj on neighboring nodes j
with large wi,j . In the graph frequency domain, it means that x mostly contains low
graph frequency components, i.e. coefficients α = U x are zeros (or mostly zeros)
for high frequencies. The smoothest signal is the constant vector – the first
eigenvector u1 for L, corresponding to the smallest eigenvalue λ1 = 0.

Mathematically, we can declare that a signal x is smooth if its graph Laplacian

regularizer (GLR) x Lx is small (Pang and Cheung 2017). GLR can be expressed
as:
 
x Lx =
2
(i,j)∈E wi,j (xi − xj ) = k λk αk2 [I.3]

Because L is PSD, x Lx is lower bounded by 0 and achieved when x = cu1 for

some scalar constant c. One can also define GLR using the normalized graph
Laplacian Ln instead of L, resulting in x Ln x. The caveats is that the constant
vector u1 – typically the most common signal in imaging – is no longer the first
eigenvector, and thus u
1 Ln u1 = 0.

In Chen et al. (2015), the adjacency matrix W is interpreted as a shift operator,

and thus, graph signal smoothness is instead defined as the difference between a signal
x and its shifted version Wx. Specifically, graph total variation (GTV) based on
lp -norm is:
 p
 1 

TVW (x) = x − Wx [I.4]
|λmax | 
p

where λmax is the eigenvalue of W with the largest magnitude (also called the
spectral radius), and p is a chosen integer. As a variant to equation [I.4], a quadratic
smoothness prior is defined in Romano et al. (2017), using a row-stochastic version
Wn = D−1 W of the adjacency matrix W:

1
S2 (x) = x − Wn x22 [I.5]
2

To avoid confusion, we will call equation [I.5] the graph shift variation (GSV)
prior. GSV is easier to use in practice than GTV, since the computation of λmax is
required for GTV. Note that GSV, as defined in equation [I.5], can also be used for
signals on directed graphs.
 Introduction xvii

I.6. References

Biyikoglu, T., Leydold, J., Stadler, P.F. (2005). Nodal domain theorems and bipartite subgraphs.
Electronic Journal of Linear Algebra, 13, 344–351.
Chen, S., Sandryhaila, A., Moura, J., Kovacevic, J. (2015). Signal recovery on graphs: Variation
minimization. IEEE Transactions on Signal Processing, 63(17), 4609–4624.
Cheung, G., Su, W.-T., Mao, Y., Lin, C.-W. (2018). Robust semisupervised graph classifier
learning with negative edge weights. IEEE Transactions on Signal and Information
Processing over Networks, 4(4), 712–726.
Chung, F. (1997). Spectral graph theory. CBMS Regional Conference Series in Mathematics,
92.
Dörfler, F. and Bullo, F. (2013). Kron reduction of graphs with applications to electrical
networks. IEEE Transactions on Circuits and Systems I: Regular Papers, 60(1), 150–163.
Lezoray, O. and Grady, L. (2012). Image Processing and Analysis with Graphs: Theory and
Practice, CRC Press, Boca Raton, Florida.
Liu, X., Cheung, G., Wu, X., Zhao, D. (2017). Random walk graph Laplacian based smoothness
prior for soft decoding of JPEG images. IEEE Transactions on Image Processing, 26(2),
509–524.
Milanfar, P. (2013a). Symmetrizing smoothing filters. SIAM Journal on Imaging Sciences,
6(1), 263–284.
Milanfar, P. (2013b). A tour of modern image filtering. IEEE Signal Processing Magazine,
30(1), 106–128.
Pang, J. and Cheung, G. (2017). Graph Laplacian regularization for image denoising: Analysis
in the continuous domain. IEEE Transactions on Image Processing, 26(4), 1770–1785.
Romano, Y., Elad, M., Milanfar, P. (2017). The little engine that could: Regularization by
denoising (RED). SIAM Journal on Imaging Sciences, 10(4), 1804–1844.
Shuman, D.I., Narang, S.K., Frossard, P., Ortega, A., Vandergheynst, P. (2013), The emerging
field of signal processing on graphs: Extending high-dimensional data analysis to networks
and other irregular domains. IEEE Signal Processing Magazine, 30(3), 83–98.
Su, W.-T., Cheung, G., Lin, C.-W. (2017). Graph Fourier transform with negative edges for
depth image coding. IEEE International Conference on Image Processing, Beijing.
Tomasi, C. and Manduchi, R. (1998), Bilateral filtering for gray and color images. IEEE
International Conference on Computer Vision, 839–846.
Vemulapalli, R., Tuzel, O., Liu, M.-Y. (2016). Deep Gaussian conditional random field
network: A model-based deep network for discriminative denoising. Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, 4801–4809.
Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L. (2017). Beyond a Gaussian denoiser:
Residual learning of deep CNN for image denoising. IEEE Transactions on Image
Processing, 26(7), 3142–3155.
 PART 1

Fundamentals of Graph
Signal Processing

Graph Spectral Image Processing,

First Edition. Gene Cheung and Enrico Magli.
© ISTE Ltd 2021. Published by ISTE Ltd and John Wiley & Sons, Inc.
 1

Graph Spectral Filtering

Yuichi TANAKA
Tokyo University of Agriculture and Technology, Japan

1.1. Introduction
The filtering of time- and spatial-domain signals is one of the fundamental
techniques for image processing and has been studied extensively to date. GSP can
treat signals with irregular structures that are mathematically represented as graphs.
Theories and methodologies for the filtering of graph signals are studied using
spectral graph theory. In image processing, graphs are strong tools for representing
structures formed by pixels, like edges and textures.
The filtering of graph signals is not only an extension of that for standard time- and
spatial-domain signals, but it also has its own interesting properties. For example, GSP
can represent traditional pixel-dependent image filtering methods as graph spectral
domain filters. Furthermore, theory and design methods for wavelets and filter banks,
which are studied extensively in signal and image processing, are also updated to treat
graph signals.
In this chapter, the spectral-domain filtering of graph signals is introduced. In
section 1.2, the filtering of time-domain signals is briefly described as a starting
point. The filtering of graph signals, both in the vertex and spectral domains, is
detailed in section 1.3, in addition to its relationship with classical filtering.
Edge-preserving image smoothing is represented as a graph filter in section 1.4.
Furthermore, a framework of filtering by multiple graph filters, i.e. graph wavelets
and filter banks, is presented in section 1.5. Eventually, section 1.6 introduces several
fast computation methods of graph filtering. Finally, the concluding remarks of this
chapter are discussed in section 1.7.

Graph Spectral Image Processing,

coordinated by Gene C HEUNG and Enrico M AGLI. © ISTE Ltd 2021
4 Graph Spectral Image Processing

1.2. Review: filtering of time-domain signals

We start by reviewing the filtering in discrete-time linear time-invariant (LTI)

systems, which has been extensively studied in literature. Suppose that a
one-dimensional discrete-time signal xn is obtained by sampling its continuous-time
counterpart x(t), with a fixed sampling period T , i.e. xn = x(nT ). A
two-dimensional image signal can be similarly obtained by performing sampling in
both the horizontal and vertical directions. In this case, the spatial sampling period
usually corresponds to the spacing between an array of photosensors.

Suppose that an impulse response of a filter hn is given a priori. The discrete-time

filtered signal yn in the LTI system is calculated from xn and hn by convolution as
follows:
∞
 ∞

yn = hn ∗ xn := hn−k xk = xn−k hk [1.1]
k=−∞ k=−∞

This equation is based on the shift of the signal or impulse response. In LTI
systems, we (implicitly) assume that the shift of a discrete-time signal is well
defined, i.e. xn−k is unique and time invariant. Therefore, equation [1.1] is
equivalently represented as
⎡ ⎤
..
⎢ . ⎥
⎢. . . h−1 h0 h1 . . . ⎥
⎢
y=⎢ ⎥ x, [1.2]
⎣ . . . h−1 h0 h1 . . .⎥⎦
..
.

where x := [. . . , x−1 , x0 , x1 , . . . ] and y := [. . . , y−1 , y0 , y1 , . . . ] . 1

In equation [1.2], the impulse response hk is invariant for n, i.e. the same filter is
used for different values of n. Instead, we can use different filters for different values
of n to yield yn , whose impulse response hk [n] is often defined in a signal-dependent
manner, i.e. hk [n] = hk [m] for m = n. It is formulated as

∞

yn := hn−k [n]xk [1.3]
k=−∞

1 Here, we assume both x and y are finite length signals and their boundaries are extended or
filtered by a boundary filter to ensure that the equation is valid.
 Graph Spectral Filtering 5

and its matrix form representation is

⎡ ⎤
..
⎢ . ⎥
⎢. . . h−1 [n − 1] h0 [n − 1] h1 [n − 1] . . . ⎥
⎢
y=⎢ ⎥ x. [1.4]
. . . h [n] h [n] h [n] . . . ⎥
⎣ −1 0 1 ⎦
..
.

Famous image processing filters in this category include the bilateral filter
(Tomasi and Manduchi 1998; Barash 2002; Durand and Dorsey 2002; Fleishman
et al. 2003), anisotropic diffusion (Weickert 1998; Desbrun et al. 1999), adaptive
directional wavelets (Chang and Girod 2007; Ding et al. 2007; Tanaka et al. 2010)
and their variants.

It is well known that convolution in the time domain equation [1.1] has an
equivalent expression in the frequency (i.e. Fourier) domain as follows:

ŷ(ω) = ĥ(ω)x̂(ω), [1.5]

where
∞

x̂(ω) := xn e−jωn . [1.6]
n=−∞

Here, x̂(ω) is the discrete-time Fourier transform (DTFT) of xn . We utilize the

fact that convolution in the time domain is identical to multiplication in the frequency
domain. Note that the fixed filter has a corresponding fixed frequency response, and
thus, we can intuitively understand the filter characteristics from the frequency
response. In contrast, the frequency response of a signal-dependent filter is not
always clear in general. Fortunately, this drawback can be partially solved with a
graph spectral domain perspective, which is described further.

1.3. Filtering of graph signals

In this chapter, we consider linear graph filters. Readers can find nonlinear graph
filters, like one used in deep learning, in the following chapters, specifically
Chapter 10.

Let us denote a graph filter as H ∈ RN ×N , where its elements are typically derived
from G and x. As in the LTI system, the filtered signal is represented as

y = Hx. [1.7]
Visit https://ebookmass.com today to explore
a vast collection of ebooks across various
genres, available in popular formats like
PDF, EPUB, and MOBI, fully compatible with
all devices. Enjoy a seamless reading
experience and effortlessly download high-
quality materials in just a few simple steps.
Plus, don’t miss out on exciting offers that
let you access a wealth of knowledge at the
best prices!
Exploring the Variety of Random
Documents with Different Content
back
back
back
back
back
back
back
back
back
back
back
back
back
back
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookmasss.com