Interdisciplinary Mathematical Sciences -
Vol. 19
Kernel-based Approximation
Methods using MATLAB
Gregory Fasshauer
Illinois Institute of Technology, USA
Michael McCourt
University of Colorado Denver, USA
^ World Scientific
BEIJING SHANGHAI HONG KONG TAIPEI CHENNAI
LONDON SINGAPORE
• •
•
NEW JERSEY • • • •
Contents
Preface vii
An Introduction to Kernel-Based Approximation Methods
and Their Stable Computation 1
1. Introduction 3
1.1 Positive Definite Kernels: Where Do They Fit in the Mathematical
Landscape? 3
1.2 A Historical Perspective 5
1.3 The Fundamental Application: Scattered Data Fitting 7
1.3.1 The Haar-Mairhuber-Curtis theorem: Why using kernels
is a "natural" approach 9
1.3.2 Variations of scattered data fitting 11
1.4 Other Applications 12
1.4.1 Statistical data
fitting 12
1.4.2 Machine learning 13
1.4.3 Numerical solution of PDEs 13
1.4.4 Computational finance 14
1.5 Topics We Do Not Cover 15
2. Positive Definite Kernels and Reproducing Kernel Hilbert Spaces 17
2.1 Positive Definite Kernels 17
2.2 Hilbert-Schmidt, Mercer and Karhunen-Loeve Series 20
2.2.1 Hilbert-Schmidt operators 20
2.2.2 The Hilbert Schmidt eigenvalue problem 22
2.2.3 Mercers theorem 24
2.2.4 Examples of Hilbert-Schmidt integral eigenvalue problems
and Mercer series 25
2.2.5 Iterated kernels 30
xi
Contents
xii
31
2.2.6 Fourier and Karhunen-Loeve expansions
Kernel Hilbert 32
2.3 Reproducing Spaces
Feature 36
2.4 Maps
41
3. Examples of Kernels
41
3.1 Radial Kernels
kernels 41
3.1.1 Isotropic radial
radial kernels 44
3.1.2 Anisotropic
Translation Invariant Kernels 45
3.2
Series Kernels 46
3.3
Power series and series kernels 47
3.3.1 Taylor
3.3.2 Other series kernels 48
General Kernels 49
3.4 Anisotropic
kernels 49
3.4.1 Dot product
3.4.2 Zonal kernels 50
kernels 52
3.4.3 Tensor product
Radial Kernels 53
3.5 Compactly Supported
Multiscale Kernels 54
3.6
55
3.7 Space-Time Kernels
Learned Kernels 56
3.8
56
3.9 Designer Kernels
3.9.1 Periodic, kernels 57
58
3.9.2 Chebyshev kernels
Kernels in Matlab 61
4.
Radial Kernels in MATLAB 62
4.1
distance matrices in Matlab 63
4.1.1 Symmetric
4.1.2 General distance matrices in Matlab 64
in Matlab 66
4.1.3 Anisotropic distance matrices
Matlab 68
4.1.4 Evaluating radial kernels and interpolants in . .
.
Kernels in Matlab 72
4.2 Compactly Supported
Zonal Kernels in Matlab 76
4.3
4.4 Tensor Product Kernels in Matlab 77
4.5 Series Kernels in Matlab 79
The Connection to
89
5. Kriging
Random Fields and Random Variables 90
5.1
of Spaces 94
5.2 Duality
and Prediction via Kriging 96
5.3 Modeling
5.3.1 best linear unbiased predictor 96
Kriging as
99
5.3.2 Bayesian framework
101
5.3.3 Confidence intervals
xiii
Contents
105
5.3.4 Semi-variograms
5.4 Karhunen-Loeve Expansions and Polynomial Chaos 106
107
5.5 Generalized Polynomial Chaos
The Connection to Green's Kernels 111
6.
6.1 Introduction Ill
6.2 Green's Kernels Defined 112
6.3 Differential Eigenvalue Problems 114
115
6.4 Computing Green's Kernels
An Computing the Brownian bridge kernel as
6.4.1 example:
115
Green's kernel
Generalizations of the Brownian bridge kernel 117
6.4.2
118
6.5 Classical Examples of Green's Kernels
120
6.6 Sturm-Liouville Theory
121
6.7 Eigenfunction Expansions
6.8 The Connection Between Hilbert-Schmidt and Sturm-Liouville
123
Eigenvalue Problems
6.9 Limitations 124
6.10 125
Summary
Iterated Brownian Bridge Kernels: A Green's Kernel Example 127
7.
Kernels 127
7.1 Derivation of Piecewise Polynomial Spline
127
7.1.1 Recall some special Green's kernels
7.1.2 A family of piecewise polynomial splines of arbitrary odd
129
degree
7.1.3 Benefits of using a kernel representation for piecewise
131
polynomial splines
7.2 Derivation of General Iterated Brownian Bridge Kernels 132
Kernels 134
7.3 Properties of Iterated Brownian Bridge
134
7.3.1 Truncation of the Mercer series
136
7.3.2 Effects of the boundary conditions
orders 139
7.3.3 Convergence
kernels bounded domains 139
7.3.4 Iterated Brownian bridge on . .
7.3.5 "Flat" limits 143
7.3.6 Summary for functions satisfying homogeneous boundary-
conditions 146
Generalized Sobolev 147
8. Spaces
How Native Were Viewed Until Recently 147
8.1 Spaces
8.2 Generalized Sobolev Spaces on the Full Space Rd 152
8.2.1 Two different kernels for H2(E) 155
156
8.2.2 Higher-dimensional examples
xiv Contents
8.2.3 Summary for full-space generalized Sobolev spaces 158
8.3 Generalized Sobolev Spaces on Bounded Domains 158
8.3.1 Modifications of the Brownian bridge kernel: A detailed
investigation 160
8.3.2 Summary for generalized Sobolev spaces on bounded
domains 167
8.3.3 An alternative framework for boundary value problems on
167
[a,b]
8.4 Conclusions 168
9. Accuracy and Optimality of Reproducing Kernel Hilbert
Space Methods 171
171
9.1 Optimality
Different of Error 172
9.2 Types
9.3 The "Standard" Error Bound 172
175
9.4 Error Bounds via Sampling Inequalities
175
9.4.1 How sampling inequalities lead to error bounds
9.4.2 Univariate sampling inequalities and error bounds 176
9.4.3 to iterated Brownian bridge kernels 181
Application
in dimensions 183
9.4.4 Sampling inequalities higher
9.5 Dimension-independent error bounds 184
9.5.1 Traditional dimension-dependent error bounds 185
9.5.2 Worst-case weighted L2 error bounds 185
10. "Flat" Limits 189
10.1 Introduction 189
191
10.2 Kernels with Infinite Smoothness
193
10.3 Kernels with Finite Smoothness
197
10.4 Summary and Outlook
An Unfortunate Misconception 199
11. The Uncertainty Principle -
199
11.1 Accuracy vs. Stability
11.2 and Stability 201
Accuracy
12. Alternate Bases 203
Basis Functions 204
12.1 Data-dependent
Standard basis functions 204
12.1.1
12.1.2 Cardinal basis functions 206
12.1.3 Alternate bases via matrix factorization 208
210
12.1.4 Newton-type basis functions
12.1.5 SVD and weighted SVD bases 215
and Numerical Eigenfunctions 217
12.2 Analytical
xv
Contents
12.2.1 Eigenfunctions given analytically 218
221
12.2.2 Eigenfunctions obtained computationally
226
12.3 Approximation Using Eigenfunctions
12.4 Other Recent Preconditioning and Alternate Basis Techniques . .
230
Stable via the Hilbert-Schmidt SVD 231
13. Computation
13.1 A Formal Matrix Decomposition of K 232
13.2 Obtaining a Stable Alternate Basis via the Hilbert Schmidt SVD .
235
13.2.1 Summary: How to use the Hilbert-Schmidt SVD 241
13.3 Iterated Brownian Bridge Kernels via the Hilbert-Schmidt SVD .
243
13.4 Issues with the Hilbert-Schmidt SVD 248
13.4.1 Truncation of the Hilbert-Schmidt series 248
13.4.2 Invertibility of 4>i 250
of Alternate Bases for Gaussian Kernels 252
13.5 Comparison
255
14. Parameter Optimization
14.1 Modified Golomb-Weinberger Bound and Kriging Variance ....
256
14.1.1 How to avoid cancelation while computing the power
257
function (kriging variance)
14.1.2 How to stably compute the native space norm of the
interpolant (Mahalanobis distance) 258
14.2 Cross-Validation 260
14.3 Maximum Likelihood Estimation 263
14.3.1 MLE independent of process variance 264
14.3.2 MLE with process variance 265
14.3.3 A deterministic derivation of MLE 266
to the Selection of Good Kernel Parameters 267
14.4 Other Approaches . .
14.5 Goals for Parametrization Judgment Tool 269
a
Advanced Examples 273
15. Scattered Data Fitting 275
276
15.1 Approximation Using Smoothing Splines
15.2 Low-rank Approximate Interpolation 280
286
15.3 Interpolation on the Unit Sphere
Considerations for Scattered Data Fitting 290
15.4 Computational
alternate basis 291
15.4.1 The cost of computing/implementing an . .
structure in kernel computations 292
15.4.2 Exploiting
and Surrogate Modeling 295
16. Computer Experiments
295
16.1 Surrogate Modeling
xvi Contents
16.2 Experimental Design 297
16.3 Models for Standard Test Functions 298
Surrogate
16.3.1 Piston simulation function 298
16.3.2 Borehole function 304
306
16.4 Modeling From Data
16.5 Fitting Empirical Distribution Functions 307
17. Statistical Data via Gaussian Processes 315
Fitting
17.1 Geostatistics 315
17.2 Data Fitting 324
Anisotropic
17.3 Data Fitting Using Universal Kriging and Maximum Likelihood
Estimation 327
18. Machine 335
Learning
Networks 336
18.1 Regularization
18.2 Radial Basis Function Networks 337
18.2.1 Numerical experiments for regression with RBF networks .
339
Vector Machines 343
18.3 Support
18.3.1 Linear classification 344
18.3.2 Kernel classification 346
18.3.3 Numerical experiments for classification with kernel SVMs 350
18.3.4 Computational consideration for classification with kernel
SVMs 354
18.3.5 Linear support vector regression 358
18.3.6 Nonlinear support vector regression 359
19. Derivatives of and Hermite Interpolation 361
Interpolants
362
19.1 Differentiating Interpolants
19.1.1 Cardinal function representation of derivatives 362
19.1.2 Error bounds for simultaneous approximation 363
364
19.1.3 Global differentiation matrices
19.1.4 Local differentiation matrices 369
Hermite 377
19.2 Interpolation
19.2.1 kernel-based Hermite interpolation 378
Nonsymmetric
19.2.2 kernel-based Hermite interpolation 381
Symmetric
19.2.3 Generalized Hermite interpolation via the Hilbert -Schmidt
SVD 383
Gradient 384
19.2.4 An example: interpolation
386
19.2.5 Kriging interpretation
via Derivatives of Eigenfunctions 387
19.3 Doing Hermite Interpolation . . .
19.3.1 Differentiation of a low-rank eigenfunction approximate
388
interpolant
xvii
Contents
An example: Derivatives of Gaussians eigenfunctions 389
19.3.2 . . .
19.4 Multiphysics Coupling 392
19.4.1 Meshfree coupling 395
An example: coupled 2D heat equation 396
19.4.2
considerations 401
19.4.3 Computational
20. Kernel-Based Methods for PDEs 403
403
20.1 Collocation for Linear Elliptic PDEs
collocation in the standard basis 404
20.1.1 Nonsyinmetric
407
20.1.2 Nonsymmetric collocation using the Hilbert-Schmidt SVD
20.2 Method of Lines 411
20.3 Method of Fundamental Solutions 416
20.4 Method of Particular Solutions 420
20.5 Kernel-based Finite Differences 423
Collocation 425
20.6 Space-Time
431
21. Finance
21.1 Brownian motion 431
21.1.1 Brownian motion and the Brownian motion kernel 432
21.1.2 Geometric Brownian motion 433
434
21.1.3 Pricing options and high-dimensional integration
21.1.4 A generic error formula for quasi-Monte Carlo integration
436
via reproducing kernels
of asset pricing through quasi-Monte Carlo 437
21.1.5 Example ....
21.2 Black-Scholes PDEs 440
Black-Scholes PDEs 441
21.2.1 Single-asset European option through
445
21.2.2 Pricing American options
Appendix A Collection of Positive Definite Kernels and Their
447
Known Mercer Series
A.l Piecewise Linear Kernels 447
kernel 447
A. 1.1 Brownian bridge
A.1.2 Brownian motion kernel 448
linear kernel 448
A. 1.3 Another piecewise
448
A.2 Exponential Kernel
449
A.2.1 Domain: [0,1]
449
A.2.2 Domain: [—L, L]
449
A.2.3 Domain: [0, oo)
A.3 Other Continuous Kernels 450
450
A.3.1 Tension spline kernel
A.3.2 Relaxation spline kernel 451
451
A.3.3 Legendre kernel
xviii Contents
A.4 Modified Exponential Kernel 451
A.5 Families of Iterated Kernels 452
A.5.1 Iterated Brownian bridge kernels 452
A.5.2 Periodic spline kernels 452
A.5.3 Periodic kernels 453
A.5.4 Chebyshev kernels 453
A.6 Kernel for the First Weighted Sobolev Space 454
A. 7 Gaussian Kernel 455
A.8 Sine Kernel 455
A. 9 Zonal Kernels 456
A.9.1 Spherical inverse multiquadric 456
A.9.2 Abel-Poisson kernel 456
Appendix B How To Choose the Data Sites 457
458
B. l Low Discrepancy Designs
in Statistics 460
B.2 Optimal Designs
461
B.3 Optimal Points in Approximation Theory
C A Few Facts from Analysis and Probability 463
Appendix
Appendix D The GaussQR Repository in Matlab 467
D.l Accessing GaussQR 467
D.2 Common functions in GaussQR 468
D.3 Full Hilbert-Schmidt SVD sample solver 469
473
Bibliography
Index 505
