Full download ebook at ebookmeta.

com

Kernel Methods for Machine Learning with Math

and Python: 100 Exercises for Building Logic
Joe Suzuki

https://ebookmeta.com/product/kernel-methods-for-
machine-learning-with-math-and-
python-100-exercises-for-building-logic-joe-
suzuki/

Download more ebook from https://ebookmeta.com

More products digital (pdf, epub, mobi) instant
download maybe you interests ...

Statistical Learning with Math and Python: 100

Exercises for Building Logic Joe Suzuki

https://ebookmeta.com/product/statistical-learning-with-math-and-
python-100-exercises-for-building-logic-joe-suzuki/

Sparse Estimation with Math and Python 100 Exercises

for Building Logic 1st Edition Joe Suzuki

https://ebookmeta.com/product/sparse-estimation-with-math-and-
python-100-exercises-for-building-logic-1st-edition-joe-suzuki/

Sparse Estimation with Math and Python: 100 Exercises

for Building Logic 1st Edition Joe Suzuki

https://ebookmeta.com/product/sparse-estimation-with-math-and-
python-100-exercises-for-building-logic-1st-edition-joe-suzuki-2/

Sparse Estimation with Math and R 2nd Edition Joe

Suzuki

https://ebookmeta.com/product/sparse-estimation-with-math-
and-r-2nd-edition-joe-suzuki/
Machine Learning for Financial Risk Management with
Python: Algorithms for Modeling Risk 1st Edition
Karasan

https://ebookmeta.com/product/machine-learning-for-financial-
risk-management-with-python-algorithms-for-modeling-risk-1st-
edition-karasan/

Machine Learning for Financial Risk Management with

Python: Algorithms for Modeling Risk (Early Release)
Abdullah Karasan

https://ebookmeta.com/product/machine-learning-for-financial-
risk-management-with-python-algorithms-for-modeling-risk-early-
release-abdullah-karasan/

Machine Learning for Decision Sciences with Case

Studies in Python 1st Edition S. Sumathi

https://ebookmeta.com/product/machine-learning-for-decision-
sciences-with-case-studies-in-python-1st-edition-s-sumathi/

Python for Probability Statistics and Machine Learning

2nd Edition José Unpingco

https://ebookmeta.com/product/python-for-probability-statistics-
and-machine-learning-2nd-edition-jose-unpingco/

Random Matrix Methods for Machine Learning 1st Edition

Romain Couillet

https://ebookmeta.com/product/random-matrix-methods-for-machine-
learning-1st-edition-romain-couillet/
Joe Suzuki

Kernel Methods
for Machine
Learning
with Math and
Python
100 Exercises for Building Logic
Kernel Methods for Machine Learning with Math
and Python
Joe Suzuki

Kernel Methods for Machine

Learning with Math
and Python
100 Exercises for Building Logic
Joe Suzuki
Graduate School of Engineering Science
Osaka University
Toyonaka, Osaka, Japan

ISBN 978-981-19-0400-4 ISBN 978-981-19-0401-1 (eBook)

https://doi.org/10.1007/978-981-19-0401-1

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature
Singapore Pte Ltd. 2022
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or
information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface

How to Overcome Your Kernel Weakness

Among machine learning methods, kernels have always been a particular weakness
of mine. I tried to read “Introduction to Kernel Methods” by Kenji Fukumizu (in
Japanese) but failed many times. I invited Prof. Fukumizu to give an intensive lecture
at Osaka University and listened to the course for a week with the students, but I
could not understand the book’s essence. When I first started writing this book, my
goal was to rid my sense of weakness. However, now that this book is completed, I
can tell readers how they can overcome their own kernel weaknesses.
Most people, even machine learning researchers, do not understand kernels and
use them. If you open this page, I believe you have a positive feeling that you want
to overcome your weakness.
The shortest path I would most recommend for achieving this is to learn mathe-
matics by starting from the basics. Kernels work according to the mathematics behind
them. It is essential to think through this concept until you understand it. The mathe-
matics needed to understand kernels are called functional analysis (Chap. 2). Even if
you know linear algebra or differential and integral calculus, you may be confused.
Vectors are finite dimensional, but a set of functions is infinite dimensional and can
be treated as linear algebra. If the concept of completeness is new to you, I hope you
will take the time to learn about it. However, if you get through this second chapter,
I think you will understand everything about kernels.
This book is the third volume (of six) in the 100 Exercises for Building Logic set.
Since this is a book, there must be a reason for publishing it (the so-called cause)
when existing books on kernels can be found. The following are some of the features
of this book.
1. The mathematical propositions of kernels are proven, and the correct conclu-
sions are stated so that the reader can reach the essence of kernels.

v
vi Preface

2. As in the other books in the 100 Mathematical Problems in Machine Learning

series, source programs and running examples are presented to promote under-
standing. It is not easy for readers to understand the results if only mathematical
formulas are given, and this is especially true for kernels.
3. Once the reader understands the basic topics of functional analysis (Chap. 2), the
applications in the subsequent chapters are discussed, and no prior knowledge
of mathematics is assumed.
4. This kernel considers both the kernel of the RKHS and the kernel of the Gaussian
process. A clear distinction is made between the two treatments. In this book,
the two types of kernels are discussed in Chaps. 5 and 6, respectively.
We surveyed books on kernels both in Japan and overseas but found that none satisfied
two or more of the above characteristics.
I have experienced many failures leading up to the publication of this book. Every
year, I give a lecture (at the graduate school of Osaka University). Each area of
machine learning is studied by solving 100 mathematical and programming exercises.
Sparse estimation (2018) and graphical models (2019) have gained popularity, and
the 2020 kernel lecture has more than 100 students enrolled. However, although I
prepared for the lectures for more than 2 days every week, the talks did not go well,
probably due to my weakness regarding the subject. This was evident from the class
questionnaires provided by the students. However, I analyzed each of these problems
and made improvements, and this book was born.
I hope that readers will learn about kernels efficiently without following the same
path that I took (consuming much time and energy through trial and error). Reading
this book does not mean that you will write a paper immediately, but it will give
you a solid foundation. You will be able to read kernel papers smoothly, which had
previously seemed difficult, and you will be able to see the whole kernel paradigm
from a higher level. This book is also enjoyable, even for researchers in machine
learning. We hope that you will use this book to achieve success in your respective
fields.

What Makes KMMP Unique?

I have summarized the features of this book as follows.

1. Developing logic
We mathematically formulate and solve each ML problem and build those
programs to grasp the subject’s essence. The KMMP (Kernel methods for
Machine learning with Math and Python) instills “logic” in the minds of the
readers. The reader will acquire both the knowledge and ideas of ML. Even
if new technology emerges, they will be able to follow the changes smoothly.
After solving the 100 problems, most students would say, “I learned a lot”.
Preface vii

2. Not just a story

If programming codes are available, you can immediately take action. It is
unfortunate when an ML book does not offer the source codes. Even if a package
is available, if we cannot see the inner workings of the programs, all we can do
is input data into those programs. In KMMP, the program codes are available
for most of the procedures. In cases where the reader does not understand the
math, the codes will help them know what it means.
3. Not just a how-to book: an academic book written by a university professor.
This book explains how to use the package and provides examples of executions
for those unfamiliar with them. Still, because only the inputs and outputs are
visible, we can see the procedure as a black box. In this sense, the reader will
have limited satisfaction because they will not obtain the subject’s essence.
KMMP intends to show the reader the heart of ML and is more of a full-fledged
academic book.
4. Solve 100 exercises: problems are improved with feedback from university
students
The exercises in this book have been used in university lectures and refined
based on students’ feedback. The best 100 problems were selected. Each chapter
(except the exercises) explains the solutions, and you can solve all of the
exercises by reading the book.
5. Self-contained
All of us have been discouraged by phrases such as “for the details, please refer to
the literature XX”. Unless you are an enthusiastic reader or researcher, nobody
will seek out those references. In this book, we have presented the material
so that consulting external references is not required. Additionally, the proofs
are simple derivations, and the complicated proofs are given in the appendices
at the end of each chapter. KMMP completes all discussions, including the
appendices.
6. Readers’ pages: questions, discussion, and program files The reader can ask any
question on the book via https://bayesnet.org/books.

Osaka, Japan Joe Suzuki

November 2021
Acknowledgments

The author wishes to thank Mr. Bing Yuan Zhang, Mr. Tian Le Yang, Mr. Ryosuke
Shimmura, Mr. Tomohiro Kamei, Ms. Rieko Tasaka, Mr. Keito Odajima, Mr. Daiki
Fujii, Mr. Hongming Huang, and all graduate students at Osaka University, for
pointing out logical errors in mathematical expressions and programs. Furthermore, I
would like to take this opportunity to thank Dr. Hidetoshi Matsui (Shiga University),
Dr. Michio Yamamoto (Okayama University), and Dr. Yoshikazu Terada (Osaka
University) for their advice on functional data analysis in seminars and workshops.
This English book is based mainly on the Japanese book published by Kyoritsu
Shuppan Co., Ltd. in 2021. The author would like to thank Kyoritsu Shuppan Co.,
Ltd., particularly its editorial members Mr. Tetsuya Ishii and Ms. Saki Otani. The
author also appreciates Ms. Mio Sugino, Springer, preparing the publication and
providing advice on the manuscript.

Osaka, Japan Joe Suzuki

November 2021

ix
Contents

1 Positive Definite Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Positive Definiteness of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Positive Definite Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.5 Bochner’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.6 Kernels for Strings, Trees, and Graphs . . . . . . . . . . . . . . . . . . . . . . . . . 16
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Exercises 1∼15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2 Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.1 Metric Spaces and Their Completeness . . . . . . . . . . . . . . . . . . . . . . . . 29
2.2 Linear Spaces and Inner Product Spaces . . . . . . . . . . . . . . . . . . . . . . . 33
2.3 Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.4 Projection Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.5 Linear Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.6 Compact Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Appendix: Proofs of Propositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Exercises 16∼30 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3 Reproducing Kernel Hilbert Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.1 RKHSs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.2 Sobolev Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.3 Mercer’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Exercises 31∼45 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4 Kernel Computations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.1 Kernel Ridge Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.2 Kernel Principle Component Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.3 Kernel SVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
4.4 Spline Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

xi
xii Contents

4.5 Random Fourier Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

4.6 Nyström Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.7 Incomplete Cholesky Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . 118
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Exercises 46∼64 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
5 The MMD and HSIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.1 Random Variables in RKHSs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5.2 The MMD and Two-Sample Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 132
5.3 The HSIC and Independence Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
5.4 Characteristic and Universal Kernels . . . . . . . . . . . . . . . . . . . . . . . . . . 150
5.5 Introduction to Empirical Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Exercises 65∼83 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6 Gaussian Processes and Functional Data Analyses . . . . . . . . . . . . . . . . . 167
6.1 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
6.2 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
6.3 Gaussian Processes with Inducing Variables . . . . . . . . . . . . . . . . . . . . 180
6.4 Karhunen-Lóeve Expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
6.5 Functional Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Exercises 83∼100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Chapter 1
Positive Definite Kernels

In data analysis and various information processing tasks, we use kernels to evaluate
the similarities between pairs of objects. In this book, we deal with mathematically
defined kernels called positive definite kernels. Let the elements x, y of a set E
correspond to the elements (functions) (x), (y) of a linear space H called the
reproducing kernel Hilbert space. The kernel k(x, y) corresponds to the inner product
(x), (y) H in the linear space H . Additionally, by choosing a nonlinear map ,
this kernel can be applied to various problems. The set E may be a string, a tree, or a
graph, even if it is not a real-numbered vector, as long as the kernel satisfies positive
definiteness. After defining probability and Lebesgue integrals in the second half,
we will learn about kernels by using characteristic functions (Bochner’s theorem).

1.1 Positive Definiteness of a Matrix

Let n ≥ 1; we say that a square matrix A is symmetric if A ∈ Rn×n is equal to its

transpose (A = A)1 , and we say that A is nonnegative definite if all the eigenvalues
are nonnegative.
Proposition 1 (nonnegative definite matrix) The following three conditions are
equivalent for a symmetric matrix A ∈ Rn×n .
1. A matrix B ∈ Rn×n exists such that A = B  B.
2. x  Ax ≥ 0 for any x ∈ Rn .
3. The eigenvalues of A are nonnegative.
⇒2. holds because A = B  B=
Proof: 1.= ⇒x  Ax = x  B  Bx = Bx2 ≥ 0. 2.=
⇒3.

⇒ 0 ≤ y  Ay = y  λy = λy2 for
follows from the fact that x Ax ≥ 0, x ∈ Rn =

1 We write the transpose of matrix A as A .

© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2022 1
J. Suzuki, Kernel Methods for Machine Learning with Math and Python,
https://doi.org/10.1007/978-981-19-0401-1_1
2 1 Positive Definite Kernels

an eigenvalue λ of A and√ its

√ eigenvector √ y ∈ R √
n
. 3.=⇒1. holds since λ √1 , . . . , λn ≥
 
0=⇒A = P D P = P D D P = ( D P√ ) D P√ , where D and D are diag-
onal matrices with elements λ1 , . . . , λn and λ1 , . . . , λn , and P is the correspond-
ing orthogonal matrix. 
A nonnegative definite matrix A is symmetric. In this book, we say that a non-
negative definite matrix is positive definite if all of its eigenvalues are positive. In
addition, we assume that the elements of any matrix are real. However, the following
fact is often useful when we deal with complex numbers and Fourier transformations.


√ definite matrix A ∈ R , we have that z Az ≥ 0
n×n
Corollary 1 For a nonnegative
for any z ∈ C , where i = −1 is the imaginary unit, and we write the conjugate
n

x − i y of z = x + i y ∈ C with x, y ∈ R as z.
Proof: Since there exists a B ∈ Rn×n such that A = B  B for a nonnegative definite
matrix A ∈ Rn×n , we have that

z  Az = (Bz) Bz = |Bz|2 ≥ 0

for any z = [z 1 , . . . , z n ] ∈ Cn . 

Example 1

# In this chapter, we assume that the following has been executed.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import style
style.use("seaborn−ticks")

n=3
B = np.random.normal(size=n∗∗2).reshape(3, 3)
A = np.dot(B.T, B)
values, vectors = np.linalg.eig(A)
print("values:\n", values, "\n\nvectors:\n", vectors, "\n")

values:
[0.09337468 7.75678625 4.43554113]
vectors:
[[ 0.49860775 0.84350568 0.199721 ]
[ 0.39606374 -0.42663779 0.81308899]
[-0.77105371 0.32631023 0.54680692]]
1.1 Positive Definiteness of a Matrix 3

S = []
for i in range(10):
z = np.random.normal(size = n)
y = np.squeeze(z.T.dot(A.dot(z)))
S.append(y)
if (i+1) % 5 == 0:
print("S[%d:%d]:"%((i−4),i), S[i−4:i])

S[0:4]: [23.24608872999895, 6.601263342526701, 5.334515801733688, 14.886876186736613]

S[5:9]: [18.85503241886245, 34.30290091714191, 1.025291282540866, 29.59512428090335]

1.2 Kernels

Let E be a set. We often express similarity between elements x, y ∈ E by using

a bivariate function k : E × E → R not just for data analysis but also for various
information processing tasks. The larger k(x, y) is, the more similar x, y are. We
call such a function k : E × E → R a kernel.
Example 2 (Epanechnikov kernel) We use the kernel k : E × E → R such that
 
|x − y|
k(x, y) = D
λ
3
(1 − t 2 ), |t| ≤ 1
D(t) = 4
0, Other wise

for λ > 0, and we construct the following function (the Nadaraya-Watson estimator)
from observations (x1 , y1 ), . . . , (x N , y N ) ∈ E × R:
N
k(x, xi )yi
fˆ(x) = i=1
N
.
j=1 k(x, x j )

For a given input x∗ ∈ E that is different from the N pairs of inputs, we return
the weighted sum of y1 , . . . , y N ,
k(x∗ , x1 ) k(x∗ , x N )
N , . . . , N ,
j=1 k(x ∗ , x j ) j=1 k(x ∗ , x j )

as the output fˆ(x∗ ). Because we assume that a larger k(x, y) yields a more similar
x, y ∈ E, the more similar x∗ and xi are, the larger the weight of yi .
Given an input x∗ ∈ E for i = 1, . . . , N , we weight yi such that xi − λ ≤ x∗ ≤
xi + λ is proportional to k(xi , x∗ ). If we make the λ value smaller, we predict y∗ by
using only the (xi , yi ) for which xi and x∗ are close. We display the output obtained
when we execute the following code in Fig. 1.1.
4 1 Positive Definite Kernels

3
λ = 0.05
λ = 0.35
λ = 0.5
2
1
y
0
-1
-2

-3 -2 -1 0 1 2 3
x

Fig. 1.1 We use the Epanechnikov kernel and Nadaraya-Watson estimator to draw the curves for
λ = 0.05, 0.35, 0.5. Finally, we obtain the optimal λ value and present it in the same graph

n = 250
x = 2 ∗ np.random.normal(size = n)
y = np.sin(2 ∗ np.pi ∗ x) + np.random.normal(size = n) / 4 # Data Generation

def D(t): # Function Definition D

return np.maximum(0.75 ∗ (1 − t∗∗2), 0)

def k(x, y, lam): # Function Definition K

return D(np.abs((x − y) / lam))

def f(z, lam): # Function Definition f

S = 0; T = 0
for i in range(n):
S = S + k(x[i], z, lam) ∗ y[i]
T = T + k(x[i], z, lam)
return S / T

plt.figure(num=1, figsize=(15, 8),dpi=80)

plt.xlim(−3, 3); plt.ylim(−2, 3)
plt.xticks(fontsize = 14); plt.yticks(fontsize = 14)
plt.scatter(x, y, facecolors=’none’, edgecolors = "k", marker = "o")

xx = np.arange(−3, 3, 0.1)
yy = [[] for _ in range(3)]
lam = [0.05, 0.35, 0.50]
color = ["g", "b", "r"]
for i in range(3):
for zz in xx:
yy[i].append(f(zz, lam[i]))
plt.plot(xx, yy[i], c = color[i], label = lam[i])

plt.legend(loc = "upper left", frameon = True, prop={’size’:14})

plt.title("Nadaraya−Watson Estimator", fontsize = 20)
1.3 Positive Definite Kernels 5

1.3 Positive Definite Kernels

The kernels that we consider in this book satisfy the positive definiteness criterion
defined below. Suppose k : E × E → R is symmetric, i.e., k(x, y) = k(y, x), x, y ∈
E. For x1 , . . . , xn ∈ E (n ≥ 1), we say that the matrix
⎡ ⎤
k(x1 , x1 ) · · · k(x1 , xn )
⎢ .. .. .. ⎥
⎦∈R
n×n
⎣ . . . (1.1)
k(xn , x1 ) · · · k(xn , xn )

is the Gram matrix w.r.t. a k of order n. We say that k is a positive definite kernel2 if
the Gram matrix of order n is nonnegative definite for any n ≥ 1 and x1 , . . . , xn ∈ E.
Example 3 The kernel in Example 2 does not satisfy positive definiteness. In
fact, when λ = 2, n = 3, and x1 = −1, x2 = 0, x3 = 1, the matrix consisting of
K λ (xi , yi ) can be written as
⎡ ⎤ ⎡ ⎤
k(x1 , x1 ) k(x1 , x2 ) k(x1 , x3 ) 3/4 9/16 0
⎣ k(x2 , x1 ) k(x2 , x2 ) k(x2 , x3 ) ⎦ = ⎣ 9/16 3/4 9/16 ⎦
k(x3 , x1 ) k(x3 , x2 ) k(x3 , x3 ) 0 9/16 3/4

and the determinant is computed as 33 /26 − 35 /210 − 35 /210 = −33 /29 . In general,
the determinant of a matrix is the product of its eigenvalues, and we find that at least
one of the three eigenvalues is negative.
∞
Example 4 For random variables {X i }i=1 that are not necessarily independent, if
k(X i , X j ) is the covariance between X i , X j , the Gram matrix of any order is the
covariance matrix among a finite number of X j , which means that k is positive
definite. We discuss Gaussian processes based on this fact in Chap. 6.
By assuming positive definiteness, the theory of kernels will be developed in this
book. Hereafter, when we state kernels, we are referring to positive definite kernels.
Let H be a linear space (vector space) equipped with an inner product ·, · H .
Then, we often construct a positive definite kernel with

k(x, y) = (x), (y) H . (1.2)

By using an arbitrary map  : E → H . We say that such a  is a feature map. In

this chapter, we may assume that the linear space H is the Euclidean space H = Rd
of dimensionality d with the standard inner product x, yRd = x  y, x, y ∈ Rd . We
define the linear space and inner product concepts in Chap. 2.
Proposition 2 The kernel k : E × E → R defined in (1.2) is positive definite.

2 Although it seems appropriate to say “a nonnegative definite kernel”, the custom of saying “a
positive definite kernel” has been established.
Another random document with
no related content on Scribd:
side of the spinal cord; the one on the left shows a well-defined
spinal nerve (sn), which may be traced ventrally as far as the end of
the muscle plate, along whose medial side it courses. The ventral
nerve root is plainly seen; the dorsal root, in this section, less plainly.
The amnion (a) and abdominal wall are, as in the preceding figure,
torn in the region of the ventricle.
Figure 17e is a short distance posterior to the figure just
described. The liver is cut through its middle region and forms a
large, darkly staining, reticular mass on the left side of the figure.
The digestive tract is seen at two places to the right of the liver; the
smaller and more ventral of these openings (i) may be called the
intestine, while the larger is evidently the stomach (i′). The body
wall is here unfused and becomes suddenly thinner as it passes
upward into the amnion (a). The Wolffian tubules (wt) form a very
conspicuous mass on either side of the mesentery, in close
connection with the posterior cardinal veins (pc). In the mesoblast
between the dorsal aorta (ao) and the notochord are two small,
irregular, darkly stained masses (sy). These are shown in the
preceding two figures, but were not mentioned in the description.
They may be traced through a great part of the length of the
embryo back of the head region; at intervals corresponding in length
to the distance between the spinal ganglia they are enlarged, while
between these enlargements they are very small in cross-section. At
certain points a small blood-vessel is given off by the dorsal aorta to
the immediate neighborhood of each of these small areas. Although
they show no connection with the central nervous system, these
structures appear to be the rudiments of the sympathetic nerves.
Figure 17f is in the region just in front of the hind legs. The
abdominal walls are here unfused, and into the unenclosed body
cavity projects the intestine (i), supported by a narrow mesentery
and surrounded by a comparatively thick mass of mesoblast. The
Wolffian body and duct form a mass of considerable size on each
side of the mesentery. The Wolffian body is cut near its posterior
end and consists of smaller tubules than in the more anterior
regions. The Wolffian ducts (wd), on the other hand, are very large
and are much more clearly distinguishable from the Wolffian tubules
than in the preceding sections. The Wolffian ridges (wr) are very
marked projections on the sides of the body, and in a region farther
caudad become especially developed as the posterior appendages,
to be described in connection with the following section. Both spinal
ganglia are shown in this figure (sg), and in connection with the left
ganglion the spinal nerve (sn), extending ventrally as far as the level
of the Wolffian duct. The sympathetic nerve rudiments do not
extend so far caudad as the plane of this section. The dorsal end of
each muscle plate (mp) is seen, in this and other sections, to be
slightly enlarged to form a round knob; this knob contains a distinct
cavity (not shown in the figure), the myocœl.
In Figure 17g, owing to the curvature of the body, the plane of the
section passes through the body at three places: through the region
of the heart and lungs (Fig. 17d), through the region of the posterior
appendages, and through the tail. In fact, the plane of the section
represented by each of the preceding figures cut the embryo in more
than one region, but for the sake of simplicity only one region was
represented in each figure. In the figure under discussion only the
leg and tail regions have been drawn, though the latter region (t),
being cut through one of its curves, is seen as an elongated body
with a section of the spinal cord, notochord, etc., at each end. Both
regions shown in the figure are enclosed in the same fold (a) of the
amnion. Of the structures in the dorsal side of the larger or more
anterior part of this figure nothing need be said. The most striking
feature of the section is the presence of the large posterior leg
rudiments (pa). As was noted in the preceding figure, they are, as
usual, merely local enlargements or projections of the mesoblast
(covered, of course, with ectoblast) of the Wolffian ridge. They are,
as shown in this section and in the surface view of this stage (Fig.
17), bluntly pointed projections from the sides of the body. The
anterior appendage seems to be slightly more developed than the
posterior, as was noted in describing the surface view of the embryo.
The digestive tract is cut through its extreme posterior end, in the
region that may be termed the cloaca (cl), for into it at this point the
Wolffian ducts open (wdo). As the narrow cloacal chamber is
followed toward the tail, it becomes still smaller in diameter, and the
ventral depression or cleft seen in this figure gradually becomes
deeper until its walls are continuous with the ectoderm that covers
the ventral projection of mesoderm between the hind legs; no actual
opening to the exterior is present, however. There is a space of
about twenty-five or thirty sections (in a series of eight hundred)
between the posterior ends of the Wolffian bodies and the cloacal
openings of the Wolffian ducts. The body cavity (bc) and the
posterior cardinal veins (pc) are very small in this region, as might
be expected.
STAGE XV
Figure 18 (Plate XXIV.)
Only the head of this embryo is represented, as the general state
of development is about the same as in the preceding stage.
The chief object in making the figure is to show the five gill clefts
(g¹⁻⁵). The fifth cleft, though small and probably not open to the
exterior, is quite distinct in this embryo. The writer would feel more
doubt of its being a true, though rudimentary, gill cleft had not
Clarke (17) found a fifth pair of clefts in his material. The nasal pit
has advanced in development somewhat and shows the beginning of
the groove that connects it with the mouth. In this figure the
crescentic hyomandibular cleft shows its connection with the groove
between the mandibular and the hyoid folds.
STAGE XVI
Figure 19 (Plate XXIV.)
This embryo is only slightly more developed than the preceding.
Body flexure is so great that the forebrain and tail nearly touch. Only
the anterior three gill clefts are visible. The maxillary process (mx) is
longer and more narrow; the mandibular fold has not changed
appreciably. The nasal pit (n) is now connected by a distinct groove
with the stomodæum. The appendages have increased in size, the
posterior (pa) being the longer. The anterior appendage (aa) is
distinctly broadened to form the manus, while no sign of the pes is
to be seen at the extremity of the posterior appendage. The heart
(ht) is still very prominent. The stalk of the umbilicus (u), which is
quite narrow, projects from the ventral wall in the region between
the heart and the hind legs. The tail is of considerable length and is
closely coiled.
STAGE XVII
Figures 20-20j (Plates XXV., XXVI.)
The superficial changes noted in this stage chiefly concern the
head, which has increased considerably in length (Fig. 20). The
curvature of the body is slightly more marked, and the tail is more
tightly coiled at the end. There are still signs of three gill clefts. The
maxillary process (mx) is long and narrow, while the mandibular
arch (md) is still short and broad. The fronto-nasal region has
greatly increased and has the aquiline profile noted by Clarke. The
nasal groove has disappeared, and there remains the small opening
(n) at the side of the fronto-nasal region, near the end of the still
separate maxillary process. The umbilicus is in about the same
condition as in the preceding stage, but the heart is less prominent.
The outline of the manus (ma) is more definite, and the extremity of
the posterior appendage is distinctly flattened out to form the
rudimentary pes (pe). The position of the elbow-joint in the anterior
appendage is seen at the end of the reference line aa.
Typical transverse sections of this stage are shown in Figures 20a-
j.
Figure 20a is a section through the middle region of the head,
cutting the hindbrain on one side and the forebrain on the other. The
walls of the brain show rather more histological differentiation than
was seen in the preceding sections, though this cannot be shown
under the low magnification used. The hindbrain (hb), which is cut
near its anterior border, exhibits the usual membranous dorsal and
thick ventral walls. The forebrain is here seen as three distinct
cavities—a median third ventricle (tv), with a thick ventral wall, and
a thin dorsal wall extended to form a large paraphysis (epi), and two
lateral ventricles (ch), the cavities of the cerebral hemispheres,
whose walls are quite thick except on the side next the third
ventricle. The sections of this series being slightly oblique, the eye is
here cut on the right side only, where it is seen as a large,
semicircular cavity (e) with thick, dense walls. The mesoblast, in
which several blood-vessels (bv) are seen, exhibits three distinct
areas—a median, lighter zone, with a more dense area on either
side. The significance of this variation in the density of the mesoblast
is not apparent.
Figure 20b is only a few sections posterior to the section just
described. It is drawn chiefly to show the appearance of the
forebrain, the other structures being about as in the preceding
figure, except that both eyes (e) are here represented. The section
passes through the wide opening between the third (tv) and the
lateral ventricles (ch) and cuts the anterior edge of the pineal
body[8] (epi). The paraphysis is very large and is directed backward
instead of forward, as is usually the case among the lower
vertebrates (if the alligator may be so classed). It is shown in Figure
17a of a preceding stage and will be again shown in a sagittal
section to be described later. The same areas of more dense and
less dense mesoblast noted in the preceding figure are seen here.
 [8] Subsequent investigation showed that the structure here described as
the pineal body is, in reality, the paraphysis; the pineal body is absent in A.
mississippiensis.

Figure 20c, though still in the head region, shows several features
that were not seen in the preceding figures. On the left of the
hindbrain (hb) the auditory vesicle (o), which is now considerably
more advanced than in earlier figures, is seen as a larger, flask-
shaped cavity and a smaller, round one. Between the larger cavity
and the hindbrain is the root of a cranial nerve (cn), apparently the
eighth, since in another section it comes in close contact with the
wall of the larger part of the auditory vesicle just mentioned. On the
right side, ventral to the hindbrain, another and much larger nerve
(cn) is seen. Nearly in the center of the figure is seen a small,
irregular, thick-walled cavity (p); this is the pituitary body, and its
connection with the roof of the pharynx may easily be made out in
another section. The mesoblast in this region of the sections
contains numerous large and small blood-vessels and exhibits certain
denser areas which probably represent the beginnings of the cranial
cartilages. No sign of the forebrain is seen (the plane of the section
passing in front of that region), except that the tip of the wall of one
of the cerebral hemispheres (ch) is cut. The left nasal chamber (n) is
shown: it will be noted again in the following section. The eye on the
right side shows no remarkable features; its lens (ln) is large and lies
well back of the lips of the optic cup, which may now be called the
iris (ir). A thin layer of mesoblast has pushed in between the lens
and the superficial ectoderm to form the cornea, and the outer wall
of the optic cup is now distinctly pigmented. The inner wall of the
optic cup is beginning to differentiate into the retinal elements. The
eye on the left side is cut farther from its central region and has a
very different appearance from the eye just described. This unusual
appearance is due to the fact that the section passed through the
choroid fissure, which is very large and seems to be formed by the
pushing in of the walls of the cup and not by a mere cleft in these
walls. This fissure is hardly noticeable in the stage preceding the
present, and in a stage slightly older it has disappeared; so that it
would seem to be a very transient structure. It apparently is formed
at about the time that the optic stalk, as such, disappears. It is in
the region of the choroid fissure, if not through it, that the optic
nerve (on) enters the eye. Through the fissure also enters a vascular
tuft of mesoblast (pt) which may be seen projecting into the optic
cup after the disappearance of the fissure. This loop of blood-vessels
is doubtless the pecten.
Figure 20d represents a section through the hindbrain (hb),
pharynx (ph), and tip of the snout. On either side of the hindbrain
are a convoluted auditory vesicle (o), and several blood-vessels and
nerves, while ventral to it is seen the anterior end of the notochord
(nt), around which the mesoblast is somewhat more dense than
elsewhere. The pharynx (ph) sends out toward the surface a narrow
gill cleft (g′) in the neighborhood of each auditory vesicle. These
clefts connect with the exterior by very narrow slits, not seen in the
plane of this section. The opposite end of the pharynx, as seen in
this figure, opens on the left (pn) into the nasal chamber. The nasal
cavity on the right is cut in such a plane that it shows neither its
external nor its pharyngeal opening. The nasal passages are here
fairly long and nearly straight chambers; their lining epithelium is
quite thick in the middle region, but becomes thinner where it
merges into the epithelium of the pharynx at one end, and into the
superficial epithelium at the other end. The unusual appearance of
the eye (e), on the right side of the figure, is due to the fact that the
plane of the section cut tangentially through the extreme edge of
the eye in the region of the choroid fissure.
Figure 20e is only a short distance posterior to the preceding. On
the left side the pharynx (ph) is connected with the exterior through
the stomodæum, and on the right the hyomandibular cleft (g′) is cut
almost through its opening to the exterior. The auditory vesicle (o)
on the right is cut near its middle region, while that on the left is
barely touched by the plane of the section. The notochord (nt), with
its condensed area of mesoblast, is somewhat larger than in the
preceding section. The nasal canal on the right (n) is cut through
neither anterior nor posterior opening, while on the left side the
canal shows the anterior opening (an).
 Figure 20f, which is in the region of the posterior part of the
pharynx and the anterior part of the heart, shows some rather
unusual conditions.
The spinal cord (sc) and notochord (nt), with the faintly outlined
condensations of mesoblast in their region, need no special
description. The pharynx (ph) is here reduced to an irregular,
transversely elongated cavity, the lateral angles of which are
connected on each side with the exterior through a tortuous and
almost closed gill cleft (g), which must be followed through many
sections before its inner and outer openings may be determined.
Dorsal to the pharynx numerous blood-vessels (bv), both large and
small, may be seen, while ventral to it is noticed a faint
condensation of mesoblast (la), in the form of an inverted T, the
anlage of the laryngeal structures. The ventral portion of the figure
is made up of a nearly circular thin-walled cavity, the pericardium
(pr). Most of the pericardial cavity is occupied in this section by the
thick-walled ventricle (vn), above which is the bulbus (b) and the tip
of the auricle (au). The bulbus is nearly circular in outline, though its
cavity is very irregular. A few sections anterior to this, the opening of
the bulbus into the ventricle is seen.
In Figure 20g the section represented is only a short distance
posterior to the one represented by Figure 20f. The mesoblastic
structures in the neighborhood of the spinal cord (sc) and notochord
(nt) will be described in connection with the next figure, where they
are more clearly defined. The œsophagus (oe)—or posterior end of
the pharynx, whichever it may be called—is here a crescentic slit,
with its convex side upward; ventrally it opens by a narrow glottis
into the trachea (ta). The trachea is surrounded by the same
condensed area of mesoblast (la) that was mentioned in connection
with the preceding figure, but the condensation is here more
marked. From the bulbus (b) an aortic arch (ar) extends upward for
a short distance on the right side, while to the left of the œsophagus
an aortic arch (ar) is cut through the upper part of its course. Ventral
to the bulbus the ventricle (vn) and two auricles (au) are seen
surrounded by the pericardial wall.
 Figure 20h is in the region of the liver (li), which has about the
same position in relation to the auricles (au) that was occupied by
the ventricle in the last figure. The auricles are connected with each
other by a wide passage. The trachea (ta) and the œsophagus (oe)
are entirely distinct from each other; the former is a small, nearly
circular hole, while the lumen of the latter is obliterated and its walls
form a solid, bow-shaped mass of cells. Since there is a narrow
space between this mass of cells and the surrounding mesoblast, it
might be thought that the lumen of the œsophagus had been closed
by the simple shrinkage of its walls; higher magnification, however,
fails to show any sign of a collapsed lumen. It is doubtless the
problematic and temporary closure of the œsophagus that is noticed
in other forms. On each side of the œsophagus, in close relation
with the anterior cardinal vein (ac), is noticed a nerve (cn) cut
through a ganglionic enlargement. When traced forward these
nerves are seen to arise from the region of the medulla, and when
followed caudad they are found to be distributed chiefly to the
tissues surrounding the newly formed bronchi; they are doubtless
the tenth cranial nerves. On the right side of the figure the close
connection of this nerve with the near-by gill cleft is seen. Above the
paired aortæ (ao) the sympathetic nerves (sy) will be noticed. The
mesoblast surrounding the spinal cord (sc) and notochord (nt) is
distinctly condensed (more so than the figure shows) to form what
may be called the centrum (c) and neural arch (na) of the vertebræ.
The arch, owing to the slight obliquity of the section, shows here
only on one side. The spinal cord is not yet completely enclosed by
the neural arches. The muscle plates (mp) are in close connection
with the rudiments of the vertebræ just mentioned. The spinal cord
(sc) is here differentiated into three areas—a dense, deeply stained
area immediately around the neurocœl; a less dense area of cells
surrounding the inner area and extending ventralward as a rounded
projection on each side; and an outer layer, with few or on nuclei,
surrounding the inner two layers except on the dorsal side.
In Figure 20i the size and complexity of the figure are due, it will
be easily understood, to the fact that the plane of the section passed
through the curve of the body, thus practically cutting the embryo in
two regions—an anterior, where the lungs (lu) and liver (li) are seen,
and a posterior, where the Wolffian bodies (wt) are present. The
spinal cord and the surrounding structures have almost the same
characteristics at both ends of the figure, except that the primitive
spinal column is rather more distinct in the posterior end of the
section. The posterior cardinal veins (pc), Wolffian ducts (wd), and
Wolffian bodies (wt) are also prominent structures of this end of the
figure, the last being made up of a great number of tubules. The
extreme anterior ends of the Wolffian bodies are seen in the other
half of the section in the upper angles of the body cavity, dorsal to
the lung rudiments (lu). Filling most of the body cavity (bc) and
making up the greater part of the middle of the figure are the liver
(li), now a very large organ; the stomach (i′), also quite large; the
pancreas (pan), a small body lying near the stomach; and the lungs
(lu), which here consist of several thick-walled tubes, surrounded by
lobes of mesoblast. The other features of the figure need no special
mention.
Figure 20j is through the base of the posterior appendages (pa),
in which the cartilages are already being outlined by condensations
of mesoblast. The intestine (i) is cut in two regions—at a more
anterior point, where it is seen as a small, circular hole surrounded
by mesoblast and hung by a narrow mesentery, and through the
cloacal region, the larger and more ventral cavity, into which the
Wolffian ducts (wd) open a short distance caudad to this section.
The blood-vessels present a rather curious appearance. A short
distance anterior to this point the aorta has divided into three, or it
might be said that it has given off two, large branches. These two
branches, one on either side near the posterior cardinal vein, pass
toward the ventral side of the embryo on each side of the cloaca and
end at about the region represented by the present figure. The small
portion of the aorta that remains after the giving off of the two
branches just described continues, as the caudal artery (ca), into the
tail; it is a small vessel just under the notochord, and gives off small,
paired branches at regular intervals toward the vertebral region. The
posterior cardinal veins (pc), posterior to the openings of the
Wolffian ducts into the cloaca, unite to form a large caudal vein lying
just ventral to the caudal artery.
STAGE XVIII
Figure 21 (Plate XXVII.)
This embryo, as may be seen, for example, by the form of the
appendages, is slightly further developed than the one represented
in Figure 20. The figure is from a photograph of a living embryo as it
lay in the egg, a portion of the shell and shell membranes having
been removed. The embryo, which lies on its left side, is rather
faintly outlined because of the overlying allantois. The allantois has
been increasing rapidly in size, and is here so large that it extends
beneath the cut edges of the shell at all points except in the region
in front of the head of the embryo, where its border may be seen.
Its blood-vessels, especially the one that crosses the head just back
of the eye, are clearly shown in the figure, and in the living
specimen, when filled with the bright red blood, they form a most
beautiful demonstration. As in the chick, the allantois lies close
beneath the shell membranes and is easily torn in removing them.
STAGE XIX
Figure 22 (Plate XXVII.)
Figure 22 is a photograph of a somewhat older embryo, removed
from the egg and freed of the fetal membranes. The appendages
show the position of both elbow and knee joints, and in the paddle-
shaped manus and pes the digits may be faintly seen. The tail is
very long and is spirally coiled, the outer spiral being in contact with
the frontal region of the head. The jaws are completely formed, the
upper projecting far beyond the lower. The elliptical outline of the
eyes is noticeable, but the lids are still too little developed to be seen
in this figure. The surface of the embryo is still smooth and white.
STAGE XX
Figures 23-23b (Plate XXVII.)
 In this surface view (Fig. 23) several changes are seen, though no
very great advance in development has taken place. The outlines of
the digits (five in the manus and four in the pes) are now well de
fined; they even project slightly beyond the general outline of the
paddle-shaped part. The tail has begun to straighten out, and it now
extends across the front of the face. The lower jaw has increased in
length, but is still shorter than the upper. The eyelids, especially the
upper, are beginning to be discernible in surface view. Though still
without pigment, the surface of the body is beginning to show by
faint transverse lines the development of scales; these lines are most
evident in this figure in the middle region of the tail, just before it
crosses the nose.
A sagittal section of the entire embryo (except the tail) of this age
is shown in Figure 23a. In the head region the section is nearly
median, while the posterior part of the body is cut slightly to one
side of the middle line. At the tip of the now well-developed snout is
seen one of the nostrils (an), cut through the edge; its connection
with the complicated nasal chamber (n) is not here seen, nor is the
connection of the nasal chamber with the posterior nares (pn). The
pharynx (ph), is anteriorly connected with the exterior through the
mouth (m) and the nares, while posteriorly it opens into the
œsophagus (oe); the trachea (ta), though distinct from the
œsophagus, does not yet open into the pharynx. In the lower jaw
two masses of cartilage are seen, one near the symphysis (mk) and
one near the wall of the trachea, doubtless the rudiment of the
hyoid. The deep groove back of the Meckel’s cartilage (mk) marks
the tip of the developing tongue, which here forms the thick mass
on the floor of the mouth cavity. Dorsal to the pharynx a mass of
cartilage (se) is developing in the sphenethmoid region. This being a
median section, the ventricles of the fore- (fb), mid- (mb), and
hindbrain (hb) are seen as large cavities, while the cerebral
hemispheres (ch) appear nearly solid, only a small portion of a
lateral ventricle showing. The paraphysis (epi) is cut a little to one
side of the middle and so does not show its connection with the
brain. At the base of the brain the infundibulum (in) is seen as an
elongated cavity whose ventral wall is in close contact with a group
of small, darkly staining alveoli (p), the pituitary body. Extending
posteriorly from the pituitary body is a gradually thickening mass of
cartilage (bp), which surrounds the anterior end of the notochord
(nt) and may be called the basilar plate. In its anterior region, where
the section is nearly median, the spinal column shows its canal, with
the enclosed spinal cord, while toward the posterior end of the
figure the vertebræ are cut to one side of the middle line, and hence
show the neural arches (na) with the alternating spinal ganglia (sg).
Near the posterior end of the figure the pelvic girdle (pl) is seen. The
largest organ of the embryo, as seen in this section, is the heart, of
which the ventricle (vn) seems to be closely surrounded, both in
front and behind, by the auricles (au). The liver (li) is the large,
reticular mass back of the heart. Dorsal and anterior to the liver is
the lung (lu), now of considerable size and development. The
enteron is cut in several places (oe, i) and its walls are beginning to
show some differentiation, though this cannot be seen under the
magnification here used. One of the Wolffian bodies is seen as a
huge mass of tubules (wt) extending from the pelvic region, where
the mass is greatest, to the region of the lungs. The Wolffian tubules
stain darkly and the whole structure forms a very striking feature of
the section. Dorsal to the posterior end of the Wolffian body is a
small, oval mass of very fine tubules (k), which do not stain so
darkly as do the Wolffian tubules; this mass is apparently the
beginning of the permanent kidney, the metanephros. Its tubules,
though their origin has not been determined, seem to be entirely
distinct from the tubules of the Wolffian body.
A single vertical section through the anterior part of the head of
an embryo of this age has been represented in Figure 23b. On the
right side the plane of the section cut through the lens of the eye
(ln); on the left side the section was anterior to the lens. The upper
(ul) and lower (ll) eyelids are more evident here than in the surface
view. Owing to the hardness of the lens, its supporting structures
were torn away in sectioning. The vitreous humor is not represented
in the figure. The superior (ur) and inferior (lr) recti muscles are well
shown on the right side; they are attached to the median part of a
Y-shaped mass of cartilage (se), which may be termed the
sphenethmoidal cartilage. Between the branches of this Y-shaped
cartilage the anterior ends of the cerebral hemispheres (ch)—better
called, perhaps, the olfactory lobes—are seen. Between the lower
end of the sphenethmoidal cartilage and a dorsally evaginated part
of the pharynx are two small openings (pn); when traced forward
these tubes are found to open into the convoluted nasal chamber,
while a short distance posterior to the plane of this figure they unite
with each other and open almost immediately into the pharynx. The
rather complicated structures of the nasal passages of the alligator
have been described by the writer in another paper (57). In the
lower jaw the cartilage (mk) is seen on either side and several bands
of muscle are developing in the mesoblast. Two deep grooves give
form to what may be called the rudimentary tongue (tn). In both
jaws one or two tooth rudiments (to) may be distinguished as small
invaginations of ectoderm.
STAGE XXI
Figure 24 (Plate XXVII.)
In this stage the curvature of the body and tail is less marked than
was seen in the last surface view. The body has increased greatly in
size, so that the size of the head is relatively not so great. The size
of the eye in relation to that of the head is much diminished also.
The five anterior and four posterior digits are well formed, and their
claws are of considerable size, though of course not present on all
the digits. The outlines of scales may be traced from the tip of the
tail to the skull; they are especially prominent along the dorsal
profile. The skin is just beginning to show traces of pigment, which
is, however, not shown in the photograph. The umbilical stalk is seen
projecting with a loop of the intestine from the abdominal wall; this
is shown more clearly in the next stage. The embryo now begins to
exhibit some of the external characteristics of the adult alligator.
STAGE XXII
Figure 25 (Plate XXVIII.)
 This embryo needs no particular description. It has reached in its
external appearance practically the adult condition, although there is
still considerable yolk (not shown in the figure) to be absorbed, and
the embryo would not have hatched for many days. Pigmentation,
begun in the last stage, is now complete. The umbilical stalk is
clearly seen projecting from a large opening in the body wall. The
long loop of the intestine that extends down into the yolk sac is here
evident, and it is hard to understand how it can all be drawn up into
the body cavity when the umbilical stalk is withdrawn. No sharp
shell-tooth at the tip of the snout, such as is described by Voeltzkow
(78) in the crocodile, is here seen.
STAGE XXIII
Figure 26 (Plate XXVIII.)
This figure shows the relative sizes of the just-hatched alligator
and the egg from which it came. It also shows the position of the
young alligator in the egg, half of the shell having been removed for
that purpose. The blotchy appearance of the unopened egg is due
chiefly to stains produced by the decayed vegetation of the nest. At
hatching the young alligator is about 20 cm. long, nearly three times
the length of the egg; but the tail is so compressed that, though it
makes up about half of the length of the animal, it takes up very
little room in the egg.
SUMMARY
Owing to the fact that the embryo may undergo considerable
development before the egg is laid, and also to the unusual difficulty
of removing the very young embryos, the earlier stages of
development are very difficult to obtain.
The mesoderm seems to be derived chiefly by proliferation from
the entoderm, in which way all of that anterior to the blastopore
arises. Posterior to the blastopore the mesoderm is proliferated from
the lower side of the ectoderm in the usual way. No distinction can
be made between the mesoderm derived from the ectoderm and
that derived from the entoderm.
The ectoderm shows during the earlier stages a very great
increase in thickness along the median longitudinal axis of the
embryo.
The notochord is apparently of entodermal origin, though in the
posterior regions, where the germ layers are continuous with each
other, it is difficult to decide with certainty.
The medullary folds have a curious origin, difficult to explain
without the use of figures. They are continuous posteriorly with the
primitive streak, so that it is impossible to tell where the medullary
groove ends and the primitive groove begins, unless the dorsal
opening of the blastopore be taken as the dividing point.
The amnion develops rapidly, and entirely from the anterior end.
The blastopore or neurenteric canal is a very distinct feature of all
the earlier stages up to about the time of closure of the medullary
canal.
Preceding the ordinary cranial flexure there is a sort of temporary
bending of the head region, due apparently to the formation of the
head-fold.
During the earlier stages of development the anterior end of the
embryo is pushed under the surface of the blastoderm, and is hence
not seen from above.
Body torsion is not so definite in direction as in the chick, some
embryos lying on the right side, others on the left.
Of the gill clefts, three clearly open to the exterior and probably a
fourth also. A probable fifth cleft was seen in sections and in one
surface view.
The first trace of the urinary system is seen as a dorsally
projecting, solid ridge of mesoblast in the middle region of the
embryo, which ridge soon becomes hollowed out to form the
Wolffian duct.
The origin of the hypophysis and paraphysis is clearly seen; the
latter projects backward.
No connection can be seen between the first rudiments of the
sympathetic nerves and the central nervous system.
 The lumen of the œsophagus is for a time obliterated as in other
forms.
The choroid fissure is a very transitory but well-marked feature of
the eye.

LETTERING FOR ALL FIGURES ON PLATES VI.-

XXVIII.
a, head-fold of amnion.
aa, anterior appendage.
ac, anterior cardinal vein.
al, allantois.
an, anterior nares.
ao, aorta.
aop, area opaca.
ap, area pellucida.
ar, aortic arch.
au, auricle.
b, bulbus arteriosus.
bc, body cavity.
blp, blastopore.
bp, basilar plate.
bv, blood-vessel.
c, centrum of vertebra.
ca, caudal artery.
ch, cerebral hemisphere.
cl, cloaca.
cn, cranial nerve.
cp, posterior choroid plexus.
cv, cardinal veins.
dc, ductus Cuvieri.
e, eye.
ec, ectoderm.
ec′, thickening of ectoderm,
en, entoderm.
en′, endocardium.
ent, enteron.
ep, epidermal layer of ectoderm.
epi, paraphysis.
es, embryonic shield.
f, fronto-nasal process.
fb, forebrain.
fg, foregut.
g¹⁻⁵, gill clefts.
gf¹⁻⁶, gill folds.
gl, glomerulus.
h, head-fold.
hb, hindbrain.
ht, heart.
i, intestine.
i′, stomach.
in, infundibulum.
ir, iris.
it, iter.
k, kidney (metanephros).
l, remains of groove between secondary folds.
la, larynx (cartilages of).
li, liver.
ll, lower lid of eye.
ln, lens.
lr, inferior rectus muscle of eye.
lu, lungs.
lv, lens vesicle.
m, mouth.
ma, manus.
mb, midbrain.
mc, medullary canal.
me′, tip end of medullary canal.
md, mandibular fold.
mes, mesoderm.
mes′, myocardium.
mf, medullary fold.
mg, medullary groove.
mk, Meckel’s cartilage.
mp, muscle plate.
ms, mesentery.
mv, meatus venosus.
mx, maxillary fold.
myc, myocœl.
n, nasal invagination or cavity.
na, neural arch of vertebra.
nc, neurenteric canal.
nl, nervous layer of ectoderm.
nt, notochord.
o, ear vesicle.
oc, optic cup.
oe, œsophagus.
on, optic nerve.
os, optic stalk.
ov, optic vesicle.
p, pituitary body.
pa, posterior appendage.
pan, pancreas.
pc, posterior cardinal vein.
pe, pes.
pg, primitive groove.
ph, pharynx.
pl, pelvis.
pn, posterior nares.
pr, pericardial cavity.
ps, primitive streak.
pt, pecten.
rt, retina.
s, somites.
sc, spinal cord.
se, sphenethmoid cartilage.
sf, secondary fold.
sg, spinal ganglion.
sm, splanchnic mesoblast.
sn, spinal nerve.
so, somatic mesoblast.
st, stomodæum.
sy, sympathetic nervous system.
t, tail.
ta, trachea.
tg, thyroid gland.
th, thickening and posterior limit of sf.
tn, tongue.
to, tooth anlage.
tr, truncus arteriosus.
tv, third ventricle of brain.
tv′, third ventricle of brain.
u, umbilical stalk.
ul, upper lid of eye.
ur, superior rectus muscle of eye.
v′-″-‴, first, second, and third cerebral vesicles.
 va, vascular area.
vm, vitelline membrane.
vn, ventricle of heart.
vv, vitelline blood-vessels.
wd, Wolffian duct.
wdo, opening of Wolffian duct.
wr, Wolffian ridge.
wt, Wolffian tubules.
y, yolk.

EXPLANATION OF FIGURES 1-26 ON PLATES VI.-

XXVIII.
All of the figures, with the exception of the photographs and those copied by
permission from S. F. Clarke, were drawn under a camera lucida.
The magnification of each figure, except those from Clarke, is indicated below.
The photographs were made by the author, and were enlarged for reproduction
by the photographic department of the Smithsonian Institution. The other surface
views were made, under the author’s direction, by Miss C. M. Reese.
With the exception of Stage III., all of the figures of any one stage are given the
same number, followed where necessary by a distinguishing letter, so that it is
possible to tell at a glance which section and surface views belong together. The
transverse sections are all cut in series from anterior to posterior.
Figure 1. Surface view of egg. × ²⁄₃.
1a. Egg with part of the shell removed to show the chalky band in the
shell membrane. × ²⁄₃.
Figures 2 and 2a. Dorsal and ventral views respectively of the blastoderm
before the formation of the notochord, medullary folds, etc.
After Clarke.
2b-2f. Transverse sections of an embryo of the age represented in
Figures 2 and 2a. × 43.
3 and 3a. Ventral and dorsal views respectively of an embryo a few
days older than that represented in Figures 2 and 2a. After
Clarke.
3b-3m. Transverse sections of an embryo of the age shown in Figures
3 and 3a. × 43.
Figures 3n and 3o. Two sagittal sections of an embryo of the same stage as
Figures 3 and 3a. × 43.
 4 and 4a. Dorsal and ventral views respectively of a slightly older
embryo than the one shown in Figures 3 and 3a. Figure 4a
shows only the head region. After Clarke.
5 and 5a. Dorsal and ventral views respectively of an embryo of
almost the same age as the preceding, to show the further
development of the medullary folds. After Clarke.
Figure 6. Dorsal view of an embryo only a day or two older than the
preceding. After Clarke.
Figures 6a-6i. A series of transverse sections of this stage. × 43.
Figures 7a-7h. A series of transverse sections of an embryo slightly older than
the one shown in Figures 4-6. × 43. (No surface view of this
stage is figured.)
8 and 8a. Dorsal and ventral views respectively of an embryo with
five pairs of mesoblastic somites. × 20. (Drawn by transmitted
light.)
8b and 8c. Two sagittal sections of an embryo of this stage. × 43.
Figures 8d-8j. A series of transverse sections of the embryo represented in
Figures 8 and 8a. × 43.
9a-9m. A series of transverse sections of an embryo somewhat more
advanced in development than the one represented in the last
series. × 43.
Figures 10 and 10a. Dorsal and ventral views respectively of an embryo with
eight pairs of mesoblastic somites. × 20. (Drawn chiefly by
transmitted light.)
Figure 11. Dorsal view of an embryo with fourteen pairs of mesoblastic
somites. The area pellucida and the developing vascular area
are shown, the latter having a mottled appearance. The
pushing of the head under the blastoderm is also shown. × 20.
(Drawn chiefly by transmitted light.)
Figures 11a-11k. A series of transverse sections of an embryo of this stage. ×
43.
Figure 12. Dorsal view of an embryo with about seventeen pairs of
mesoblastic somites. Part of the area pellucida is represented.
(Both transmitted and reflected light were used in making the
drawing.) × 13.
Figures 12a-12g. A series of transverse sections of an embryo of this stage. ×
43.
Figure 13. Surface view of an embryo with about twenty pairs of mesoblastic
somites. × (about) 15. (Drawn with both reflected and
transmitted light.)
Figures 13a-13f. A series of transverse sections of an embryo slightly more
developed than the one shown in Figure 13. × 20.
Figure 13g. A sagittal section of an embryo of about the age of the one
represented in Figure 13. × 20.
14. Head of an embryo with one pair of gill clefts; ventro-lateral view.
× 13.
15. Profile view of the head of an embryo with three pairs of gill
clefts. × 13.
Figures 15a-15e. A series of transverse sections of an embryo of about the
age of the one represented in Figure 15. × 20.
Figure 15f. A horizontal section through the anterior region of an embryo of
the age of that shown in Figure 15. × 20.
16. Surface view in profile of an embryo with four pairs of gill clefts.
× (about) 12.
Figures 16a-16f. A series of transverse sections of an embryo of the
approximate age of the one represented in Figure 16. × 20.
Figure 16g. A sagittal section of an embryo of the age (possibly slightly
younger) of the one represented in Figure 16. × 20.
17. Surface view in profile of an embryo at the time of origin of the
limbs. × (about) 5.
Figures 17a-17g. A series of transverse sections of an embryo of the age of
the one represented in Figure 17. × 7.
Figure 18. Surface view in profile of the head of an embryo slightly larger
than, though of about the same state of development as, the
one represented in Figure 17. Reproduced here chiefly to show
the gill clefts. × (about) 3.
19. Surface view of an embryo somewhat more developed than the
one just described. × (about) 3.
Figure 20. Surface view of an embryo older than the one represented in
Figure 19; with well-developed manus and pes. × (about) 5.
Figures 20a-20j. A series of transverse sections of an embryo of the age of
the one represented in Figure 20. × 7.
Figure 21. A photograph of a living embryo in the egg, showing the allantois,
yolk mass, etc. The embryo is somewhat more developed than
the one shown in Figure 20. × ²⁄₃.
22. A photograph of a still larger embryo, removed from the shell and
freed from the fetal membranes. × (about) 1.
23. A photograph of a still more advanced embryo, in which the digits
are quite evident and the scales are beginning to show. ×
 (about) 1.
23a. A sagittal section of an embryo of the age of the one
represented in Figure 23; the tail has not been shown in this
figure. × (about) 3.
23b. A vertical section through the head of an embryo of about the
size (perhaps slightly smaller) of the one shown in Figure 23. ×
(about) 3.
24. A photograph of an older embryo in which the pigmentation of
the scales is evident, though not shown in the figure. × (about)
1.
25. A photograph of an embryo in which the pigmentation and the
development of the body form are practically complete. The
allantois, unabsorbed yolk, etc., have been removed. × (about)
³⁄₄.
26. A photograph of a just-hatched alligator, of an alligator egg, and
of a young alligator in the egg just before hatching. × (about)
³⁄₇.
Plate VI. 1, 1a, The Egg; 2, 2a, Stage I.
Plate VII. 2b-2f, Stage I. 3, 3b, Stage II.