Get ebook downloads in full at ebookmeta.

com

Deep Learning Patterns and Practices 1st Edition

Andrew Ferlitsch

https://ebookmeta.com/product/deep-learning-patterns-and-
practices-1st-edition-andrew-ferlitsch/

OR CLICK BUTTON

DOWNLOAD NOW

Explore and download more ebook at https://ebookmeta.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Grokking Deep Learning MEAP Version 7 Andrew W Trask

https://ebookmeta.com/product/grokking-deep-learning-meap-
version-7-andrew-w-trask/

ebookmeta.com

Deep Learning in Data Analytics: Recent Techniques,

Practices and Applications 1st Edition Debi Prasanna
Acharjya
https://ebookmeta.com/product/deep-learning-in-data-analytics-recent-
techniques-practices-and-applications-1st-edition-debi-prasanna-
acharjya/
ebookmeta.com

Fluent C: Principles, Practices, and Patterns 1st Edition

Christopher Preschern

https://ebookmeta.com/product/fluent-c-principles-practices-and-
patterns-1st-edition-christopher-preschern/

ebookmeta.com

Third Person Singular A Mars Bahr Mystery Mars Bahr

Mysteries Book 1 K J Erickson

https://ebookmeta.com/product/third-person-singular-a-mars-bahr-
mystery-mars-bahr-mysteries-book-1-k-j-erickson/

ebookmeta.com
Quarkus in Action (MEAP V03) Martin Štefanko

https://ebookmeta.com/product/quarkus-in-action-meap-v03-martin-
stefanko/

ebookmeta.com

A Bluestocking for a Baron A Duke s Daughters The Elbury

Bouquet 3 1st Edition Arietta Richmond

https://ebookmeta.com/product/a-bluestocking-for-a-baron-a-duke-s-
daughters-the-elbury-bouquet-3-1st-edition-arietta-richmond/

ebookmeta.com

The Next Generation of Biomedical and Behavioral Sciences

Researchers Breaking Through 1st Edition And Medicine
Engineering National Academies Of Sciences
https://ebookmeta.com/product/the-next-generation-of-biomedical-and-
behavioral-sciences-researchers-breaking-through-1st-edition-and-
medicine-engineering-national-academies-of-sciences/
ebookmeta.com

Frontier Midwife Amanda Kate Bridges

https://ebookmeta.com/product/frontier-midwife-amanda-kate-bridges/

ebookmeta.com

Calculus on Heisenberg Manifolds AM 119 1st Edition

Richard Beals

https://ebookmeta.com/product/calculus-on-heisenberg-manifolds-
am-119-1st-edition-richard-beals/

ebookmeta.com
Collective Bargaining Among Photo Engravers in
Philadelphia Ordinary Methods Applied to an Occupation
Which Is Both an Art and a Manual Trade Charles Leese
https://ebookmeta.com/product/collective-bargaining-among-photo-
engravers-in-philadelphia-ordinary-methods-applied-to-an-occupation-
which-is-both-an-art-and-a-manual-trade-charles-leese/
ebookmeta.com
Andrew Ferlitsch

MANNING
Deep Learning Patterns and Practices
 Deep Learning
Patterns and Practices
ANDREW FERLITSCH

MANNING
SHELTER ISLAND
For online information and ordering of this and other Manning books, please visit
www.manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
20 Baldwin Road
PO Box 761
Shelter Island, NY 11964
Email: [email protected]

©2021 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in

any form or by means electronic, mechanical, photocopying, or otherwise, without prior written
permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are
claimed as trademarks. Where those designations appear in the book, and Manning
Publications was aware of a trademark claim, the designations have been printed in initial caps
or all caps.

Recognizing the importance of preserving what has been written, it is Manning’s policy to have
the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books are
printed on paper that is at least 15 percent recycled and processed without the use of elemental
chlorine.

The author and publisher have made every effort to ensure that the information in this book
was correct at press time. The author and publisher do not assume and hereby disclaim any
liability to any party for any loss, damage, or disruption caused by errors or omissions, whether
such errors or omissions result from negligence, accident, or any other cause, or from an usage
of the information herein.

Manning Publications Co. Development editor: Frances Lefkowitz

20 Baldwin Road Technical development editor: Al Krinker
PO Box 761 Review editor: Aleksandar Dragosavljević
Shelter Island, NY 11964 Production editor: Deirdre S. Hiam
Copy editor: Sharon Wilkey
Proofreader: Keri Hales
Technical proofreader: Karsten Strobaek
Typesetter: Gordan Salinovic
Cover designer: Marija Tudor

ISBN 9781617298264
Printed in the United States of America
 brief contents
PART 1 DEEP LEARNING FUNDAMENTALS ..........................................1
1 ■ Designing modern machine learning 3
2 ■ Deep neural networks 19
3 ■ Convolutional and residual neural networks 43
4 ■ Training fundamentals 70

PART 2 BASIC DESIGN PATTERN ..................................................119

5 ■ Procedural design pattern 121
6 ■ Wide convolutional neural networks 150
7 ■ Alternative connectivity patterns 183
8 ■ Mobile convolutional neural networks 209
9 ■ Autoencoders 250

PART 3 WORKING WITH PIPELINES ..............................................273

10 ■ Hyperparameter tuning 275
11 ■ Transfer learning 302
12 ■ Data distributions 325
13 ■ Data pipeline 346
14 ■ Training and deployment pipeline 386

v
 contents
preface xiii
acknowledgments xiv
about this book xv
about the author xx
about the cover illustration xxii

PART 1 DEEP LEARNING FUNDAMENTALS ................................1

1 Designing modern machine learning

1.1 A focus on adaptability 4
3

Computer vision leading the way 5 ■

Beyond computer vision:
NLP, NLU, structured data 6
1.2 The evolution in machine learning approaches 7
Classical AI vs. narrow AI 7 ■
Next steps in computer
learning 10
1.3 The benefits of design patterns 16

vii
viii CONTENTS

2 Deep neural networks

2.1 Neural network basics
19
20
Input layer 20 Deep neural networks 22 Feed-forward
■ ■

networks 23 Sequential API method 23 Functional API

■ ■

method 24 Input shape vs. input layer 24 Dense layer 25

■ ■

Activation functions 27 Shorthand syntax 30 Improving

■ ■

accuracy with an optimizer 30

2.2 DNN binary classifier 31
2.3 DNN multiclass classifier 34
2.4 DNN multilabel multiclass classifier 35
2.5 Simple image classifier 38
Flattening 38 ■
Overfitting and dropout 39

3 Convolutional and residual neural networks

3.1 Convolutional neural networks 44
43

Why we use a CNN over a DNN for image models 45

Downsampling (resizing) 45 Feature detection 46 ■

Pooling 48 Flattening 49
■

3.2 The ConvNet design for a CNN 50

3.3 VGG networks 54
3.4 ResNet networks 57
Architecture 58 ■
Batch normalization 63 ■
ResNet50 64

4 Training fundamentals
4.1 Forward feeding and backward propagation
70
71
Feeding 71 ■
Backward propagation 72
4.2 Dataset splitting 74
Training and test sets 74 ■
One-hot encoding 75
4.3 Data normalization 78
Normalization 78 ■
Standardization 80
4.4 Validation and overfitting 80
Validation 80 ■
Loss monitoring 84 ■
Going deeper with
layers 84
4.5 Convergence 86
4.6 Checkpointing and early stopping 88
Checkpointing 88 ■
Early stopping 90
4.7 Hyperparameters 91
 CONTENTS ix

Epochs 91 ■
Steps 92 ■
Batch size 94 ■
Learning rate 95
4.8 Invariance 97
Translational invariance 98 Scale invariance 104 ■

TF.Keras ImageDataGenerator 106

4.9 Raw (disk) datasets 108
Directory structure 108 CSV file 110 ■ ■
JSON file 111
Reading images 111 Resizing 114 ■

4.10 Model save/restore 116

Save 116 ■
Restore 116

PART 2 BASIC DESIGN PATTERN ........................................119

5 Procedural design pattern

5.1 Basic neural network architecture
121
123
5.2 Stem component 125
VGG 125 ■
ResNet 126 ■
ResNeXt 131 ■
Xception 132
5.3 Pre-stem 133
5.4 Learner component 134
ResNet 136 ■
DenseNet 139
5.5 Task component 141
ResNet 141 ■
Multilayer output 142 ■
SqueezeNet 145
5.6 Beyond computer vision: NLP 146
Natural-language understanding 146 ■
Transformer
architecture 148

6 Wide convolutional neural networks

6.1 Inception v1 152
150

Naive inception module 152 Inception v1 module 154 ■

Stem 157 Learner 157 Auxiliary classifiers 158

■ ■

Classifier 160
6.2 Inception v2: Factoring convolutions 161
6.3 Inception v3: Architecture redesign 163
Inception groups and blocks 165 Normal convolution ■
168
Spatial separable convolution 169 Stem redesign and ■

implementation 169 Auxiliary classifier 171

■
x CONTENTS

6.4 ResNeXt: Wide residual neural networks 172

ResNeXt block 172 ■
ResNeXt architecture 175
6.5 Wide residual network 176
WRN-50-2 architecture 177 ■
Wide residual block 178
6.6 Beyond computer vision: Structured data 179

7 Alternative connectivity patterns

7.1 DenseNet: Densely connected convolutional neural
183

network 185
Dense group 185 Dense block 188 DenseNet macro-
■ ■

architecture 190 Dense transition block 191

■

7.2 Xception: Extreme Inception 192

Xception architecture 193 Entry flow of Xception 195
■

Middle flow of Xception 198 Exit flow of Xception 200

■

Depthwise separable convolution 202 Depthwise ■

convolution 202 Pointwise convolution 203

■

7.3 SE-Net: Squeeze and excitation 203

Architecture of SE-Net 204 ■
Group and block of SE-Net 205
SE link 206

8 Mobile convolutional neural networks

8.1 MobileNet v1 210
209

Architecture 211 Width multiplier 212 Resolution

■ ■

multiplier 212 Stem 214 Learner 216 Classifier

■ ■ ■
218
8.2 MobileNet v2 219
Architecture 219 ■
Stem 220 ■
Learner 221 ■
Classifier 225
8.3 SqueezeNet 226
Architecture 227 ■
Stem 228 ■
Learner 228 ■
Classifier 231
Bypass connections 233
8.4 ShuffleNet v1 237
Architecture 237 ■
Stem 238 ■
Learner 238
8.5 Deployment 246
Quantization 246 ■
TF Lite conversion and prediction 247

9 Autoencoders
9.1
250
Deep neural network autoencoders 251
Autoencoder architecture 251 ■
Encoder 252 ■
Decoder 253
Training 254
 CONTENTS xi

9.2 Convolutional autoencoders 255

Architecture 256 ■
Encoder 257 ■
Decoder 258
9.3 Sparse autoencoders 259
9.4 Denoising autoencoders 260
9.5 Super-resolution 261
Pre-upsampling SR 261 ■
Post-upsampling SR 264
9.6 Pretext tasks 267
9.7 Beyond computer vision: sequence to sequence 270

PART 3 WORKING WITH PIPELINES ....................................273

10 Hyperparameter tuning
10.1 Weight initialization
275
277
Weight distributions 277 ■
Lottery hypothesis 278 ■
Warm-up
(numerical stability) 280
10.2 Hyperparameter search fundamentals 283
Manual method for hyperparameter search 284 Grid ■

search 285 Random search 286 KerasTuner 289

■ ■

10.3 Learning rate scheduler 291

Keras decay parameter 292 ■
Keras learning rate scheduler 292
Ramp 293 Constant step
■
294 Cosine annealing 295
■

10.4 Regularization 297

Weight regularization 297 ■
Label smoothing 298
10.5 Beyond computer vision 300

11 Transfer learning
11.1
302
TF.Keras prebuilt models 304
Base model 305 Pretrained ImageNet models for
■

prediction 306 New classifier 307

■

11.2 TF Hub prebuilt models 311

Using TF Hub pretrained models 312 ■
New classifier 314
11.3 Transfer learning between domains 315
Similar tasks 315 Distinct tasks 317 Domain-specific
■ ■

weights 319 Domain transfer weight initialization 321

■

Negative transfer 323

11.4 Beyond computer vision 323
xii CONTENTS

12 Data distributions
12.1 Distribution types
325
326
Population distribution 327 Sampling distribution ■
328
Subpopulation distribution 329
12.2 Out of distribution 330
The MNIST curated dataset 330 Setting up the ■

environment 331 The challenge (“in the wild”) 331

■

Training as a DNN 332 Training as a CNN 339 Image■ ■

augmentation 342 Final test 344 ■

13 Data pipeline
13.1
346
Data formats and storage 348
Compressed and raw-image formats 348 ■
HDF5 format 352
DICOM format 356 TFRecord format ■
358
13.2 Data feeding 364
NumPy 364 ■
TFRecord 366
13.3 Data preprocessing 368
Preprocessing with a pre-stem 368 ■
Preprocessing with TF
Extended 376
13.4 Data augmentation 381
Invariance 381 ■
Augmentation with tf.data 383 ■
Pre-stem 384

14 Training and deployment pipeline

14.1 Model feeding 388
386

Model feeding with tf.data.Dataset 392 Distributed feeding with ■

tf.Strategy 395 Model feeding with TFX 397

■

14.2 Training schedulers 404

Pipeline versioning 406 ■
Metadata 408 ■
History 410
14.3 Model evaluations 412
Candidate vs. blessed model 412 ■
TFX evaluation 416
14.4 Serving predictions 419
On-demand (live) serving 419 Batch prediction 422 TFX ■ ■

pipeline components for deployment 423 A/B testing 425 ■

Load balancing 427 Continuous evaluation 429

■

14.5 Evolution in production pipeline design 430

Machine learning as a pipeline 430 Machine learning as a CI/CD ■

production process 431 Model amalgamation in production 431

■

index 435
 preface
As a Googler, one of my duties is to educate software engineers on how to use machine
learning. I already had experience creating online tutorials, meetups, conference pre-
sentations, training workshops, and coursework for private coding schools and univer-
sity graduate studies, but I am always looking for new ways to effectively teach.
Prior to Google, I worked in Japanese IT as a principal research scientist for 20
years—all without deep learning. Almost everything I see today, we were doing in
innovation labs 15 years ago; the difference is we needed a room full of scientists and
a vast budget. It’s incredible how things have so rapidly changed as a result of deep
learning.
Back in the late 2000s, I was working with small structured datasets with geospatial
data from national and international sources all over the world. Coworkers called me
a data scientist, but nobody knew what a data scientist really was. Then came big data,
and I didn’t know the big data tools and frameworks, and suddenly I wasn’t a data sci-
entist. What? I had to scramble and learn the tools and concepts behind big data and
once again I was a data scientist.
Then emerged machine learning on big datasets, like linear/logistic regression
and CART analysis, and I hadn’t used statistics since graduate school decades ago, and
once again I was not a data scientist. What? I had to scramble to learn statistics all over
again, and once again I was a data scientist. Then came deep learning, and I didn’t
know the theory and frameworks for neural networks and suddenly I wasn’t a data sci-
entist. What? I scrambled again and learned deep learning theory and other deep
learning frameworks. And once again, I am a data scientist.

xiii
 acknowledgments
I would like to thank all those at Manning who helped throughout this process. Fran-
ces Lefkowitz, my development editor; Deirdre Hiam, my project editor; Sharon
Wilkey, my copyeditor; Keri Hales, my proofreader; and Aleksandar Dragosavljević,
my reviewing editor.
To all the reviewers: Ariel Gamino, Arne Peter Raulf, Barry Siegel, Brian R. Gaines,
Christopher Marshall, Curtis Bates, Eros Pedrini, Hilde Van Gysel, Ishan Khurana, Jen
Lee, Karthikeyarajan Rajendran, Michael Kareev, Muhammad Sohaib Arif, Nick
Vazquez, Ninoslav Cerkez, Oliver Korten, Piyush Mehta, Richard Tobias, Romit Sing-
hai, Sayak Paul, Sergio Govoni, Simone Sguazza, Udendran Mudaliyar, Vishwesh Ravi
Shrimali, and Viton Vitanis, your suggestions helped make this a better book.
To all Google Cloud AI staff who have shared their personal and customer insights,
your insights helped the book cover a broader audience.

xiv
Other documents randomly have
different content
presence in the Synagogue is of comparatively modern date. The
ner tamid of the Sanctuary, however, is explained to be a lamp
burning “from evening to morning” (Exod. xxvii. 21). A golden
candlestick standing in the Sanctuary against the south side, with its
seven branches arranged from east to west, served this purpose.
According to Tradition it was the second branch, counting from east
to west, which really burnt continually; but this was not considered as
implied in the term ner tamid, which only meant “a continual lamp” in
the sense of a lamp that burns regularly every night.

The ner tamid in the Synagogue, which burns continually day and
night, is not mentioned by any of the earlier Rabbinical authors. It
has been introduced as a symbol representing our conviction that
from the Synagogue shall continually come forth the light of
instruction, the light of comfort and blessing, and the light of love and
peace.

In the Synagogue women are separated from men. There was also
in the Temple an ‫‏עזרת נשים‬‎“court of the women,” distinguished from
the ‫‏עזרת אנשים‬‎“court of the men,” to which women had no access.
During the Feast of Tabernacles, when the great rejoicings in the
Temple attracted a large assembly, special care was taken (‫‏תקון גדול‬
‫היה שם‬‎) that the separation of the sexes should be maintained
(Mishnah, Succah v. 2; and [427]Talm. B., Succah 51b). This
precedent has been followed in the Synagogue, and has been
accepted as law up to this day.

Reservedness and modesty (‫‏צניעות‬‎) have always been the pride and
ornament of Jewish women, both in their homes and in the
Synagogue; hence also their taking a silent part in the public
devotion is an honour to them, and by no means derogatory.

In addition to the above-mentioned points, a Synagogue ought to be

distinguished by the greatest possible simplicity, by the absence of
all kinds of images, portraits, or statues representing living beings,
whether real or imaginary. The Jewish religion is void of every visible
symbol; and the so-called magen-david (the double triangle) is
probably not of Jewish origin, and has no connection with our holy
religion. It is not a symbol of this kind, but some inscription of a
passage from the Scriptures that in most houses of worship reminds
us of the sacredness of the place. We enter it with due reverence,
manifesting it outwardly, in our peculiar traditional manner, by
keeping the head covered. It is our ancient custom to cover the head
when engaged in prayers, in reading the Bible or Talmud and their
commentaries. This outward sign serves to remind us that not only
our Service but even our literature is something holy, and its study a
religious act (‫‏מצוה‬‎). 119

Before we proceed to describe the details of our [428]ritual, we

mention one important point in which the present Synagogal Service
differs from the ancient Service in the Temple. From what we are told
in the Scriptures and in the Talmud, we learn that instrumental music
was an essential element in the Service, and that King David and his
successors paid great attention to it, whilst, with a few exceptions, it
is almost entirely absent from our Synagogues. The principal reason
why instrumental music is excluded from the Synagogue is its
prohibition on Sabbaths and Holy-days by Rabbinical law (Babyl. T.,
Erubin 104a). This prohibition, like many other enactments, did not
apply to the Temple Service; for the sacrificial laws had to be
obeyed, irrespective of the fact that they involved acts which, if
performed apart from the Temple Service, would constitute a breach
of the Sabbath laws. Apart from the Temple Service the Sabbath
laws remained in full force for the priests as well as for the general
public.

There were also other considerations that helped to keep

instrumental music out of the Synagogue Service. Its absence,
though not directly a sign of mourning, served to preserve the
memory of the destruction of the Temple, and to strengthen our
longing for its restoration. It is also urged that the introduction of
instrumental music into the Service would not satisfy any real want of
Jewish worshippers, but would merely be a concession to the desire
to assimilate our Divine Service to that of our non-Jewish
neighbours, contrary to the prohibition of chukkoth haggoyim
contained in the words, “Ye shall not walk in their statutes” (Lev. xviii.
3), i.e., in the statutes of the Gentiles. But, on [429]the other hand, it
has been argued that the feeling once expressed by the nation in the
words “This is my God, and I will worship him in a beautiful manner”
(Exod. xv. 2), still animates us. It is said that it is our duty to make
our Service as beautiful and as attractive as possible. This argument
deserves consideration, and might even outweigh some of the
above-mentioned arguments against the introduction of music into
our Service, if we were sure of the result of such introduction. But
this is by no means the case, for the experiment, where tried, has
not been successful if judged by the most practical test. The number
of worshippers has not been increased, and discontent has not been
removed. Whether the devotion of the worshippers has been
improved, refined, or intensified by music is a question that cannot
be answered with certainty. Even if the answer were satisfactory, it
could only apply to the introduction of instrumental music into our
Service on week-days, on Friday evening before the commencement
of Sabbath, but not on Sabbaths and Holy-days.

The Ritual.

In the Bible there is no indication of a fixed ritual; there are, however,

a few instances of forms of prayer prescribed for certain occasions.
There is the priests’ blessing (Num. vi. 24–26); the thanksgiving on
bringing the first-fruit offering to the Temple (Deut. xxvi. 3–10); prayer
on distributing the tithes which accumulated in three years (ibid. 13–
15). David (Ps. lv. 18) says, “Evening, and morning, and at
[430]noonday do I pray;” Daniel “kneeled upon his knees three times
a day and prayed, and gave thanks before his God, as he did
aforetime” (Dan. vi. 11); but nothing is said about the form and the
contents of these prayers. The Mishnah first speaks of certain fixed
forms of prayer: the “Eighteen” (‫‏שמונה עשרה‬‎), the reading of Shema
(‫‏קריאת שמע‬‎), and Benedictions (‫‏ברכות‬‎). The composition of the
tefillah, “Prayer” par excellence, is attributed to the Men of the Great
Synagogue (‫‏אנשי כנסת הגדולה‬‎), but only in its outlines. The number of
the paragraphs, the theme of each paragraph, and the formula by
which it is concluded may then have been fixed, the rest being left to
be filled up by each supplicant according to his capacity. It was but
natural that prayers uttered repeatedly by men eminent for their piety
should be eagerly copied by others, and gradually become, to some
extent at least, fixed forms of prayer. The tefillah, however, in the
time of the Mishnah was by no means identical with the tefillah of the
Men of the Great Synagogue. The destruction of the Temple
necessitated several changes; e.g., the prayers for the welfare of
Jerusalem, for the prosperity of Israel and of the Holy Land, and for
the acceptance of the Service in the Temple were altered in
accordance with the new state of affairs.

The Mishnah speaks of the tefillah as a well-known existing

institution; it seems that it was the regular prayer in the Synagogue
Service, and the discussion whether the tefillah should be repeated
every day in extenso or in an abbreviated form (Mishnah, Berachoth
iv. 3) refers probably to the prayer recited privatim (‫‏תפלת יחיד‬‎), and
not to the Service in the Synagogue. [431]The prescribed “Eighteen
Blessings” were the framework, into which each man was expected
to fit in his peculiar, individual supplications; whilst in the public
Service the tefillah remained uniform. In the days of Rabban Gamliel
of Jamnia, and with his sanction, an important addition was made by
Samuel: a prayer for the discomfiture of those who by slander,
denunciation, or other wicked means attempt to undermine the
existence of the Jewish religion and community (‫‏ברכת הצדוקים‬‎or
‫‏ברכת המינים‬‎). 120 In some congregations two other paragraphs (‫‏את‬
‫צמח‬‎and ‫‏ולירושלם‬‎) were at the same time combined into one, in order
to keep to the traditional “Eighteen Blessings.” 121

The reading of shema in the evening and in the morning, the three
sections constituting the shema, and the order of these sections, are
assumed in the Mishnah as fully established by law and usage. Only
a few regulations are discussed concerning the time and the mode of
the reading. There was this difference between the custom of the
Babylonian Jews and that of their brethren in Palestine, that the
latter omitted in the evening the passage referring to tsitsith. Later
on, however, the Palestine Jews conformed to the Babylonian
custom. Suggestions have been made [432]to substitute other Biblical
passages for shema, but they have been rejected. Several attempts
have been made to introduce, as an addition to the three sections of
shema, the reading of the Decalogue; the addition was disallowed,
lest people should be misled to think that the Ten Commandments
alone were to be observed, and that the other laws were not binding
(Babyl. T., Berachoth 12a). 122

The Benedictions which precede and follow the reading of shema

were fixed in the time of the Mishnah as regards number, order, and
form; but the contents were left unsettled for some time (Mishnah,
Berachoth i. 4); in the Gemara their wording is still a subject for
discussion. The same can be said with regard to the relative order of
shema and tefillah. For the Evening Service the tefillah seems to
have generally been considered as optional. As to Benedictions in
general, their obligatory character is assumed in the Mishnah as
admitted by all, and only their form seems to have been fixed by the
regulations mentioned in Berachoth vi.–ix.
The Mishnah (Megillah iii. 4–iv. 10) includes a number of regulations
concerning the reading of the Law, the Prophets, and the Book of
Esther. Detailed rules were laid down for the reader and the
translator (methurgeman), pointing out which passages should be
omitted in the translation, and which should be omitted [433]even in
the original. It seems that there was a regular, consecutive reading,
which was interrupted on extraordinary days by the reading of
passages referring to these days.

The ritual which was adopted for the priests in the Temple was an
abridged form of the ritual then in general use. It was as follows:
They commenced with a benediction—the first of those which
precede the shema (‫‏יוצר אור‬‎); then they read the Decalogue, shema
(the three paragraphs), and three further benedictions, ‫‏אמת ויציב‬‎,
‫‏עבודה‬‎(corresponding to ‫‏רצה‬‎in our prayer), and the blessing of the
priests (Mishnah, Tamid v. 1).

A special ritual is also mentioned in the Talmud (Mishnah, Taanith iv.

2) for the Maamadoth and the Fast-days. 123 There were four
Services daily, as on the Day of Atonement. The principal feature in
the Service of the Maamadoth was the reading of the first chapter of
Genesis.

At the conclusion of the Talmud (about 500 c.e.) the essential parts
of our present ritual were already in a settled state; the shema with
the benedictions preceding and following, the tefillah with its
variations for New-moon, Sabbath, and Holy-days, the reading from
the Law and the prophets, and Hallel. The Seder evening Service
was complete in its main parts. [434]Of the Benedictions (Berachoth)
on various occasions both form and contents were fixed, and the rule
was laid down by Rabbi Meir (Berachoth 40a) that he who uttered a
berachah in a form different from that fixed by our Sages has not
fulfilled his duty (or, according to Maimonides, Hilchoth Berachoth i.
5, is in error). Notwithstanding this rule, however, changes were
made; new benedictions were introduced and old ones
discontinued. 124—Kaddish and Kedushah seem to have formed part
of the Service; of the latter the Talmud mentions the name, of the
former the response: “May his great Name be praised” ‫‏יהא שמו הגדול‬
‫מבורך‬‎or ‫‏יהא שמיה רבא מ׳‬‎(Comp. Babyl. Talm., Berachoth 3a).

In the next period, that of the Geonim, we meet with the complete
Siddur, “Arrangement” or “Order” of Service for ordinary days, for
Sabbaths and Festivals, Benedictions for all occasions, and Piyyutim
as optional additions. Such a Siddur was arranged by the Gaon
Rabbenu Saadiah (892–942), and another by the Gaon Rabbenu
Amram (about 880). Henceforth the principal prayers underwent only
insignificant alterations. Of the next period the most important
Siddurim are those included in the Mishneh-torah, at the end of the
second book, and that contained in the Machzor Vitry. 125

The additional prayers and piyyutim, being optional, varied according

to the taste of each congregation and [435]its leaders; in course of
time these variations became permanent; the same was the case
with minor changes, especially in the less essential elements of the
Service, and thus the various Minhagim (Rites) of the various
congregations came into existence. The principal Minhagim of
importance for us are: the Polish, the Sephardic, the German, and
the Italian Rites. 126 In the following description of the Ritual only the
two rites adopted in the principal Synagogues of the Anglo-Jewish
congregations in England will be noted.

Prayers at Fixed Times.

Although we constantly enjoy the blessings of God, the very breath

we breathe being the gift of our Heavenly Father, yet certain seasons
of the day, of the week, of the month, and of the year have been
selected as especially fit for reminding us of God’s kindness, and
predisposing our heart to devotion. Thus in the day, morning, noon
and evening have been fixed for prayer; in the week, Sabbath; in the
month, New-moon; in the year, the Festivals.

We have three daily Services: Maaribh, “Evening prayer;”

Shacharith, “Morning prayer,” 127 and Minchah, “Afternoon prayer.”
On Sabbath, New-moon, and Festivals an “Additional prayer,” Musaf,
is inserted [436]between the Morning and the Afternoon Services, and
on the Day of Atonement, Neïlah, “Concluding Service,” is added
after Minchah.

The two most essential elements in these Services are: (1) the
Reading of Shema (‫‏קריאת שמע‬‎), in the Maaribh and the Shacharith;
(2) the Tefillah or Amidah, common to all the Services.

1. The Reading of Shema.

In obedience to the precept, “Thou shalt speak of them,” i.e., of “the

words which I command thee this day—when thou liest down and
when thou risest up,” three sections of the Law are read daily in the
morning and in the evening, viz., (1) Deut. vi. 4–9, beginning ‫‏שמע‬‎
“Hear;” (2) Ibid. xi. 13–21, beginning ‫‏והיה אם שמע תשמעו‬‎“And it shall
be if ye will diligently hearken;” (3) Num. xv. 37–41, beginning ‫‏ויאמר יי‬‎
“And the Lord said.” The first section teaches the Unity of God, and
our duty to love this One God with all our heart, to make His Word
the subject of our constant meditation, and to instil it into the heart of
the young.—The second section contains the lesson of reward and
punishment: that our success depends on our obedience to the Will
of God. This important truth must constantly be kept before our eyes
and before the eyes of our children.—The third section contains the
commandment of tsitsith, the object of which is to remind us of God’s
precepts: “Ye shall see it and remember all the commandments of
the Lord and do them, and that ye seek not after your own heart and
your own eyes, after which ye use to go astray, that [437]you
remember and do all my commandments, and be holy unto your
God.”

The reading of the shema is preceded by two berachoth: (1) ‫‏יוצר אור‬‎
Praise of the Creator for the regular sequence of day and night, light
and darkness; (2) ‫‏אהבה רבה‬‎or ‫‏אהבת עולם‬‎Praise of His goodness in
giving us the Torah, and prayer for His assistance in the study of the
Torah. The shema is followed by a berachah on the Redemption of
Israel; it contains a reflection on the last words of shema, “I am the
Lord your God,” an expression of our faith in the truth of these
words, which strengthen our belief in the future Redemption of
Israel. 128 In the Evening Service a second berachah follows,
beginning ‫‏השכיבנו‬‎, and containing a prayer for protection during the
night. 129

2. The Tefillah or Amidah.

The Tefillah, “Prayer” par excellence, is called Amidah (lit.,

“standing”), because the worshipper stands during the time he offers
it up. It is also called Shemoneh-esreh, “Eighteen,” because it
contains on most occasions eighteen (or nineteen, comp. p. 431)
paragraphs, each concluding with a benediction. [438]

The first three paragraphs contain praise of God’s goodness to us,

the descendants of the pious patriarchs (1), His omnipotence (2),
and His holiness (3).

The next thirteen paragraphs are petitions for our individual and
national well-being. For our individual well-being (4–9), namely, for
reason and wisdom (4), assistance in our endeavour to return to God
(5), forgiveness of our sins (6), deliverance from trouble (7), from
illness (8), and from want (9).—For our national well-being (10–15),
namely, for the gathering of those who are scattered (10), under
good leaders (11), protected from the evil designs of our foes (12),
for the support of the faithful (13), the rebuilding of Jerusalem (14),
and the advent of Messiah (15). The sixteenth paragraph is a prayer
that our petition may be accepted.—The last three paragraphs
include a petition for the re-establishment of Divine Service in the
Temple of Jerusalem (17), thanksgiving (18), and prayer for peace
and prosperity (19). When the prayer is finished we express the wish
that our lips, from which prayer to God has come forth, may not be
defiled by unworthy language.

On Sabbaths, Holy-days, and in every Musaph the thirteen middle

paragraphs are replaced by one in which reference is made to the
characteristic feature of the day; in the Musaph of New-year three
berachoth (p. 404) are substituted for the thirteen middle berachoth
of the ordinary tefillah. The thirteen paragraphs have been
eliminated in order that we should not be reminded on Sabbath and
Holy-days of our failings, wants, and troubles; that those seasons
should be [439]marked by a happier and more cheerful mood than
ordinary days (supra, p. 354).

There are two shorter forms of the tefillah for urgent occasions: the
one is a substitute for the “Eighteen,” in which the middle thirteen
paragraphs are contracted into one; it is called ‫‏הביננו‬‎(the first word
of this middle section), or ‫‏מעין שמונה עשרה‬‎“abstract of the
‘Eighteen.’ ” The other is a contraction of the Friday evening tefillah,
and is called ‫‏מעין שבע‬‎“abstract of the ‘Seven’ ” (scil., paragraphs
forming the tefillah), originally intended for those who were too late
for the full Service. 130

Each of the above Services ends with a prayer called after its initial
word alenu, “It is our duty.” In this prayer we thank God that we have
the privilege of proclaiming His Unity, and express our hope to see
the worship of the One God adopted by all mankind. It is omitted
between two Services following closely the one upon the other.

In addition to the above, the Service contains the following parts:—

(1.) ‫‏ברכות השחר‬‎“Blessings of the Morning,” forming the first part of

the Morning Service. It contains benedictions, reflections, and
prayers suggested by the change from night to day, from sleep to
wakefulness, from rest to activity.

(2.) Psalms.—Our Service contains various groups of psalms: chief

among them the mizmorim or pesuke dezimrah (“songs” or “verses
of song”), and shir shel yom (“song of the day”), in the Morning
Service. The former include Ps. cxlv. to cl., some other psalms, and
[440]the song of Moses (Exod. xv.). The latter correspond to the
songs of the Levites in the Temple, and consist of Ps. xxiv. (for
Sunday), xlviii. (for Monday), lxxxii. (for Tuesday), xciv. (for
Wednesday), lxxxi. (for Thursday), xciii. (for Friday), and xcii. (for
Saturday).—The repetition of Ps. cxlv. three times a day, twice during
Shacharith and once during Minchah, is an old minhag (Babyl. Talm.,
Berachoth, p. 4b).

(3.) Supplications (‫‏תחנונים‬‎) added in the Morning and the Afternoon

Services after the tefillah.

(4.) Readings from the Bible and Post-Biblical Sacred Literature,

such as Num. vi. 22 sqq. (priests’ blessing); Gen. xxii. (binding of
Isaac); Exod. xvi. (manna); Mishnah, Peah i. 1, and Babyl. T.,
Shabbath 127a, in the earlier part of the Morning Service; and words
of comfort (beginning ‫‏ובא לציון‬‎) from the Prophets after the
“Supplications.” Originally an exposition of the Written and the Oral
Law followed the “Supplications,” and concluded with Messianic
prophecies, recited in Hebrew and in the Chaldee Version.
(5.) Biblical and Post-Biblical passages referring to the Sacrificial
Service, in the Morning and the Afternoon Services.

In addition to the above Services, read either in the Synagogue or

privately at home, there is a special prayer read by us before retiring
to rest. The chief element in it is the first section of shema; hence the
name ‫‏קריאת שמע שעל המטה‬‎“Reading of shema before going to bed.”
Some psalms and supplications are generally added. [441]

Public Service, ‫‏תפלה בצבור‬‎. 131

The following points mark off the Public Service from the various
forms of private prayer:—

(1.) Kaddish, “Sanctification,” a prayer for the universal sanctification

of God’s name, which will distinguish the age of Messiah. In the
second part of the Kaddish we pray for the Messianic peace, and in
the last sentence express our hope that it may soon be granted.

Formerly the Kaddish concluded the Service; at present it is recited

at the end of the Service in its full form (‫‏קדיש שלם‬‎“the whole
Kaddish”); the first half (‫‏חצי קדיש‬‎“half-Kaddish”) has its place at the
end of a section of the Service—e.g., after the ‫‏פסוקי דזמרה‬‎in the
Morning Service; a third form is recited by mourners after ‫‏עלינו‬‎and
after special hymns or psalms; it is the whole Kaddish with the
omission of the sentence beginning ‫‏תתקבל‬‎. It is called ‫‏קדיש יתום‬‎
“Kaddish of the orphan,” and is intended to express the mourner’s
faith in God and his resignation to His Will.—Sometimes a special
Kaddish, called Kaddish dirabbanan, is recited after the reading of
some Talmudic or Midrashic passages. It is the same as Kaddish
shalem, except that the sentence beginning ‫‏תתקבל‬‎is replaced by a
prayer for the welfare of the scholars, the Rabbis, and their pupils.
[442]

(2.) Repetition of the tefillah by the Reader, with the addition of

Kedushah before the third paragraph, and the Priests’ Blessing
before the last paragraph, of the tefillah. The Kedushah,
“Proclamation of the Holiness of God,” is based on the visions of Isa.
vi. and of Ezek. iii., with citation of three verses, Isa. vi. 3, Ezek. iii.
12, and Ps. cxlvi. 10, in which the Holiness, Glory, and Kingdom of
God are proclaimed.

The Priests’ Blessing, originally spoken by priests, descendants of

Aaron, is now in most Synagogues included by the Reader in the
tefillah by way of quotation; only on Holy-days it is pronounced by
the priests.

(3.) ‫‏קריאת התורה‬‎“the Reading of the Law,” and the “Lessons from the
Prophets” (‫‏הפטרה‬‎), with the benedictions preceding and following
(supra, p. 348).

Occasional Prayers—Benedictions, ‫‏ברכות‬‎.

The feeling of our dependence on the goodness of God must

constantly be present to our mind. Whatever we enjoy, be it in the
form of eating or drinking, or some pleasing or remarkable sight, an
agreeable smell, a festivity on a joyful event, or the performance of a
Divine commandment (‫‏מצוה‬‎); whatever befall us, whether it be
pleasant or unpleasant—all this we consider as sent to us by the Will
of the Almighty, and we express our conviction by a suitable
berachah. The general rule is thus laid down by our Sages: It is
unlawful for man to enjoy anything on earth without previously
acknowledging by a berachah that God is the source whence the
enjoyment is derived. For [443]different cases different forms of
“blessings” have been fixed by our Sages. In some cases the
enjoyment is also followed by a prayer of thanksgiving, the most
important being the prayer after meals, called ‫‏ברכת המזון‬‎
“Benediction for food or Grace.” 132 In the Mishnah it is called “Three
Blessings” (‫‏שלש ברכות‬‎), because it consisted originally of three
paragraphs, each ending with a benediction. The three paragraphs
are the following: (1) ‫‏ברכת הזן‬‎(“Benediction commencing ‫‏הזן‬‎”), in
which we praise God’s providential care of all creatures. (2) ‫‏הודאה‬‎
“Thanksgiving,” or ‫‏ברכת הארץ‬‎(Benediction referring to Palestine). In
this paragraph we offer thanks for our individual sustenance, as well
as for our national gifts: Palestine, the Covenant, and the Law. (3)
Prayer for the restoration of Zion and the rebuilding of the Temple
(‫‏בנין ירושלם‬‎).—Subsequently a fourth paragraph was added (‫‏הטוב‬
‫והמטיב‬‎“who is good and causes His creatures to be good”) in
commemoration of the relief given to the Jews after the close of the
war with Hadrian. 133—On certain occasions, e.g., at a wedding
repast, suitable additions are made. Besides these, various
supplications have been added in later times.

There are various short forms of this ‫‏ברכת המזון‬‎; the shortest is that
for children, “Blessed be the Merciful, the Giver of this bread.” 134—
When three grown-up male persons or more have their meal
together, [444]a special introductory form is used, called ‫‏זמון‬‎
“summons to prayer,” one of the company acting as Reader, and the
rest forming the congregation.

Another form of thanksgiving is the “Abstract of the Three Blessings”

(‫‏מעין שלש‬‎), consisting of one paragraph which contains the whole of
the Grace in a contracted form, and is used after cake, wine, and the
like.
No restriction is enforced upon us if we desire on our part to give
expression to our feeling of gratitude and reverence toward the
Almighty in our own words on occasions not provided for in the
ancient forms of benedictions and prayers. In order, however, to
make a distinction between the forms of obligatory berachoth fixed
by our Sages and the optional ones introduced by ourselves, we do
not employ the words, “O Lord, our God, King of the Universe,”
which are essential in the former.
[Contents]

Notes.

1. On Page 424 sqq.

Among the different minhagim observed in the Synagogue the

following are noteworthy:—The head is kept covered, the hands
uncovered; gloves are generally taken off before the beginning of the
Service. It was customary to spread forth the hands during prayer,
and the phrase “spreading forth the hands” is used in the Bible in the
sense of “praying.” The priests still raise their hands when
pronouncing the blessing. Isaiah, rebuking those who prayed to God
without seeking purification from evil deeds, says, “And when you
spread forth your hands I will hide mine eyes from you; yea, when ye
make many prayers I will not hear; your hands are full of blood” (Isa.
i. 15). Following the example of the Psalmist, “I will wash my hands
in innocency, so will I compass thine altar” (Ps. xxvi. 6), we wash our
hands [445]before prayer, as a symbol of the duty of purifying our
conscience from guilt before approaching the Almighty with our
petitions. We thus uncover our hands as if to say, “The reproach of
Isaiah does not apply to us; we have tried to free our heart and our
hands from guilt.”

A custom frequently animadverted upon is the habit which many

Jews have adopted of swinging their bodies forward and backward
during prayer. We consider it a more decent way to stand or sit still
when communing with the Supreme Being. Both ways find support in
the Talmud (Babyl. T., Berachoth 31a, and Shabbath 10a); whilst the
one stands like “a servant in the presence of his master,” the other
gives way to his emotions and excitement. The Magen Abraham, on
chap. xlviii. 4, says: “He who follows the one example is right, and he
who follows the other is likewise right: all depends on the devotion of
the heart.” Rabbi Jehudah ha-Levi in his Cuzari (Book II. chap. xli.)
mentions and explains the custom of shaking during prayer. The
habit of accompanying the emotions of our heart by corresponding
motions of our body has produced the custom of raising the whole
body upwards when uttering the word “holy” in the kedushah.

During tefillah we remain standing in the same place; at the end,

when we have finished our petition, we retire slowly a few steps
backward; the same is done by the Reader during the last paragraph
of the kaddish. It is as if, our petition ended, we reverently withdrew
from the heavenly King who has given us audience during the
prayer.

We bend the knee, incline our head, and bow down on certain
occasions during the Service, but we do not kneel during prayer.—It
has perhaps been avoided as an idolatrous practice, with reference
to Judges vii. 5.

When the Ark is opened and the Sefer is taken out or put back, we
stand and show our respect for the Word of God in various ways.
Some bow the head; others, considering this as worship, kiss the
Sefer, or otherwise express their reverence.

The traditional way in which the kohanim proceed to bless the

people is this: they remove their shoes, as the priests did who
ministered in the Temple; water is then poured over their hands by
the Levites, the ablutions of the ancient priests being thus imitated to
some extent (see Exod. xxx. 20). It is a holy act, and is done in the
Synagogue generally in front of the Ark. [446]The priests ascend the
steps of the hechal and wait till called upon by the Reader to
pronounce the blessing. They turn toward the congregation, spread
forth their hands in the traditional manner, and cover head and face
with the talith, in order not to be disturbed in their devotion by the
sight of the congregation before them; the Reader dictates the words
of the benediction to them to guard against any mistakes being
made by them. The congregation, giving special importance to each
word, add Biblical quotations and special supplications during the
interval between one word and the other. Of greater importance,
however, is respectful listening to the words uttered by the priests,
and chanted by them in a peculiar traditional tune. The priests turn to
all directions while pronouncing the blessing, expressing thereby that
they would have no one excluded from the blessing.

Some kohanim refuse to perform this duty, pretending or believing

that they are unworthy to bless the congregation. This is a mistake.
Those who feel that they are unworthy must try by improved conduct
to render themselves worthy, but disobedience to the direct
commandment of the Divine Law is certainly not the beginning of
improvement. Others object to the singing, in which they are unable
to join; others to taking off their boots. The excuses are certainly
insufficient. But as these two elements are less essential, they might,
if necessary, be dispensed with if the fulfilment of the commandment
is secured thereby.

2. On Page 439.

There are various parts in our Service which originally seem to have
formed a substitute, under certain circumstances, for a section of the
Service or for the whole of it, but were subsequently, when the
circumstances altered, embodied as an integral part of the Service in
addition to the sections which they had replaced.

In the Morning Service there is in the section called ‫‏ברכות השחר‬‎a

prayer beginning ‫‏לעולם יהא אדם‬‎and concluding ‫‏לעיניכם אמר יי‬. This
prayer, preceded by an exhortation to be God-fearing in secret—
when persecution prevented public worship of God—contains an
expression of pride in our history, and of gratitude to God that we
have the privilege to proclaim the [447]Unity of God in the words,
“Hear, O Israel,” &c., and a petition for the fulfilment of the Messianic
prophecies.

In the Maaribh the part beginning ‫‏ברוך יי לעולם‬‎and ending ‫‏על כל מעשיו‬‎
was originally a substitute for the Amidah, and the conclusion of the
Evening Service for those who considered the tefillah in the evening
optional. Similarly, on Friday evening the contracted tefillah was
originally a substitute for the tefillah for those who came late. In both
these and similar cases the substitute and its original have been
retained as integral parts of the Service.

3. On Page 442 (2).

In the public Service the tefillah is repeated by the Reader after the
silent prayer (‫‏בלתש‬‎) of the congregation. This minhag must have
been introduced very early. In the Talmud (end of Rosh ha-shanah) it
is spoken of as a regular institution, its purpose is discussed, and the
reason stated why we should not dispense with the Reader’s
repetition or with the silent tefillah. It seems that there was, on the
one hand, a desire on the part of the congregants to have an
uninterrupted silent tefillah in which they could give suitable
expression each one to his personal and peculiar wants and wishes.
On the other hand, there was also a desire felt by many to be guided
in their devotions by the Reader. Our minhag satisfies both
requirements. But it is a grave error to think, as unfortunately many
do, that, while the Reader repeats the tefillah, the congregants may
turn their heart and mind to other things, however holy these be. The
congregation and the Reader must be united in devotion during ‫‏תפלה‬
‫בצבור‬‎, and where the continued concentration of thought during the
tefillah and its repetition seems unattainable, it would be better to
sacrifice the minhag of repeating the tefillah rather than to have the
repetition of the prayer without the participation of the congregants,
or even without decorum.

In the repetition of the tefillah the kedushah forms an important

addition. The essential idea of the kedushah is repeated thrice
during the Morning Service, viz., in the first of the benedictions
preceding the shema, in the tefillah or Amidah, and in the concluding
section commencing ‫‏ובא לציון‬‎.

In the first kedushah (called ‫‏קדושה דיוצר‬‎), while praising God [448]as
the Creator of light and of the heavenly luminaries, we introduce
these, as proclaiming, as it were, the holiness and glory of God in
the words of the Prophets. In the last kedushah (called ‫‏קדושה דסידרא‬‎
or ‫‏סידרא דקדושה‬‎) we merely read, among other passages from the
Prophets, those verses of Isaiah and Ezekiel which contain the chief
sentences of the kedushah. In the kedushah of the tefillah the
Reader summons the congregation to proclaim the sanctification of
God in the manner of the angels above; it has therefore its place
only in Public Worship, whilst as to the other two kedushoth there is
no difference whether a person prays by himself or in a congregation
of worshippers.

As regards ‫‏קדיש וברכו‬‎, two prayers generally united, it must be

remarked that in their meaning they are disunited: the half-kaddish is
the conclusion of the mizmorim or pesuke dezimrah, and ‫‏ברכו‬‎is the
commencement of the next section: shema, with its benedictions.
The half-kaddish, wherever it occurs, concludes some section of the
Service. In shacharith, after Amidah, or after “Supplications,” or after
the Reading of the Law; at Musaf, Minchah, Neïlah, and Maaribh
after the introductory psalms. The half-kaddish before the Amidah in
the Maaribh is probably a remnant of the whole kaddish that used to