Effective Kafka A Hands on Guide to Building

Robust and Scalable Event Driven Applications

with Code Examples in Java 1st Edition Emil
Koutanov pdf download
https://ebookfinal.com/download/effective-kafka-a-hands-on-guide-
to-building-robust-and-scalable-event-driven-applications-with-
code-examples-in-java-1st-edition-emil-koutanov/

Explore and download more ebooks or textbooks

at ebookfinal.com
Here are some recommended products for you. Click the link to
download, or explore more at ebookfinal

Microservices for Java developers a hands on introduction

to frameworks and containers First Edition. Edition Posta

https://ebookfinal.com/download/microservices-for-java-developers-a-
hands-on-introduction-to-frameworks-and-containers-first-edition-
edition-posta/

Cloud Native DevOps Building Scalable and Reliable

Applications 1st Edition Mohammed Ilyas Ahmed

https://ebookfinal.com/download/cloud-native-devops-building-scalable-
and-reliable-applications-1st-edition-mohammed-ilyas-ahmed/

Test driven infrastructure with Chef bring behavior driven

development to infrastructure as code 2nd Edition Stephen
Nelson-Smith
https://ebookfinal.com/download/test-driven-infrastructure-with-chef-
bring-behavior-driven-development-to-infrastructure-as-code-2nd-
edition-stephen-nelson-smith/

J2EE Technology in Practice Building Business Applications

with the Java 2 Platform 1st Edition Rick Cattell

https://ebookfinal.com/download/j2ee-technology-in-practice-building-
business-applications-with-the-java-2-platform-1st-edition-rick-
cattell/
Node Up and Running Scalable Server Side Code with
JavaScript 1st Edition Tom Hughes-Croucher

https://ebookfinal.com/download/node-up-and-running-scalable-server-
side-code-with-javascript-1st-edition-tom-hughes-croucher/

Hands On Guide to Video Blogging and Podcasting Emerging

Media Tools for Business Communication Hands On Guide
Series 1st Edition Lionel Felix
https://ebookfinal.com/download/hands-on-guide-to-video-blogging-and-
podcasting-emerging-media-tools-for-business-communication-hands-on-
guide-series-1st-edition-lionel-felix/

Effective Unit Testing A guide for Java Developers 1st

Edition Lasse Koskela

https://ebookfinal.com/download/effective-unit-testing-a-guide-for-
java-developers-1st-edition-lasse-koskela/

The Hands on Guide to Diabetes Care in Hospital 1st

Edition David Levy

https://ebookfinal.com/download/the-hands-on-guide-to-diabetes-care-
in-hospital-1st-edition-david-levy/

Map Scripting 101 An Example Driven Guide to Building

Interactive Maps with Bing Yahoo and Google Maps 1st
Edition Adam Duvander
https://ebookfinal.com/download/map-scripting-101-an-example-driven-
guide-to-building-interactive-maps-with-bing-yahoo-and-google-
maps-1st-edition-adam-duvander/
Effective Kafka A Hands on Guide to Building Robust
and Scalable Event Driven Applications with Code
Examples in Java 1st Edition Emil Koutanov Digital
Instant Download
Author(s): Emil Koutanov
ISBN(s): 9798628558515, 8628558519
Edition: 1
File Details: PDF, 13.74 MB
Year: 2020
Language: english
Effective Kafka
A Hands-On Guide to Building Robust and Scalable
Event-Driven Applications with Code Examples in Java

Emil Koutanov
This book is for sale at http://leanpub.com/effectivekafka

This version was published on 2021-01-05

This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing
process. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools and
many iterations to get reader feedback, pivot until you have the right book and build traction once
you do.

© 2019 - 2021 Emil Koutanov

Tweet This Book!
Please help Emil Koutanov by spreading the word about this book on Twitter!
The suggested hashtag for this book is #ApacheKafka.
Find out what other people are saying about the book by clicking on this link to search for this
hashtag on Twitter:
#ApacheKafka
Dedicated to my family, who have unrelentlessly supported me as I disappeared for countless hours
in my study, producing reams of mildly cohesive text.
Contents

Chapter 1: Event Streaming Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

The real challenges of distributed systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Event-Driven Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
What is event streaming? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Chapter 2: Introducing Apache Kafka . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

The history of Kafka . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
The present day . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Uses of Kafka . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Chapter 3: Architecture and Core Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Architecture Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Total and partial order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Consumer groups and load balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Free consumers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Summary of core concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

Chapter 4: Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Installing Kafka and ZooKeeper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
Launching Kafka and ZooKeeper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Running in the background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Installing Kafdrop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

Chapter 5: Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Publishing and consuming using the CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
A basic Java producer and consumer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

Chapter 6: Design Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79

Roles and responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
Parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
Idempotence and exactly-once delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
CONTENTS

Chapter 7: Serialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Key and value serializer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Key and value deserializer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Chapter 8: Bootstrapping and Advertised Listeners . . . . . . . . . . . . . . . . . . . . . . . . . 124

A gentle introduction to bootstrapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
A simple scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Multiple listeners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
Listeners and the Docker Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

Chapter 9: Broker Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

Entity types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
Dynamic update modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
Configuration precedence and defaults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Applying broker configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Applying topic configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
Users and Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

Chapter 10: Client Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154

Configuration gotchas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
Applying client configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
Common configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
Producer configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
Consumer configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Admin client configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

Chapter 11: Robust Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184

Using constants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
Type-safe configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

Chapter 12: Batching and Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

Comparing disk and network I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Producer record batching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Chapter 13: Replication and Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

Replication basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Leader election . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Setting the initial replication factor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
Changing the replication factor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Decommissioning broker nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

Chapter 14: Data Retention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218

Kafka storage internals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
CONTENTS

Deletion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Compaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Combining compaction with deletion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232

Chapter 15: Group Membership and Partition Assignment . . . . . . . . . . . . . . . . . . . . 235

Group membership basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
Liveness and safety . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Partition assignment strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

Chapter 16: Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

State of security in Kafka . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
Target state security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
Network traffic policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
Confidentiality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
Authorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318

Chapter 17: Quotas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339

The rationale behind quotas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
Types of quotas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
Subject affinity and precedence order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
Applying quotas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
Buffering and timeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
Sensing quota enforcement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
Tuning the duration and number of sampling windows . . . . . . . . . . . . . . . . . . . . . 360

Chapter 18: Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366

Preamble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
The rationale behind transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
Transactions under the hood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
Simple stream processing example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384
Are transactions over-hyped? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
Chapter 1: Event Streaming
Fundamentals
It is amazing how the software engineering landscape has transformed over the last decade. Not
long ago, applications were largely monolithic in nature, internally-layered, typically hosted within
application servers and backed by ‘big iron’ relational databases with hundreds or thousands of
interrelated tables. Distributed applications were the ‘gold standard’ by those measures — coarse-
grained deployable units scattered among a static cluster of application servers, hosted on a fleet of
virtual machines and communicating over SOAP-based APIs or message queues. Containerisation,
cloud computing, elasticity, ephemeral computing, functions-as-a-service, immutable infrastructure
— all niche concepts that were just starting to surface, making minor, barely perceptible ripples in
an architectural institution that was otherwise well-set in its ways.
That was then. Today, these concepts are profoundly commonplace. Engineers are often heard
interleaving several such terms in the same sentence; it would seem that the engineering community
had miraculously stumbled upon an elixir that has all but cured us of our prior burdens — at least
when it comes to developer velocity, time-to-market, system availability, scalability, and just about
every other material concern that had kept the engineering manager of yore awake at night. Today,
we have microservices in the cloud. Problem solved. Next question.
Except no such event actually occurred. We did not discover a solution to the problem; we merely
shifted the problem. Aspects of software development that used to be straightforward in the ‘old
world’, such as debugging, profiling, performance management, and state consistency — are now
an order of magnitude more complex. On top of this, a microservices architecture brings its own
unique woes. Services are more fluid and elastic, and tracking of their instances, their versions
and dependencies is a Herculean challenge that balloons in complexity as the component landscape
evolves. To top this off, services will fail in isolation, further exacerbated by unreliable networks,
potentially leaving some activities in a state of partial completeness. Given a large enough system,
parts of it may be suffering a minor outage at any given point in time, potentially impacting a subset
of users, quite often without the operator’s awareness.
With so many ‘moving parts’, how does one stay on top of these challenges? How does one make
the engineering process sustainable? Or should we just write off the metamorphosis of the recent
decade as a failed experiment?

The real challenges of distributed systems

If there is one thing to be learned from the opening gambit, it is that there is no ‘silver bullet’.
Architectural paradigms are somewhat like design patterns, but broader scoped, more subjective,
Chapter 1: Event Streaming Fundamentals 2

and far less prescriptive. However fashionable and blogged-about these paradigms might be, they
only offer partial solutions to common problems. One must be mindful of the context at all times,
and apply the model judiciously. And crucially, one must understand the deficiencies of the proposed
approach, being able to reason about the implications of its adoption — both immediate and long-
term.
The principal inconvenience of a distributed system is that it shifts the complexity from the innards
of a service implementation to the notional fabric that spans across services. Some might say, it
lifts the complexity from the micro level to the macro level. In doing so, it does not reduce the net
complexity; on the contrary, it increases the aggregate complexity of the combined solution. An
astute engineering leader is well-aware of this. The reason why a distributed architecture is often
chosen — assuming it is chosen correctly — is to enable the compartmentalisation of the problem
domain. It can, if necessary, be decomposed into smaller chunks and solved in partial isolation,
typically by different teams — then progressively integrated into a complete whole. In some cases,
this decomposition is deliberate, where teams are organised around the problem. In other, less-
than-ideal cases, the breakdown of the problem is a reflection of Conway’s Law, conforming to
organisational structures. Presumed is the role an architect, or a senior engineering figure that
orchestrates the decomposition, assuming the responsibility for ensuring the conceptual integrity
and efficacy of the overall solution. Centralised coordination may not always be present — some
organisations have opted for a more democratic style, whereby teams act in concert to decompose
the problem organically, with little outside influence.

Coupling
Whichever the style of decomposition, the notion of macro complexity cannot be escaped. Funda-
mentally, components must communicate in one manner or another, and therein lies the problem:
components are often inadvertently made aware of each other. This is called coupling — the degree of
interdependence between software components. The lower the coupling, the greater the propensity
of the system to evolve to meet new requirements, performance demands, and operational challenges.
Conversely, tight coupling shackles the components of the system, increasing their mutual reliance
and impeding their evolution.
There are known ways for alleviating the problem of coupling, such as the use of an asynchronous
communication style and message-oriented middleware to segregate components. These techniques
have been used to varying degrees of success; there are times where message-based communication
has created a false economy — collaborating components may still be transitively dependent upon
one another in spite of their designers’ best efforts to forge opaque conduits between them.

Resilience
It would be rather nice if computers never failed and networks were reliable; as it happens, reality
differs. The problem is exacerbated in a distributed context: the likelihood of any one component
experiencing an isolated failure increases with the total number of components, which carries
negative ramifications if components are interdependent.
Chapter 1: Event Streaming Fundamentals 3

Distributed systems typically require a different approach to resilience compared to their centralised
counterparts. The quantity and makeup of failure scenarios is often much more daunting in dis-
tributed systems. Failures in centralised systems are mostly characterised as fail-stop scenarios —
where a process fails totally and permanently, or a network partition occurs, which separates the
entirety of the system from one or more clients, or the system from its dependencies. At either rate,
the failure modes are trivially understood. By contrast, distributed systems introduce the concept of
partial failures, intermittent failures, and, in the more extreme cases, Byzantine failures. The latter
represents a special class of failures where processes submit incorrect or misleading information to
unsuspecting peers.

Consistency
Ensuring state consistency in a distributed system is perhaps the most difficult aspect to get right.
One can think of a distributed system as a vast state machine, with some elements of it being updated
independently of others. There are varying levels of consistency, and different applications may
demand specific forms of consistency to satisfy their requirements. The stronger the consistency
level, the more synchronisation is necessary to maintain it. Synchronisation is generally regarded
as a difficult problem; it is also expensive — requiring additional resources and impacting the
performance of the system.
Cost being held a constant, the greater the requirement for consistency, the less distributed a system
will be. There is also a natural counterbalance between consistency and availability, identified by
Eric Brewer in 1998. The essence of it is in the following: distributed systems must be tolerant of
network partitions, but in achieving this tolerance, they will have to either give up consistency
or availability guarantees. Note, this conjecture does not claim that a consistent system cannot
simultaneously be highly available, only that it must give up availability if a network partition
does occur.
By comparison, centralised systems are not bound by the same laws, as they don’t have to contend
with network partitions. They can also take advantage of the underlying hardware, such as CPU
cache lines and atomic operations, to ensure that individual threads within a process maintain
consistency of shared data. When they do fail, they typically fail as a unit — losing any ephemeral
state and leaving the persistent state as it was just before failure.

Event-Driven Architecture
Event-Driven Architecture (EDA) is a paradigm promoting the production, detection, consumption
of, and reaction to events. An event is a significant state in change, that may be of interest within
the domain where this state change occurred, or outside of that domain. Interested parties can be
notified of an event by having the originating domain publish some canonical depiction of the event
to a well-known conduit — a message broker, a ledger, or a shared datastore of some sort. Note, the
event itself does not travel — only its notification; however, we often metonymically refer to the
notification of the event as the event. (While formally incorrect, it is convenient.)
Chapter 1: Event Streaming Fundamentals 4

An event-driven system formally consists of emitters (also known as producers and agents), con-
sumers (also known as subscribers and sinks), and channels (also known as brokers). We also use
the term upstream — to refer to the elements prior to a given element in the emitter-consumer
relation, and downstream — to refer to the subsequent elements.
An emitter of an event is not aware of any of the event’s downstream consumers. This statement
captures the essence of an event-driven architecture. An emitter does not even know whether
a consumer exists; every transmission of an event is effectively a ‘blind’ broadcast. Likewise,
consumers react to specific events without the knowledge of the particular emitter that published
the event. A consumer need not be the final destination of the event; the event notification may
be persisted or transformed by the consumer before being broadcast to the next stage in a notional
pipeline. In other words, an event may spawn other events; elements in an event-driven architecture
may combine the roles of emitters and consumers, simultaneously acting as both.
Event notifications are immutable. An element cannot modify an event’s representation once it has
been emitted, not even if it is the originally emitter. At most, it can emit new notifications relating
to that event — enriching, refining, or superseding the original notification.

Coupling
Elements within EDA are exceedingly loosely coupled, to the point that they are largely unaware
of one another. Emitters and consumers are only coupled to the intermediate channels, as well as to
the representations of events — schemas. While some coupling invariably remains, in practice, EDA
offers the lowest degree of coupling of any practical system. The collaborating components become
largely autonomous, standalone systems that operate in their own right — each with their individual
set of stakeholders, operational teams, and governance mechanisms.
By way of an example, an e-commerce system might emit events for each product purchase, detailing
the time, product type, quantity, the identity of the customer, and so on. Downstream of the
emitter, two systems — a business intelligence (BI) platform and an enterprise resource planning
(ERP) platform — might react to the sales events and build their own sets of materialised views.
(In effect, view-only projections of the emitter’s state.) Each of these platforms are completely
independent systems with their own stakeholders: the BI system satisfies the business reporting
and analytics requirements for the marketing business unit, while the ERP system supports supply
chain management and capacity planning — the remit of an entirely different business unit.
To put things into perspective, we shall consider the potential solutions to this problem in the absence
of EDA. There are several ways one could have approached the solution; each approach commonly
found in the industry to this day:

1. Build a monolith. Conceptually, the simplest approach, requiring a system to fulfill all
requirements and cater to all stakeholders as an indivisible unit.
2. Integration. Allow the systems to invoke one another via some form of an API. Either the
e-commerce platform could invoke the BI and ERP platforms at the point of sale, or the BI and
ERP platforms could invoke the e-commerce platform APIs just before generating a business
Chapter 1: Event Streaming Fundamentals 5

report or supplier request. Some variations of this model use message queues for systems to
send commands and queries to one another.
3. Data decapsulation. If system integrators were cowboys, this would be their prairie. Data
decapsulation (a coined term, if one were to ask) sees systems ‘reaching over’ into each other’s
‘backyard’, so to speak, to retrieve data directly from the source (for example, from an SQL
database) — without asking the owner of the data, and oftentimes without their awareness.
4. Shared data. Build separate applications that share the same datastore. Each application is
aware of all data, and can both read and modify any data element. Some variations of this
scheme use database-level permissions to restrict access to the data based on an application’s
role, thereby binding the scope of each application.

Once laid out, the drawbacks of each model become apparent. The first approach — the proverbial
monolith — suffers from uncontrolled complexity growth. In effect, it has to satisfy everyone and
everything. This also makes it very difficult to change. From a reliability standpoint, it is the
equivalent of putting all of one’s eggs in one basket — if the monolith were to fail, it will impact all
stakeholders simultaneously.
The second approach — integrate everything — is what these days is becoming more commonly
known as the ‘distributed monolith’, especially when it is being discussed in the context of mi-
croservices. While the systems (or services, as the case may be) appear to be standalone — they
might even be independently sourced and maintained — they are by no means autonomous, as they
cannot change freely without impacting their peers.
The third approach — read others’ data — is the architectural equivalent of a ‘get rich quick scheme’
that always ends in tears. It takes the path of least resistance, making it highly alluring. However,
the model creates the tightest possible level of coupling, making it very difficult to change the
parties down the track. It is also brittle — a minor and seemingly benign change to the internal
data representation in one system could have a catastrophic effect on another system.
The final model — the use of a shared datastore — is a more civilised variation of the third approach.
While it may be easier to govern, especially with the aid of database-level access control — the
negative attributes are largely the same.
Now imagine that the business operates multiple disparate e-commerce platforms, located in dif-
ferent geographic regions or selling different sorts of products. And to top it off, the business now
needs a separate data warehouse for long-term data collection and analysis. The addition of each
new component significantly increases the complexity of the above solutions; in other words, they
do not scale. By comparison, EDA scales perfectly linearly. Systems are unaware of one another and
react to discrete events — the origin of an event is largely circumstantial. This level of autonomy
permits the components to evolve rapidly in isolation, meeting new functional and non-functional
requirements as necessary.

Resilience
The autonomy created by the use of EDA ensures that, as a whole, the system is less prone to outage
if any of its individual components suffer a catastrophic failure. How is this achieved?
Chapter 1: Event Streaming Fundamentals 6

Integrated systems, and generally, any topological arrangement that exhibits a high degree of
component coupling is prone to correlated failure — whereby the failure of one component can
take down an entire system. In a tightly coupled system, components directly rely on one another
to jointly achieve some goal. If one of these components fails, then the remaining components that
depend on it may also cease to function; at minimum, they will not be able to carry out those
operations that depend on the failed component.
In the case of a monolith, the failure assertion is trivial — if a fail-stop scenario occurs, the entire
process is affected.
Under EDA, enduring a component failure implies the inability to either emit events or consume
them. In the event of emitter failure, consumers may still operate freely, albeit without a facility
for reacting to new events. Using our earlier example, if the e-commerce engine fails, none of the
downstream processes will be affected — the business can still run analytical queries and attend to
resource planning concerns. Conversely, if the ERP system fails, the business will still make sales;
however, some products might not be placed on back-order in time, potentially leading to low stock
levels. Furthermore, provided the event channel is durable, the e-commerce engine will continue to
publish sales events, which will eventually be processed by the ERP system when it is restored. The
failure of an event channel can be countered by implementing a local, stateful buffer on the emitter,
so that any backlogged events can be published when the channel has been restored. In other words,
not only is an event-driven system more resilient by retaining limited operational status during
component failure, it is also capable of self-healing when failed components are replaced.
In practice, systems may suffer from soft failures, where components are saturated beyond their
capacity to process requests, creating a cascading effect. In networking, this phenomenon is called
‘congestive collapse’. In effect, components appear to be online, but are stressed — unable to turn
around some fraction of requests within acceptable time frames. In turn, the requesting components
— having detected a timeout — retransmit requests, hoping to eventually get a response. This in-
creases pressure on the stressed components, exacerbating the situation. Often, the missed response
is merely an indication of receiving the request — in effect, the requester is simply piling on duplicate
work.
Under EDA, requesters do not require a confirmation from downstream consumers — a simple
acknowledgement from the event channel is sufficient to assume that the event has been stably
enqueued and that the consumer(s) will get to it at some future point in time.

Consistency
EDA ameliorates the problem of distributed consistency by attributing explicit mastership to state,
such that any stateful element can only be manipulated by at most one system — its designated
owner. This is also referred to as the originating domain of the event. Other domains may only react
to the event; for example, they may reduce the event stream to a local projection of the emitter’s
state.
Under this model, consistency within the originating domain is trivially maintained by enforcing
the single writer principle. External to the domain, the events can be replayed in the exact order
Chapter 1: Event Streaming Fundamentals 7

they were observed on the emitter, creating sequential consistency — a model of consistency where
updates do not have to be seen instantaneously, but must be presented in the same order to all
observers, which is also the order they were observed on the emitter. Alternatively, events may
be emitted in causal order, categorising them into multiple related sequences, where events within
any sequence are related amongst themselves, but unrelated to events in another sequence. This is
a slight relaxation of sequential consistency to allow for safe parallelism, and is sufficient in the
overwhelming majority of use cases.

Applicability
For all its outstanding benefits, EDA is not a panacea and cannot supplant integrated or monolithic
systems in all cases. For instances, EDA is not well-suited to synchronous interactions, as mutual
or unilateral awareness among collaborating parties runs contrary to the grain of EDA and negates
most of its benefits.
EDA is not a general-purpose architectural paradigm. It is designed to be used in conjunction with
other paradigms and design patterns, such as synchronous request-response style messaging, to
solve more general problems. In the areas where it can be applied, it ordinarily leads to significant
improvements in the system’s non-functional characteristics. Therefore, one should seek to max-
imise opportunities for event-driven compositions, refactoring the architecture to that extent.

What is event streaming?

Finally, we arrive at the central question: What is event streaming? And frankly, there is little left
to explain. There is but one shortfall in the earlier narrative: EDA is an architectural paradigm —
it does not prescribe the particular semantics of the event interchange. Events could be broadcast
among parties using different mechanisms, all potentially satisfying the basic tenets of EDA.
Event streaming is a mechanism that can be used to realise the event channel element in EDA. It is
primarily concerned with the following aspects of event propagation:

• Interface between the emitter and the channel, and the consumer and the channel;
• Cardinality of the emitter and consumer elements that interact with a common channel;
• Delivery semantics;
• Enabling parallelism in the handling of event notifications;
• Persistence, durability, and retention of event records; and
• Ordering of events and associated consistency models.

The focal point of event streaming is, unsurprisingly, an event stream. At minimum, an event stream
is a durable, totally-ordered, unbounded sequence of immutable event records, delivered at least once
to its subscriber(s). An event streaming platform is a concrete technology that implements the event
streaming model, addressing the points enumerated above. It interfaces with emitter and consumer
Chapter 1: Event Streaming Fundamentals 8

ecosystems, hosts event streams, and may provide additional functionality beyond the essential set
of event streaming capabilities. For example, an event streaming platform may offer end-to-end
compression and encryption of event records, which is not essential in the construction of event-
driven systems, but is convenient nonetheless.
It is worth noting that event streaming is not required to implement the event channel element of
EDA. Other transports, such as message queues, may be used to fulfill similar objectives. In fact,
there is nothing to say that EDA is exclusive to distributed systems; the earliest forms of EDA were
realised within the confines of a single process, using purely in-memory data structures. It may seem
banal in comparison, but even UI frameworks of the bygone era, such as Java Swing, draw on the
foundations of EDA, as do their more contemporary counterparts, such as React.
When operating in the context of a distributed system, the primary reason for choosing event
streaming over the competing alternatives is that the former was designed specifically for use in
EDA, and its various implementations — event streaming platforms — offer a host of capabilities
that streamline their adoption in EDA. A well-designed event streaming platform provides direct
correspondence with native EDA concepts. For example, it takes care of event immutability, record
ordering, and supports multiple independent consumers — concepts that might not necessarily be
endemic to alternate solutions, such as message queues.

This chapter has furnished an overview of the challenges of engineering distributed systems,
contrasted with the building of monolithic business applications. The numerous drawbacks of dis-
tributed systems increase their cost and complicate their upkeep. Generally speaking, the components
of a complex system are distributed out of necessity — namely, the requirement to scale in both the
performance plane and in the engineering capacity to deliver change.
We looked at how the state of the art has progressed since the mass adoption of the principles of
distributed computing in mainstream software engineering. Specifically, we explored Event-Driven
Architecture as a highly effective paradigm for reducing coupling, bolstering resilience, and avoiding
the complexities of maintaining a globally consistent state.
Finally, we touched upon event streaming, which is a rendition of the event channel element of
EDA. We also learned why event streaming is the preferred approach for persisting and transporting
event notifications. In no uncertain terms, event streaming is the most straightforward path for the
construction of event-driven systems.
Chapter 2: Introducing Apache Kafka
Apache Kafka (or simply Kafka) is an event streaming platform. But it is also more than that. It is
an entire ecosystem of technologies designed to assist in the construction of complete event-driven
systems. Kafka goes above and beyond the essential set of event streaming capabilities, providing
rich event persistence, transformation, and processing semantics.
Event streaming platforms are a comparatively recent paradigm within the broader message-oriented
middleware class. There are only a handful of mainstream implementations available, compared to
hundreds of MQ-style brokers, some going back to the 1980s (for example, Tuxedo). Compared to
established messaging standards such as AMQP, MQTT, XMPP, and JMS, there are no equivalent
standards in the streaming space. Kafka is a leader in the area of event streaming, and more broadly,
event-driven architecture. While there is no de jure standard in event streaming, Kafka is the
benchmark to which most competing products orient themselves. To this effect, several competitors
— such as Azure Event Hubs and Apache Pulsar — offer APIs that mimic Kafka.

Event streaming platforms are an active area of continuous research and experimentation.
In spite of this, event streaming platforms aren’t just a niche concept or an academic idea
with few esoteric use cases; they can be applied effectively to a broad range of messaging
and eventing scenarios, routinely displacing their more traditional counterparts.

Kafka is written in Java, meaning it can run comfortably on most operating systems and hardware
configurations. It can equally be deployed on bare metal, in the Cloud, and a Kubernetes cluster.
And finally, Kafka has libraries written for just about every programming language, meaning that
virtually every developer can start taking advantage of event streaming and push their application
architecture to the next level of resilience and scalability.

The history of Kafka

Apache Kafka was originally developed by LinkedIn, and was subsequently open-sourced in early
2011. The name ‘Kafka’ was chosen by one of its founders — Jay Kreps. Kreps chose to name the
software after the famous 20ʰ-century author Franz Kafka because it was “a system optimised for
writing”. Kafka gained the full Apache Software Foundation project status in October 2012, having
graduated from the Apache Incubator program.
Kafka was born out of a need to track and process large volumes of site events, such as page
views and user actions, as well as for the aggregation log data. Before Kafka, LinkedIn maintained
several disparate data pipelines, which presented a challenge from both complexity and operational
scalability perspectives. In July 2011, having consolidated the individual platforms, Kafka was
Chapter 2: Introducing Apache Kafka 10

processing approximately one billion events per day. By 2012, this number had risen to 20 billion.
By July 2013, Kafka was carrying 200 billion events per day. Two years later, in 2015, Kafka was
turning over one trillion events per day, with peaks of up to 4.5 million events per second.
Over the four years of 2011 to 2015, the volume of records has grown by three orders of magnitude.
By the end of this period, LinkedIn was moving well over a petabyte of event data per week. By all
means, this level of growth could not be attributed to Kafka alone; however, Kafka was undoubtedly
a key enabler from an infrastructure perspective.
As of October 2019, LinkedIn maintains over 100 Kafka clusters, comprising more than 4,000 brokers.
These collectively serve more than 100,000 topics and 7 million partitions. The total number of
records handled by Kafka has surpassed 7 trillion per day.

The present day

The industry adoption of Kafka has been nothing short of phenomenal. The list of tech giants that
heavily rely on Kafka is impressive in itself. To name just a few:

• Yahoo uses Kafka for real-time analytics, handling up to 20 gigabits of uncompressed event
data per second in 2015. Yahoo is also a major contributor to the Kafka ecosystem, having
open-sourced its in-house Cluster Manager for Apache Kafka (CMAK) product.
• Twitter heavily relies on Kafka for its mobile application performance management and
analytics product, which has been clocked at five billion sessions per day in February 2015.
Twitter processes this stream using a combination of Apache Storm, Hadoop, and AWS Elastic
MapReduce.
• Netflix uses Kafka as the messaging backbone for its Keystone pipeline — a unified event
publishing, collection, and routing infrastructure for both batch and stream processing. As of
2016, Keystone comprises over 4,000 brokers deployed entirely in the Cloud, which collectively
handle more than 700 billion events per day.
• Tumblr relies on Kafka as an integral part of its event processing pipeline, capturing up 500
million page views a day back in 2012.
• Square uses Kafka as the underlying bus to facilitate stream processing, website activity
tracking, metrics collection and monitoring, log aggregation, real-time analytics, and complex
event processing.
• Pinterest employs Kafka for its real-time advertising platform, with 100 clusters comprising
over 2,000 brokers deployed in AWS. Pinterest is turning over in excess of 800 billion events
per day, peaking at 15 million per second.
• Uber is among the most prominent of Kafka adopters, processing in excess of a trillion
events per day — mostly for data ingestion, event stream processing, database changelogs, log
aggregation, and general-purpose publish-subscribe message exchanges. In addition, Uber is
an avid open-source contributor — having released its in-house cluster replication solution
uReplicator into the wild.
Chapter 2: Introducing Apache Kafka 11

And it’s not just the engineering-focused organisations that have adopted Kafka — by some esti-
mates, up a third of Fortune 500 companies use Kafka to fulfill their event streaming and processing
needs.
There are good reasons for this level of industry adoption. As it happens, Kafka is one of the
most well-supported and well-regarded event streaming platforms, boasting an impressive number
of open-source projects that integrate with Kafka. Some of the big names include Apache Storm,
Apache Flink, Apache Hadoop, LogStash and the Elasticsearch Stack, to name a few There are also
Kafka Connect integrations with every major SQL database, and most NoSQL ones too. At the time
of writing, there are circa one hundred supported off-the-shelf connectors, which does not include
custom connectors that have been independently developed.

Uses of Kafka
Chapter 1: Event Streaming Fundamentals has provided the necessary background, fitting Kafka as
an event streaming platform within a larger event-driven system.
There are several use cases falling within the scope of EDA that are well-served by Apache Kafka.
This section covers some of these scenarios, illustrating how Kafka may be used to address them.

Publish-subscribe

Pub-Sub

Any messaging scenario where producers are generally unaware of consumers, and instead publish
messages to well-known aggregations called topics. Conversely, consumers are generally unaware of
the producers but are instead concerned with specific content categories. The producer and consumer
Chapter 2: Introducing Apache Kafka 12

ecosystems are loosely-coupled, being aware of only the common topic(s) and messaging schema(s).
This pattern is commonly used in the construction of loosely-coupled microservices.
When Kafka is used for general-purpose publish-subscribe messaging, it will be competing with its
‘enterprise’ counterparts, such as message brokers and service buses. Admittedly, Kafka might not
have all the features of some of these middleware platforms — such as message deletion, priority
levels, producer flow control, distributed transactions, or dead-letter queues. On the other hand,
these features are mostly representative of traditional messaging paradigms — intrinsic to how
these platforms are commonly used. Kafka works in its own idiomatic way — optimised around
unbounded sequences of immutable events. As long as a publish-subscribe relationship can be
represented as such, then Kafka is fit for the task.

Log aggregation

Log aggregation

Dealing with large volumes of log-structured events, typically emitted by application or infrastruc-
ture components. Logs may be generated at burst rates that significantly outstrip the ability of query-
centric datastores to keep up with log ingestion and indexing, which are regarded as ‘expensive’
operations. Kafka can act as a buffer, offering an intermediate, durable datastore. The ingestion
process will act as a sink, eventually collating the logs into a read-optimised database (for example,
Elasticsearch or HBase).
A log aggregation pipeline may also contain intermediate steps, each adding value en route to the
final destination; for example, to compress log data, encrypt log content, normalise the logs into a
canonical form, or sanitise the log entries — scrubbing them of personally-identifiable information.
Chapter 2: Introducing Apache Kafka 13

Log shipping

Log shipping

While sounding vaguely similar to log aggregation, the shipping of logs is a vastly different concept.
Essentially, this involves the real-time copying of journal entries from a master data-centric system
to one or more read-only replicas. Assuming stage changes are fully captured as journal records,
replaying those records allows the replicas to accurately mimic the state of the master, albeit with
some lag.
Kafka’s optional ability to partition records within a topic to create independent, causally ordered
sequences of events allows for replicas to operate in one of sequential or causal consistency models —
depending on the chosen partitioning scheme. The various consistency models were briefly covered
in Chapter 1: Event Streaming Fundamentals. Both consistency models are sufficient for creating
read-only copies of the original data.
Log shipping is a key enabler for another related architectural pattern — event sourcing. Kafka will
act as a durable event store, allowing any number of consumers to rebuild a point-in-time snapshot
of their application state by replaying all records up to that point in time. Loss of state information
in any of the downstream consumers can be recovered by replaying the events from the last stable
checkpoint, thereby reducing the need to take frequent backups.

SEDA pipelines

SEDA pipeline
Discovering Diverse Content Through
Random Scribd Documents
we can see how this loving obedience was in reality a
storing up of energy for the great revolution of which
she had caught the earliest intimations.
It is a pleasant thought to take in passing that this
good sister—happier than many—had brothers
equally good. If she was all that a sister could be she
found in them good brothers, who were friends and
fellow-workers, helping her in all the great aims of
her life. Her eldest brother, the Rev. Alfred J. Buss, as
clerk to the governing body of the schools, quite
relieved her mind from all anxiety concerning
business arrangements; whilst the religious
instruction given by the Rev. Septimus Buss carried
on the early tradition of the school. There was a wide
gap between the eldest of the family and number
seven, so that her relation with this brother, after the
mother’s death, was half maternal as well as half
sisterly. When he early became engaged to her pupil,
cousin, and friend, and thus gave her the truest and
most tender of sisters, the bond was doubled, and
the children of this beloved pair—her namesake
Francis, especially—became as her very own. Her
letters are full of allusion to “my boy,” who was her
joy from his peculiarly engaging babyhood till he
fulfilled her heart’s desire by taking Holy Orders. His
next brother followed in this example, first set by the
son of the Rev. A. J. Buss, now Minor Canon of
Lincoln.
This clerical bent was very strong in the family. As
a boy, Alfred Joseph Buss shared his sister’s
enthusiasm for teaching, and for any hope of head-
mastership Holy Orders were essential. Before he
was out of his teens he became the first assistant-
master in the then newly opened North London
Collegiate School for Boys. He was also English tutor
at one time to the young Orleans princes. But later in
life he found himself drawn most strongly to the
work of the parish priest. Septimus Buss inherited so
much of his father’s genius, that he seemed destined
for art, having a picture in the Royal Academy whilst
only nineteen years of age. But, though in obedience
to his father he worked hard at painting, he still had
his own intentions, and worked harder at Greek and
Latin. Knowing, however, that there was at that time
an extra strain upon the family finances, he bravely
kept his own wishes to himself till he had earned the
means of carrying them out. The story of these two
brothers is among the helpful and instructive tales
that ought some day to be written, to show what can
be done by high aims and resolute will. Of both it
may be said that they are all the stronger as fighters
in their splendid battle against East End misery,
because, in their own boyhood, they knew how “to
endure hardness as good soldiers.”
This attraction to the clerical profession was a
natural sequence to early associations. The most
powerful influence of Miss Buss’ girlish life was
undoubtedly that of her revered friend of whom Mrs.
Septimus Buss writes, when alluding to—

“the earnest spiritual influence of the Rev. David Laing, who

built the church and schools of Holy Trinity, Kentish Town,
giving his whole fortune and his life to found the parish. His
teaching by precept and practice was self-sacrifice, and the
large-hearted charity that beareth all things, believeth all
things, hopeth all things, coupled with the wide culture that
welcomed new thought, and proved all things. His hospitable
home was constantly open to his parishioners, where he
received them among his cultured circle of literary, scientific,
and artistic friends. He at once took his stand by the North
London Collegiate School, while others waited till its success
was sure. We, oldest of old pupils, still thrill with somewhat
of the past enthusiasm when we recall his inspiring teaching.
The band of devoted workers he gathered round him in his
parish—which was then almost unique for the number of
works of charity carried on in it, and for the weekly lectures
by Mr. S. C. Hall and others—testified to his personal
influence, the motive power of which was not what he saw,
but what he was.”

In memory of her lamented friend, Miss Buss, after

his death, established six “Laing Scholarships,” by
which so many girls who needed this help received a
free education in her school. Thus for ten years Mr.
Laing’s memory was kept in mind. With the changes
of 1870 these Scholarships ceased, but Miss Buss’
devotion to Mrs. Laing knew no intermission till her
old friend’s death in 1876; and Miss Fawcett has an
interesting little comment on this unfailing
thoughtfulness—

“All associated with our dear friend must have been struck
with her loyalty and faithfulness to her old friends. I am
thinking especially of her treatment of Mrs. Laing, for so
many years. Sunday by Sunday she went to see her after
morning service as regularly as the day came round; flowers
were sent to her very frequently, also nice books to read. On
her birthday Miss Buss never failed to see her before the
school-work began.”

Among the school records there is a letter which is

of interest as showing the close relations which
existed between Mr. and Mrs. Laing and the school.
It is addressed to the chairman presiding at the first
prize-day after the double loss which made so sad a
change for the young head-mistress—the death
within a year of her mother and of Mr. Laing—

“Rev. and dear Sir,

 “May I beg you to express my great regret at the
impossibility of my being at your meeting to-day? I do not
say that it would not have been very painful to attend, when
two so loved and honoured are missing since we last
assembled for the same purpose; but it is still more painful
to stay away. I wished to show my true interest in the cause
Mr. Laing had so much at heart; my warm regard for the
friends he so much valued; my deep sense of the respect
and affection shown to his memory in the establishment of
the Laing Scholarships.
“Many to-day will remember how in much pain and
weakness he filled his place last year, but a few days before
he took to the bed whence he was to rise no more. It was
the last evidence he was permitted to give of his feeling with
regard to the work carried on here; and I feel I can do
nothing better than adopt that which in various ways he has
so often said to me, ‘Miss Buss is doing a great and good
work. Hundreds will rise up and call her blessed.’
“I am, yours faithfully,
“Mary E. Laing.”

To the influence of Mr. Laing, and of his no less

admirable wife, Miss Buss owed much of the mental
and moral breadth for which she was afterwards so
distinguished. In their home she was always
welcome, finding a never-failing sympathy and
encouragement. Often in our quiet talks she
delighted to refer to these early memories, speaking
of the advantage such a friendship had been to her
in her young life; and to this grateful memory it is
probable that many of her own young assistants,
especially those least fortunate in their social
surroundings, may have owed much of the
thoughtful kindness so valuable to girls beginning
their career as teachers.
With the knowledge of the satisfaction she would
have felt in fuller recognition of Mr. Laing’s services
to education in general, as well as in particular to her
own school, it will not be out of place here to give
some notes supplied by the Rev. A. J. Buss, with his
own comment on them—

“There is much that I would say about the connection with

Mr. Laing—about himself as a great leader (almost
unacknowledged) in the educational movement of the latter
half of this century. To me the question is an interesting one,
for I loved Mr. Laing as a young man, and cherish his
memory as most precious now that I am advanced in life. It
is at least remarkable that he who, as honorary secretary
and a member of the Board of Management of the
Governesses’ Benevolent Institution, took some part in the
foundation of Queen’s, should have been a prime mover in
the foundation of that school which has become the North
London Collegiate School for Girls, and has rendered
possible, and given such impetus to, the higher education of
girls and women.”

The story of the rise of Queen’s College is of

interest from many points of view, beyond that
concerning our present purpose of showing the
influences that inspired Frances Mary Buss with her
special zeal for education. In knowing Mr. Laing she
came into direct touch with the newest educational
effort, and must have heard the whole question
discussed from all sides.
Mr. Laing, in 1843, rescued the Governesses’
Benevolent Institution from decay, remaining its
active honorary secretary till his death in 1860. This
society was formed—

“with the idea of benefiting governesses in every possible

way; to help in temporary difficulty; to provide annuities for
aged governesses; to help the younger to help themselves;
to provide a home for governesses during engagements, and
an asylum for the aged; also a system of registration, free of
expense, to those seeking engagements.”

The whole of these objects were contemplated in

1843, and, in 1844, were a matter of negotiation
with the National Society, with the Committee of
Council, and with the heads of the Church.
In giving an account of the early work—as a reply
to an article in Fraser’s Magazine (July, 1849),
commenting unfavourably on the efforts that were
then made—Mr. Laing shows that with the
foundation of the Governesses’ Benevolent
Institution the first principles of all future movements
were really incorporated. He says—

“In undertaking an institution for the benefit of

governesses, it was felt to be absurd and short-sighted to
remedy existing evils without an attempt at their removal....
To do this the character of the whole class must be raised,
and there was the bright thought that to raise the character
of governesses as a class was to raise the whole tone of
Christian society throughout the country.”

But it was easier to plan such a college than to

carry out these plans, and several years passed
without practical results. Reference is made, year by
year, on the subject, in the annual reports of the
Governesses’ Benevolent Institution.
In that for 1845, we find that “difficulties which
the committee had not anticipated, have arisen with
the several authorities, from whom Boards of
Examiners, with power to grant a diploma of
qualification, might originate.”
In the report for 1846, “an act of incorporation and
arrangements for a diploma” are still “subjects of
consideration, upon which the committee are
prepared to enter into communication with all parties
friendly to the cause. Unexpected difficulties still
intervene.”
 It was in 1848 that the Governesses’ Benevolent
Institution received a royal charter of incorporation,
thus worded—

“We have been graciously pleased to permit the name of

Queen’s College, in which certificates of qualification are
granted to governesses, and in which arrangements have
been made with professors of high talent and standing in
society to open classes in all branches of female education.”

Queen’s College was governed by a council of

gentlemen, and its first principal, Professor Maurice,
was followed by Professor Plumptre. A committee of
lady-visitors was formed, but the duties of these
ladies was merely to be present while the teaching
was done by men. Among them we find the familiar
names of Mrs. S. C. Hall, Mrs. Marcet, Miss Maurice,
Mrs. Kay Shuttleworth, and Mrs. Hensleigh
Wedgwood.
It would appear, from the report of 1849, that
while the Governesses’ Benevolent Institution was
thus working for better education for women and
girls, other schemes had been proposed, first by Miss
Murray, one of her Majesty’s ladies in waiting, and
then by the professors of King’s College. Eventually,
the formation of a Committee of Education, of which
Mr. Laing and Professors Maurice and Nicolay were
active members, brought things to a practical point,
as Professor Nicolay states[4] that the “Committee of
Education,” thus formed, did its work in connection
with, if not actually for, the Governesses’ Benevolent
Institution.
4. In the English Education Journal, 1849.

In his inaugural lecture at Hanover Square, in

1848, Professor Maurice shows how this institution,
beginning with a provision for distress among
governesses, came to associate distress with
incompetency, and hence to provide better
instruction. In like manner, beginning as examiners,
the professors soon found that before they could
examine they must first teach, and for this purpose
organized the classes that grew into Queen’s College.
In Fraser’s Magazine, early in the fifties, are to be
found several papers concerning the foundation of
Queen’s College, thus finally summed up by the
editor—

“With reference to the article on Queen’s College in our

last number, Mr. Laing, as Hon. Sec. to the Governesses’
Benevolent Institution, desires us to state that the society
was in communication with the Government and other
parties respecting the establishment of the college as early
as 1844, whilst there was no communication with the
present professors until 1847; and that her Majesty granted
to the society the permission to use the Royal name for the
 college before any connection was formed with the present
professors.
“Whilst, therefore, the success of the college is wholly
attributable to the character and talents of its teachers, the
college would have existed under any circumstances.”

In the same year, six months later, Bedford College

was founded, mainly by Mrs. Reid and Miss Bostock,
and among the ladies interested we find many
names afterwards prominent in the movement for
opening the Universities to women, as those of Lady
Romilly, Lady Belcher, Mrs. (afterwards Lady)
Crompton, Mrs. (afterwards Lady) Goldsmid, Mrs.
Jameson, Mrs. Bryan Walter Procter, Lady Pollock,
Miss Julia Smith, Mrs. Strutt (afterwards Lady
Belper), Miss Emily Davies, Miss Anna Swanwick, and
Mrs. Hensleigh Wedgwood.
One distinct difference between Queen’s College
and Bedford College is that the first was managed by
men, with a man as the principal and women only as
lady-visitors. Bedford College had from the first a
mixed committee, and the visitor who represented
the head might be of either sex. Latterly Miss Anna
Swanwick has held this post. Results seem to
indicate the advantage of giving women an equal
share in the education of girls.
 It was by Mr. Laing’s introduction that Miss Buss
became one of the first pupils of the evening classes
at Queen’s College. The Queen’s College of that day
(1848) bore little resemblance to the colleges of a
quarter of a century later, but there was an
enormous stride onwards in the curriculum offered to
its first pupils.
In her “History of Cheltenham College,” Miss Beale
gives us a glimpse of these classes—

“Queen’s College offered to grant certificates to

governesses.... My sisters and I were amongst some of the
first to offer ourselves for examination. For Holy Scripture the
examiner was the Rev. E. H. Plumptre, afterwards Dean of
Wells, so well known for his Biblical Commentaries, his great
learning, and his translations of the Greek dramatists and
Dante. He also examined in classics. In modern history and
literature we had the pleasure of being examined by
Professor Maurice. The viva voce was a delightful
conversation; he led us on by his sympathetic manner and
kindly appreciation so that we hardly remembered he was an
examiner. For French and German our examiners were
Professors Brasseur and Bernays; for mathematics, Professor
Hall and Mr. Cock; for music, Sterndale Bennett; and for
pedagogy, the head of the Battersea Training College.”

The names of the Rev. Charles Kingsley, for English

literature and composition; of Professor Nicolay, for
history and geography; and of Professor Hullah, for
vocal music, also appear on the list.
It was of classes like these that, as a girl of
twenty-one, Frances Mary Buss became a happy
pupil. Her father’s interest in art and science had
prepared her to enter into the spirit of such teaching,
and to profit by the influence of the great men who
threw their whole souls into their work. What this
meant to the girls thus privileged is shown in lives
like those of Miss Buss, Miss Beale, Miss Frances
Martin, or Miss Julia Wedgwood, and many more
perhaps less known to fame.
A memory comes back to me of an evening in
1881, spent at Myra Lodge, where the difference
between the old and the new order of things was
emphasized in a marked degree. Standing out from
the far past, as precursors of the new era, were Miss
Buss herself, Miss Beale, and Miss Frances Martin;
midway, as a Schools Inquiry Commissioner, was Mr.
J. G. Fitch; while the moderns bloomed out in Dr.
Sophie Bryant, one of the earliest Cambridge Local
candidates, and the very first woman-Doctor of
Science; Miss Rose Aitkin, B.A., stood for the arts;
and, I think, Miss Sara A. Burstall (since B.A.) as the
first girl who had, like her brothers, educated herself
by her brains, passing, largely by scholarships, up
from the Camden School, through the Upper School,
and on to Girton.
It was a thing to remember to hear how the three
elder women spoke of the old and new days, and
then to see what had been done for the girls through
their efforts. Miss Buss told us many things of her
girlhood, and her difficulties in fitting herself for her
work; and especially of the stimulus and delight of
the new world of thought and feeling opened by
those first lectures. Miss Beale and Miss Martin,
coming later, had enjoyed all the advantages of
Queen’s College, but they did not the less appreciate
those first lectures. As they spoke in glowing terms
of Professor Maurice, one could not but wish that he
might have been there to see the three grand
women who have done so much for womanhood—
pupils worthy of even such a master.
The picture fixed itself in my mind of Frances Mary
Buss, in the first ardour of this new intellectual
awakening. She was teaching all day in her own
school, so that she could take only the evening
classes. There were at that time no omnibuses, and
night after night, her day’s work done, the
enthusiastic girl walked from Camden Town to
Queen’s College and back. Night after night she sat
up into the small hours, entranced by her new
studies, preparing thus not only for the papers which
won for her the desired certificates, but for that
greater future of which she did not then even dream.
In her Autobiography, Miss Cobbe gives a very
telling summary of the education of the earlier part
of this century, in her account of the particular school
in which her own education had been, as it was
called, “finished,” at a cost, for two years, of £1000.
How she began it for herself afterwards she also
tells, but of this finished portion she thus writes—

“Nobody dreamed that any one of us could, in later life, be

more or less than an ornament to Society. That a pupil in
that school should become an artist or authoress would have
been regarded as a deplorable dereliction. Not that which
was good or useful to the community, or even that which
would be delightful to ourselves, but that which would make
us admired in society was the raison d’être of such
requirement.
“The education of women was probably at its lowest ebb
about half a century ago. It was at that period more
pretentious than it had ever been before, and infinitely more
costly; and it was likewise more shallow and senseless than
can easily be believed. To inspire young women with due
gratitude for their present privileges, won for them by my
contemporaries, I can think of nothing better than to
acquaint them with some of the features of school-life in
England in the days of their mothers. I say advisedly in those
of their mothers, for in those of their grandmothers things
were by no means equally bad. There was much less
pretence, and more genuine instruction, so far as it
extended.”
 We are justified in the conclusion that Mrs.
Wyand’s school, in which Frances Mary Buss received
her training, as pupil and then as assistant, was one
of the survivals from this olden time. From one of the
pupils, who was there as a child while Miss Buss was
assistant-mistress, we have a sketch of Mrs. Wyand
as a slight, erect little lady, with very dark eyes, and
with black hair, in the ringlets of that era, confined
on each side by tortoiseshell side-combs. She always
wore long rustling silk gowns, and altogether was an
impressive personage, before whom the most volatile
schoolgirl at once grew staid and sober. Mention of
Miss Buss herself seems limited to a certain
satisfaction in having carried provocation to so great
an extent as to make the young teacher cry. But we
may easily imagine that before the end of that
encounter the tables were turned, and that then may
have begun the treatment of “naughty girls” so
successful in later life.
Thanks to the good training received under Mrs.
Wyand, Miss Buss was able, at the age of eighteen,
to take an active part in the school opened by Mrs.
Buss in Clarence Road. Before she was twenty-three
she had gained the Queen’s College Diploma, and
she then became the head of the new school in
Camden Street, which was the outcome of this first
venture.
 The course of instruction included most of the
subjects now taught, and Miss Eleanor Begbie—who
claims to have been the first pupil in Camden Street,
and who has been superintendent of the Sandall
Road School, familiar, therefore, with all new
methods—affirms confidently that the Science and
Art classes taken by Mr. Buss were “as good, and
quite as interesting, as anything given now.”
This is confirmed by Mrs. Pierson, who says of
these very happy school-days—

“Her dear father greatly added to the enjoyment of school

life by giving us courses of lectures illustrated by diagrams
on geology, astronomy, botany, zoology, and chemistry, quite
equal to those given by highly paid professors of the present
day, and he gave them for love, and nothing extra was put
down in the bills, although each course was an education by
itself, given in his lucid and most interesting way.”

These lectures, as Mrs. S. Buss says in her

reminiscences—

“awakened in many a pupil the thirst for reading and

study. His artistic talent, and the pleasant excursions for
sketching from Nature, were novel inspirations in the days
when the ordinary girlish specimens of copied drawings
resembled nothing in Nature. A good elocutionist himself, he
taught us to read and recite with expression.”
 His daughter had the same gift, inherited or
acquired, and her school has always been specially
distinguished in all examinations for the excellence of
the reading.
Mrs. S. Buss mentions, in addition to Mrs. Laing, as
also specially interested in the school—

“the Rev. Canon Dale, Vicar of St. Pancras, and his two
sons, Pelham and Lawford Dale; the Rev. Cornelius Hart,
Vicar of Old St. Pancras; the Rev. R. P. Clemenger, Vicar of
St. Thomas’, Agar Town; the Revs. E. Spooner and Charles
Lee, the immediate successors of Mr. Laing; the Countess of
Hardwicke, one of the earliest and most faithful friends of
the school, whose daughter, Lady Elizabeth Biddulph, still
continues the yearly prize for good conduct, and whose
warm letter of sympathy, in January, was one of the many
we received. We all remember, too, Judge Payne, and his
witty impromptu verses at so many prize-givings.”

When we listen to these memories of the earlier

school-days, we cannot dispute the position that—

“The foundation of the North London Collegiate School for

Ladies was not merely the commencement of one special
school, but was an era in education. If we very old pupils can
carry our mind back to the time when the ‘Guide to
Knowledge’ and ‘Mangnall’s Questions’ were the chief
standard school-books for most of the scientific and historical
instruction that girls received; when the mildest form of
gymnastics (such as jumping over a stick held a few inches
above the ground) was deemed so unladylike that some girls
were withdrawn from the earliest classes formed; when the
study of the most rudimentary physiology horrified the Mrs.
Grundies of the period, who would not permit their
daughters to continue the course after the first lesson (like
the mother of later times at the primary school, who wrote
to the teacher, ‘Mrs. S—— asks that my Mary Jane do not go
again to those lessons where they talk about their bodies:
first, which it is nasty; and second, which it is rude!’); the
time when we learnt pages of dictionary, with meanings, in
the first class, and rules of dry-as-dust grammar, without any
meaning to us for years afterwards; the time when it was
asserted and believed, that a girl’s mind was incapable of
grasping any mathematical knowledge beyond the first four
rules of arithmetic;—we can, remembering those good old
times, see what a wonderful stride was taken in girls’
education by the North London Collegiate School, even in its
infancy. Can we not recall those long tramps, to and fro,
when the present North London Railway ran only between
Chalk Farm and Fenchurch Street, and when there was no
service of omnibuses between the various districts? Fares,
even when a conveyance could be had, were fares, sixpence
or a shilling. Do we not remember the overskirts insisted on
by Miss Buss as a protection from the wet, at a time when
waterproof clothing was unknown? What dressing and
undressing went on round the stove, where Miss Reneau sat
with the default list, to put down the name of any too riotous
girl! What a delight the giant strides and see-saws were to
the athletic young damsels of the period, while the more
staid elders waited anxiously for the chance of a turn with
the dear head-mistress, who gave up her hour of leisure to
talk and walk with us on the playground, and give us a word
of sympathy, counsel, or encouragement, or tell some funny
story, or teach some new game, sharing her brimming cup of
life with us all—ever regardless of her own need of rest!”
 From letters at this period we have a glimpse of
this young head-mistress at work and at play, both of
which she did very thoroughly. The work must have
been rather overdone, and we may admire the self-
control which is remembered as so marked a
characteristic, when we see that it came from real
self-conquest. In 1859 she writes to her brother
Septimus, speaking of herself and her cousin Maria
(Mrs. Septimus Buss)—

“As usual at this period—and, for that matter, at most

periods—of the year, we are over-worked. At times I am so
irritable I feel inclined to throw things at people, and twice
this week I have allowed myself to be provoked into a fit of
temper. It is so grievous afterwards to reflect upon. Why was
I made so gunpowdery? I do think, however, the provocation
was very great, though that, of course, is no excuse.”

The next letter is to her father in holiday-time:—

“Dinan, 1860.
“Everything has combined to make this holiday delightful,
and I am so well and happy, that I feel as if I was only
twenty years of age, instead of a hundred, as I do in
Camden Street. I find myself talking slang to the boys, and
actually shouting fag-ends of absurd choruses from mere
lightheartedness.
“I am very sorry to say that I do not feel any more
industrious, though doubtless I shall have to recover from
that complaint in London. Also I regret to say that I have to-
day incurred the severe displeasure of our wee blue-eyed
laddie!”
 CHAPTER III.

INFLUENCE.

“You were the sower of a deathless seed,

The reaper of a glorious harvest, too;
But man is greater than his greatest deed,
And nobler than your noblest work were you!”
Emily Hickey.

“I am always thinking of the first time I ever saw her—in

the old house in Camden Street, when I was seven years old,
a timid child, sent upstairs with a message, which I stood
and mumbled at the door. I remember her now—an elegant
dark young lady, she seemed to me—with curls and a low-
necked dress, as we all had then. She told me to come
forward and deliver my message as if I wasn’t frightened;
and I remember now how her vigorous intensity seemed to
sweep me up like a strong wind. And that is forty-four years
ago!”

This graphic sketch, from the pen of Mrs. Alfred

Marks, gives us the young head of the new school as
she must have looked in 1850, when the first venture
in Clarence Road became the North London
Collegiate School for Ladies, reconstructed after the
lines of Queen’s College, founded two years before.
Among the many appreciative notices with which
the entire press of England met the news of the
death of one of the foremost educators of the time,
none went so straight to the mark as that of a
country paper, the Bath Herald, which seized on the
most distinctive point of this remarkable personality.
After observing that it is rare for the influence of a
school-mistress to be felt beyond her immediate
circle, it thus proceeds—

“There is not a county of her native country, not a colony

of its empire, where the news of this death will not have
saddened the hearts of pupils and friends.
“When she began her great work the matter of girls’
education was still a ‘question.’ Miss Buss solved it in the
most direct and practical fashion; and every college for
women, and every high school for girls, is a memorial of her
labours. A personality of singular charm, and of what the
slang of the day calls ‘magnetism,’ wholly without pedantry
or self-consciousness, persuaded Royal Commissioners, City
Companies, Lord Mayors and Royal Princesses, physicians,
and even Universities, that women might be thoroughly
educated without any danger to themselves or the State. To
mention her name to any one of the many thousand pupils
scattered over the face of the earth, was to raise constantly
emotions of affection and pride. Undoubtedly she was one of
the ‘pioneers’ of the century, and is secure of a niche in the
temples of memory and of fame.”
 These words are written at the end of her career,
but they were true from the beginning. It is most
truly characteristic of her that her power was
exercised without self-consciousness. On one
occasion I had remarked on her wonderful influence,
and find her answer in a brief sentence, after which
she turns to some more practical subject with her
instinctive distaste for introspection or self-
dissection: “What you say about personal influence
strikes me curiously. I cannot possibly measure it or
even understand it. To a certain extent I am
conscious of an influence over young girls, but am
not able to explain it.”
To those who knew her well, the explanation
comes readily enough as we find her power of
impressing others to be the result of the vividness of
her sympathy, and of the imagination which,
transcending mere personal limitations, is able
actually to enter into the life of others, no matter
how diverse in temperament or in circumstances.
Speaking of her as she was in middle life, Mrs.
Marks offers a suggestion full of interest, as she says
—

“Her utter spontaneity, her sense of people and things in

their living essences, made a very deep and lasting
impression on me. And some kind words she said to me—
 which showed she had seen into my very heart—were a
greater encouragement to me than I can express. Their
meaning was that she felt I was spontaneous, and had not
settled down into conventionality; and as things were very
real to me, it was a comfort to know that she too thought
them so.”

It was doubtless as a direct consequence of this

vision of the “soul of things” that the mere names of
things meant so much more to Miss Buss than to
most of us, to whom in general a name is the mere
husk of the thing it stands for. Seeing through these
names as she did, they stood to her for all the living
reality of which they were the symbols. With the
name, she came into possession of all that went to
make up the personality represented by it.
Surroundings, time, place, with every other relation,
became an inseparable part of any name that once
fixed itself in this truly royal memory. To every one
who met her it was a standing wonder how she
could know so much of the thousands of girls who
had passed under her care. That she did know them
is a fact that comes into almost every memorial
relating to her, from those first simple days when she
gave herself without stint to the little band of pupils,
up to the very last, when her circle of influence was
bounded only by the bounds of the empire itself.
 It is not surprising that so many of these girls
should bear for life the impress of this strong
influence. But still there is something to call for
comment in the depth of the feeling thus aroused.
Before even the suggestion of approaching death
had lifted the veil of commonplace, which so often
hides from us the beauty of those with whom we
walk the dusty path of everyday life, there came, in
answer to questions about the “story of the school,”
so many reminiscences of the early days, giving the
freshness of early enthusiasm, all undimmed by the
daily intercourse of nearly fifty years, that one could
not but marvel.
Many of those first pupils have remained as
teachers, many others have settled in the
neighbourhood as friends, and to not a few this deep
affection has been the master-passion of their lives.
In the wisdom of these later times it is thought well
to chill the fervour of the too engrossing devotion to
which very young enthusiasts are prone. But nothing
seems to have checked the ardour of these early
days, while only good has resulted from a love which
has moulded so many lives to strength and beauty.
One of the old pupils says of this time—

“She was true, so staunch, so utterly wanting in all the

little pettinesses that so often mar even noble characters,
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebookfinal.com