Kafka Using Spring Boot v2
Kafka Using Spring Boot v2
Using
Spring Boot
Dilip Sundarraj
About Me
• Dilip
• Knowledge about building Kafka Clients using Producer and Consumer API
Order Payment
Service Service
Order Payment
Service Service
Retail App
Inventory Noti cation
Service Service Inventory Delivery
Service Service
Noti cation
Service
fi
fi
MicroServices Architecture
API API API
Inventory Cart
Order Service
Service Service
Streaming
Producer Consumer
platform
1 2 3 4 N
………
Events
Interface Events
Interface
Payment
Service Noti cation Delivery
API Service Service
API
fi
Traditional Kafka
Messaging System Streaming Platform
• Transient Message Persistance • Stores events based on a retention
ddd time. Events are Immutable
Kafka Cluster
Source Kafka Kafka
Connector Broker1 Broker2
Kafka Cluster Kafka Streams
Kafka Kafka
Broker3 Broker4
Sink StreamsAPI
Connector
File
DB System
Kafka
Consumers ConsumerAPI
ConnectAPI
Download Kafka
Kafka Topics
&
Partitions
Kafka Topics
• Topic is an Entity in Kafka with a name
Kafka Topics
• Topic is an Entity in Kafka with a name
Kafka Broker
Send to Poll
Kafka Producer TopicA TopicA Kafka Consumer
TopicA
ABC ABC
Topic and Partitions
• Partition is where the message lives inside the topic
0 1 2 3 4 5
Send to
Partition 0
Kafka Producer TopicA
Partition 1 0 1 2 3 4 5 6 7
Setting up
Zookeeper
&
Kafka Broker
Setting up Kafka in Local
Broker registered
with zookeeper
Sending
Kafka Messages
With
Key and Value
Kafka Message
• Kafka Message these sent from producer has two properties
• Key (optional)
• Value
Sending Message Without Key
test-topic
Partition 0 Apple
Send to
Kafka Producer TopicA
Partitioner
Partition 1 Adam
Apple Adam
Partition 2 Alpha
Alpha Angel
Partition 3 Angel
Sending Message With Key
test-topic
K:A K: A K: A K: A
Send to Partition 0 V:Apple V: Adam V:Alpha V: Angel
Kafka Producer test-topic
Partitioner
K:B K: B K: B K: B
Partition 1 V:Boy V: Ben V:Beta V: Becky
Key: A Key: A
Apple Adam
K:C K: C K: C K: C
Partition 2 V:Cat V: Carl V:Cam V: Cathy
Key: A Key: A
Alpha Angel
Partition 3
Key: B Boy Key: B Ben
From beginning
From beginning
Producer
Partition 0 Partition 1 Partition 2 Partition 3
P0 P1
P2 P3
Consumer Groups
test-topic
Consumer A Consumer A
group.id = group1 group.id = group1
P0 P1 P2 P3
Consumer Groups
test-topic
P0 P1 P2 P3
Consumer Groups
test-topic
group.id = group1 group.id = group1 group.id = group1 group.id = group1 group.id = group1
P0 P1 P2 P3 Idle
Consumer Groups
test-topic
Consumer A Consumer A
P2 P3
Group2
Group1
Consumer Groups : Summary
• Consumer Groups are used for scalable message consumption
Send to Poll
Kafka Producer TopicA TopicA Kafka Consumer
TopicA
ABC
Bytes
00000000000000000000.log
Retention Policy
• Determines how long the message is retained ?
System 1
Client Client
System 2 System 3
Characteristics of Distributed System
Client
• Availability and Fault Tolerance
• Easily Scalable
System 2 System 3
Client System 4
Kafka as a Distributed System
Producer Consumer
Kafka Broker
Producer Producer Consumer Consumer
Consumer Consumer
Producer Producer
File File File
System System System
./kafka-server-start.sh ../config/server.properties
Setting up Kafka Cluster
- New server.properties les with the new broker details.
broker.id=<unique-broker-d>
listeners=PLAINTEXT://localhost:<unique-port>
log.dirs=/tmp/<unique-kafka-folder>
auto.create.topics.enable=false(optional)
Example: server-1.properties
broker.id=1
listeners=PLAINTEXT://localhost:9093
log.dirs=/tmp/kafka-logs-1
auto.create.topics.enable=false(optional)
fi
How Kafka Distributes the
Client Requests?
How Topics are distributed?
./kafka-topics.sh -
-create --topic test-topic-replicated
-zookeeper localhost:2181
--replication-factor 3
--partitions 3
Controller
test-topic-replicated
Kafka Cluster
How Kafka Distributes Client Requests?
Kafka Producer
Partition-0
Kafka Send to
test-topic- Partitioner Partition-1
Producer replicated Leader of Leader of Leader of
Partition-2 Partition0 Partition1 Partition2
ABC GHI
DEF
test-topic-replicated
Kafka Cluster
How Kafka Distributes Client Requests?
Kafka Consumer
Kafka Cluster
Leader of Leader of Leader of
Partition0 Partition1 Partition2
Poll
test-topic- Kafka Consumer
replicated
Records processed
Partition-0 Partition-1 Partition-2 Successfully
test-topic-replicated
Records processed
Successfully
Poll
test-topic Kafka Consumer
Partition-0 Partition-1 Partition-2 (partition-1) group.id=group1
Poll
Broker 1 Broker 2 Broker 3 test-topic
Kafka Consumer
(partition-2) group.id=group1
File System File System File System
Records processed
Successfully
Summary : How Kafka Distributes the Client
Requests?
• Partition leaders are assigned during topic Creation
• Clients will only invoke leader of the partition to produce and consume
data
test-topic-replicated
Replication factor = 3
Partition-0 Partition-1 Partition-2
Leader Replica
Kafka Partition-1 Partition-0 Partition-0
Producer (Follower) (Follower) (Follower)
Librarian
Library Inventory Architecture
MicroService 1
Librarian
Kafka
API Producer
Library-events
MicroService 2
Kafka H2
Consumer (In-memory)
Library-events
MicroService 2
Kafka H2
Consumer (In-memory)
Library-events
MicroService 2
Kafka H2
Consumer (In-memory)
Library-events
MicroService 2
Kafka H2
Consumer (In-memory)
Librarian MicroService 1
POST Kafka
Producer
New Book API
PUT
Update Book
Library Events Producer
Library-events
KafkaTemplate
Kafka Producer in Spring
KafkaTemplate
KafkaTemplate.send() Library-events
KafkaTemplate.send()
Mandatory Values:
bootstrap-servers: localhost:9092,localhost:9093,localhost:9094
key-serializer: org.apache.kafka.common.serialization.IntegerSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
fi
KafkaTemplate AutoCon guration
application.yml
spring:
profiles: local
kafka:
producer:
bootstrap-servers: localhost:9092,localhost:9093,localhost:9094
key-serializer: org.apache.kafka.common.serialization.IntegerSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
fi
Library Inventory Architecture
Librarian MicroService 1
Library-events
MicroService 2
Kafka H2
Consumer (In-memory)
fi
fi
Introduction
To
Automated Tests
Why Automated Tests ?
• Manual testing is time consuming
• UnitTest
• Integration Tests
• Spock
Integration Tests
Using
JUnit5
What is Integration Test?
• Test combines the di erent layers of the code and verify the behavior is
working as expected.
MicroService 1
1 2
MicroService 1
Embedded
LibraryControllerIntegrationTest
Kafka
Why Embedded Kafka ?
Library Events Producer
Embedded
• Test all the code as like you interact with LibraryControllerIntegrationTest
Kafka
Kafka
Unit Tests
Using
JUnit5
What is Unit Test?
• Test the just focuses on a single unit (method)
MicroService 1
MicroService 1
Controller Mock
Kafka Producer
Mockito
• Unit Tests cover scenarios that’s not possible with Integration tests
Library Events Producer API
Librarian MicroService 1
POST Kafka
Producer
New Book API
PUT
Update Book
Library Events Producer
Library-events
PUT - “/v1/libraryevent”
• libraryEventId is a mandatory eld
{
"libraryEventId": 123,
"eventStatus": null,
"book": {
"bookId": 456,
"bookName": "Kafka Using Spring Boot",
"bookAuthor": "Dilip"
}
}
fi
Kafka Producer Con gurations
fi
Kafka Producer Con gurations
• acks
fi
Kafka Producer Con gurations
• retries
• retry.backo .ms
Library-events
Library Events Producer
MicroService 2
Kafka H2
Consumer (In-memory)
Poll
library-events Kafka Consumer
Records processed
Successfully
Library-events
Spring Kafka Consumer
• MessageListenerContainer
• KafkaMessageListenerContainer
• ConcurrentMessageListenerContainer
• @KafkaLisener Annotation
• Single Threaded
ff
ConcurrentMessageListenerContainer
@Configuration
@EnableKafka
@Slf4j
public class LibraryEventsConsumerConfig {
fi
KafkaConsumer Con g
key-deserializer: org.apache.kafka.common.serialization.IntegerDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
group-id: library-events-listener-group
fi
Consumer Groups
&
Rebalance
Consumer Groups
Multiple instances of the same application with the same group id.
Rebalance
• Changing the partition ownership from one consumer to another
Library-events -consumer
group.id : Library-events-listener-group
Library-events
P0 P1 P2
Rebalance
• Changing the partition ownership from one consumer to another
Library-events -consumer
group.id : Library-events-listener-group
Library-events
P0 P2
Group-Co-ordinator
P1
Committing Offsets
Poll
library-events Kafka Consumer
Records processed
Successfully
Library-events
MicroService 2
Kafka Consumer H2
(In-memory)
Exhausted
RECOVERY
Retry and Recovery
Recovery - Type 1
Exhausted
Recovery - Type 2
Exhausted
Issues with Recovery ?
Producer
Exhausted
Error Handling
in
Kafka Producer
Library Events Producer API
Librarian MicroService 1
POST Kafka
Producer
New Book API Library-events
PUT
Update Book
Library Events Producer
Kafka Producer Errors
• Kafka Cluster is not available
PUT Library-events
min.insync.replicas = 2
Kafka Cluster
• Banking
• Retail
• Insurance
• Encryption
• Authentication
Without SSL Encryption
Client
Server
(amazon.com)
Plaintext
?
private key
Public Key Cryptography
SSL Authentication
• SSL Authentication is to make sure you are talking to the correct server
Server
Client
(amazon.com)
Hacker
(amazon.com)
How SSL Authentication works ?
Secure
Server
(amazon.com)
SSL cert
SSL Requests by Enterprise
Secure
Client abc.com
SSl Cert
One Way Authentication
Signed Certi cate
keystore le
TrustStore
(CA Root Certi cate)
Requesting CSR
TrustStore
Secure
SSL Cert
TrustStore KeyStore
KeyStore TrustStore
Kafka SSL SetUp
SSL Set Up Steps
1. Generate server.keystore.jks
Kafka Kafka
2. SetUp Local Certi cate Authority Clients Cluster
6. Con gure the SSL cert in our Kafka Broker Local Certi cate Authority