Confluent Platform Reference Architecture Kubernetes
Confluent Platform Reference Architecture Kubernetes
Viktor Gamov
August 2018
NOTE: All recommendations a valid (or not) for Kubernetes 1.9.x+ and Confluent Platform 5.0.x
Prerequisites
This document doesn’t provide a comprehensive overview of all Kubernetes features. If you’re not familiar with Kubernetes, we
recommend starting with the official Kubernetes Getting started guide.
To get more context about running a Kafka-based streaming platform, we recommend reading Gwen Shapira’s Recommendations
for Deploying Apache Kafka on Kubernetes
From that document, you will learn Kubernetes concepts that you need to be familiar with in order to understand this
document context.
• Pod
• ReplicationController
• StatefulSets
• Persistent Volumes
• Service, HeadlessService
CPU
ZooKeeper is not a CPU intensive application. For a production deployment, you should start with 2 CPUs and adjust as necessary.
For a demonstration deployment, you can set the CPUs as low as 0.5. The amount of CPU is configured by setting the StatefulSet’s
spec.template.containers[0].resources.requests.cpus.
ZooKeeper StatefulSet
An example of ZooKeeper StatefulSet configuration
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
...
spec:
...
template:
metadata:
....
spec:
containers:
...
image: "confluentinc/cp-zookeeper" (1)
imagePullPolicy: "OrderedReady" (2)
livenessProbe:
exec:
command: ['/bin/bash', '-c', 'echo "ruok" | nc -w 2 -q 2 localhost 2181 | grep imok']
initialDelaySeconds: 1
timeoutSeconds: 3
Kafka Brokers
Kafka brokers are the main storage and messaging components of Apache Kafka. Kafka is a streaming platform that uses
messaging semantics. The Kafka cluster maintains streams of messages called Topics; the topics are sharded into Partitions
(ordered, immutable logs of messages) and the partitions are replicated and distributed for high availability. The servers that
run the Kafka cluster are called Brokers.
Cluster Size
The size of the Kafka cluster, the number of brokers, is controlled by the .spec.replicas field of the StatefulSet. You should ensure that the
size of the cluster supports your planned throughput and latency requirements for all topics. If the size of the cluster gets too large, you
should consider segregating it into multiple smaller clusters.
We recommend having at least 3 Kafka brokers in a cluster, each running on a separate server. This way you can replicate each Kafka
partition at least 3 times and have a cluster that survives a failure of 2 nodes without data loss.
With 3 Kafka brokers, if any broker is not available, you won’t be able to create new topics with 3 replicas until all brokers are available again.
For this reason, if you have use-cases that require creating new topics frequently, we recommend running at least 4 brokers in a cluster.
If the Kafka cluster is not going to be highly loaded, it is acceptable to run Kafka brokers on the same servers as the ZooKeeper nodes.
In this case, it is recommended to allocate separate disks for ZooKeeper (as we’ll specify in the hardware recommendations below). For
high-throughput use cases, we do recommend installing Kafka brokers on separate nodes.
Disk
Disk throughput is the most common bottleneck that users encounter with Kafka. Given that Network Attached Storage backs
Persistent Volumes, the throughput is, in most cases, capped on a per Node basis without respect to the number of Persistent Volumes
that are attached to the Node. For instance, if you are deploying Kafka onto a GKE or GCP based Kubernetes cluster, and if you use the
standard PD type, your maximum sustained per instance throughput is 120 MB/s (Write) and 180 MB/s (Read). If you have multiple
applications, each with a Persistent Volume mounted, these numbers represent the total achievable throughput.
You can control the amount of disk allocated by your provisioner using the
.spec.volume.volumeClaimTemplates[0].resources.requests.storage field.
Memory
Kafka utilizes the OS page cache heavily to buffer data.
To understand the interaction of Kafka and Linux containers you should read the Kafka file system design and memory cgroups
documentation.
Keep in mind that page cache is managed by the kernel, which is used by all pods.
The JVM heap size of the Kafka brokers is controlled by `KAFKA_HEAP_OPTS` environment variable. "-Xms=2G -Xmx=2G" is sufficient
for most deployments.
NOTE: Currently, CP helm charts use Zulu OpenJDK build with the support of CGroups limits. JVM Heap settings will be
honored inside the container https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits
Manifests
A Kafka Kubernetes deployment consists of
• Kafka StatefulSet
• Kafka Service
• Kafka Headless Service
References
Kubernetes Storage Volumes
How to use SSD persistent disks on Google Kubernetes Engine
10 Apache, Apache Kafka, Kafka, and associated open source project names are trademarks of the Apache Software Foundation. ©2018 Confluent, Inc.