Scribd
Upload
42 views1 page

Kafka

Kafka uses topics which are divided into partitions to distribute data across a cluster. Partitions allow for parallelism in consumption and help determine which broker a message gets sent to. Records in a partition are written sequentially in an append-only fashion with each record having a unique offset. Consumer groups ensure that partitions are consumed by only one consumer, allowing groups to scale by adding more partitions and consumers.

Uploaded by

engelo89
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
42 views1 page

Kafka

Kafka uses topics which are divided into partitions to distribute data across a cluster. Partitions allow for parallelism in consumption and help determine which broker a message gets sent to. Records in a partition are written sequentially in an append-only fashion with each record having a unique offset. Consumer groups ensure that partitions are consumed by only one consumer, allowing groups to scale by adding more partitions and consumers.

Uploaded by

engelo89
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 1

partition number = expectation


throughput/partition speed (10MB/s)

No message key  RoundRobin


Which partition a record should go 
Has message key  Hash key

Increate number of partition, can not


decrease

Each partition is a single log file where


Kafka Cluster Brokers Topic Partition  records are written to it in an append-only File descriptor index and actual data
fashion

Offset which is unique for each record within A partition can be consumed by one or more
Central Topic the partition consumers, each reading at different offsets.

When a consumer group consumes the


partitions of a topic, Kafka makes sure that
each partition is consumed by exactly one
The consumer group concept ensures that a consumer in the group.
message is only ever read by a single
consumer in the group
However, the maximum parallelism of a
group will be equal to the number of
partitions of that topic

You might also like