Scribd
Upload
14 views

Kafka Topic and its parameters

Uploaded by

Krishnaraghava
Copyright
© © All Rights Reserved
Available Formats
Download as DOC, PDF, TXT or read online on Scribd
14 views

Kafka Topic and its parameters

Uploaded by

Krishnaraghava
Copyright
© © All Rights Reserved
Available Formats
Download as DOC, PDF, TXT or read online on Scribd
You are on page 1/ 2

What is a Kafka topic and what are its parameters

A Kafka topic is a category or feed name to which records (messages) are sent by
producers and from which records are retrieved by consumers in Apache Kafka. It serves
as the primary mechanism for organizing data in Kafka. Each topic acts as a log, storing
data in an ordered, append-only fashion across multiple partitions.

Key Features of Kafka Topics:

 Partitions: Topics are divided into partitions for scalability and fault tolerance.
 Replication: Each partition can be replicated across multiple brokers to ensure
reliability.
 Retention: Messages in a topic are stored for a configurable duration, regardless
of whether they are consumed.

Parameters of a Kafka Topic

1. Partitions:
o Determines how the topic is split across brokers.
o Partitions are used to distribute messages for scalability.
o Each partition is an ordered log of messages.
2. Replication Factor:

o Specifies the number of copies of a partition across brokers.


o A higher replication factor increases fault tolerance.
3. Retention Time:

o Defines how long messages are retained in the topic.


o Configured in milliseconds (e.g., retention.ms).
4. Retention Size:

oSets the maximum size of data to retain per partition.


oOnce this limit is reached, older messages are deleted (retention.bytes).
5. Cleanup Policy:

o Controls how messages are retained or deleted:


 delete: Removes messages after retention criteria are met.
 compact: Retains the latest value for each key (log compaction).
6. Min In-Sync Replicas:

o Defines the minimum number of replicas that must acknowledge a write


for it to be considered successful (min.insync.replicas).
7. Segment Size:

o Determines the size of log segments for partitions (segment.bytes).


8. Compression Type:

o Specifies the compression algorithm for messages (e.g., gzip, snappy, lz4, or
zstd).

9. Message Size:
o Defines the maximum size of a single message (max.message.bytes).
10. Replication Acknowledgment:

o Configured using acks in the producer:


 0: No acknowledgment required.
 1: Leader partition acknowledges.
 all: All in-sync replicas must acknowledge.
11. Default Message Format:

o Determines the version of the message format (log.message.format.version).


12. Unclean Leader Election:

o Defines whether a new leader can be elected from out-of-sync replicas in


case of failure (unclean.leader.election.enable).
13. Throttle Parameters:

o Controls limits on producers or consumers (e.g., quota.producer.byte-rate).

By adjusting these parameters, Kafka topics can be fine-tuned for specific requirements
like throughput, fault tolerance, and message durability.

You might also like