Scribd
Upload
14 views2 pages

Kafka

Uploaded by

aghakhania54
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
14 views2 pages

Kafka

Uploaded by

aghakhania54
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

md2pdf - Markdown to PDF https://md2pdf.netlify.

app/

Consuming
1. Handle errors and logs :

using var consumer = new ConsumerBuilder<Ignore,string>(_consumerConfig)


.SetErrorHandler((_,error)=> HandleKafkaError(error))
.SetLogHandler((_,log)=> HandleLog(log)).Build();

2. AutoOffset Rest :

• latest : If set to "latest," the consumer will start consuming from the latest available offset in the
partition. This means it will read only the messages that are produced after the consumer starts.
• earliest : If set to "earliest," the consumer will start consuming from the earliest available offset
in the partition. This means it will read from the beginning of the partition's log.

3. Partitions: Kafka uses topic partitioning to improve scalability. In partitioning a topic, Kafka breaks
it into fractions and stores each of them in different nodes of its distributed system. That number
of fractions is determined by us or by the cluster default configurations. ==Kafka guarantees the
order of the events within the same topic partition==

4. Parallelism: Kafka allows you to process multiple messages in parallel by having multiple
partitions. Each partition can be consumed by a separate consumer, enabling concurrent
processing of messages within a topic.

5. Consumer Group: Consumers in Kafka are part of a consumer group. Each partition in a topic
can be consumed by ==only one consumer within a consumer group==. However, multiple
consumer groups can consume from the same topic independently, enabling different
applications or processing pipelines to process the same data.

Partitioning
The producer will decide target partition to place any message, depending on:

• Partition id, if it's specified within the message


• MessageKey
• Round robin if neither partition id nor message key is available in the message means only the
value is available

In one consumer group, each partition will be processed by one consumer only. These are the

1 of 2 11/15/23, 11:51
md2pdf - Markdown to PDF https://md2pdf.netlify.app/

possible scenarios :

• If the number of consumers is less than the number of topic partitions, then multiple partitions
can be assigned to one of the consumers in the group.
• If the number of consumers is the same as the number of topic partitions, the every consumer
gets to consume messages of only one partition
• if the number of consumers is higher than the number of topic partitions, then partition and
consumer mapping can be as seen below, Not effective

2 of 2 11/15/23, 11:51

You might also like