Scribd
Upload
192 views3 pages

Top 10 Kafka Problems

The document outlines 10 common problems encountered when using Apache Kafka as a distributed messaging platform. These include alerts when topics are under-replicated, issues with broker liveness checks affecting automation, new brokers impacting performance during partition reassignment, questions around its long-term storage capabilities, challenges finding optimal data retention settings, complex data transformations, difficulties with upscaling and rebalancing topics, MirrorMaker not replicating offsets between clusters, its limited support for some messaging paradigms, and reduced performance when modifying messages. The document recommends checking Apache Pulsar as an alternative that addresses many of these issues.

Uploaded by

gogula sivannarayana
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
192 views3 pages

Top 10 Kafka Problems

The document outlines 10 common problems encountered when using Apache Kafka as a distributed messaging platform. These include alerts when topics are under-replicated, issues with broker liveness checks affecting automation, new brokers impacting performance during partition reassignment, questions around its long-term storage capabilities, challenges finding optimal data retention settings, complex data transformations, difficulties with upscaling and rebalancing topics, MirrorMaker not replicating offsets between clusters, its limited support for some messaging paradigms, and reduced performance when modifying messages. The document recommends checking Apache Pulsar as an alternative that addresses many of these issues.

Uploaded by

gogula sivannarayana
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
You are on page 1/ 3

https://pandio.

com/blog/top-10-problems-when-using-apache-kafka/
Top 10 Problems When Using Apache Kafka
Apache Kafka is one of the most popularly used open-
source(https://opensource.com/resources/what-open-source) distributed event
streaming platforms. Its use cases range from enabling mission-critical apps to
building and maintaining high-performance data pipelines. If you consider whether
to use Apache Kafka for your future projects, you should know all about the pros
and cons of using it. In today’s article, we will stick to the cons. While Apache
Kafka is a solid distributed messaging platform, it has some limitations. To paint
the picture, we have put together the top 10 problems when using Apache Kafka.

1. In Sync Replica Alerts


Kafka In Sync Replica Alert tells you that some of the topics are under-
replicated. The data is simply not being replicated to brokers. These alerts
indicate a potentially serious problem because the probability of data being lost
becomes higher. It can happen entirely unexpectedly, even if you do nothing on your
side. It usually takes place when downlevel clients affect the volume of data.

A spike in data volume causes the Kafka broker to back up message conversion.
However, the problem has to be addressed as soon as possible. Usually, the
questionable broker has to be fixed for the entire system to be operational again.

2. Kafka Liveness Check Problems and Automation


The Kafka liveness check problems can quickly occur if the host where the
liveness check is running cannot reach the host where the broker is running. If
this happens, the broker will keep on restarting. Meanwhile, all the downlevel
clients won’t be able to run their apps. It can become a real nuance if you want to
automate some of your tasks on Kafka.

Why? Because you need to enable liveness check to streamline automation and
make sure that the broker’s client-serving port is open. You can simply write a
piece of code to restart the broker when the port is not open. But if the broker
falls into a dead-loop and keeps restarting, your entire infrastructure is rendered
useless. Is there a quick fix? Simply turn off the liveness check.

3. New Brokers Can Impact the Performance


Staging a new cluster and installing the broker software on Apache Kafka
(https://pandio.com/blog/how-to-fix-apache-kafka-error-nobrokersavailable/) is
straightforward. Adding new brokers should not cause any problems, right? Pushing a
new Kafka broker into production can potentially impact the performance and cause
serious latency and missing file problems.

The broker can work properly before the partition reassign process is
completed. Devs usually forget about it and use the default commands from the
documentation. Moving thousands of partitions to the staging cluster can take
hours. And, until all the partitions have been moved, its performance will suffer.
This is why you should be careful and have a plan when you want to add a new broker
to the infrastructure.

4. Questionable Long-Term Storage Solution


If you are working with large sets of data, using Apache Kafka to store it
might cause you several problems. The major problem comes from Kafka storing
redundant copies of data. It can affect the performance, but, more importantly, it
can significantly increase your storage costs.

The best solution would be to use Kafka only for storing data for a brief
period and migrate data to a relational or non-relational database, depending on
your specific requirements.
5. Finding Perfect Data Retention Settings
While we are discussing long-term storage solution
problems(https://pandio.com/blog/zero-data-loss-a-reality-in-apache-pulsar-not-
true-with-kafka/), let’s point out one additional issue related to it. The
downstream clients often have completely unpredictable data request patterns. This
makes finding the perfect and most optimal data retention settings somewhat of a
problem.

Kafka stores messages in topics. This data can take up significant disk space
on your brokers. To dump the data, you need to set the retention period or
configurable size. If you don’t tune the data
retention(https://pandio.com/blog/pulsar-with-pandio-dont-use-apache-kafka/)
settings correctly, you risk either rendering data useless or paying way too much
for storage than you should have to in the first place.

6. Overly Complex Data Transformations on-Fly


Using Apache Kafka on big data integration and migration projects can become
too complex. How come? Kafka was built to streamline delivering messages, and the
platform excels at it. However, you will run into some problems if you want to
transform data on-fly.

Even with Kafka Stream API, you will have to spend days building complex data
pipelines(https://pandio.com/blog/challenges-building-big-data-pipelines/) and
managing the interaction between data producers and data consumers. Not to mention
having to deal with and manage a system this complex. There are other distributed
messaging systems that are much better for streamlining
ETL(https://pandio.com/blog/what-is-etl-benefits-challenges-recent-advances/) jobs,
such as Apache Pulsar.

7. Upscaling and Topic Rebalancing


The volume of your data streams can go in both directions. This is why it is
crucial to choose a distributed messaging platform easy to scale up and down. With
Kafka, this is a problem because you need to balance things manually to reduce
resource bottlenecks.

You will have to do it every time a major change in the data stream occurs.
And do it both via partition leadership balancing and Kafka reassign partition
script. At the same time, with stateless brokers, Apache Pulsar makes the scale-out
process (https://pandio.com/blog/how-apache-pulsar-solves-kafkas-scalability-
issues/) significantly easier.

8. MirrorMaker Doesn’t Replicate the Topic Offsets


MirrorMaker is one of Kafka’s features that allows you to make copies of your
clusters. This would be a great disaster recovery plan if it weren’t for one
downside. MirrorMaker doesn’t replicate the topic offsets between the clusters. You
will have to create unique keys in messages to overcome this problem which can
become a daunting task when you are working at scale.

9. Not All Messaging Paradigms Are Included


While Apache Kafka comes with many messaging paradigms, some are still
missing.This can turn into a real problem if you need to extend your infrastructure
use case.It limits the Kafka capability to support building complex data pipelines.

Two major messaging paradigms not supported in Kafka are point-to-point


queues and request/reply queues

10. Changing Messages Reduces Performance


If you want to use Apache Kafka to deliver messages as they are, you will
have no issues performance-wise. However, the problem occurs once you wish to
modify the messages before you deliver them.

Manipulating data on the fly is possible with Kafka, but the system it uses
has some limits. It uses system calls to do it, and modifying messages makes the
entire platform perform significantly slower.

Nevertheless, many giants across industries use Apache Kafka, including


Twitter, Netflix, and LinkedIn. These ten problems are quite specific to Kafka, and
they might affect your implementation of the distributed messaging solution in a
specific case. Feel free to check Pandio if you want to learn more about Apache
Pulsar, a distributed messaging platform (https://pandio.com/blog/apache-kafka-or-
apache-pulsar-which-is-better-and-why/) that outperforms Kafka in almost every
possible use case and is positioned for the ML workloads of the future.

callHome ams 5f492ceeba1fc80006f6af54

You might also like