How to install Kafka with Zookeeper on Ubuntu?

Apache Kafka is an open-source distributed event streaming platform used to handle real-time data feeds. It is designed to handle a high volume, high throughput, and low latency data streams, and can be used to process, store, and analyze data in real-time. Kafka can be used to build real-time streaming data pipelines and applications and is often used in combination with Apache Storm, Apache Hadoop, and Apache Spark.

Install Kafka with Zookeeper on Ubuntu

Follow the below steps to install Kafka with Zookeeper on Ubuntu.

Step 1: Install Java JDK version 11

Ensure that Java is installed on your system. If it is not, you can install it by running the command to install java on your machine. 

sudo apt-get install openjdk-11-jdk 

Step  2: Go to https://kafka.apache.org/downloads and download the binary version of Apache Kafka.

Go to Apache Kafka's website and select one of the binary downloads. It is recommended to choose the most recent version that corresponds to the Scala version you are using (for example, version 2.13).

Download the binary and extract its contents to a directory of your choice (for example, ~/kafka_2.13-3.2.3).

Step 3: Extract the contents of the downloaded file to your desired location

Open a terminal and navigate to the root directory of the extracted Apache Kafka folder. If you have extracted it in the directory ~/kafka_2.13-3.2.3, navigate to that directory.

ls

Step  4: Use the extracted binaries to start Zookeeper

Apache Kafka relies on Zookeeper for cluster management, so before starting Kafka, Zookeeper must be started first. Zookeeper is included with Apache Kafka, so there is no need to separately install it.

To start Zookeeper, navigate to the root directory of Apache Kafka and run the following command:

bin/zookeeper-server-start.sh config/zookeeper.properties

Step 5: In another process, use the same binaries to start Kafka.

In a separate terminal window, navigate to the root directory of Apache Kafka and execute the following command to start Apache Kafka.

bin/kafka-server-start.sh config/server.properties

Make sure to keep both terminal windows open, otherwise, you will shut down both Kafka and Zookeeper.

Step 6: Lastly, set up the environment variables for easy access to the Kafka binaries by adding the path to the $PATH.

To simplify access to the Kafka binaries, you can update your PATH variable by adding the following line (adjust the path to match your system) to the file that runs system commands on your machine (for example, ~/.zshrc if you are using zshrc):

PATH="$PATH:/Users/stephanemaarek/kafka_2.13-3.0.0/bin"

This allows you to run Kafka commands without prefixing them with the path. Once you reload your terminal, you will be able to run the following commands from any directory.

Testing Kafka Setup

We will create a topic called "topic1" to test that everything is functioning properly, after starting both ZooKeeper and Kafka

You can see that Kafka has successfully created the topic "topic1". 

There is one more command to check if the zookeeper and Kafka are started successfully. Make sure that the zookeeper is running on port 2181.

echo dump | nc localhost 2181

Conclusion : 

In conclusion, installing Apache Kafka with Zookeeper on Ubuntu involves several steps: first, installing Java JDK version 11, then downloading and extracting the binary version of Apache Kafka from the official website, starting Zookeeper, and Kafka, and finally, setting up the environment variables for easy access to the Kafka binaries by adding the path to the $PATH. It is important to ensure that Java is installed on your system before proceeding with the installation. Additionally, Zookeeper is included with Apache Kafka and must be started before starting Kafka. By following these steps, you will be able to successfully install and run Apache Kafka and Zookeeper on Ubuntu.