Scribd
Upload
22 views

Kafka Python

Uploaded by

oumarous
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
22 views

Kafka Python

Uploaded by

oumarous
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 5

30/08/2024 11:28 about:blank

Hands-on Lab: Kafka Python Client


Estimated time needed: 30 minutes

Objectives
After completing this lab, you will be able to:

Use kafka-python to interact with Kafka server in Python


Send and receive messages through the Kafka-python client

About Skills Network Cloud IDE


Skills Network Cloud IDE (based on Theia and Docker) provides an environment for hands on labs for
course and project-related labs. Theia is an open-source IDE (Integrated Development Environment) that can
be run on desktop or on the cloud. To complete this lab, we will be using the Cloud IDE based on Theia
running in a Docker container.

Important notice about this lab environment


Please be aware that sessions for this lab environment are not persistent. A new environment is created for
you every time you connect to this lab. Any data you may have saved in an earlier session will get lost. To
avoid losing your data, please plan to complete these labs in a single session.

Exercise 1: Download and extract Kafka


1. Open a new terminal by clicking the menu bar and selecting Terminal->New Terminal.

This will open a new terminal at the bottom of the screen.

Run the commands below on the newly opened terminal.

Note: You can copy the code by clicking the little copy button on the bottom right of the code
snippet below and then paste it, wherever you wish.

2. Download Kafka by running the command below:


1. 1

1. wget https://downloads.apache.org/kafka/3.8.0/kafka_2.13-3.8.0.tgz

Copied! Executed!

3. Extract Kafka from the zip file by running the command below.
1. 1

1. tar -xzf kafka_2.13-3.8.0.tgz

Copied! Executed!

about:blank 1/5
30/08/2024 11:28 about:blank
This creates a new directory kafka_2.13-3.8.0 in the current directory.

Exercise 2: Configure KRaft and start server


1. Change to the kafka_2.13-3.8.0 directory.
1. 1

1. cd kafka_2.13-3.8.0

Copied! Executed!

2. Generate a Cluster UUID that will uniquely identify the Kafka cluster.
1. 1

1. KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

Copied! Executed!

Note: This cluster ID will be used by the KRaft controller.

3. KRaft requires the log directories to be configured. Run the following command to configure the log
directories passing the cluster ID.

1. 1

1. bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties

Copied! Executed!

4. Now that KRaft is configured, you can start the Kafka server by running the following command.

1. 1

1. bin/kafka-server-start.sh config/kraft/server.properties

Copied!

Note: You can be sure it has started when you see an output contains messages that confirm the
Kafka Server started successfully.

Exercise 3: Create a topic in the admin.py file


Install kafka-python
1. Open a new terminal and navigate to the kafka_2.13-3.8.0 directory.
1. 1

1. cd kafka_2.13-3.8.0

Copied! Executed!

2. Install the kafka-python package by running the following command.

about:blank 2/5
30/08/2024 11:28 about:blank
1. 1

1. pip3 install kafka-python

Copied!

3. Create a file named admin.py by running the following command.

1. 1

1. touch admin.py

Copied! Executed!

4. Click the button below to open the file in edit mode and paste the following content in the file and save
it.

Open admin.py in IDE

1. 1
2. 2
3. 3
4. 4
5. 5
6. 6

1. from kafka.admin import KafkaAdminClient,NewTopic


2. admin_client = KafkaAdminClient(bootstrap_servers="localhost:9092", client_id='test')
3. topic_list = []
4. new_topic = NewTopic(name="bankbranch", num_partitions= 2, replication_factor=1)
5. topic_list.append(new_topic)
6. admin_client.create_topics(new_topics=topic_list)

Copied!

Note: We are creating a topic "bankbranch" through this code.

Exercise 4: Create the producer.py file


You need a producer to send messages to Kafka. You will find the code for the producer in the producer.py
file.

1. Create a file named producer.py by running the following command.


1. 1

1. touch producer.py

Copied! Executed!

2. Click the button below to open the file in edit mode and paste the following content in the file and save
it.

Open producer.py in IDE

1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8

about:blank 3/5
30/08/2024 11:28 about:blank
9. 9

1. from kafka import KafkaProducer


2. import json
3. producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))
4. producer.send("bankbranch", {'atmid':1, 'transid':100})
5. producer.send("bankbranch", {'atmid':2, 'transid':101})
6.
7. producer.flush()
8.
9. producer.close()

Copied!

In the above code, the producer is sending across two messages through this code. These messages will be
received by the consumer.

Exercise 5: Create the consumer.py file


You need a consumer to read messages from Kafka. The code for consumer will be written in the
consumer.py file.

1. Create a file named consumer.py by running the following command.

1. 1

1. touch consumer.py

Copied! Executed!

2. Click the button below to open the file in edit mode and paste the following content in the file and save
it.

Open consumer.py in IDE

1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
9. 9
10. 10

1. from kafka import KafkaConsumer


2. consumer = KafkaConsumer('bankbranch',
3. group_id=None,
4. bootstrap_servers=['localhost:9092'],
5. auto_offset_reset = 'earliest')
6. print("Hello")
7. print(consumer)
8.
9. for msg in consumer:
10. print(msg.value.decode("utf-8"))

Copied!

Exercise 6: Execute the three Python files


1. Execute admin.py and producer.py using the following commands in terminal:
about:blank 4/5
30/08/2024 11:28 about:blank
1. 1
2. 2

1. python3 admin.py
2. python3 producer.py

Copied! Executed!

2. Open a new terminal and execute the following commands to run consumer.py:

1. 1
2. 2

1. cd kafka_2.13-3.8.0
2. python3 consumer.py

Copied! Executed!

3. Your consumer should print the messages sent by the producer as follows:

Practice Exercise
1. Create a new producer from bankbranch in a file named new_producer.py which will take user input
as long as the user wants and accept user input for the ATM number they want to transact with (1 or 2)
and stream the transaction.

2. Observe the consumer getting the events streamed by the producer in real time.

Click here for the solution.

Authors
Ramesh Sana Reddy
Lavanya T S
Shreya Khurana

© IBM Corporation. All rights reserved.

about:blank 5/5

You might also like