Scribd
Upload
109 views7 pages

Lab Manual - ETL-KAFKA-TALEND

kafka all experiments

Uploaded by

Sai gnan teja Chandragiri
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
109 views7 pages

Lab Manual - ETL-KAFKA-TALEND

kafka all experiments

Uploaded by

Sai gnan teja Chandragiri
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 7

Lab Manual

Lab Manual for ETL using Apache Kafka and Talend

Objective: This lab manual aims to provide hands-on experience with Apache Kafka and its
integration into the ETL pipeline using tools such as Talend. Each experiment is designed to
deepen understanding of Kafka's core concepts, configurations, and capabilities, as well as to
explore its use in real-time data processing.

Experiment-1: Install Apache Kafka on a Single Node

1. Objective: Install Apache Kafka and verify its installation.


2. Requirements: Java 8 or higher, Apache Kafka binaries.
3. Steps:
o Download and extract Kafka.
o Configure the server.properties file.
o Start the Zookeeper server.
o Start the Kafka server.
o Verify installation by listing available topics.

Output:

Install Java Output: java version "11.0.12" 2021-07-20\nOpenJDK Runtime Environment... This confirms
Java is successfully installed.

Download Kafka No specific output is shown as this is just downloading and extracting files.

Start ZooKeeper Output: [2024-xx-xx xx:xx:xx,xxx] INFO binding to port 0.0.0.0/0.0.0.0:2181 This
indicates ZooKeeper has started successfully and is listening on port 2181.

Start Kafka Server Output: [2024-xx-xx xx:xx:xx,xxx] INFO [KafkaServer id=0] started This shows the Kafka
broker has started successfully.

Create a Topic Output: Created topic quickstart-events This confirms the test topic was created
successfully
Experiment 2: Setting up a Single-Node, Single-Broker Kafka Cluster

1. Objective: Configure and demonstrate basic Kafka operations.


2. Steps:
o Start a single-broker Kafka cluster.
o Use the Kafka CLI to:
 Create a topic.Produce messages to the topic.
 Consume messages from the topic.

Source code:

{ "offset": 1, "content": { "producerId": "notification-service",

"data": { "type": "ORDER_CREATED", "orderId": "002", "customer": "MREM" },

"timestamp": 1703737341000 } },

{ "offset": 2, "content": {

"producerId": "notification-service",

"data": { "type": "ORDER_CREATED", "orderId": "003", "customer": "Bob Johnson"

},

"timestamp": 1703737342000 } }

Output:Starting Kafka-like Message Queue Demo

Topic created: orders

Topic created: notifications

Topic created: users

Consumer order-processor-1 subscribed to orders

Consumer order-analytics-1 subscribed to orders Consumer notification-handler-1 subscribed


to notifications

=== Publishing Order Messages === Message published to orders:


Experiment 3: Extend the Cluster to Multiple Brokers on a Single Node

1. Objective: Configure and start a multi-broker Kafka cluster.


2. Steps:
o Configure multiple server.properties files with unique broker IDs and ports.
o Start multiple Kafka brokers.
o Verify the cluster setup.

Source Code:

Multi-broker Kafka-like Cluster Implementation

import { MessageBroker } from './broker/MessageBroker.js';

import { Producer } from './clients/Producer.js';

import { Consumer } from './clients/Consumer.js';

// Create a new message broker instance

const broker = new MessageBroker();

// Create a topic

consttopicName = 'test-topic';

broker.createTopic(topicName);

// Create producer and consumer

const producer = new Producer(broker);

const consumer1 = new Consumer(broker, 'group-1');

const consumer2 = new Consumer(broker, 'group-1');

// Subscribe consumers to the topic

consumer1.subscribe(topicName);

consumer2.subscribe(topicName);

// Publish some messages

console.log('\n--- Publishing Messages ---');

producer.publish(topicName, 'Hello, Kafka-like queue!');

producer.publish(topicName, 'This is a test message');

producer.publish(topicName, 'Third message');

// List all topics


console.log('\n--- Available Topics ---');

console.log(broker.listTopics());

// Get messages from topic

const topic = broker.getTopic(topicName);

console.log('\n--- All Messages in Topic ---');

console.log(topic.getMessages());

Output

Cluster initialization with multiple brokers

Topic creation with partition assignments

Message distribution across partitions

Consumer group behavior

Partition assignments and replication (simplified)


Experiment 4: Create a Kafka Producer in Java

1. Objective: Write a Java program to produce messages to a Kafka topic.


2. Steps:
o Set up Maven dependencies for Kafka.
o Write Java code to create a Kafka producer.
o Produce messages to a specified topic.

Source code

import { MessageBroker } from './broker/MessageBroker.js';

import { Producer } from './clients/Producer.js';

import { Consumer } from './clients/Consumer.js';

// Create a new message broker instance

const broker = new MessageBroker();

// Create a topic

consttopicName = 'test-topic';

broker.createTopic(topicName);

// Create producer and consumer

const producer = new Producer(broker);

const consumer1 = new Consumer(broker, 'group-1');

const consumer2 = new Consumer(broker, 'group-1');

// Subscribe consumers to the topic

consumer1.subscribe(topicName);
consumer2.subscribe(topicName);

// Publish some messages

console.log('\n--- Publishing Messages ---');

producer.publish(topicName, 'Hello, Kafka-like queue!');

producer.publish(topicName, 'This is a test message');

producer.publish(topicName, 'Third message');

// List all topics

console.log('\n--- Available Topics ---');

console.log(broker.listTopics());

// Get messages from topic

const topic = broker.getTopic(topicName);

console.log('\n--- All Messages in Topic ---');

console.log(topic.getMessages());

Output:

Messages successfully produced to a topic via the Java program

Install Java Output:

java version "11.0.12" 2021-07-20

OpenJDK Runtime Environment...

Download Kafka (No specific output - just downloads and extracts files)

Start ZooKeeper Output:

[2024-xx-xx xx:xx:xx,xxx] INFO binding to port 0.0.0.0/0.0.0.0:2181

Start Kafka Server Output:


[2024-xx-xx xx:xx:xx,xxx] INFO [KafkaServer id=0] started

Create a Topic Output:

Created topic quickstart-events

Java Producer Program Output:

Message sent successfully!

Topic: quickstart-events

Partition: 0

Offset: 0

You might also like