Spring Boot - Create and Configure Topics in Apache Kafka

Topics are a special and essential component of Apache Kafka that are used to organize events or messages. In other words, Kafka Topics enable simple data transmission and reception across Kafka Servers by acting as Virtual Groups or Logs that store messages and events in a logical sequence. In this article, we will discuss about how to configure Kafka topics from the spring boot application.

Step by Step Implementation

First, our spring boot project needs Kafka dependencies. Include this part in your pom.xml file. Any Kafka version is accessible. We would prefer that you use the most recent Kafka client. For additional information on the Kafka version.

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>2.9.0</version>
    <scope>compile</scope>
</dependency>

Create a class with whatever name you prefer, but I used TopicConfiguration.java. We need to convert this class into a bean, and since we are configuring Kafka topics, we included the @Configuration annotation from Spring Boot. 

import java.io.*;

@Configuration
public class TopicConfiguration {
  
}

The process of adding new topics to our broker is handled by a KafkaAdmin bean. A KafkaAdmin bean is automatically registered with Spring Boot. The KafkaAdmin bean has to be explicitly registered for non-Spring Boot applications. The next step is to construct a new bean that tells KafkaAdmin what our Kafka URL is. BOOTSTRAP_SERVERS_CONFIG: Host and port on where Kafka broker is running.

BOOTSTRAP_SERVERS_CONFIG: <kafka_url>:<port>

@Bean 
public KafkaAdmin admin()
{
    Map<String, Object> configs = new HashMap<>();
    configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
                "localhost:9092");
    return new KafkaAdmin(configs);
}

We can now configure and create Kafka topics. A lot of various settings can be used right here while creating a topic. For more information about TopicBuilder.

import java.io.*;

@Bean 
public NewTopic topic1()
{
    return TopicBuilder.name("TOPIC-1")
        .partitions(1)
        .replicas(1)
        .build();
}

@Bean
public NewTopic topic2()
{
    return TopicBuilder.name("TOPIC-2")
        .partitions(1)
        .replicas(1)
        .build();
}

File: TopicConfiguration.java

import java.io.*;

@Configuration
public class TopicConfiguration {

    @Bean 
      public KafkaAdmin admin()
    {
        Map<String, Object> configs = new HashMap<>();
        configs.put(
            AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
            "localhost:9092");
        return new KafkaAdmin(configs);
    }

    @Bean 
      public NewTopic topic1()
    {
        return TopicBuilder.name("TOPIC-1")
            .partitions(1)
            .replicas(1)
            .build();
    }

    @Bean 
      public NewTopic topic2()
    {
        return TopicBuilder.name("TOPIC-2")
            .partitions(1)
            .replicas(1)
            .build();
    }
}

In the Springboot application, Kafka topics can be created and configured this way. The spring boot application will automatically create Kafka topics on the specified Kafka broker when it is launched. To get the topic configuration details on the server, run this command.

kafka-topics.sh --bootstrap-server localhost:9092 --topic <topic_name> --describe

Output:

This is the output from the server.