Building Real-Time

Data Pipelines with

Apache Kafka

B6/702/A
Introduction
Chapter 1
Introduction
▪ Introduction

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

01-1
Not to be reproduced in any form without prior written consent.
About You
▪ Raise your hand if…

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

01-3
Not to be reproduced in any form without prior written consent.
Agenda
▪ During this tutorial, we will:
– Make sure we’re all familiar with Apache Kafka
– Investigate Kafka Connect for data ingest and export
– Investigate Kafka Streams, the Kafka API for building stream processing applications
– Combine the two to create a complete streaming data processing pipeline
▪ As we go, we have Hands-On Exercises so you can try this out yourself

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

01-4
Not to be reproduced in any form without prior written consent.
Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.
01-5
Not to be reproduced in any form without prior written consent.
The Motivation for Apache Kafka
Chapter 2
Course Contents
01: Introduction

>>> 02: The Motivation for Apache Kafka

03: Kafka Fundamentals

04: Ingesting Data with Kafka Connect

05: Kafka Streams

06: Conclusion

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-1
Not to be reproduced in any form without prior written consent.
The Motivation for Apache Kafka
▪ In this chapter you will learn:
– Some of the problems encountered when multiple complex systems must be
integrated
– How processing stream data is preferable to batch processing
– The key features provided by Apache Kafka

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-2
Not to be reproduced in any form without prior written consent.
The Motivation for Apache Kafka
▪ Systems Complexity
▪ Real-Time Processing is Becoming Prevalent
▪ Kafka: A Stream Data Platform

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-3
Not to be reproduced in any form without prior written consent.
Simple Data Pipelines
▪ Data pipelines typically start out simply
– A single place where all data resides
– A single ETL (Extract, Transform, Load) process to move data to that location
▪ Data pipelines inevitably grow over time
– New systems are added
– Each new system requires its own ETL procedures
▪ Systems and ETL become increasingly hard to manage
– Codebase grows
– Data storage formats diverge

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-4
Not to be reproduced in any form without prior written consent.
Small Numbers of Systems are Easy to Integrate
It is (relatively) easy to connect just a few systems together

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-5
Not to be reproduced in any form without prior written consent.
More Systems Rapidly Introduce Complexity (1)
As we add more systems, complexity increases dramatically

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-6
Not to be reproduced in any form without prior written consent.
More Systems Rapidly Introduce Complexity (2)
…until eventually things become unmanageable

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-7
Not to be reproduced in any form without prior written consent.
The Motivation for Apache Kafka
▪ Systems Complexity
▪ Real-Time Processing is Becoming Prevalent
▪ Kafka: A Stream Data Platform

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-8
Not to be reproduced in any form without prior written consent.
Batch Processing: The Traditional Approach
▪ Traditionally, almost all data processing was batch-oriented
– Daily, weekly, monthly…
▪ This is inherently limiting
– “I can’t start to analyze today’s data until the overnight ingest process has run”

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-9
Not to be reproduced in any form without prior written consent.
Real-Time Processing: Often a Better Approach
▪ These days, it is often beneficial to process data as it is being generated
– Real-time processing allows real-time decisions
▪ Examples:
– Fraud detection
– Recommender systems for e-commerce web sites
– Log monitoring and fault diagnosis
– etc.
▪ Of course, many legacy systems still rely on batch processing
– However, this is changing over time, as more ‘stream processing’ systems emerge
– Kafka Streams
– Apache Spark Streaming
– Apache Storm
– Apache Samza
– etc.

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-10
Not to be reproduced in any form without prior written consent.
The Motivation for Apache Kafka
▪ Systems Complexity
▪ Real-Time Processing is Becoming Prevalent
▪ Kafka: A Stream Data Platform

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-11
Not to be reproduced in any form without prior written consent.
Kafka’s Origins
▪ Kafka was designed to solve both problems
– Simplifying data pipelines
– Handling streaming data
▪ Originally created at LinkedIn in 2010
– Designed to support batch and real-time analytics
– Kafka is now at the core of LinkedIn’s architecture
– Performs extremely well at very large scale
– Processes over 1.4 trillion messages per day

▪ An open source, top-level Apache project since 2012

▪ In use at many organizations
– Twitter, Netflix, Goldman Sachs, Hotels.com, IBM, Spotify, Uber, Square, Cisco…
▪ Confluent was founded by Kafka’s original authors to provide commercial support,
training, and consulting for Kafka

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-12
Not to be reproduced in any form without prior written consent.
A Universal Pipeline for Data
▪ Kafka decouples data source and destination systems
– Via a publish/subscribe architecture
▪ All data sources write their data to the Kafka cluster
▪ All systems wishing to use the data read from Kafka
▪ Stream data platform
– Data integration: capture streams of events
– Stream processing: continuous, real-time data processing and transformation

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-13
Not to be reproduced in any form without prior written consent.
Use Cases
▪ Here are some example scenarios where Kafka can be used
– Real-time event processing
– Log aggregation
– Operational metrics and analytics
– Stream processing
– Messaging

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-14
Not to be reproduced in any form without prior written consent.
Chapter Review
▪ Kafka was designed to simplify data pipelines, and to provide a way for systems to
process streaming data
▪ Kafka can support batch and real-time analytics
▪ Kafka was started at LinkedIn and is now an open source Apache project with broad
deployment

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

02-15
Not to be reproduced in any form without prior written consent.
Kafka Fundamentals
Chapter 3
Course Contents
01: Introduction

02: The Motivation for Apache Kafka

>>> 03: Kafka Fundamentals

04: Ingesting Data with Kafka Connect

05: Kafka Streams

06: Conclusion

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-1
Not to be reproduced in any form without prior written consent.
Kafka Fundamentals
▪ In this chapter you will learn:
– How Producers write data to a Kafka cluster
– How data is divided into partitions, and then stored on Brokers
– How Consumers read data from the cluster

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-2
Not to be reproduced in any form without prior written consent.
Kafka Fundamentals
▪ An Overview of Kafka
▪ Kafka Producers
▪ Kafka Brokers
▪ Kafka Consumers
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-3
Not to be reproduced in any form without prior written consent.
Reprise: A Very High-Level View of Kafka
▪ Producers send data to the Kafka cluster
▪ Consumers read data from the Kafka cluster
▪ Brokers are the main storage and messaging components of the Kafka cluster

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-4
Not to be reproduced in any form without prior written consent.
Kafka Messages
▪ The basic unit of data in Kafka is a message
– Message is sometimes used interchangeably with record
– Producers write messages to Brokers
– Consumers read messages from Brokers

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-5
Not to be reproduced in any form without prior written consent.
Key-Value Pairs
▪ A message is a key-value pair
– All data is stored in Kafka as byte arrays
– Producer provides serializers to convert the key and value to byte arrays
– Key and value can be any data type

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-6
Not to be reproduced in any form without prior written consent.
Topics
▪ Kafka maintains streams of messages called Topics
– Logical representation
– They categorize messages into groups

▪ Developers decide which Topics exist

– By default, a Topic is auto-created when it is first used
▪ One or more Producers can write to one or more Topics
▪ There is no limit to the number of Topics that can be used

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-7
Not to be reproduced in any form without prior written consent.
Partitioned Data Ingestion
▪ Producers shard data over a set of Partitions
– Each Partition contains a subset of the Topic’s messages
– Each Partition is an ordered, immutable log of messages
▪ Partitions are distributed across the Brokers
▪ Typically, the message key is used to determine which Partition a message is
assigned to
– This can be overridden by the Producer

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-8
Not to be reproduced in any form without prior written consent.
Kafka Components
▪ There are four key components in a Kafka system
– Producers
– Brokers
– Consumers
– ZooKeeper
▪ We will now investigate each of these in turn

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-9
Not to be reproduced in any form without prior written consent.
Kafka Fundamentals
▪ An Overview of Kafka
▪ Kafka Producers
▪ Kafka Brokers
▪ Kafka Consumers
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-10
Not to be reproduced in any form without prior written consent.
Producer Basics
▪ Producers write data in the form of messages to the Kafka cluster
▪ Producers can be written in any language
– Native Java, C, Python, and Go clients are supported by Confluent
– Clients for many other languages exist
– Confluent develops and supports a REST (REpresentational State Transfer) server
which can be used by clients written in any language
▪ A command-line Producer tool exists to send messages to the cluster
– Useful for testing, debugging, etc.

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-11
Not to be reproduced in any form without prior written consent.
Load Balancing and Semantic Partitioning
▪ Producers use a partitioning strategy to assign each message to a Partition
▪ Having a partitioning strategy serves two purposes
– Load balancing: shares the load across the Brokers
– Semantic partitioning: user-specified key allows locality-sensitive message
processing
▪ The partitioning strategy is specified by the Producer
– Default strategy is a hash of the message key
– hash(key) % number_of_partitions

– If a key is not specified, messages are sent to Partitions on a round-robin basis

▪ Developers can provide a custom partitioner class

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-12
Not to be reproduced in any form without prior written consent.
Kafka Fundamentals
▪ An Overview of Kafka
▪ Kafka Producers
▪ Kafka Brokers
▪ Kafka Consumers
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-13
Not to be reproduced in any form without prior written consent.
Broker Basics
▪ Brokers receive and store messages when they are sent by the Producers
▪ A Kafka cluster will typically have multiple Brokers
– Each can handle hundreds of thousands, or millions, of messages per second
▪ Each Broker manages multiple Partitions

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-14
Not to be reproduced in any form without prior written consent.
Brokers Manage Partitions
▪ Messages in a Topic are spread across Partitions in different Brokers
▪ Typically, a Broker will handle many Partitions
▪ Each Partition is stored on the Broker’s disk as one or more log files
– Not to be confused with log4j files used for monitoring

▪ Each message in the log is identified by its offset

– A monotonically increasing value
▪ Kafka provides a configurable retention policy for messages to manage log file
growth

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-15
Not to be reproduced in any form without prior written consent.
Messages are Stored in a Persistent Log

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-16
Not to be reproduced in any form without prior written consent.
Fault Tolerance via a Replicated Log
▪ Partitions can be replicated across multiple Brokers
▪ Replication provides fault tolerance in case a Broker goes down
– Kafka automatically handles the replication

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-17
Not to be reproduced in any form without prior written consent.
Kafka Fundamentals
▪ An Overview of Kafka
▪ Kafka Producers
▪ Kafka Brokers
▪ Kafka Consumers
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-18
Not to be reproduced in any form without prior written consent.
Consumer Basics
▪ Consumers pull messages from one or more Topics in the cluster
– As messages are written to a Topic, the Consumer will automatically retrieve them
▪ The Consumer Offset keeps track of the latest message read
– If necessary, the Consumer Offset can be changed
– For example, to reread messages

▪ The Consumer Offset is stored in a special Kafka Topic

▪ A command-line Consumer tool exists to read messages from the cluster
– Useful for testing, debugging, etc.

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-19
Not to be reproduced in any form without prior written consent.
Distributed Consumption
▪ Different Consumers can read data from the same Topic
– By default, each Consumer will receive all the messages in the Topic
▪ Multiple Consumers can be combined into a Consumer Group
– Consumer Groups provide scaling capabilities
– Each Consumer is assigned a subset of Partitions for consumption

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-20
Not to be reproduced in any form without prior written consent.
What is ZooKeeper?
▪ ZooKeeper is a centralized service that can be used by distributed applications
– Open source Apache project
– Enables highly reliable distributed coordination
– Maintains configuration information
– Provides distributed synchronization
▪ Used by many projects
– Including Kafka and Hadoop
▪ Typically consists of three or five servers in a quorum
– This provides resiliency should a machine fail

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-21
Not to be reproduced in any form without prior written consent.
How Kafka Uses ZooKeeper
▪ Kafka Brokers use ZooKeeper for a number of important internal features
– Cluster management
– Failure detection and recovery
– Access Control List (ACL) storage
▪ In earlier versions of Kafka, the Consumer needed access to the ZooKeeper quorum
– This is no longer the case

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-22
Not to be reproduced in any form without prior written consent.
Hands-On Exercise: Using Kafka’s Command-Line Tools
▪ In this Hands-On Exercise you will use Kafka’s command-line tools to Produce and
Consume data
▪ Please refer to the Hands-On Exercise Manual

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-23
Not to be reproduced in any form without prior written consent.
Kafka Fundamentals
▪ An Overview of Kafka
▪ Kafka Producers
▪ Kafka Brokers
▪ Kafka Consumers
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-24
Not to be reproduced in any form without prior written consent.
Chapter Review
▪ A Kafka system is made up of Producers, Consumers, and Brokers
– ZooKeeper provides co-ordination services for the Brokers
▪ Producers write messages to Topics
– Topics are broken down into partitions for scalability
▪ Consumers read data from one or more Topics

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

03-25
Not to be reproduced in any form without prior written consent.
Ingesting Data with Kafka Connect
Chapter 4
Course Contents
01: Introduction

02: The Motivation for Apache Kafka

03: Kafka Fundamentals

>>> 04: Ingesting Data with Kafka Connect

05: Kafka Streams

06: Conclusion

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-1
Not to be reproduced in any form without prior written consent.
Ingesting Data with Kafka Connect
▪ In this chapter you will learn:
– The motivation for Kafka Connect
– What standard Connectors are provided
– The differences between standalone and distributed mode
– How to configure and use Kafka Connect
– How Kafka Connect compares to writing your own data transfer system

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-2
Not to be reproduced in any form without prior written consent.
Ingesting Data with Kafka Connect
▪ The Motivation for Kafka Connect
▪ Standalone and Distributed Modes
▪ Configuring the Connectors
▪ Hands-On Exercise: Running Kafka Connect
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-3
Not to be reproduced in any form without prior written consent.
What is Kafka Connect?
▪ Kafka Connect is a framework for streaming data between Apache Kafka and other
data systems
▪ Kafka Connect is open source, and is part of the Apache Kafka distribution
▪ It is simple, scalable, and reliable

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-4
Not to be reproduced in any form without prior written consent.
Example Use Cases
▪ Example use cases for Kafka Connect include:
– Stream an entire SQL database into Kafka
– Stream Kafka topics into HDFS for batch processing
– Stream Kafka topics into Elasticsearch for secondary indexing
– …

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-5
Not to be reproduced in any form without prior written consent.
Why Not Just Use Producers and Consumers?
▪ Internally, Kafka Connect is a Kafka client using the standard Producer and
Consumer APIs
▪ Kafka Connect has benefits over ‘do-it-yourself’ Producers and Consumers:
– Off-the-shelf, tested Connectors for common data sources are available
– Features fault tolerance and automatic load balancing when running in distributed
mode
– No coding required
– Just write configuration files for Kafka Connect
– Pluggable/extendable by developers

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-6
Not to be reproduced in any form without prior written consent.
Connect Basics
▪ Connectors are logical jobs responsible for managing the copying of data between
Kafka and another system
▪ Connector Sources read data from an external data system into Kafka
– Internally, a connector source is a Kafka Producer client
▪ Connector Sinks write Kafka data to an external data system
– Internally, a connector sink is a Kafka Consumer client

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-7
Not to be reproduced in any form without prior written consent.
Providing Parallelism and Scalability
▪ Splitting the workload into smaller pieces provides the parallelism and scalability
▪ Connector jobs are broken down into tasks that do the actual copying of the data
▪ Workers are processes running one or more tasks in different threads
▪ Input stream can be partitioned for parallelism, for example:
– File input: Partition → file
– Database input: Partition → table

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-8
Not to be reproduced in any form without prior written consent.
Converting Data
▪ Converters provide the data format written to or read from Kafka (like Serializers)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-9
Not to be reproduced in any form without prior written consent.
Converter Data Formats
▪ Converters apply to both the key and value of the message
– Key and value converters can be set independently
– key.converter
– value.converter

▪ Pre-defined data formats for Converter

– Avro: AvroConverter
– JSON: JsonConverter
– String: StringConverter

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-10
Not to be reproduced in any form without prior written consent.
Avro Converter as a Best Practice
▪ Best Practice is to use an Avro Converter and Schema Registry with the Connectors

key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://schemaregistry1:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://schemaregistry1:8081

▪ Benefits
– Provides a data structure format
– Supports code generation of data types
– Avro data is binary, so stores data efficiently
– Type checking is performed at write time
▪ Avro schemas evolve as updates to code happen
– Connectors may support schema evolution and react to schema changes in a
configurable way
▪ Schemas can be centrally managed in a Schema Registry

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-11
Not to be reproduced in any form without prior written consent.
Off-The-Shelf Connectors
▪ Confluent Open Source ships with commonly used Connectors
– FileStream
– JDBC
– HDFS
– Elasticsearch
– AWS S3
▪ Confluent Enterprise includes additional connectors
– Replicator
– JMS
▪ Many other certified Connectors are available
– See https://www.confluent.io/product/connectors/

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-12
Not to be reproduced in any form without prior written consent.
JDBC Source Connector: Overview
▪ JDBC Source periodically polls a relational database for new or recently modified
rows
– Creates a record for each row, and Produces that record as a Kafka message
▪ Records from each table are Produced to their own Kafka topic
▪ New and deleted tables are handled automatically

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-13
Not to be reproduced in any form without prior written consent.
JDBC Source Connector: Detecting New and Updated Rows
▪ The Connector can detect new and updated rows in several ways:

Incremental query made Description

Incrementing column Check a single column where newer rows
have a larger, autoincremented ID. Does
not support updated rows
Timestamp column Checks a single ‘last modified’ column.
Can’t guarantee reading all updates
Timestamp and incrementing column Combination of the two methods above.
Guarantees that all updates are read
Custom query Used in conjunction with the options
above for custom filtering

▪ Alternative: bulk mode for one-time load, not incremental, unfiltered

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-14
Not to be reproduced in any form without prior written consent.
JDBC Source Connector: Configuration
Parameter Description
connection.url The JDBC connection URL for the database
topic.prefix The prefix to prepend to table names to
generate the Kafka topic name
mode The mode for detecting table changes.
Options are bulk, incrementing,
timestamp, timestamp+incrementing
query The custom query to run, if specified
poll.interval.ms The frequency in milliseconds to poll for
new data in each table
table.blacklist A list of tables to ignore and not import. If
specified, tables.whitelist cannot be
specified
table.whitelist A list of tables to import. If specified,
tables.blacklist cannot be specified
▪ Note: This is not a complete list. See http://docs.confluent.io

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-15
Not to be reproduced in any form without prior written consent.
HDFS Sink Connector: Overview
▪ Continuously polls from Kafka and writes to HDFS (Hadoop Distributed File System)
▪ Integrates with Hive
– Auto table creation
– Schema evolution with Avro
▪ Works with secure HDFS and the Hive Metastore, using Kerberos
▪ Provides exactly once delivery
▪ Data format is extensible
– Avro, Parquet, custom formats
▪ Pluggable Partitioner, supporting:
– Kafka Partitioner (default)
– Field Partitioner
– Time Partitioner
– Custom Partitioners

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-16
Not to be reproduced in any form without prior written consent.
Ingesting Data with Kafka Connect
▪ The Motivation for Kafka Connect
▪ Standalone and Distributed Modes
▪ Configuring the Connectors
▪ Hands-On Exercise: Running Kafka Connect
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-17
Not to be reproduced in any form without prior written consent.
Two Modes: Standalone and Distributed
▪ Kafka Connect can be run in two modes
– Standalone mode
– Single worker process on a single machine
– Use case: testing and development, or when a process should not be distributed
(e.g. tail a log file)
– Distributed mode
– Multiple worker processes on one or more machines
– Use Case: requirements for fault tolerance and scalability

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-18
Not to be reproduced in any form without prior written consent.
Running in Standalone Mode
▪ To run in standalone mode, start a process by providing as arguments
– Standalone configuration properties file
– One or more connector configuration files
– Each connector instance will be run in its own thread

$ connect-standalone connect-standalone.properties \
connector1.properties [connector2.properties connector3.properties ...]

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-19
Not to be reproduced in any form without prior written consent.
Running in Distributed Mode
▪ To run in distributed mode, start Kafka Connect on each worker node

$ connect-distributed worker.properties

▪ Group coordination
– Connect leverages Kafka’s group membership protocol
– Configure workers with the same group.id

– Workers distribute load within this Kafka Connect “cluster”

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-20
Not to be reproduced in any form without prior written consent.
Providing Fault Tolerance in Distributed Mode (1)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-21
Not to be reproduced in any form without prior written consent.
Providing Fault Tolerance in Distributed Mode (2)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-22
Not to be reproduced in any form without prior written consent.
Providing Fault Tolerance in Distributed Mode (3)

▪ Tasks have no state stored within them

– Task state is stored in special Topics in Kafka

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-23
Not to be reproduced in any form without prior written consent.
Ingesting Data with Kafka Connect
▪ The Motivation for Kafka Connect
▪ Standalone and Distributed Modes
▪ Configuring the Connectors
▪ Hands-On Exercise: Running Kafka Connect
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-24
Not to be reproduced in any form without prior written consent.
Transforming Data with Connect
▪ Kafka 0.10.2.0 provides the ability to transform data a message-at-a-time
– Configured at the connector-level
– Applied to the message key or value
▪ A subset of available transformations:
– InsertField: insert a field using attributes from the message metadata or from a
configured static value
– ReplaceField: rename fields, or apply a blacklist or whitelist to filter
– ValueToKey: replace the key with a new key formed from a subset of fields in the
value payload
– TimestampRouter: update the topic field as a function of the original topic value
and timestamp
▪ More information on Connect Transformations can be found at
http://kafka.apache.org/documentation.html#connect_transforms

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-25
Not to be reproduced in any form without prior written consent.
The REST API
▪ Connectors can be added, modified, and deleted via a REST API on port 8083
– The REST requests can be made to any worker
▪ In standalone mode, configurations can be done via a REST API
– This is optional to the other way of modifying the standalone configuration
properties file
– Changes made this way will not persist after worker restart
▪ In distributed mode, configurations can be done only via a REST API
– Changes made this way will persist after a worker process restart
– Connector configuration data is stored in a special Kafka topic

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-26
Not to be reproduced in any form without prior written consent.
Ingesting Data with Kafka Connect
▪ The Motivation for Kafka Connect
▪ Standalone and Distributed Modes
▪ Configuring the Connectors
▪ Hands-On Exercise: Running Kafka Connect
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-27
Not to be reproduced in any form without prior written consent.
Hands-On Exercise: Running Kafka Connect
▪ In this Hands-On Exercise, you will run Kafka Connect to pull data from a MySQL
database into a Kafka topic
▪ Please refer to the Hands-On Exercise Manual

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-28
Not to be reproduced in any form without prior written consent.
Ingesting Data with Kafka Connect
▪ The Motivation for Kafka Connect
▪ Standalone and Distributed Modes
▪ Configuring the Connectors
▪ Hands-On Exercise: Running Kafka Connect
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-29
Not to be reproduced in any form without prior written consent.
Chapter Review
▪ Kafka Connect provides a scalable, reliable way to transfer data from external
systems into Kafka, and vice versa
▪ Many off-the-shelf Connectors are provided by Confluent, and many others are
under development by third parties

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

04-30
Not to be reproduced in any form without prior written consent.
Kafka Streams
Chapter 5
Course Contents
01: Introduction

02: The Motivation for Apache Kafka

03: Kafka Fundamentals

04: Ingesting Data with Kafka Connect

>>> 05: Kafka Streams

06: Conclusion

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-1
Not to be reproduced in any form without prior written consent.
Kafka Streams
▪ In this chapter you will learn:
– What Kafka Streams is
– How to create a Kafka Streams Application
– How the KStream and KTable abstractions differ

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-2
Not to be reproduced in any form without prior written consent.
Kafka Streams
▪ Kafka Streams Basics
▪ Kafka Streams Basic Concepts
▪ Key Features of Kafka Streams
▪ Kafka Streams APIs
▪ Hands-On Exercise: Writing a Kafka Streams Application
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-3
Not to be reproduced in any form without prior written consent.
What Is Kafka Streams?
▪ Kafka Streams is a lightweight Java library for building distributed stream processing
applications using Kafka
– Easy to embed in your own applications
▪ No external dependencies, other than Kafka
▪ Supports event-at-a-time processing (not microbatching) with millisecond latency
▪ Provides a table-like model for data
▪ Supports windowing operations, and stateful processing including distributed joins
and aggregation
▪ Has fault-tolerance and supports distributed processing
▪ Includes both a Domain-Specific Language (DSL) and a low-level API

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-4
Not to be reproduced in any form without prior written consent.
A Library, Not a Framework
▪ Kafka Streams is an alternative to streaming frameworks such as
– Spark Streaming
– Apache Storm
– Apache Samza
– etc.
▪ Unlike these, it does not require its own cluster
– Can run on a stand-alone machine, or multiple machines

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-5
Not to be reproduced in any form without prior written consent.
When to Use Kafka Streams
▪ Mainstream Application Development
▪ When running a cluster would be too painful
▪ Fast Data apps for small and big data
▪ Reactive and stateful applications
▪ Continuous transformations
▪ Continuous queries
▪ Microservices
▪ Event-driven systems
▪ The “T” in ETL
▪ …and more

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-6
Not to be reproduced in any form without prior written consent.
Why Not Just Build Your Own?
▪ Many people are currently building their own stream processing applications
– Using the Producer and Consumer APIs
▪ Using Kafka Streams is much easier than taking the ‘do it yourself’ approach
– Well-designed, well-tested, robust
– Means you can focus on the application logic, not the low-level plumbing

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-7
Not to be reproduced in any form without prior written consent.
Installing Kafka Streams
▪ There is no ‘installation’!
▪ Kafka Streams is a library. Add it to your application like any other library

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-8
Not to be reproduced in any form without prior written consent.
But… Where’s THE CLUSTER to Process the Data?
▪ There is no cluster!
▪ Unlearn your bad habits: ‘do cool stuff with data’ != ‘must have a cluster’

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-9
Not to be reproduced in any form without prior written consent.
Kafka Streams
▪ Kafka Streams Basics
▪ Kafka Streams Basic Concepts
▪ Key Features of Kafka Streams
▪ Kafka Streams APIs
▪ Hands-On Exercise: Writing a Kafka Streams Application
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-10
Not to be reproduced in any form without prior written consent.
Kafka Streams Basic Concepts (1)
▪ A stream is an unbounded, continuously updating data set

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-11
Not to be reproduced in any form without prior written consent.
Kafka Streams Basic Concepts (2)
▪ A stream processor transforms data in a stream
▪ A processor topology defines the data flow through the stream processors

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-12
Not to be reproduced in any form without prior written consent.
KStreams and KTables
▪ A KStream is an abstraction of a record stream
– Each record represents a self-contained piece of data in the unbounded data set
▪ A KTable is an abstraction of a changelog stream
– Each record represents an update
▪ Example: We send two records to the stream
– ('apple', 1), and ('apple', 5)

▪ If we were to treat the stream as a KStream and sum up the values for apple, the
result would be 6
▪ If we were to treat the stream as a KTable and sum up the values for apple, the
result would be 5
– The second record is treated as an update to the first, because they have the same
key
▪ Typically, if you are going to treat a topic as a KTable it makes sense to configure
log compaction on the topic
Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.
05-13
Not to be reproduced in any form without prior written consent.
KStreams and KTables: Example

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-14
Not to be reproduced in any form without prior written consent.
Motivating Example: Users Per Region

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-15
Not to be reproduced in any form without prior written consent.
Motivating Example: Users Per Region

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-16
Not to be reproduced in any form without prior written consent.
Different Use Cases, Different Interpretations (1)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-17
Not to be reproduced in any form without prior written consent.
Different Use Cases, Different Interpretations (2)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-18
Not to be reproduced in any form without prior written consent.
Streams, Meet Tables
Example When you Read the Topic Data interpreted Messages
need… into as interpreted as
All the places All the values of KStream record stream INSERT (append)
Alice has ever a key
been
Where Alice is Latest value of a KTable changelog UPSERT
right now key stream (overwrite
existing)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-19
Not to be reproduced in any form without prior written consent.
Kafka Streams
▪ Kafka Streams Basics
▪ Kafka Streams Basic Concepts
▪ Key Features of Kafka Streams
▪ Kafka Streams APIs
▪ Hands-On Exercise: Writing a Kafka Streams Application
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-20
Not to be reproduced in any form without prior written consent.
Key Kafka Streams Features
▪ Key Kafka Streams features
– Reliable
– Fault-tolerant
– Scalable
– Stateful and Stateless Computations
– Interactive Queries

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-21
Not to be reproduced in any form without prior written consent.
Fault-Tolerance and Scalability (1)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-22
Not to be reproduced in any form without prior written consent.
Fault-Tolerance and Scalability (2)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-23
Not to be reproduced in any form without prior written consent.
Fault-Tolerance and Scalability (3)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-24
Not to be reproduced in any form without prior written consent.
Fault-Tolerance and Scalability (4)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-25
Not to be reproduced in any form without prior written consent.
Stateful Computations
▪ Stateful computations like aggregations or joins require state
▪ Example: count() will cause the creation of a state store to keep track of counts
▪ State stores in Kafka Streams…
– are local for best performance
– are replicated to Kafka for elasticity and for fault-tolerance
▪ Pluggable storage engines
– Default: RocksDB (key-value store) to allow for local state that is larger than
available RAM
– Further built-in options available: in-memory store
– You can also use your own, custom storage engine

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-26
Not to be reproduced in any form without prior written consent.
State Management (1)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-27
Not to be reproduced in any form without prior written consent.
State Management (2)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-28
Not to be reproduced in any form without prior written consent.
State Management (3)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-29
Not to be reproduced in any form without prior written consent.
State Management (4)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-30
Not to be reproduced in any form without prior written consent.
Interactive Queries

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-31
Not to be reproduced in any form without prior written consent.
Kafka Streams
▪ Kafka Streams Basics
▪ Kafka Streams Basic Concepts
▪ Key Features of Kafka Streams
▪ Kafka Streams APIs
▪ Hands-On Exercise: Writing a Kafka Streams Application
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-32
Not to be reproduced in any form without prior written consent.
API Option 1: Processor API
▪ Flexible, but requires more manual work
▪ May appeal to people who are familiar with Storm, Samza etc
– Or for people who require functionality that is not yet available in the DSL

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-33
Not to be reproduced in any form without prior written consent.
API Option 2: Kafka Streams DSL
▪ The preferred API for most use-cases
▪ Will appeal to people who are used to Spark, Flink,…
▪ This is the API we’ll concentrate on in the tutorial

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-34
Not to be reproduced in any form without prior written consent.
Windowing, Joins, and Aggregations
▪ Kafka Streams allows us to window the stream of data by time
– To divide it up into ‘time buckets’
▪ We can join, or merge, two streams together based on their keys
– Typically we do this on windows of data
▪ We can aggregate records
– Combine multiple input records together in some way into a single output record
– Examples: sum, count
– Again, this is usually done on a windowed basis

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-35
Not to be reproduced in any form without prior written consent.
Windowing: Example
▪ Windowing: Group events in a stream using time-based windows
▪ Use case examples:
– Time-based analysis of ad impressions (“number of ads clicked in the past hour”)
– Monitoring statistics of telemetry data (“1min/5min/15min averages”)

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-36
Not to be reproduced in any form without prior written consent.
Tumbling and Hopping Windows

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-37
Not to be reproduced in any form without prior written consent.
New in Kafka 10.2.0: Session Windows
▪ Session: periods of activity separated by a defined gap of inactivity
▪ Use for behavior analysis

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-38
Not to be reproduced in any form without prior written consent.
Configuring the Application
▪ Kafka Streams configuration is specified with a StreamsConfig instance
▪ This is typically created from a java.util.Properties instance
▪ Example:

Properties streamsConfiguration = new Properties();

streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "simple-example");
streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
streamsConfiguration.put(StreamsConfig.ZOOKEEPER_CONNECT_CONFIG, "zookeeper1:2181");

// Continue to specify configuration options

streamsConfiguration.put(..., ...);

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-39
Not to be reproduced in any form without prior written consent.
Serializers and Deserializers (Serdes)
▪ Each Kafka Streams application must provide serdes (serializers and deserializers)
for the data types of the keys and values of the records
– These will be used by some of the operations which transform the data
▪ Set default serdes in the configuration of your application
– These can be overridden in individual methods by specifying explicit serdes
▪ Configuration example:

streamsConfiguration.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().
getName());
streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass().
getName());

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-40
Not to be reproduced in any form without prior written consent.
Available Serdes
▪ Kafka includes a variety of serdes in the kafka-clients Maven artifact

Data type Serde

byte[] Serdes.ByteArray(), Serdes.Bytes()
(Bytes wraps Java’s byte[] and supports
equality and ordering semantics)
ByteBuffer Serdes.ByteBuffer()
Double Serdes.Double()
Integer Serdes.Integer()
Long Serdes.Long()
String Serdes.String()
▪ Confluent also provides GenericAvroSerde and SpecificAvroSerde as examples
– See https://github.com/confluentinc/examples/tree/kafka-0.10.0.0-cp-
3.0.0/kafka-
streams/src/main/java/io/confluent/examples/streams/utils

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-41
Not to be reproduced in any form without prior written consent.
Creating the Processing Topology
▪ Create a KStream or KTable object from one or more Kafka Topics using
KStreamBuilder or KTableBuilder
– For KTableBuilder only a single Topic can be specified

▪ Example:

KStreamBuilder builder = new KStreamBuilder();

KStream<String, Long> purchases = builder.stream("PurchaseTopic");

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-42
Not to be reproduced in any form without prior written consent.
Transforming a Stream
▪ Data can be transformed using a number of different operators
▪ Some operations result in a new KStream object
– For example, filter or map

▪ Some operations result in a KTable object

– For example, an aggregation operation

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-43
Not to be reproduced in any form without prior written consent.
Some Stateless Transformation Operations (1)
▪ Examples of stateless transformation operations:
– filter
– Creates a new KStream containing only records from the previous KStream
which meet some specified criteria
– map
– Creates a new KStream by transforming each element in the current stream into
a different element in the new stream
– mapValues
– Creates a new KStream by transforming the value of each element in the current
stream into a different element in the new stream

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-44
Not to be reproduced in any form without prior written consent.
Some Stateless Transformation Operations (2)
▪ Examples of stateless transformation operations (cont’d):
– flatMap
– Creates a new KStream by transforming each element in the current stream into
zero or more different elements in the new stream
– flatMapValues
– Creates a new KStream by transforming the value of each element in the current
stream into zero or more different elements in the new stream

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-45
Not to be reproduced in any form without prior written consent.
Some Stateful Transformation Operations
▪ Examples of stateful transformation operations:
– countByKey
– Counts the number of instances of each key in the stream; results in a new, ever-
updating KTable
– reduceByKey
– Combines values of the stream using a supplied Reducer into a new, ever-
updating KTable

▪ For a full list of operations, see the JavaDocs at

http://http://docs.confluent.io/3.0.0/streams/javadocs/index.html

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-46
Not to be reproduced in any form without prior written consent.
Writing Streams Back to Kafka
▪ Streams can be written to Kafka topics using the to method
– Example:

myNewStream.to("NewTopic");

▪ We often want to write to a topic but then continue to process the data
– Do this using the through method

myNewStream.through("NewTopic").flatMap(...)...;

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-47
Not to be reproduced in any form without prior written consent.
Printing A Stream’s Contents
▪ It is sometimes useful to be able to see what a stream contains
– Especially when testing and debugging
▪ print() writes the contents of the stream to System.out

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-48
Not to be reproduced in any form without prior written consent.
Running the Application
▪ To start processing the stream, create a KafkaStreams object
– Configure it with your KStreamBuilder or KTableBuilder and your configuration

▪ Then call the start() method

▪ Example:

KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration);

streams.start();

▪ If you need to stop the application, call streams.close()

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-49
Not to be reproduced in any form without prior written consent.
A Simple Kafka Streams Example (1)
1 public class SimpleStreamsExample {
2
3
4 public static void main(String[] args) throws Exception {
5 Properties streamsConfiguration = new Properties();
6 // Give the Streams application a unique name. The name must be unique in the Kafka cluster
7 streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "simple-streams-example");
8 streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
9 streamsConfiguration.put(StreamsConfig.ZOOKEEPER_CONNECT_CONFIG, "zookeeper1:2181");
10 streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
11 // Specify default (de)serializers for record keys and for record values.
12 streamsConfiguration.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.ByteArray().getClass
().getName());
13 streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass
().getName());

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-50
Not to be reproduced in any form without prior written consent.
A Simple Kafka Streams Example (2)
1 KStreamBuilder builder = new KStreamBuilder();
2
3 // Read the input Kafka topic into a KStream instance.
4 KStream<byte[], String> textLines = builder.stream("TextLinesTopic");
5
6 // Convert to upper case (:: is Java 8 syntax)
7 KStream<byte[], String> uppercasedWithMapValues = textLines.mapValues(String::toUpperCase);
8
9 // Write the results to a new Kafka topic called "UppercasedTextLinesTopic".
10 uppercasedWithMapValues.to("UppercasedTextLinesTopic");
11
12 KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration);
13 streams.start();
14 }
15 }

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-51
Not to be reproduced in any form without prior written consent.
A More Complex Kafka Streams Application (1)
1 public class WordCountLambdaExample {
2 public static void main(String[] args) throws Exception {
3 Properties streamsConfiguration = new Properties();
4 // Application name must be unique in the Kafka cluster
5 streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "wordcount-lambda-example");
6
7 streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092");
8 streamsConfiguration.put(StreamsConfig.ZOOKEEPER_CONNECT_CONFIG, "zookeeper1:2181");
9 // Specify default (de)serializers for record keys and for record values.
10 streamsConfiguration.put(StreamsConfig.KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass
().getName());
11 streamsConfiguration.put(StreamsConfig.VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass
().getName());
12
13 KStreamBuilder builder = new KStreamBuilder();

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-52
Not to be reproduced in any form without prior written consent.
A More Complex Kafka Streams Application (2)
1 // Construct a KStream from the input topic "TextLinesTopic"
2 KStream<String, String> textLines = builder.stream("TextLinesTopic");
3
4 KStream<String, Long> wordCounts = textLines
5 .flatMapValues(value -> Arrays.asList(value.toLowerCase().split("\\W+")))
6 .map((key, word) -> new KeyValue<>(word, word))
7 // Count the occurrences of each word (record key).
8 // This will change the stream type from KStream<String, String> to
9 // KTable<String, Long> (word -> count).
10 .countByKey("Counts")
11 // Convert the KTable<String, Long> into a `KStream<String, Long>`.
12 .toStream();
13
14 // Write the `KStream<String, Long>` to the output topic.
15 wordCounts.to(Serdes.String(), Serdes.Long(), "WordsWithCountsTopic");
16
17 // Now that we have finished the definition of the processing topology we can actually run
18 // it via `start()`. The Streams application as a whole can be launched just like any
19 // normal Java application that has a `main()` method.
20 KafkaStreams streams = new KafkaStreams(builder, streamsConfiguration);
21 streams.start();
22 }
23 }

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-53
Not to be reproduced in any form without prior written consent.
Processing Guarantees
▪ Kafka streams supports at-least-once processing
▪ With no failures, it will process data exactly once
▪ If a machine fails, it is possible that some records may be processed more than once
– Whether this is acceptable or not depends on the use-case
▪ Exactly-once processing semantics will be supported in the next major release of
Kafka

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-54
Not to be reproduced in any form without prior written consent.
More Kafka Streams Features
▪ We have only scratched the surface of Kafka Streams
▪ Check the documentation to learn about processing windowed tables, joining
tables, joining streams and tables…

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-55
Not to be reproduced in any form without prior written consent.
Kafka Streams
▪ Kafka Streams Basics
▪ Kafka Streams Basic Concepts
▪ Key Features of Kafka Streams
▪ Kafka Streams APIs
▪ Hands-On Exercise: Writing a Kafka Streams Application
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-56
Not to be reproduced in any form without prior written consent.
Hands-On Exercise: Writing a Kafka Streams Application
▪ In this Hands-On Exercise, you will write a Kafka Streams application
▪ Please refer to the Hands-On Exercise Manual

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-57
Not to be reproduced in any form without prior written consent.
Kafka Streams
▪ Kafka Streams Basics
▪ Kafka Streams Basic Concepts
▪ Key Features of Kafka Streams
▪ Kafka Streams APIs
▪ Hands-On Exercise: Writing a Kafka Streams Application
▪ Chapter Review

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-58
Not to be reproduced in any form without prior written consent.
Chapter Review
▪ Kafka Streams provides a way of writing streaming data applications that is
– Simple
– Powerful
– Flexible
– Scalable

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

05-59
Not to be reproduced in any form without prior written consent.
Conclusion
Chapter 6
Course Contents
01: Introduction

02: The Motivation for Apache Kafka

03: Kafka Fundamentals

04: Ingesting Data with Kafka Connect

05: Kafka Streams

>>> 06: Conclusion

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

06-1
Not to be reproduced in any form without prior written consent.
Kafka: A Complete Streaming Data Platform
▪ Kafka Connect ingests and exports data, with no code required
▪ Kafka Streams provides powerful, flexible, scalable stream data processing
▪ Together, they give us a way to import data from external systems, manipulate or
modify that data in real time, and then export it to destination systems for storage
or further processing

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

06-2
Not to be reproduced in any form without prior written consent.
Thanks!
▪ Thank you for coming!
▪ If you have feedback, suggestions, or questions, please e-mail [email protected]
▪ Obligatory commercial messages:
– Confluent offers Developer and Operations training for Kafka. See
http://confluent.io/training
– We’re hiring! See http://confluent.io/careers
– Especially: We’re hiring instructors!

Copyright © 2015, 2016, 2017 Confluent, Inc. All rights reserved.

06-3
Not to be reproduced in any form without prior written consent.
 Discount code: kafstrata
Special Strata Attendee discount code = 25% off
www.kafka-summit.org
Kafka Summit New York: May 8
Kafka Summit San Francisco: August 28

Presented by