Chapter 7 Flink, Stream and Batch

Processing in a Single Engine

0 Huawei Confidential
 Objectives

On completion of this course, you will be able to:

 Master the core technologies and architecture of Flink.
 Understand the time and window mechanisms of Flink.
 Know the fault tolerance mechanism of Flink.

1 Huawei Confidential
 Contents

1. Principles and Architecture of Flink

2. Flink Time & Window

3. Flink Watermark

4. Fault Tolerance of Flink

2 Huawei Confidential
Overview
 Apache Flink is an open-source stream processing framework for distributed, high-performance
stream processing applications. Flink not only supports real-time computing with both high
throughput and exactly-once semantics, but also provides batch data processing.
 Compared with other data processing engines in the market, Flink and Spark support both stream
processing and batch processing. From the perspective of the technical philosophy, Spark
simulates stream computing based on batch computing. Flink is the opposite. It simulates batch
computing based on stream computing.

Flink
3 Huawei Confidential
Key Concepts
 Stateful stream processing, event time, window mechanism for continuous processing
of streaming data, and checkpoints for state snapshots.

Time Window

State Checkpoint

4 Huawei Confidential
Core Ideas
 The biggest difference between Flink and other stream computing engines is
state management.
 Flink provides built-in state management. You can store states in Flink instead
of storing them in an external system. Doing this can:
 Reduce the dependency of the computing engine on external systems, simplifying
deployment and O&M.
 Significantly improve performance.

5 Huawei Confidential
Overall Architecture of Flink Runtime

6 Huawei Confidential
Key Concept - DataStream
 DataStream: represents streaming data. You can think of DataStream as immutable
collections of data that can contain duplicates. The number of elements in DataStream
are unlimited.

7 Huawei Confidential
Flink Program
 Flink programs consist of three parts: Source, Transformation, and Sink. Source is
responsible for reading data from data sources such as HDFS, Kafka, and text.
Transformation is responsible for data transformation operations. Sink is responsible for
final data outputs (such as HDFS, Kafka, and text). Data that flows between parts is
called stream.

9 Huawei Confidential
Flink Data Sources
 Batch processing  Stream processing
 Files  Files
 HDFS, Local file system, MapR file system  JDBC
 Text, CSV, Avro, Hadoop input formats  Socket streams
 JDBC  Kafka
 RabbitMQ
 HBase
 Flume
 Collections
 Collections
 Implement your own
 SourceFunction.collect
10 Huawei Confidential
Flink Program Running Diagram

11 Huawei Confidential
Flink Job Running Process
 A user submits a Flink program to JobClient. JobClient processes, parses, and optimizes the
program, and then submits the program to JobManager. TaskManager runs the task.

4. Submit the
task.
1. Submit the Flink 2. Send the
program. program.

3. Return a message of program

submitted to JobClient.

6. Return task results to

JobClient.

5. Regularly report the

task status.

12 Huawei Confidential
Bounded Stream and Unbounded Stream
 Unbounded stream: Only the start of a stream is defined. Data sources generate data endlessly. The data of
unbounded streams must be processed continuously. That is, the data must be processed immediately after
being read. You cannot wait until all data arrives before processing, because the input is infinite and cannot
be completed at any time. Processing unbounded stream data typically requires ingesting events in a
particular sequence, such as the sequence of the events that occurred, so that the integrity of the results can
be inferred.
 Bounded stream: Both the start and end of a stream are defined. Bounded streams can be computed after all
data is read. All data in a bounded stream can be sorted, without ingesting events in an order. Bounded
stream processing is often referred to as batch processing.

16 Huawei Confidential
Batch Processing Example (1)
 Batch processing is a very special case of stream processing. Instead of defining a
sliding or tumbling window over the data and producing results every time the window
slides, a global window is defined, with all records belonging to the same window. For
example, a simple Flink program that counts visitors in a website every hour, grouped
by region continuously, is the following:

val counts = visits

.keyBy("region")
.timeWindow(Time.hours(1))
.sum("visits")

17 Huawei Confidential
Batch Processing Example (2)
 If you know that the input data is bounded, you can implement batch processing using
the following code:
val counts = visits
.keyBy("region")
.window(GlobalWindows.create)
.trigger(EndOfTimeTrigger.create)
.sum("visits")

 If the input data is bounded, the result of the following code is the same as that of the
preceding code:
val counts = visits
.groupBy("region")
.sum("visits")
18 Huawei Confidential
Flink Processing Model
 Flink uses a bottom-layer engine to support both stream and batch processing.

DataStream API DataSet API

Backtracking for
Checkpoints, state
scheduling and recovery,
management,
special memory data
watermark, window,
structures, and query
and trigger
optimization

Stream processing engine

19 Huawei Confidential
Stream and Batch Processing Mechanisms
 The two sets of Flink mechanisms correspond to their respective APIs
(DataStream API and DataSet API). When creating a Flink job, you cannot
combine the two sets of mechanisms to use all Flink functions at the same
time.
 Flink supports two types of relational APIs: Table API and SQL. Both of these
APIs are for unified batch and stream processing, which means that relational
APIs execute queries with the same semantics and produce the same results on
unbounded data streams and bounded data streams.
 The Table API and SQL APIs are becoming the main APIs for analytical use cases.
 The DataStream API is the primary API for data-driven applications and pipelines.

20 Huawei Confidential
 Contents

1. Principles and Architecture of Flink

2. Flink Time & Window

3. Flink Watermark

4. Fault Tolerance of Flink

21 Huawei Confidential
Time Background
 In stream processing programming, time processing is very critical. For example,
event stream data (such as server log data, web page click data, and
transaction data) is continuously generated. In this case, you need to use keys
to group events and count the events corresponding to each key at a specified
interval. This is our known "big data" application.

22 Huawei Confidential
Time Classification in Stream Processing
 During stream processing, the system time (processing time) is often used as
the time of an event. However, the system time (processing time) is the time
that is imposed on an event. Due to reasons such as network delay, the
processing time cannot reflect the sequence of events.
 In actual cases, the time of each event can be classified into the following types:
 Event time: time when an event occurs
 Ingestion time: time when an event arrives at the stream processing system
 Processing time: time when an event is processed by the system

23 Huawei Confidential
Differences Between Three Time
 In the actual situation, the sequence of events is different from the system time. These
differences are caused by network delay and processing time. See the following figure.
 The horizontal coordinate indicates the event
time, and the vertical coordinate indicates the
processing time. Ideally, the coordinate formed by
the event time and processing time should form a
line with a tilt angle of 45 degrees. However, the
processing time is later than the event time. As a
result, the sequence of events is inconsistent.

25 Huawei Confidential
Window Overview
 Streaming system is a data processing engine designed to process infinite data
sets. Infinite data sets refer to constantly growing data sets. Window is a
method for splitting infinite data sets into finite blocks for processing.
 Windows are at the heart of processing infinite streams. Windows split the
stream into "buckets" of finite size, over which we can apply computations.

27 Huawei Confidential
Window Types
 Windows can be classified into two types:
 Count Window: Data-driven window is generated based on the specified number of
data records, which is irrelevant to time.
 Time Window: Time-driven window is generated based on time.
 Apache Flink is a distributed computing framework that supports infinite stream
processing. In Flink, Window can divide infinite streams into finite streams.

28 Huawei Confidential
Time Window Types
 According to the window implementation principles, Time Window can be
classified into tumbling window, sliding window, and session window.

29 Huawei Confidential
Tumbling Window
 The data is sliced according to a fixed window length. Characteristics: The time is aligned, the
window length is fixed, and the windows do not overlap.
 Application scenario: BI statistics collection (aggregation calculation in each time period)
 For example, assume that you want to count the values emitted by a sensor. A tumbling window
of 1 minute collects the values of the last minute, and emits their sum at the end of the minute.
See the following figure.

Input

Tumbling
window

Output

30 Huawei Confidential
Sliding Window
 The sliding window is a more generalized form of a fixed window. It consists of a fixed window
length and a sliding interval. Characteristics: The time is aligned, the window length is fixed, and
the windows can be overlapping.
 Application scenario: Collect statistics on the failure rate of an interface over the past 5 minutes
to determine whether to report an alarm.
 A sliding window of 1 minute that slides every half minute counts the values of the last minute,
emitting the count every half minute. See the following figure.

Input
Sliding
window

Output

31 Huawei Confidential
Session Window
 A session window consists of a series of events and a timeout interval of a specified
duration. It is similar to a web application session. That is, a new window is generated if
no new data is received within a period of time. Characteristics: The time is not aligned.

32 Huawei Confidential
 Contents

1. Principles and Architecture of Flink

2. Flink Time & Window

3. Flink Watermark

4. Fault Tolerance of Flink

34 Huawei Confidential
Out-of-Order Problem
 There is a process from event generation to stream processing through the source and
then to the operator. In most cases, data is sent to the operator based on the time
when the event is generated. However, out-of-order data may be generated, such as,
events received by Flink are not sorted strictly based on the event time due to network
problems.

Ideal situation

Actual situation
35 Huawei Confidential
Why Are Watermarks Needed?
 For infinite data sets, there is no effective way to determine data integrity.
Therefore, watermark is a concept based on event time to describe the integrity
of data streams. If events are measured by processing time, everything is
ordered and perfect. Therefore, watermarks are not required. In other words,
event time causes disorder. Watermarks are used to solve the disorder problem.
The so-called disorder is actually an event delay. For a delayed element, it is
impossible to wait indefinitely. A mechanism must be provided to ensure that
the window is triggered for calculation after a specific time. This special
mechanism is watermark, which tells operators that delayed messages should
no longer be received.

37 Huawei Confidential
Watermark Principles (1)
 How does Flink ensure that all data has been processed when the event time-based
window is destroyed? This is a watermark does. A watermark is a monotonically
increasing timestamp t. Watermark(t) indicates that all data whose timestamp is less
than or equal to t has arrived, and data whose timestamp is less than or equal to t will
not be received in the future. Therefore, the window can be safely triggered and
destroyed.

38 Huawei Confidential
Watermark Principles (2)
 The following figure shows the watermark of ordered streams (Watermark is set to 0).

 The following figure shows the watermark of out-of-order streams (Watermark is set
to 2).

39 Huawei Confidential
Delayed Data
 Watermark is a means of coping with out-of-order data, but in the real world we
cannot obtain a perfect watermark value - either it cannot be obtained, or it is too
costly. Therefore, in actual use, we will use an approximate Watermark(t) value, but
there is still a low probability that the data before the timestamp t will be received. In
Flink, the data is defined as late elements. Similarly, we can specify the maximum
latency allowed in the window (0 by default), which can be set using the following
code:

input .keyBy(<key selector>)

.window(<window assigner>)
.allowedLateness(<time>)
.<windowed transformation>(<window function>);

40 Huawei Confidential
Delayed Data Processing Mechanism
 Delayed events are special out-of-order events. Different from common out-of-
order events, their out-of-order degree exceeds that watermarks can predict. As
a result, the window is closed before they arrive.
 In this case, you can use any of the following methods to solve the problem:
 Reactivate the closed windows and recalculate to correct the results.
 Collect the delayed events and process them separately.
 Consider delay events as error messages and discard them.
 By default, Flink discards the third method and uses Side Output and Allowed
Lateness instead.

41 Huawei Confidential
 Contents

1. Principles and Architecture of Flink

2. Flink Time & Window

3. Flink Watermark

4. Fault Tolerance of Flink

45 Huawei Confidential
Checkpoint Mechanism
 Apache Flink offers a lightweight fault tolerance mechanism based on distributed
checkpoints. A checkpoint is an automatic, asynchronous snapshot of task/operator
state. Flink generates checkpoint barriers at intervals on the input data set and uses
barriers to divide the data during the interval into the corresponding checkpoints. When
an application error occurs, the states of all operators can be restored from the
previous snapshot to ensure data consistency.
 For applications with small state, these snapshots are very light-weight and can be
drawn frequently without impacting the performance much. During checkpointing, the
state is stored at a configurable place (such as the JobManager node or HDFS).

47 Huawei Confidential
How Do I Restore Job Data?
 The checkpoint can be retained in an external media when a job is cancelled.
Flink also has another mechanism, savepoint, to restore job data.
 Savepoints are a special implementation of checkpoints. The underlying layer
uses the checkpoint mechanism. Savepoints are manually triggered by running
commands and results are saved to a specified storage path so that the system
can be restored to the original computing status during system upgrade and
maintenance and end-to-end exactly-once semantics can be ensured.

54 Huawei Confidential
Savepoint and Checkpoint

Checkpoint Savepoint
Triggering and Automatically triggered and Manually triggered and
management managed by Flink managed by users
Quickly restores tasks from Backs up data as planned, for
Purpose failures, for example, timeout example, by modifying code or
due to network jitter. adjusting concurrency.
• Persistent
• Lightweight
• Stored in a standard format
• Automatic recovery from
and allows code or
Features failures
configuration changes.
• State is cleared by default
• Manually restores data from
after a job is stopped.
savepoints.

55 Huawei Confidential
State Storage Method - MemoryStateBackend
 Constructor
 MemoryStateBackend(int maxStateSize, boolean asynchronousSnapshots)
 Storage Methods
 State: TaskManager memory
 Checkpoint: JobManager memory
 Capacity Limit
 The default value of maxStateSize for a single state is 5 MB.
 maxStateSize ≤ akka.framesize (Default value: 10 MB)
 The total size cannot exceed the memory of JobManager.
 The MemoryStateBackend is encouraged for local testing and jobs that hold little
state, such as ETL.

56 Huawei Confidential
State Storage Method - FsStateBackend
 Constructor
 FsStateBackend(URI checkpointDataUri ,boolean asynchronousSnapshots)
 Storage Methods
 State: TaskManager memory
 Checkpoint: external file storage system (local or HDFS)
 Capacity Limit
 The amount of state on a single TaskManager cannot exceed its memory.
 The total size cannot exceed the configured file system capacity.
 The FsStateBackend is encouraged for jobs with large state, such as aggregation at
the minute-window level and join, and jobs requiring all high-availability setups. It
can be used in production scenarios.

57 Huawei Confidential
State Storage Method - RocksDBStateBackend
 Constructor
 RocksDBStateBackend(URI checkpointDataUri ,boolean enableIncrementalCheckpointing)
 Storage Methods
 State: KV database on the TaskManager (used memory + disk)
 Checkpoint: external file storage system (local or HDFS)
 Capacity Limit
 The amount of state on a single TaskManager cannot exceed the memory and disk size of the TaskManager.
 The maximum size of a key is 2 GB.
 The total size cannot exceed the configured file system capacity.
 The RocksDBStateBackend is encouraged for jobs with very large state, for example, aggregation
at the day-window level, jobs requiring high-availability setups, and jobs that do not require high
read/write performance. It can be used in production scenarios.

58 Huawei Confidential
 Summary

 This course explains the architecture and technical principles of Flink and
the running process of Flink programs. The focus is on the difference
between Flink stream processing and batch processing. In the long run, the
DataStream API should contain the DataSet API through bounded data
streams.

59 Huawei Confidential
 Quiz

1. What are the four key concepts of Flink?

2. What are the two types of APIs for Flink stream processing and batch processing?

60 Huawei Confidential
 Recommendations

 Huawei Cloud Official Web Link:

 https://www.huaweicloud.com/intl/en-us/
 Huawei MRS Documentation:
 https://www.huaweicloud.com/intl/en-us/product/mrs.html
 Huawei TALENT ONLINE:
 https://e.huawei.com/en/talent/#/

61 Huawei Confidential
Thank you.

62 Huawei Confidential