2010 IBM Corporation

Information Management
Advanced DataStage Workshop
Module 06 Buffering in Parallel Jobs
2
2010 IBM Corporation
Information Management
After completing this module, you should be able to:
Explain what is buffering in parallel jobs
Understand the effect of buffer between stages
Tune buffering when it is necessary
Avoid buffers with certain job designs using different technics
Module Summary
2010 IBM Corporation
Information Management
At runtime, buffer operators are inserted to prevent deadlocks and to
optimize performance
Provide resistance for incoming rows
For fork-joins, buffer operators are inserted on all
inputs to the downstream join operator
Buffer operators may also be inserted in an
attempt to match producer and consumer rates
Data is never repartitioned across
a buffer operator
First-in, first-out row processing
Some stages (e.g. Sort, Hash Aggregator) internally buffer the entire
dataset before outputting a row
Buffer operators are never inserted after these stages
Introducing the Buffer Operator
Stage 3
Buffer
Buffer
Stage 1
Stage 2
2010 IBM Corporation
Information Management
Identifying Buffer Operators
At runtime, buffers are
identified in the operators
section of the job SCORE
It has 6 operators:
op0[1p] {(sequential Row_Generator_0)
on nodes (
ecc3671[op0,p0]
)}
op1[1p] {(sequential Row_Generator_1)
on nodes (
ecc3672[op1,p0]
)}
op2[1p] {(parallel APT_LUTCreateImpl in Lookup_3)
on nodes (
ecc3671[op2,p0]
)}
op3[4p] {(parallel buffer(0))
on nodes (
ecc3671[op3,p0]
ecc3672[op3,p1]
ecc3673[op3,p2]
ecc3674[op3,p3]
)}
op4[4p] {(parallel APT_CombinedOperatorController:
(APT_LUTProcessImpl in Lookup_3)
(APT_TransformOperatorImplV0S7_cpLookupTest1_Transformer_7
in Transformer_7)
(PeekNull)
) on nodes (
ecc3671[op4,p0]
ecc3672[op4,p1]
ecc3673[op4,p2]
ecc3674[op4,p3]
)}
op5[1p] {(sequential APT_RealFileExportOperator in
Sequential_File_12)
on nodes (
ecc3672[op5,p0]
)}
It runs 12 processes on 4 nodes.
2010 IBM Corporation
Information Management
How Buffer Operators Work
The primary goal of a buffer operator is to prevent deadlocks
This is accomplished by holding rows until the downstream operator
is ready to process them
Rows are held in memory up to size defined by $APT_BUFFER_MAXIMUM_MEMORY
default is 3MB per buffer per partition
When buffer memory is filled, rows are spilled to disk
By default, it will use up to amount of available scratch disk unless QUEUE UPPER
BOUND limit has been set
Note that limiting the queue upper bound can cause deadlocks!
Buffer
Producer Consumer
2010 IBM Corporation
Information Management
Buffer Flow Control
When buffer memory usage reaches $APT_BUFFER_FREE_RUN the buffer
operator will offer resistance to the new rows, slowing down the rate of
upstream producer
Default 0.5 = 50%
Setting $APT_BUFFER_FREE_RUN > 100% (1) prevents the buffer from
slowing down the upstream producer until data size of
$APT_BUFFER_MAXIMUM_MEMORY * $APT_BUFFER_FREE_RUN has been buffered
Assumes that the overhead of disk I/O for buffer scratch usage is less than the impact
of slowing down upstream operator
Producer Consumer Buffer
$APT_BUFFER_FREE_RUN
Buffer will offer resistance to
new rows, slowing down
upstream producer
2010 IBM Corporation
Information Management
Tuning Buffer Settings Environment Variables
On a per-job basis through environment variables
$APT_BUFFER_MAXIMUM_MEMORY
$APT_BUFFER_FREE_RUN
$APT_BUFFER_DISK_WRITE_INCREMENT
And many other advanced options
In general, buffer tuning is an advanced topic. The default settings
should be appropriate for most job flows.
For very wide rows, it may be necessary to increase default buffer size to
handle more rows in memory
Calculate total record width using internal storage for each data type / length /
scale. For variable-length (varchar) columns, use maximum length.
2010 IBM Corporation
Information Management
Tuning Buffer Settings Environment Variables
On a per-link basis
(Inputs/Outputs ->Advanced)
Buffer options are defined per link
(virtual dataset)
Hence the output of one stage is
the input of the following stage
In general, Auto Buffering
(default) is recommended
Dont change unless you really
understand your job flow and
data!
Disabling buffering may cause the
job to deadlock
2010 IBM Corporation
Information Management
Buffer Resource Usage
By default, each buffer operator uses 3MB per partition of virtual
memory
Can be changed through Advanced link properties, or globally using
$APT_BUFFER_MAXIMUM_MEMORY
When buffer memory is filled, temporary disk space is used in the
following order:
Scratch disks in the $APT_CONFIG_FILE buffer named disk pool
Scratch disks in the $APT_CONFIG_FILE default disk pool
The default directory specified by $TMPDIR
The UNIX /tmp directory
2010 IBM Corporation
Information Management
End of Data / End of Group
Stages that process groups of data (Join, Merge, Aggregator in Sort
mode) cannot output a row until:
Data in the grouping key column changes (End of Group)
All rows have been processed (End of Data)
Rows are buffered in memory until the End of Group/Data
Some stages (e.g. Sort, Aggregator in Hash mode) must read the
entire input before outputting a single record
Setting Dont Sort, Previously Sorted key option changes Sort behavior to
output on groups instead of entire dataset
2010 IBM Corporation
Information Management
Join Stage: Internal Buffering
Even for inner Joins, there is a difference between the inputs of a Join
stage!
The first link (#0, LEFT within Link Ordering) establishes driver input
Rows are read one at a time
The second link (#1, RIGHT by Link Ordering) buffers all rows within
the group
2010 IBM Corporation
Information Management
Avoiding Buffer Contention
Datasets do not buffer
There is no upstream operation that would prevent rows from being output
In some cases, the best solution to avoiding fork-join buffer contention
is to split the job, landing results to intermediate datasets
Develop a single job first
If performance / volume testing indicates a buffering-related performance
issue that cannot be resolved by adjusting buffering settings, then split the
job across intermediate datasets
2010 IBM Corporation
Information Management
For large data volumes, buffering introduces a possible problem with
this solution:
At runtime, buffer operators are inserted for this scenario
The Join stage, operating on key-column groups, is unable to output rows
until end of group or data
Generating one header row with no subsequent change in join
column, data is buffered until end of group
Problem: Processing is halted until all rows in the group are read
Header
Detail
Src Out
Buffer
Buffer
Revisiting the Header Detail Job Design
2010 IBM Corporation
Information Management
Do the join in the Transformer using stage variables
Stage variables store the information of the header records
Data is hash partitioned to ensure that header and detail records in a
group are not spread across different partitions
Buffering Solution
2010 IBM Corporation
Information Management
Redesigned Header Detail Processing Job
Parse out OrderNum and
RecType columns
Store Header info
in stage variables
16
2010 IBM Corporation
Information Management
Impact of Buffering
Consider maixmum row width
For very wide row, it may be necessary to increase buffer size to hold more
rows in memory
Default is 3 MB per partition
Set in stage properties
For entire job, set with $APT_BUFFER_MAXIMUM_MEMORY
Tune all other factors before tune buffer settings
Disabling buffer may cause dead lock
Best solution might be not to use fork-join design pattern that will have
inserted buffer operators
17
2010 IBM Corporation
Information Management
Isolating Buffers
Buffer operators may make it difficult to identify performance bottleneck
The following environment variables effectively isolate each stage (by
inserting buffers), and prevent the buffers from slowing down upstream
stages
$APT_BUFFERING_POLICY=FORCE
Insert buffer between each operator (isolates)
$APT_BUFFER_FREE_RUN=1000
Writes excess buffers to disk instead of slowing down producer
Buffer will not slow down producer until it has written
1000*$APT_MAXIMUM_MEMORY to disk
These settings will generate a significant amount of disk I/O therefore DO NOT
use these setting for production jobs
18
2010 IBM Corporation
Information Management
After completing this module, you should be able to:
Explain what is buffering in parallel jobs
Understand the effect of buffer between stages
Tune buffering when it is necessary
Avoid buffers with certain job designs using different technics
Module Summary