Ab Initio

1
Confidential
 Ab Initio

 Data processing tool from Ab Initio software

corporation (http://www.abinitio.com)
 Latin for “from the beginning”
 Designed to support largest and most complex
business applications
 Graphical, intuitive, and “fits the way your
business works”text

2
Confidential
 Ab Initio

Graphical Development Environment

(GDE)
FTP
TELNET
REXEC
RSH
DCOM

Co-operating System
On a typical installation, the Co-operating system is
installed on a Unix or Windows NT server while the
GDE is installed on a Pentium PC.

3
Confidential
 Ab Initio

 Layered on the top of the operating system

 Unites a network of computing resources – CPUs, storage disks, programs,
datasets into a data-processing system with scalable performance
 Co>Operating system runs on …

IBM AIX 4.1,4.2,4.3….

Sun Solaris 2.5.1,2.6,2.7….
HP HPUX 10, 11….
Sequent DYNIX/ptx 4.4.1, 4.4.2….
Alpha Digital Unix 4.0D, 4.4.2….
Pyramid Reliant Unix 5.43 ….
Intel Windows NT 4.0

4
Confidential
 Ab Initio

The GDE …
 can talk to the Co-operating system using several protocols like Telnet,

Rexec and FTP

 is GUI for building applications in Ab Initio

 and the Co-operating system have different release mechanisms, making

Co-operating system upgradation possible without change in the GDE

release
Note: During deployment, GDE sets AB_COMPATIBILITY to the Co>Operating System version number. So, a change in the

Co>Operating System release may require a re-deployment

5
Confidential
 Ab Initio

A Graph

 is the logical modular unit of an application.

 consists of several components that forms the
building blocks of an Ab Initio application
A Component

 is a program that does a specific type of job and can

be controlled by its parameter settings.
A Component Organizer

 groups all components under different categories.

6
Confidential
 Ab Initio

A Sample Graph …
Datasets
Dataset
Components

L1

L1* L1* out*

L1 Good
Score Select
deselect* Customers
Customers
L1

Other
Customers
Flows

7
Confidential
 Ab Initio

A Sample Graph …
Expression Metadata

Record format metadata

Ports

Layout

8
Confidential
 Ab Initio

 Files
 Formats
 Components
 Flows
 Layouts
 Building with mp job
 Building with mp run

9
Confidential
 Ab Initio

 Setup Command
◦ Ab Initio Host (AIH) file
◦ Builds up the environment to run an Ab Initio application.

 Start Script (Host Setup)

◦ Local to the Graph

 Graph
 End Script
◦ Local to the Graph

10
Confidential
 Ab Initio

 A graph, after development, is deployed to the back-end

server as a Unix shell script or Windows NT batch file.
 This becomes the executable to run at the back-end
with the help of the Co-operating system.
 The execution can be done from the GDE itself or
manually from the back-end
 Ab Initio runtime environment is different from the
development environment.

11
Confidential
 Ab Initio

12
Confidential
 Ab Initio

 DML
◦ Ab Initio stores metadata in the form of record formats.

◦ Metadata can be embedded within a component or can be

stored external to the graph in a file with a “.dml” extension.

 XFR
◦ Data can be transformed with the help of transform functions.

◦ Transform functions can be embedded within a component or

can be stored external to the graph in a file with a “.xfr”
extension.

13
Confidential
 Ab Initio

Record Format Metadata in DML

0345John Smith

0212Sam Spade

Field Names 0322Elvis Jones

0492Sue West
Data Types 0221William Black

record

decimal(4) id;
DML BLOCK string(6) first_name;

string(6) last_name;

end

14
Confidential
 Ab Initio

DML Syntax

 Record types begin with record and end with end

 Fields are declared: data_type(len) field_name;

 Field names consist of letters(a…z,A…Z),digits(0…9) and

underscores(_) and are Case sensitive

 Keywords/Reserved words are record, end, date….

15
Confidential
 Ab Initio

More DML Types

0345,01-09-02,1000.00John,Smith
Delimiters
0212,05-07-03, 950.00Sam,Spade

0322,17-01-00, 890.50Elvis,Jones

0492,25-12-02,1000.00Sue,West

record 0221,28-02-03, 500.00William,Black

decimal(“,”) id;

date(“DD-MM-YY”)(“,”) join_date;

decimal(7,2) salary_per_day;
Precision string(“,”) first_name;
& Scale
string(“\n”) last_name;

end

16
Confidential
 Ab Initio

Null in Oracle  Missing Information

Null in Ab Initio  Non-existence of coulmns

Ab Initio uses Conditional DML and “is_defined” function

in order to take care of NULL values.
Records Table Data

Header Header NULL NULL

Body NULL Body NULL

Trailer NULL NULL Trailer

17
Confidential
 Ab Initio

record
0345,090297John,Smith; decimal(7) id;
date(“MMDDYY”) join_date;
string(“,”) first_name;
Drop string(“;”) last_name;
end

Reformat
Reformat Reorder

id+1000000

record
decimal(7) id;
string(8) last_name;
date(“DD-MM-YY”)(“,”) join_date; 1000345,Smith 1997-09-02
end

18
Confidential
 Ab Initio

 User-defined function producing one or more output from one or more

input
 Associated with transform components
 Rules that computes expression from input values and local variable and
assigns the result to output objects
 Syntax
◦ Functions :
output-records : : function-name (input-records) =
begin
assignments
end;
◦ Assignments :
output-records.field : : expression;

19
Confidential
 Ab Initio

Ab Initio >

DAY 2

20
Confidential
 Ab Initio

 Filter by Expression

 Reformat

 Redefine Format

 Sort

 Join

 Replicate

 Dedup

 Aggregate

 Rollup

 Scan

21
Confidential
 Ab Initio

1. Reads record from input port

2. Evaluate the select_expr
3. If result is true, record written to out port
4. If result is false, record written to deselect port
Input port

expr
true?
Yes No

Out port Deselect

22
Confidential
 Ab Initio

 REJECT
◦ Input records that caused error

 ERROR
◦ Associated error message

 LOG
◦ Logging records

Note: Every transform component has got diagnostic ports

23
Confidential
 Ab Initio

1. Reads record from input port

2. Record passes as argument to transform
function or xfr
3. Records written to out ports, if the function
returns a success status
4. Records written to reject ports, if the function
returns a failure status

24
Confidential
 Ab Initio

Parameters of Reformat Component

 Count

 Transform (Xfr) Function

 Reject-Threshold

◦ Abort

◦ Never Abort

◦ Use Limit & Ramp

 Limit

 Ramp

25
Confidential
 Ab Initio

 Limit
◦ Number of errors to tolerate

 Ramp
◦ Scale of errors to tolerate per input

 Tolerance value=limit + ramp*total number of records read

 Typical Limit and Ramp settings . .
◦ Limit = 0 Ramp = 0.0  Abort on any error
◦ Limit = 50 Ramp = 0.0  Abort after 50 errors
◦ Limit = 1 Ramp = 0.01  Abort if more than 1 in 100 records causes error
◦ Limit = 1 Ramp = 1  Never Abort

26
Confidential
 Ab Initio

Keys
A key identifies a field or set of fields to organize a dataset
◦ Single Field: employee_number
◦ Multiple field or Composite key: (last_name; first_name)
◦ Modifiers: employee_number descending
A surrogate key is a substitution for the natural primary key.
 It is just a unique identifier or number for each record like ROWID of an Oracle table

Sort Component
Reads records from input port, sorts them by key, writes result to output port
Parameters

◦ Key
◦ Max-core

27
Confidential
 Ab Initio

1. Reads records from multiple input ports

2. Operates on records with matching keys using a multi-input transform
function
3. Writes result to the output port

PORTS PARAMETERS
in count
out key
unused override key
reject (optional) transform
error (optional) limit
log (optional) ramp

28
Confidential
 Ab Initio

Join Types
◦ Inner
◦ Outer
◦ Explicit

Join Methods
◦ Merge Join
 Using sorted inputs

◦ Hash Join
 Using in-memory hash tables to group input

29
Confidential
 Ab Initio

The Priority is the order of evaluation of rules in a

transform function.

An example

A join component may have a transform

function with prioritized rules as

out.ssn :1: in1.ssn;

out.ssn :2: in2.ssn;

out.ssn :3: "999999999";

30
Confidential
 Ab Initio

 Data transformation in multiple stages following several sets of

rules
 Each set of rule form one transform function
 Information is passed across stages by temporary variables
 Stages include initialization, iteration, finalization and more
 Few multistage components are aggregate,rollup,scan

Aggregate/Rollup/Scan
Generates summary records for group of input records

31
Confidential
 Ab Initio

Name Description
Normalize Generates multiple data records from each input data record
Separate a data record with a vector field into several individual records, each containing
one element of the vector.
Denormalize Consolidates groups of related data records into a single output record with a vector
Sorted field for each group
Requires Grouped Input
Validate Separates valid data records from invalid data records
Records
Check Order Tests whether data records are sorted according to a key-specifier.

Compare Compares data records from two flows one by one

Records
Generate Generates a specified number of data records with fields of specified lengths and types.
Records
Gather Logs Collects the output from the log ports of components for analysis of a graph after
execution
Sample Selects a specified number of data records at random from one or multiple input flows

32
Confidential
 Ab Initio

Ab Initio built-in functions are DML expressions that

◦ can manipulate strings, dates, and numbers
◦ access system properties
Function categories
◦ Date functions : now(), today(), date_to_int(), ..
◦ Inquiry and error functions: is_defined(), is_valid(),
force_error(), ..
◦ Lookup functions: lookup(), lookup_local(), ..
◦ Math functions: ceiling(), floor(), ..
◦ Miscellaneous functions:decimal_round(), hash_value(), ..
◦ String functions: string_substring(), is_blank(), ..

33
Confidential
 Ab Initio

Perform pattern matching operations on strings using regular expressions

Name Function Example
re_get_match() Returns the first string that •re_get_match("man on the moon", "oo")  “oo”
matches a regular expression. •re_get_match("1234 Milldale Ave.", "[a-zA-Z]+") 
"Milldale"
•re_get_match("1234 Milldale Ave.", "^[0-9]+") 
"1234“
•re_get_match("Fourteen 35th St.", "^[0-9]+")  NULL
re_index() Returns the index of the first re_index("man on the moon", "oo")  13
character of a string matching re_index("1234 Milldale Ave.", "[a-zA-Z]+")  6
a regular expression. re_index("1234 Milldale Ave.", "^[0-9]+")  1
re_index("Fourteen 35th St.", "^[0-9]+")  0
re_replace() Returns a string after replacing re_replace("man on the moon", "m[aeiou]+n", "X") 
all the substrings matching a “ X on X”
regular expression
re_replace_first() Returns a string after replacing re_replace_first("man on the moon", "m[aeiou]+n", "X")
the first substring matching a  "X on the moon"
regular expression.

34
Confidential
 Ab Initio

A Vector is

an array of same type of elements that is repeated

number of repeats is a constant integer or the value of another field in the record.

record
string(20) cust_id;

decimal(3) num_purchases;

decimal(8.2)[num_purchases]purchase_amt;
end;

NOTE: purchase_amount field is repeated depending on the value in

the number_of_purchases field

Initializing a Vector
let decimal(‘,’)[100] values = make_constant_vector(100, 0);

Converting a string into a vector

let decimal(‘,’)[3] values = reinterpret_as(val_str, “,”, 3);

35
Confidential
 Ab Initio

db_config_utility : Generate interface file to the database

Input Table

◦ unloads data records from a database into an Ab Initio graph

◦ Source : DB table or SQL statement to SELECT from table

Output Table

◦ loads data records into a database

◦ Destination : DB table or SQL statement to INSERT into table

Update Table

◦ executes UPDATE or INSERT statements in embedded SQL format to modify a

DB table

36
Confidential
 Ab Initio

Truncate Table

◦ deletes all the rows in a specified DB table

Run SQL

◦ executes SQL statements in a DB

37
Confidential
 Ab Initio

Ab Initio >

DAY 3

38
Confidential
 Ab Initio

Parallel Runtime Environment

Where some or all of the components of an application – datasets and processing
modules are replicated into a number of partitions, each spawning a process.

Ab Initio can process data in parallel runtime

environment
Forms of Parallelism
Inherent in Ab Initio
◦ Component Parallelism
◦ Pipeline Parallelism
◦ Data Parallelism

39
Confidential
 Ab Initio

When different instances of same component run on separate data sets

Sorting Customers

Sorting Transactions

40
Confidential
 Ab Initio

When multiple components run on same data set

Processing Record 100

Processing Record 99

41
Confidential
 Ab Initio

When data is divided into segments or partitions and processes run simultaneously on each partition

Expanded View

Global View

Multifile

NOTE : # of processes per component = # of partitions

42
Confidential
 Ab Initio

Multifiles

A global view of a set of ordinary files called partitions

usually located on different disks or systems
Ab Initio provides shell level utilities called “m_ commands”
for handling multifiles (copy, delete,move etc.)
Multifiles reside on Multidirectories
Each is represented using URL notation with “mfile” as the
protocol part:
 mfile://pluto.us.com/usr/ed/mfs1/new.dat

43
Confidential
 Ab Initio

A directory spanning across partitions on different hosts

mfile://host1/u/jo/mfs/mydir

//host1/u1/jo/mfs
//host1/vol4/pA/mydir //host2/vol3/pB/mydir //host3/vol7/pC/mydir

<.mdir>

Control Data Data Data

Partition Partition Partition Partition
on Host1 on Host2 on Host3

44
Confidential
 Ab Initio

A file spanning across partitions on different hosts

mfile://host1/u/jo/mfs/mydir/myfile.dat

//host1/u1/jo/mfs/mydir
/myfile.dat //host1/vol4/pA/mydir //host2/vol3/pB/mydir //host3/vol7/pC/mydir
/myfile.dat /myfile.dat /myfile.dat

Control Data Data Data

Partition Partition Partition Partition
on Host1 on Host2 on Host3

45
Confidential
 Ab Initio

Host Node  Agent Nodes 

A multidirectory

A multifile

Control file
Partitions (Serial Files)

Multidirectories  dat, s95

Multifiles  t.out, new.dat

46
Confidential
 Ab Initio

Data parallelism scales with data and requires data

partitioning
Data can be partitioned using different partitioning
methods.
The actual way of working in a parallel runtime
environment is transparent to the application developer.
It can be decided at runtime whether to work in serial
or in parallel, as well as to determine the degree of
parallelism

47
Confidential
 Ab Initio

Data can be partitioned using

Partition by Round-robin

Broadcast

Partition by Key

Partition by Expression

Partition by Range

Partition by Percentage

Partition by Load Balance

48
Confidential
 Ab Initio

Partition0 Partition1 Partition2

Writes records to the flow partitions in
A A
round-robin way, with block-size B B
C C
records going into one partition before D D
E E
moving on to the next F F
C C
D D
B B
G G
B B
Partition0 Partition1 Partition2 A A
A B C A A
D E F D D
C D B F F
G B A E E
A D F A A
E A D D D

49
Confidential
 Ab Initio

Key dependent data parallelism requires data partitioning

based on key value

Partition by key

◦ Generates a hash code based on the key

◦ Hash code determines which partition a record will go into

◦ Records with same key value goes into the same partition

Partition by Key is also called Hash Partition

50
Confidential
 Ab Initio

Partition0 Partition1 Partition2

Partition0 Partition1 Partition2
A A
A B D
B B
C E F
C C
C B D
D D
A G D
E E
A B F
F F
C C A E D
D D
B B
Hash + Sort = Parallel Grouping
G G
B B
A A
A A Partition0 Partition1 Partition2

D D A B D
A B D
F F
A B D
E E
A E D
A A
C E F
D D
C G F

51
Confidential
 Ab Initio

In key-dependent data parallelism,identical key values must be

in the same partition
Layouts determine the degree of parallelism
Fan-out and All-to-All flows with respect to data partitioning

52
Confidential
 Ab Initio

Partition by Range
◦ partitions according to the ranges of key values specified
for each partition
◦ Splitters or split port
◦ Partition by Range + Sort  Global Ordering

Partition by Percentage
◦ distributes a specified percentage of the total number of
input data records to each output flow
◦ Pct port

53
Confidential
 Ab Initio

Partition by Expression
◦ partitions according to a specified hash function or DML
expression

Partition by Load Balance

◦ distributes data records to its output flow partitions, writing
more records to the flow partitions that consume records
faster

Broadcast

◦ Acts like a partitioning component when the layout changes

54
Confidential
 Ab Initio

Method Key-Based Balancing Uses

Roundrobin No Good Record-independent
parallelism
Hash Yes Good Key-dependent
parallelism
Function Yes Depends on data and Application specific
function
Range Yes Depends on splitters Key-dependent
parallelism, Global
Ordering
Load-level No Depends on load Record-independent
parallelism

55
Confidential
 Ab Initio

A A
A A
A A p0 A A
B A A
C B C A p0
D B D
A B p1 A B
D D B p1
C C D
D C D C p2
B C p2 B
C D D
B D B D
A D A D
D p3 D
D p3
Partitions evenly balanced Partitions skewed

Skew determines the distribution of data on different partitions.

Skew for a data storage partition is defined as: (N – Average)/Max
where N is the number of bytes in that partition
AVERAGE is the total number of bytes in all the partitions divided by number of partitions
MAX is the number of bytes in the partition with the most bytes.

Common Source of Skew – large key groups

56
Confidential
 Ab Initio

Data can be de-partitioned using

Gather

Concatenate

Merge
Global View

Interleave

Expanded View

57
Confidential
 Ab Initio

Gather

◦ Reads data records from the flows connected to the input port

◦ Combines the records arbitrarily and writes to the output

Concatenate

◦ Concatenate appends multiple flow partitions of data records one after

another

Merge

◦ Combines data records from multiple flow partitions that have been sorted on

a key

◦ Maintains the sort order

58
Confidential
 Ab Initio

Summary of Departitioning Methods

Method Key-Based Ordering Uses
Concatenate No Global Creating serial flows
from partitioned data
Interleave No Inverse of Round Creating serial flows
Robin partition from partitioned data
Merge Yes Sorted Creating ordered
serial flows
Gather No Arbitrary Unordered
departitioning

59
Confidential
 Ab Initio

Ab Initio >

DAY 4

60
Confidential
 Ab Initio

Serial or Multifiles

Held in main memory

Searching and Retrieval is key-based and faster as compared to files stored on disks

associates key values with corresponding data values to index records and retrieve them

Lookup parameters

◦ Key

◦ Record Format

61
Confidential
 Ab Initio

Storage Methods

◦ Serial lookup : lookup()

 whole file replicated to each partition

◦ Parallel lookup : lookup_local()

 file partitions held separately

Lookup Functions

Name Arguments Purpose

lookup() File Label and Expression. Returns a data record from a Lookup File which matches with the
values of the expression argument

lookup_count() - do - Returns the number of matching data records in a Lookup File.

lookup_next() File Label Returns successive data records from a Lookup File.
lookup_local File Label and Expression. Returns a data record from a partition of a Lookup File.
lookup_count_local() - do - Same as lookup_count but for a single partition
lookup_next_local() File Label Same as lookup_count but for a single partition

NOTE: Data needs to be partitioned on same key before using lookup local functions

62
Confidential
 Ab Initio

In Parallel Environment unique keys are generated using the

functions

next_in_sequence()

this_partition()

number_of_partitions()

Key = number_of_partitions() * next_in_sequence() + this_partition()

last_generated_key

63
Confidential
 Ab Initio

A Deadlock happens when

◦ when the graph stops progressing because of mutual dependency of data among

components

◦ Identified when record count in does NOT change over a period of time

Deadlock can be avoided using

◦ Phasing

◦ Checkpointing

◦ Flow Buffering

Components likely to cause deadlock

Join Concatenate Compare records

Merge Interleave Compare Checksums

NOTE : Too many all-to-all flows likely to cause deadlocks

64
Confidential
 Ab Initio

Blocking on read

Blocking on write

65
Confidential
 Ab Initio

Ab Initio >

Progressing to the Next Level:

Performance

66
Confidential
 Ab Initio

Data Volume : Files

◦ Source/Target files

 DB Tables, Flat files

◦ Temporary files

 Phases

 Checkpoints

 Sort, Merge etc.

 Buffered Flows

Data Volume : Skew

◦ reflects data load balance among partitions

67
Confidential
 Ab Initio

Data Volume: I/O

◦ reading from/writing to files on disk

 datasets

 phases

 checkpoints

◦ loading/unloading tables from database

◦ "spilling" to disk

 In-memory join or rollup

 sort

68
Confidential
 Ab Initio

Data Volume: .WORK

◦ Temp files in parallel layouts are written to .WORK

subdirectories of the multifile system associated with the layout

◦ These files are cleaned up on successful job completion or

after rolling back the job with m_rollback -d.

◦ WARNING: .WORK contains hidden files and should never be

manually removed and recreated.

69
Confidential
 Ab Initio

Memory: Consumers

◦ Lookup Tables

 Serial lookup: lookup()

 whole table replicated into each partition

 Parallel lookup: lookup_local()

 table partitions held separately

◦ In-memory components

 Rollup, scan, normalize

 (temporary data) x (number of key groups)

 Join

 non-driving input(s) loaded into memory

70
Confidential
 Ab Initio

Memory: max_core

◦ Memory Allocated by component to hold data

◦ Exceeding max-core

 Disk “spilling” if memory is used up writing to disks

 Graph aborts if memory not available

 Performance bottleneck: slower execution

◦ Issues

 Changing data volume over time

◦ Rule of thumb : available max-core = (total memory)/(2* # partitions)

 shared among components that allocate max-core in one phase

71
Confidential
 Ab Initio

Rule of Thumb: GO parallel as soon as possible

◦ Processing data serially causes I/O bottleneck

Rule of Thumb: Minimize Serial reads

◦ Reading multiple serial reads simultaneously is faster than reading them

separately

Rule of Thumb: Minimize Database access

◦ Unload DB tables once (if possible) – SQL Loader is slow

◦ If multiple graphs require same database table, unload to a file and replicate.

72
Confidential
 Ab Initio

Purpose:Controlling # simultaneous processes

Performance Enhancements:

◦ CPU Usage:Optimum resource utilization - limit the number of active

components and thus the number of processes.

◦ Memory usage:

 Allocate max-core across components in phase

 Place components which need large max-core in separate phases

 - trade-off: writing to disk at phase boundary

 Sort: Set max-core to 100 MB

 Safety Cushion: DO NOT over allocate memory in a phase

73
Confidential
 Ab Initio

Performance Enhancements

◦ Communication bottleneck : ALL-to-ALL flows

 N-way to N-way: Uses N² network resources

 Limit the number of All-to-All flows by using phases

 Safety cushion: <=4 per phase

 Alternative: Use Two-Stage-Communication

 Uses 2Nx√(N) channels of communication

 applicable if depth is >= 30

74
Confidential
 Ab Initio

Rule of thumb: Placement of phases

◦ DO NOT put phases after Replicates, across all-to-all

flows, after temp files, after sorts

◦ Data on all flows crossing the current phase boundary

is written to disk, so place phases to minimize writing

of data to disks

75
Confidential
 Ab Initio

Purpose:

◦ Provide same functionality as phase

◦ Additional: Provide restart capability

How does it work ?

◦ At job start, output datasets are copied to temporary files (in .WORK dirs)

◦ At checkpoint completion, intermediate datasets and job state are stored in

temporary files

◦ Recovery information is stored in host and vnode directories of /var/abinitio

76
Confidential
 Ab Initio

Rule of thumb : Placement of Checkpoint

◦ Same as phase

Rule of thumb : Minimize the amount of data to be held on temporary storage

Rule of thumb : Recovery files

◦ Should be explicitly deleted/rolled back when a checkpointed graph is aborted

 m_rollback –d graphname.rec

 WARNING: rolling back old .rec files will restore the output files to old state

77
Confidential
 Ab Initio

insert temporary files to check/capture intermediate data

set reject to Never Abort, capture and inspect rejected records

attach Logger component and inspect log messages

advanced: use write_to_log() / validate to debug dml code

advanced: use hash_value() to check partitioning

Use check order component to check sorting

Using m_dump command to view data

78
Confidential
 Ab Initio

79
Confidential
 Ab Initio

Reading the Log : CPU

◦ CPU time: total processing for component

◦ Status: [ Running : Finished ]

◦ Skew: among CPU times of each partition

◦ Vertex: component

Reading the Log : DATA

◦ Data bytes: # processed

◦ Records: # processed

◦ Status: [ unopened : opened : closed ]

◦ Skew: among data bytes in partitions

80
Confidential
 Ab Initio

Reading the Log : DATA

◦ Flow: link between components

 data tracking info is displayed on flows in GDE

◦ Vertex: component

◦ Port: of component

Interpreting the log

◦ Compute data bytes/sec through component, in each partition

◦ Look for serialization: effective CPU = (cpu time)/(elapsed time)

◦ compare open vs. closed partitions:serialized when some partitions remain open long after

others have closed  data skew

◦ Deadlock:no change in record counts over couple of intervals

81
Confidential
 Ab Initio

Avoid Sorts

Use Lookups

Use In-memory Join/Rollup

Assign Driving Port of Join correctly

Allocate memory correctly

Phasing

82
Confidential
 Ab Initio

THANK YOU

83
Confidential