Ab Initio Training

Lookup Files and Reusability Features

Gokulnath J
 Objectives

 Lookup Files
 Other lookup functions
 Using lookup_local()
 Other local lookup functions
 Lookup Modifiers
 Conditional Components
 Sub graphs

Confidential ©2012 Syntel, Inc.

 Lookup File
• Lookup File can make graph processing faster and
What is a DML?
more efficient.
• Unlike other dataset components, Lookup File is not
connected to other components in a graph. However,
its contents are accessible from other components in a
graph.
• A file you want to use as a Lookup File must be small
enough to fit into main memory. If a file is too large to
fit into memory, use Input File followed by Join.
• Information about Lookup File components is stored in
a catalog, and thus you can share the components with
other graphs.

Confidential ©2012 Syntel, Inc.

 Contd…

 DML provides a facility for looking up records in a dataset

based on a key:
record lookup(string file, expression
[, expression ...] )

 The data is read from a file into memory.

 The GDE provides a Lookup File component as a special

dataset with no ports.

Confidential ©2012 Syntel, Inc.

Using lookup instead of Sort/Join

Using Last-Visits
as a lookup file

Confidential ©2012 Syntel, Inc.

 Configuring a Lookup File

1. Label used as name in 3. Set record format

lookup expression

2. Browse for pathname 4. Set key

Confidential ©2012 Syntel, Inc.

 Using lookup in a Transform Function

Input 0 record format: Output record format:

record record
decimal(4) id; decimal(4) id;
string(6) name; string(8) city;
string(8) city; decimal(3) amount;
decimal(3) amount; date(”YYYY/MM/DD”) dt;
end end

Transform function:
out :: lookup_info(in) =
begin
out.id : : in.id;
out.city : : in.city;
out.amount : : in.amount;
out.dt :1 : lookup(”Last-Visits”, in.id).dt;
out.dt :2 : ”1900/01/01”;
end;

Confidential ©2012 Syntel, Inc.

 Other lookup functions

 int lookup_count (string file, expression [,expression...] )

Returns the number of records from Lookup File file that
match the given expression(s). Returns 0 instead of NULL
if no match is found.

 record lookup_next (string file )

Used after lookup_count or after a successful call to lookup,
the function lookup_next returns successive records from
file that match the values of the expression arguments
given in the prior lookup_count or lookup.

Confidential ©2012 Syntel, Inc.

 Other lookup functions…

 lookup_match(string file_label[, expression[, expression ...]])

Performance considerations:

You can use lookup_match to quickly determine whether or not a particular key value is
contained in a lookup file. This function is faster than lookup and lookup_count, and is useful
when you do not care about the matched record or how many matching records there are.
In these circumstances, instead of using the following code construct:

if (is_defined(lookup(file, key)))
statement1
else
statement2

You can use this approach:

if (lookup_match(file, key))
statement1
else
statement2

Confidential ©2012 Syntel, Inc.

 Using lookup_local()

 If lookup files grow large, it may be useful to partition them

as multifiles on their key.

 The function lookup_local() is identical to lookup(), except

that only the local partition will be examined.

 The input data must be partitioned on the same key that is

used for the lookup.

 Usage:
 record lookup_local (string file, expression [,
expression ...] )

Confidential ©2012 Syntel, Inc.

 Pop Quiz

What’s wrong with this picture?

Confidential ©2012 Syntel, Inc.

Example of lookup_local()

Multifile
Multifilepartitioned
partitionedby
byfield
fieldAA
Input
Inputdata
datapartitioned
partitionedby
byfield
fieldAA

lookup_local(“Lookup
lookup_local(“LookupFile”,
File”,in.A)
in.A)

Confidential ©2012 Syntel, Inc.

 Other local lookup functions

 int lookup_local_count (string file, expression

[,expression...] )

 record lookup_local_next (string file )

 These are identical to the non-local versions,

except that they only operate on the local
partition.

Confidential ©2012 Syntel, Inc.

 Lookup Modifiers

 The Ab Initio Lookup File component provides a “search

within a range” capability, also known as “interval lookup.”
Each record in the file specifies a range of values that the
lookup key expressions may take and still be considered to
match that record.

 When a Lookup File component is configured for interval

lookup, the key specifier declares which fields are the
endpoints of the range with a modifier.

Confidential ©2012 Syntel, Inc.

 Lookup Modifiers (Contd…)

 interval_bottom:Marks field as lower endpoint of interval

 interval_top: Marks field as upper endpoint of
interval
 interval: Field holds both lower (inclusive) and
upper (exclusive) values

Modifiers to interval_top and interval_bottom:

 inclusive: Specifies that interval includes field value
(this is the default)
 exclusive: Specifies that the interval does not
include field value

Confidential ©2012 Syntel, Inc.

 Lookup Modifier Example 1

 {income_low interval_bottom inclusive; income_high

interval_top exclusive}

 If the values of income_low and income_high were 50 and 75

respectively, the record would match lookup key
expressions with values from 50 to 74.

Confidential ©2012 Syntel, Inc.

 Lookup Modifier Example 2

 {west_longitude interval}

 If the Lookup File contains four records with the values 75,
90, 105, and 180 in the west_longitude field, then the
intervals corresponding to the first three records are 75..89,
90..104, and 105..179:

Confidential ©2012 Syntel, Inc.

Reusability Features

Confidential ©2012 Syntel, Inc.

 Reusability Features Topics

 Conditional Components

 Subgraphs

 Linked Subgraphs

Confidential ©2012 Syntel, Inc.

 Conditional Components

 The GDE supports Conditional Components where a

shell expression determines, at runtime, whether or
not to include certain components.

 To turn on this feature:

 File -> Preferences -> “Parameters” section of

dialog

 Check “Conditional Components”

 The “Condition” tab appears on all components

Confidential ©2012 Syntel, Inc.

Confidential ©2012 Syntel, Inc.
 The Condition expression is a shell expression that, if it
evaluates to “0” or “false”, will cause the component to not
exist. Any other value means the component will exist.

 Components which are conditioned out can be replaced by

a flow or removed completely. When removing completely,
make sure you don’t leave any required ports unconnected.

 To apply the same condition to more than one component,

make them a subgraph and condition the subgraph.

Confidential ©2012 Syntel, Inc.

 How to condition two components on one parameter, which
chooses one of the two components and disables another:

Component 1 condition:
  $( if [[ $DO_SORT = "1" ]];  then print 0;  else print 1;  fi )

Component 2 condition:
  $( if [[ $DO_SORT = "1" ]];  then print 1;  else print 0;  fi )

Confidential ©2012 Syntel, Inc.

 Sub Graph

 From a subgraph’s Properties -> Parameters tab, you can

export subgraph parameters to graph parameters.

 Parameterized subgraphs can be made into components:

Partition by Key and Sort is a subgraph.

Confidential ©2012 Syntel, Inc.

 Linked Subgraphs

 Linked subgraphs can be used for:

– Writing reusable pieces of library code

– Sharing development of a graph across many developers

– Breaking out pieces of functionality for separate unit testing

Confidential ©2012 Syntel, Inc.

 Creating Linked Subgraphs

 Select the components you wish to make into a component.

 Make into a subgraph:

Edit -> Subgraph -> Create From Selection

 Double-click into subgraph

 Save subgraph:
– File -> Save Subgraph “name” As …
– This must be saved to shared directory
– Usually best to copy *.mp file for subgraph to a directory on the
server, for shared access

Confidential ©2012 Syntel, Inc.

 Linked Subgraphs (cont.)

 To use that subgraph inside another graph:

– choose Insert  Component and browse for your subgraph.

 To get changes from the linked subgraph back into the

main graph, right-click Update on subgraph component.

 The subgraph must not be open while updating (in older

GDE versions)

Confidential ©2012 Syntel, Inc.

 Questions
???????????????

Confidential ©2012 Syntel, Inc.

Thank You!

Confidential ©2012 Syntel, Inc.