Lookup File

Lookup File represents one or multiple serial files or a multifile of data records small enough to be held in main memory, letting a
transform function retrieve records much more quickly than it could retrieve them if they were stored on disk.
Lookup File associates key values with corresponding data values to index records and retrieve them.

Click the following for:

Ports
Parameters
Details
Shell programming syntax
Example
See also

Ports
None

Parameters
key Required Key specifier

Name(s) of the key field(s) against which Lookup File

matches its arguments.

RecordFormat Required Record format

The record format you want Lookup File to use when

returning data records.

Details
Unlike other datasets, you do not connect Lookup Files to other components.
Although a Lookup File appears in the graph without any connected flows, its contents are accessible from other components in the
same or later phases.
You use the Lookup File in other components by calling one of the DML functions lookup , lookup_count , or lookup_next in any
transform function or expression parameter.
The first argument to these lookup functions is the name of the Lookup File; the remaining arguments are values to be matched
against the fields named by the key parameter. The lookup functions return a record that matches the key values and has the
format given by the RecordFormat parameter.
A file you want to use as a Lookup File must fit into memory. If a file is too large to fit into memory, use Input File followed by
Match Sorted or Join instead.
Information about Lookup Files is stored in a catalog, which allows you to share them with other graphs.
If the key specifier for a Lookup File contains the word interval, interval_bottom, or interval_top, then the Lookup File is an
"interval lookup." The following special rules apply:
 Each record in the Lookup File represents an interval, that is, a range of values. The lower and upper bounds of the range are
usually given in two separate fields of the same type.
A key field marked interval_bottom holds the lower endpoint of the interval
A key field marked interval_top holds the upper endpoint.
For example, in a Lookup File that is an interval lookup, the following DML function returns the record, if any, for which arg is
between the lower and upper endpoints.
lookup("Lookup_File_name",arg)
By default, the interval endpoints are inclusive, but you may add the modifier exclusive to specify otherwise. For example, suppose
Lookup File insurance_coverage has the following key specifier:
"{coverage_start interval_bottom exclusive; coverage_end interval_top}"
This identifies the fields coverage_start and coverage_end as the endpoints of the interval.
The following DML function returns record R if R.coverage_start is less than arg and less than or equal to R.coverage_end.
lookup("insurance_coverage",arg)
Your Lookup File must contain well-formed intervals: for each record, the value of the interval_bottom field(s) must be less than or
equal to the value of the interval_top field(s). The intervals must not overlap and must be sorted into ascending order.
Sometimes it is not convenient to specify both endpoints of an interval in every record. You may specify interval boundaries with
only one key field f by setting the key specifier to {f interval}. In this case, a given record's value for field f is interpreted as the
bottom (inclusive) of the interval for that record. The value in the subsequent record's field f is interpreted as the top (exclusive) of
the interval. The last record in the table is an orphan: it specifies no interval because there is no subsequent record from which to
determine the upper endpoint.
If a field in the Lookup File's key specifier is marked interval, it must be the only key for that Lookup File.
See "Advanced lookup modifiers" for more information.
You can convert an Output File generated in one phase of a graph to a Lookup File used in a later phase. To do this:
1. Create an Output File to contain the data records you want to use as a Lookup File.
2. On the Description tab of the File Properties dialog box for that Output File, check Add to Catalog.

Shell programming syntax

To call the Lookup File component in the SDE, use the m_catalog_add command.
Following is the command syntax:
m_catalog_add -catalog catalog_name -table label ⇒
-key key_specifier -data url ⇒
-metadata .dml_filename
Following is a list of the arguments to the command, with a brief description of each:
-catalog catalog_name Name of the catalog you create with the
m_mkcatalog command (see the Co>Operating
System Administrator's guide for details).

-table label Name you want to assign to this particular

instance of this component. Must be identical to
the first argument to the lookup function.

-key key_specifier See key parameter.

-data url URL of the data file for this component.

-metadata ⇒ Name of the .dml file containing the record

.dml_filename format you want Lookup file to use when
returning records.

Example
This example shows how to use a Lookup File to improve the performance of a graph.
The first part of the example (see the following graph) joins the Visits dataset with the Last Visits dataset to produce the Merged
Visits dataset
In this graph, Match Sorted has a transform function which assigns values from the Last Visits dataset to the lastdate field of the
Merged Visits dataset, as follows:
out :: merge(visits, lastvisits) =
begin
out.id : : visits.id;
out.lastdate :1: lastvisits.dt;
out.lastdate : : 0;
end;
You can improve the performance of the graph in the first part of this example as follows:
If the Last Visits dataset is small enough to fit into memory, use it as a Lookup File and call the lookup function in the Reformat
transform function.

The Reformat component's transform function also sets the lastdate field as follows:
out :: reformat(visits) =
begin
out.id :: visits.id;
out.lastdate :: if (lookup("Last Visits", visits.id))
lookup("Last Visits", visits.id).dt
else
0;
end;

See also
Dataset components
Reformat for further details on the use of the Reformat component
Sort for further details on the use of the Sort component
"Advanced lookup modifiers" .

Copyright © 2002 Ab Initio Software Corporation, Confidential and Proprietary. All rights reserved.