Orchestrate 7.

0
Developer Guide

Ascential Software Corporation

Orchestrate 7.0 Developer Guide
Version 3.1 Mode: Print Size: 7x9x11 8/28/03 15:21:22

Orchestrate 7.0 Developer Guide

September 2003
© 2003, 1995-2002 Ascential Software Corporation. All rights reserved. Orchestrate, Ascential, Ascential Software,
DataStage, MetaStage, MetaBroker, and Axielle are trademarks of Ascential Software Corporation or its affiliates
and may be registered in the United States or other jurisdictions. Adobe Acrobat is a trademark of Adobe
Systems, Inc. HP and Tru64 is either a registered trademark or trademark of Hewlett-Packard Company. AIX,
DB2, DB2 Universal Database, IBM, Informix, MQSeries, Red Brick, UniData, UniVerse, and WebSphere are either
registered trademarks or trademarks of IBM Corporation. Microsoft, Windows, Windows NT, and Windows
Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other
countries. Teradata is a registered trademark of NCR International, Inc. Oracle, Oracle8i, and Oracle 9i are either
registered trademarks or trademarks of Oracle Corporation. Solaris, Sun, and Sun Microsystems are either
trademarks or service marks of Sun Microsystems, Inc. Adaptive Server, Open Client, and Sybase are either
registered trademarks or trademarks of Sybase, Inc. Linux is a trademark of Linus Torvalds. WinZip is a
registered trademark of WinZip Company, Inc. UNIX is a registered trademark in the United States and other
countries, licensed exclusively through X/Open Company, Ltd. Other marks mentioned are the property of the
owners of those marks.
Ascential Software Corporation.
50 Washington Street
Westboro, MA 01581-1021
508 366-3888
508 389-8955 FAX
For technical support, send e-mail to: [email protected].
 Orchestrate 7.0 Developer Guide

Contents
Preface xi
About This Guide xii
Compiling Your Operators xii
Sun Solaris xii
HP-UX xii
Compaq Tru-64 xiii
AIX xiii
Linux Redhat xiii
Class and Member Function Descriptions xiii
HTML-Documented Header Files xiii
Text Header Files xiv
The Orchestrate Documentation Set xiv
Typographic Conventions xv
Typographic Formats xv
Cross-References xvi
Syntax Conventions xvi
Using the Adobe Acrobat Reader xvi
Searching for Text in Orchestrate Documents xvii
Assistance and Additional Information xviii

Chapter 1 Creating Operators 1 1

Overview 1 2
Derivation Requirements 1 3
The APT_Operator Class Interface 1 4
APT_Operator Derivation Examples 1 5
APT_Operator Derivation With No Arguments 1 6
APT_Operator Derivation With Arguments 1 10

iii
Contents

Including Additional APT_Operator Member Functions 1 14

Defining Parallel and Sequential Operators 1 15
Creating Parallel Operators 1 16
Creating Sequential Operators 1 17
Creating Operators that can be Either Parallel or Sequential 1 17
Specifying Input and Output Interface Schemas 1 18
Introducing Schema Variables 1 19
Introducing Operators with Dynamic Interface Schemas 1 20
Specifying a Partitioning Method 1 20
Specifying Partitioning Style and Sorting Requirements 1 23
Specifying Partitioning Style 1 23
Specifying Sort Keys 1 24
Specifying a Collection Method 1 25
Using Cursors 1 26
Using Field Accessors 1 29
Field Accessors and Schema Variables 1 32
Field Accessors and Dynamic Operators 1 33
Examples of Operator Derivations With Options 1 33
Including Schema Variables in an Interface Schema 1 34
Working with a Dynamic Interface Schemas 1 48
Customizing Job Monitoring Messages 1 55
Customizing Metadata and Summary Response Messages 1 55
Generating Custom Messages Asynchronously 1 57
Declaring External Data for MetaStage 1 57

Chapter 2 Creating Composite Operators 2 1

Overview 2 1
A Composite Operator with Two Suboperators 2 2
The APT_CompositeOperator Class Interface 2 4
Composite Operator Example Derivation 2 5

Chapter 3 Creating Subprocess Operators 3 1

Overview 3 1
The APT_SubProcessOperator Class Interface 3 2
Deriving from APT_SubProcessOperator 3 3
APT_SubProcessOperator::runSource() 3 3
APT_SubProcessOperator::runSink() 3 4

iv Orchestrate 7.0 Developer Guide

Contents

APT_SubProcessOperator::commandLine() 3 5
Subprocess Operator Example Derivation 3 5

Chapter 4 Localizing Messages 4 1

Introduction 4 2
Using the Messaging Macros 4 2
Including Messaging Macros in Your Source File 4 2
Descriptions of the Message Macros 4 3
Message Environment Variables 4 5
Localizing Message Code 4 5
Using the APT_MSG() Macro 4 5
Example Conversions 4 6
Steps for Converting Your Pre-NLS Messages 4 7
Eliminating Deprecated Interfaces 4 9

Chapter 5 Orchestrate’s Argument-List Processor 5 1

Overview 5 2
The Advantages of Argument-List Processing 5 2
Argument-List Descriptions 5 3
Supplying an Argument-List Description 5 3
Argument-List Description Elements 5 3
Structure of the Orchestrate-Generated Property List 5 4
Argument-List Description Syntax 5 5
Argument-List Description Examples 5 16
Property List Encoding of Operator Arguments 5 19
Example Property List 5 19
Traversing a Property List 5 19
Error Handling 5 20
Argument-Description Syntax Errors 5 20
Command-Line Argument Errors 5 20
Usage Strings 5 21
Converting Your Wrappered Operators 5 22
Converting a Composite Operator 5 23
Registering Your Operator Library 5 25
Executing Your Custom Operator 5 25

Orchestrate 7.0 Developer Guide v

Contents

Chapter 6 Type Conversions 6 1

Default Type Conversions 6 2
Converting Between ustring and string Types 6 2
Non-Default Pre-Defined Type Conversions 6 2
Defining Custom Type-Conversion Functions 6 5
The APT_FieldConversionRegistry Class Interface 6 5
The APT_FieldConversion Class Interface 6 5
Example 1: Using Pre-Defined Type Conversions 6 6
Example 2: Creating New Type Conversions 6 9

Chapter 7 Using Cursors and Accessors 7 1

Overview 7 2
How Record Fields Are Referenced 7 2
Cursors 7 3
The APT_InputCursor and APT_OutputCursor Class Interfaces 7 3
Example of Accessing Records with Cursors 7 5
Field Accessors 7 7
Defining Field Accessors 7 8
Using Field Accessors 7 9

Chapter 8 Creating Partitioners 8 1

Overview 8 1
Setting the Preserve-Partitioning Flag 8 3
Choosing a Partitioning Method 8 3
Using the Default Partitioning Method 8 3
Using a Keyless Partitioning Method 8 4
Using the Class APT_HashPartitioner 8 5
Using The Class APT_RangePartitioner 8 7
The APT_Partitioner Class Interface 8 8
Creating a Custom Partitioner 8 8
Overriding APT_Partitioner::describePartitioner() 8 9
Overriding APT_Partitioner::setupInputs() 8 9
Overriding APT_Partitioner::partitionInput() 8 10
Example Partitioning Method Definition 8 11
Orchestrate Hashing Functions 8 15
Using a View Adapter with a Partitioner 8 16

vi Orchestrate 7.0 Developer Guide

Contents

Chapter 9 Creating Collectors 9 1

Overview 9 1
Choosing a Collection Method 9 2
Using the Default Collection Method 9 3
Using a Keyless Collection Method 9 3
Using The Sorted-Merge Collector 9 4
An APT_SortedMergeCollector Example 9 4
The APT_Collector Class Interface 9 6
Creating a Custom Collector 9 6
How APT_Collector Operates 9 7
Overriding APT_Collector::describeCollector() 9 8
Overriding APT_Collector::setupInputs() 9 9
Overriding APT_Collector::selectInput() 9 10
Example Collector Derivation 9 11
Example Collector with a View Adapter 9 16

Chapter 10 Using Field Lists 10 1

Overview 10 1
Creating a Field List 10 2
The APT_FieldList Class Interface 10 2
Expanding a Field List 10 3
Using Field Lists with Operators 10 4

Chapter 11 Enabling Object Persistence 11 1

Overview 11 2
Simple Persistence and Complex Persistence 11 2
Storing and Loading Simple and Complex Persistent Objects 11 3
Creating Archives 11 3
The APT_Archive Class Interface 11 4
Using the Serialization Operators 11 4
Defining Simple-Persistent Classes 11 7
Defining Complex-Persistent Classes 11 8
The APT_Persistent Class Interface 11 9
Implementing APT_Persistent::serialize() 11 10

Orchestrate 7.0 Developer Guide vii

Contents

Serializing Pointers 11 11
Serializing Arrays 11 11
Persistence Macros 11 13

Chapter 12 Run-Time Type Information 12 1

Overview 12 1
Determining the Run-Time Data Type of an Object 12 2
Performing Casts 12 3
Adding RTTI Support to Derived Classes 12 3
RTTI Macros 12 5
Derivation Macros 12 5
Macro APT_IMPLEMENT_RTTI_ONEBASE() 12 6
Application Macros 12 6

Chapter 13 Debugging Your Applications 13 1

Overview 13 1
Choosing a Debugger 13 2
Specifying Program Execution Mode 13 2
Modifying the Configuration File 13 3
Debugging With Third Degree on Tru64 13 4
Debugger Options with dbx 13 5
Debugging in Sequential Mode 13 5
Debugging in Parallel Mode 13 9
Profiling Your Application 13 11
The Score Dump 13 12

Chapter 14 Using the Error Log 14 1

Overview 14 1
The APT_ErrorLog Class Interface 14 2
Base Class Error Log Functions 14 3
The Error Log Compared to the Assertion Failures Facility 14 3
Using the Error Log 14 4
Error Log Structure 14 4
Defining Message Identifiers 14 5

viii Orchestrate 7.0 Developer Guide

Contents

Using the Error Log in a Derived Operator 14 6

Chapter 15 Advanced Features 15 1

Generic Functions, Accessors, and Dynamic Schema 15 1
Unit-of-Work Batch Processing 15 9
Defining Units-of-Work 15 10
Resetting Operator State 15 11
Per-Partition Processing 15 11
Combinable Operators 15 11
When to Use Combinable Operators 15 11
Using Combinable Operators 15 12
Methods Used by Combinable Operators 15 13
The Combinable Controller 15 17
Limitations and Risks 15 17
Data Set Buffering 15 18

Index Index 1

Orchestrate 7.0 Developer Guide ix

Contents

x Orchestrate 7.0 Developer Guide

List of Tables
Table 1 Typographic Formats xv
Table 2 APT_Operator Derivation With No Arguments 1 6
Table 3 APT_Operator Derivation With Arguments: Header File 1 10
Table 4 APT_Operator Derivation With Arguments: .C File 1 11
Table 5 Override of APT_Operator::runlocally() 1 28
Table 6 Override of APT_Operator::describeOperator() 1 30
Table 7 Override of APT_Operator::runLocally() 1 31
Table 8 Override of describeOperator() With a Transfer 1 37
Table 9 Override of runLocally() With a Transfer 1 38
Table 10 Example of Using Schema Variables in a Filter Operator 1 40
Table 11 Example of a Two-Input Operator 1 44
Table 12 Example of an Operator with a Dynamic Interface 1 52
Table 13 Metadata for Orchestrate Source/Sink Operators 1 59
Table 14 APT_CompositeOperator Derivation() 2 5
Table 15 APT_SubProcessorOperator Derivation() 3 6
Table 16 Argument Description Syntax 5 6
Table 17 Argument Description Syntax Comments 5 11
Table 18 Example Operator with Pre-Defined Type-Conversion 6 6
Table 19 Example Operator with Custom Type Conversion 6 9
Table 20 Accessing Records with Cursors in runLocally() 7 5
Table 21 Field Accessor Classes 7 8
Table 22 Accessors to Numeric Data Types in describeOperator() 7 11
Table 23 Creating Accessors in runLocally() 7 11
Table 24 Accessors to Fixed-Length Vector Fields in runLocally() 7 14
Table 25 Accessors to Variable-Length Vector Fields in runLocally() 7 16
Table 26 Nullable Fields in describeOperator() Schema 7 18
Table 27 Handling Nullable Fields in runLocally() 7 20
Table 28 Handling String and Ustring Fields in runLocally() 7 23

xi
List of Tables

Table 29 Handling Raw Fields in runLocally() 7 28

Table 30 Date, Time, and Timestamp Type Field Accessor Classes 7 29
Table 31 Handling a Date Field in runLocally() 7 30
Table 32 Handling Aggregate Fields in runLocally() 7 32
Table 33 Handling Subrecord Vector Fields in runLocally() 7 35
Table 34 Partitioner Interface Schema in describeOperator() 8 6
Table 35 Partitioner Derivation 8 11
Table 36 Example APT_Collector Derivation 9 11
Table 37 Using Field Lists in a Derived Operator 10 5
Table 38 Override of APT_Persistent::serialize with Arrays 11 12
Table 39 APT_DUMP_SCORE Report 13 13
Table 40 Using the Error Log in a Derived Operator 14 6
Table 41 types.C 15 2
Table 42 types.h 15 9

xii
 Orchestrate 7.0 Developer Guide

Preface
Describes this Guide and the conventions it uses, and tells you how to find class
information in header files, how to do text searches in Orchestrate documents, and
how to contact customer support.
About This Guide xii
Compiling Your Operators xii
Sun Solaris xii
HP-UX xii
Compaq Tru-64 xiii
AIX xiii
Linux Redhat xiii
Class and Member Function Descriptions xiii
HTML-Documented Header Files xiii
Text Header Files xiv
The Orchestrate Documentation Set xiv
Typographic Conventions xv
Typographic Formats xv
Cross-References xvi
Syntax Conventions xvi
Using the Adobe Acrobat Reader xvi
Searching for Text in Orchestrate Documents xvii
Assistance and Additional Information xviii

xi
Preface Orchestrate 7.0 Developer Guide

About This Guide

This is the Orchestrate 7.0 Developer Guide. It describes how you can extend
Orchestrate’s application development libraries of predefined operators by
deriving your own component classes from the Orchestrate C++ class library. You
can then use your derived operators within an Orchestrate application.
You can derive three types of classes from the Orchestrate C++ class library:
• General Operators: You create operators to execute custom application logic
as part of an Orchestrate step. Your operators can execute in parallel or
execute sequentially.
• Partitioner Operators: You create partitioner operators to implement specific
partitioning algorithms. The derived partitioners can then be used by parallel
operators.
• Collector Operators: You create collector operators to implement specific
collection methods. A collection method defines how the partitions of a data
set are combined for input to a sequential operator.
While you derive operators from the Orchestrate C++ library, your application
must be developed in Orchestrate osh data-flow notation using the Orchestrate
Shell Interface described in the Orchestrate 7.0 User Guide.

Compiling Your Operators

Use the compiler, the debugger, and the compiler and linker options listed for
your platform:

Sun Solaris
Use the Sun Pro C++ compiler and the dbx debugger.
Use these compiler options:
-library=iostream -dalign -g -I$(APT_ORCHHOME)/include

Use these linker options:

-library=iostream -dalign -L$(APT_ORCHHOME)/lib -lorchsun4
-lorchcoresun4 -lrwtool -lsocket -lnsl -ldl -laio

HP-UX
Use the HP ANSI C++ A3.33 or A3.37 compiler and the gdb debugger.
Use these compiler options:
-g -I$(APT_ORCHHOME)/include

xii Orchestrate 7.0 Developer Guide

Preface

Use these linker options:

-L$(APT_ORCHHOME)/lib -lorchhpux -lorchcorehpux -lrwtool -lm

Compaq Tru-64
Use the Compaq C++ 6.2, 6.3, or 6.5 compiler, and the ladebug debugger.
Use these compiler options:
-stdnew -ieee -nopt -g -I$(APT_ORCHHOME)/include

Use these linker options:

-L$(APT_ORCHHOME)/lib -lorchOSF1 -lorchcoreOSF1 -lrwtool

AIX
Use IBM VisualAge(R) C++ Professional for AIX, version 5.0.20 or 6.0, and the
dbx debugger.

Use these compiler options:

-g -I$(APT_ORCHHOME)/include

Use these linker options:

-L$(APT_ORCHHOME)/lib -lorchsortaix3 -lorchgeneralaix3 -lorchaix3
-lorchcoreaix3 -lm
-lsorti686 or -lorchegenrali686

Linux Redhat
Use the gcc/g++ 2.96 compiler, and the dbx debugger.
Use these compiler options:
-g -fPIC -I$(APT_ORCHHOME)/include

Use these linker options:

-L$(APT_ORCHHOME)/lib -lorchi686 -lorchmonitori686 -lorchcorei686
-lrwtool -lsocket -lnsl -ldl -laio

Class and Member Function Descriptions

HTML-Documented Header Files
You can use HTML-documented header files to browse the Orchestrate class
hierarchy. These hyperlinked javadoc header files have been generated using the
Doxygen documentation system.

Orchestrate 7.0 Developer Guide xiii

Preface Orchestrate 7.0 Developer Guide

Use the index.html file in $APT_ORCHHOME/doc/html/ to begin your on-line

browsing.

Text Header Files

The text header files are in the $APT_ORCHHOME/include/ subdirectories of your
Orchestrate installation.
Use these quick reference cards to locate the header file for a class or macro.
• Orchestrate 7.0 C++ Classes and Macros Sorted by Name
This card maps class and macro names to their header filepaths located under
orch_master/include.

For example, the class APT_Operator maps to:

$APT_ORCHHOME/include/apt_framework/operator.h

and the macro APT_DECLARE_PERSISTENT() maps to:

$APT_ORCHHOME/include/apt_util/persist.h.

• Orchestrate 7.0 C++ Classes and Macros Sorted by Header File

This card maps header files to their declared classes and defined macros.
For example, $APT_ORCHHOME/include/apt_framework/operator.h maps to:
APT_Operator
and $APT_ORCHHOME/include/apt_util/persist.h maps to the macros
pertaining to persistence.

The Orchestrate Documentation Set

These pdf documents are located in $APT_ORCHHOME/doc/pdf/.
• Orchestrate 7.0 User Guide
• Orchestrate 7.0 Operators Reference
• Orchestrate 7.0 Installation and Administration Manual
• Orchestrate 7.0 Developer Guide
• Orchestrate 7.0 WebHouse User Guide
• Orchestrate 7.0 Record Schema
• Orchestrate 7.0 C++ Classes and Macros Sorted by Header File
• Orchestrate 7.0 C++ Classes and Macros Sorted by Name

xiv Orchestrate 7.0 Developer Guide

Preface

Typographic Conventions
Typographic Formats
Table 1 Typographic Formats

Format Term Examples

bold serif Orchestrate-defined terms APT_Operator

including class and function runLocally()
names, data types, and default APT_Operator::runLocally()
field names. cluster_predicted
uint64

Orchestrate-defined macros APT_DECLARE_PERSISTENT

Orchestrate-defined operators, The clusterquality operator takes the

operator options, and collection results and flatten options.
and partition methods
Use the any or round robin collection
method.

bold italic serif Orchestrate technical terms In pipeline parallelism, each operation
within the text that defines them runs when it has input data available.

sans serif Orchestrate menus Tools > Check Config

fixed width C++ code return APT_StatusOk;

operating-system commands $ cd $APT_ORCHHOME/bin

$ rm aptserv2

osh commands $ osh "clusterquality < kmeans.ds”

non-Orchestrate-defined C++ sprintf()

functions

italic user-defined classes, functions, MyOperator

and field names selectState()
MyOperator::selectState()
Price

Orchestrate variables Specify an insert_statement

Orchestrate 7.0 Developer Guide xv

Preface Orchestrate 7.0 Developer Guide

Cross-References
Most cross-references indicate sections located in this book. They are hotspots and
appear in blue typeface in the online version of the document. When there are
references to other books in the documentation set, their names appear in italics.

Syntax Conventions
Operator syntax is presented as you would enter it as part of an osh command.
For a description of the general syntax of an osh command, refer to the Orchestrate
7.0 User Guide.
The following syntax conventions are used throughout this book:
• A vertical bar (|) separates mutually exclusive alternatives.
• Braces ({ }) are used with vertical bars (|) to indicate that one of several
mutually exclusive alternatives are required, for example {a | b} indicates that
a or b is required.
• If one or more items are enclosed in braces ({ }) and separated by commas the
items are synonymous. Thus {a , b} indicates that a and b have exactly the
same effect.
• Brackets ([ ])indicate that the item(s) inside are optional. For example, [a | b]
indicates a, or b, or neither.
• Ellipsis (...) indicates that the preceding item occurs zero or more times. If a
user-provided string is indicated, it may represent a different item in each
occurrence. For example:
– [-keyfield ...] means zero or more occurrences of -key field, where field
may be a different string in each occurrence.
– To indicate one or more occurrences of an item, the item occurs first
without brackets and ellipsis and then with, for example
-key field [-key field ...]

Using the Adobe Acrobat Reader

To read the document set online, you need the Adobe Acrobat Reader. If you
already have Adobe Acrobat Reader installed, make sure it is version 4.05 or later
and is the version of Reader with Search. If your version of Reader does not have
Search, you will not be able to search across all the documents in the Orchestrate
set.

xvi Orchestrate 7.0 Developer Guide

Preface

To see if your copy of Reader has Search, look for the Search icon: on the
Reader toolbar, and make sure it is present and not dimmed. The Search icon
should be located alongside the somewhat similar Find icon: .
If you do not have the appropriate version of Acrobat installed, you may use the
Acrobat Reader 4.05 included with Orchestrate. Find the version you need in one
of the following platform-specific directories:
$APT_ORCHHOME/etc/acroread-sun-405.tar.gz
$APT_ORCHHOME/etc/acroread-aix-405.tar.gz
$APT_ORCHHOME/etc/acroread-osf1-405.tar.gz
$APT_ORCHHOME/etc/acroread-hpux-405.tar.gz

Use the UNIX gunzip and tar commands to unpack the files. Then cd to the
directory *.install (where * contains an abbreviation of your platform name)
and follow the instructions in the INSTGUID.TXT file.
The Orchestrate online documentation set and the full-text search index is located
in $APT_ORCHHOME/doc.

Searching for Text in Orchestrate Documents

You can find specific words or phrases in this Guide and across all Orchestrate
online documents using the Adobe Acrobat Reader.

! To find text in a single document:

1 Open the document in the Adobe Acrobat Reader.
2 Choose Edit > Find
or
Click Find on the Adobe Toolbar
3 Enter the text you want to find in the Find What field and click Find.
4 Use Ctrl+G to find the next occurrence of the text.

! To find text across all documents:

1 Choose Edit > Search > Query
or
Click Search on the Acrobat toolbar
2 In the Adobe Acrobat Search window, type the text you want to find in the Find
Results Containing Text box, then click Search.

a If the following message appears at the bottom of the Adobe Acrobat Search
window:
No selected indexes are available for search

Click Indexes... to bring up the Index Selection window.

Orchestrate 7.0 Developer Guide xvii

Preface Orchestrate 7.0 Developer Guide

b Click Add, then navigate the Add Index window to find the full-text search
index, Orchestrate.pdx, located in $APT_ORCHHOME/doc.
c Select the index, then click OK on the Index Selection window.
3 The Search Results window lists the documents that contain your search text,
ordered according to the number of search hits.
4 Select a book in the list.
5 Use the Previous Highlight and Next Highlight buttons to move to the
previous or next instances of your search text.

Assistance and Additional Information

If you require assistance or have questions about Orchestrate, you can contact
Ascential Customer Support by:
• Calling (866) INFONOW in North America or your Regional Support Center
• Writing [email protected] for any Ascential Software product
or [email protected] for Orchestrate-specific help.
• Logging onto the Ascential Support e.Service Web site at:
www.ascential.com:8080/eservice/index.html
For current information about Ascential and its products log onto:
http://www.ascential.com/

xviii Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

1
Creating Operators
Describes how to derive your own custom operators from Orchestrate’s
APT_Operator class.

Overview 1 2
Derivation Requirements 1 3
The APT_Operator Class Interface 1 4
APT_Operator Derivation Examples 1 5
APT_Operator Derivation With No Arguments 1 5
APT_Operator Derivation With Arguments 1 9
Including Additional APT_Operator Member Functions 1 14
Defining Parallel and Sequential Operators 1 15
Creating Parallel Operators 1 16
Creating Sequential Operators 1 16
Creating Operators that can be Either Parallel or Sequential 1 17
Specifying Input and Output Interface Schemas 1 17
Introducing Schema Variables 1 19
Introducing Operators with Dynamic Interface Schemas 1 19
Specifying a Partitioning Method 1 20
Specifying Partitioning Style and Sorting Requirements 1 23
Specifying Partitioning Style 1 23
Specifying Sort Keys 1 24
Specifying a Collection Method 1 25
Using Cursors 1 26
Using Field Accessors 1 29
Field Accessors and Schema Variables 1 32
Field Accessors and Dynamic Operators 1 33
Examples of Operator Derivations With Options 1 33

11
Chapter 1 Creating Operators Overview

Including Schema Variables in an Interface Schema 1 34

Background 1 34
The Transfer Mechanism 1 35
Performing Transfers 1 37
Example: Using Schema Variables in a Filter Operator 1 39
Example: Creating a Two-Input Operator 1 43
Working with a Dynamic Interface Schemas 1 48
Using Multifield Accessors 1 49
Example: Creating an Operator with a Dynamic Interface 1 52
Customizing Job Monitoring Messages 1 55
Customizing Metadata and Summary Response Messages 1 55
Generating Custom Messages Asynchronously 1 57
Declaring External Data for MetaStage 1 57

Overview
Operators are the basic functional units of an Orchestrate application. Operators
read records from input data sets, perform actions on the input records, and write
results to output data sets. An operator may perform an action as simple as
copying records from an input data set to an output data set without
modification. Alternatively, an operator may modify a record by adding,
removing, or modifying fields during execution.
You can use predefined operators in your application. Orchestrate supplies a
number of such operators, such as the copy operator (APT_CopyOperator) and
the sample operator (APT_SampleOperator) The Orchestrate 7.0 Operators
Reference describes Orchestrate’s predefined operators.
You can also define your own operators. One type of operator that you can create
wraps a standard UNIX command for execution within an Orchestrate step. This
facility is documented in the Orchestrate 7.0 User Guide.
You can also derive classes from one of three Orchestrate base classes:
APT_Operator, APT_CompositeOperator, and APT_SubProcessOperator.
These correspond to the three types of operators you can define:
• Operators
• Composite operators, which contain more than one operator
• Subprocess operators, which allow you to incorporate a third-party
executable into the framework

12 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Derivation Requirements

This chapter describes the steps required to derive new operators from
APT_Operator. You follow much the same process for operators derived from
APT_CompositeOperator and APT_SubProcessOperator. See Chapter 2,
Creating Composite Operators, and Chapter 3, Creating Subprocess Operators,
for information on these operators. This chapter also provides examples of
variations in the creation process, such as using schema variables or a dynamic
interface schema.

Derivation Requirements
Deriving an operator from the base class APT_Operator requires the following:
• Using the public and private interface of APT_Operator.
• Including support for Run-Time Type Information (RTTI) and persistence.
• Enabling Orchestrate’s argc/argv command-line argument checking facility
by providing a description of the arguments expected by your operator to the
APT_DEFINE_OSH_NAME macro, and overriding the virtual function
initializeFromArgs_() to make your operator osh aware. Orchestrate’s
argument-checking facility is described in Chapter 5, Orchestrate’s
Argument-List Processor,.
• Overriding two pure virtual functions, describeOperator() and runLocally(),
and, within those functions, specifying the interface schema that defines the
number and type of data sets that may be used as input or output and how the
records should be processed.
In addition, you can optionally specify:
– The execution mode for the operator: parallel or sequential
– The partitioning method for parallel operators
– The collection method for sequential operators
– The use of cursors, which define the current input or output record of a
data set.
– The use of field accessors, which provide access to any field within a
record; or schema variables, which simplify access to an entire input or
output record.
– Data transfers
• Including support for detecting error and warning conditions and for relaying
that information to users.
• Compiling your operator code, and including its shared library in the libraries
optionally defined by your OSH_PRELOAD_LIBS environment variable.
• Running your operator.

Orchestrate 7.0 Developer Guide 13

Chapter 1 Creating Operators The APT_Operator Class Interface

The APT_Operator Class Interface

The header file for this class, $APT_ORCHHOME/incude/apt_framework/operator.h,
contains descriptions of the functions. You can also browse the Orchestrate class
hierarchy using HTML-documented header files. Start your search with the
index.html file in $APT_ORCHHOME/doc/html.

The next figure below the interface to APT_Operator. You use the member
functions to attach data sets to the operator as well as to attach view and transfer
adapters.

APT_Operator abstract base class interface

public: public: (continued) protected: (continued)
accessArgv() setAvailableNodes() inputAdaptedSchema()
addCustomMetadata() setIdent() inputConcreteSchema()
addCustomSummary() setNodeMap() inputDataSets()
addNodeConstraint() setOutputAdapter() inputInterfaceSchema()
addResourceConstraint() setOutputInterfaceSchema() isTransferBufferFixedlength()
attachInput() setPartitionMethod() outputDataSets()
attachOutput() setRequestKind() outputInterfaceSchema()
availableNodes() setRuntimeArgs() partitionCount()
checkpointStateHandling() setTransferAdapter() partitionNumber()
currentOperator() setupInputCursor() postFinalRunLocally()
getTransferBufferSize() setupOutputCursor() postRun()
ident() setViewAdapter() preRun()
initializationArgs() transfer() reportError()
initializeFromArgs_() = 0 transferAdapter() reportInputAttachment()
input() transferAndPutRecord() reportOutputAttachment()
inputAdapter() transferFromBuffer() reportWarning()
isRequestededKindSet() transferToBuffer() runLocally() = 0
kind() viewAdapter() setCheckpointStateHan-
lookupAndInitializeFromArgv() dling()
lookupCollectionMethod() protected: setCollectionMethod()
lookupPartitionMethod() checkConfig() setInputDataSets()
modifyAdapter() checkWasSuccessful() setInputInterfaceSchema()
nodeMap() clearPreservePartitioning- setKind()
output() Flag() setOutputDataSets()
outputAdapter() declareTransfer() setOutputInterfaceSchema()
requestedKind() describeOperator() = 0 setPreservePartitioningFlag()
sendCustomInstanceReport() errorLog() setRuntimeArgs()
sendCustomReport() initializeFromArgs_()

The two pure virtual functions, describeOperator() and runLocally(), and the
virtual function initializeFromArgs_() are included in the protected interface.
You must override these three functions.
The runLocally() function is invoked in parallel for each instance of the operator.
After runLocally() has been called, the postFinalRunLocally() function is
invoked for each dataset partition. The default implementation of both functions
is a no op.

14 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators APT_Operator Derivation Examples

When unit-of-work processing is done, the runLocally() function may be called

multiple times. In this case, the postFinalRunLocally() function is called after the
last invocation of runLocally(). In a checkpointed configuration,
postFinalRunLocally() is called once for each segment, at the end of the segment.

APT_Operator Derivation Examples

This section contains two APT_Operator derivation examples:
• ExampleOperator. This operator takes no arguments and has a single dataset as
input. It changes the contents of its two record fields and writes each record to
a single output dataset. The code, which represents a single .C file, is shown in
Table 2.
• HelloWorldOp. This operator has two arguments which determine how many
times the string “hello world” should be printed and whether it should be
printed in upper or lowercase. It takes a single dataset as input and simply
copies it to output. The code is shown in two tables: Table 3 has the code from
the header file, and Table 4 has the code from the .C file.

APT_Operator Derivation With No Arguments

This figure illustrates the interface schema for the ExampleOperator operator:

input data set

iField:int32; sField:string;

ExampleOperator

iField:int32; sField:string;

output data set

Here is the definition for ExampleOperator. The operator simply changes the
contents of its two record fields and then writes each record to a single output
dataset. Comments follow the code.

Table 2 APT_Operator Derivation With No Arguments

Code

1 #include <apt_framework/orchestrate.h>

2 class ExampleOperator : public APT_Operator

{
4 APT_DECLARE_RTTI(ExampleOperator);

Orchestrate 7.0 Developer Guide 15

Chapter 1 Creating Operators APT_Operator Derivation Examples

Table 2 APT_Operator Derivation With No Arguments (continued)

5 APT_DECLARE_PERSISTENT(ExampleOperator);

public:
ExampleOperator();

protected:
9 virtual APT_Status describeOperator();
virtual APT_Status runLocally();
11 virtual APT_Status initializeFromArgs_(const APT_PropertyList &args,
APT_Operator::InitializeContext context);

12 #define ARGS_DESC "{. . .}”

13 APT_DEFINE_OSH_NAME(ExampleOperator, exampleOp, ARGS_DESC);
14 APT_IMPLEMENT_RTTI_ONEBASE(ExampleOperator, APT_Operator);
15 APT_IMPLEMENT_PERSISTENT(ExampleOperator);

16 ExampleOperator::ExampleOperator()
{}

18 APT_Status ExampleOperator::initializeFromArgs_(const APT_PropertyList

&args,
APT_Operator::InitializeContext context)
{
return APT_StatusOk;
}

22 void ExampleOperator::serialize(APT_Archive& archive, APT_UInt8)

{}

24 APT_Status ExampleOperator::describeOperator()
{
setKind(APT_Operator::eParallel);
setInputDataSets(1);
setOutputDataSets(1);
setInputInterfaceSchema("record (iField:int32; sField:string)”, 0);
setOutputInterfaceSchema(“record (iField:int32; sField:string)”, 0);
return APT_StatusOk;
}

33 APT_Status ExampleOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;
setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);
APT_InputAccessorToInt32 iFieldInAcc(“iField”, &inCur);
APT_InputAccessorToString sFieldInAcc(“sField”, &inCur);
APT_OutputAccessorToInt32 iFieldOutAcc(“iField”, &outCur);

16 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators APT_Operator Derivation Examples

Table 2 APT_Operator Derivation With No Arguments (continued)

APT_OutputAccessorToString sFieldOutAcc(“sField”, &outCur);

while (inCur.getRecord())
{
cout << “*sFieldInAcc = << *sFieldInAcc << endl;
cout << “*iFieldInAcc = “ << *iFieldInAcc << endl;
*sFieldOutAcc = “XXXXX” + *sFieldInAcc;
*iFieldOutAcc = *iFieldInAcc + 100;
cout << “*FieldOutAcc = << *iFieldOutAcc << endl;
outCur.putRecord();
};

Return APT_StatusOK;
}

Comments

1 Include the Orchestrate header file.

2 All operators are derived, directly or indirectly, from APT_Operator.

4 Using the required macro, APT_DECLARE_RTTI, you declare run-time type

information for the operator. This facility is described in Chapter 12, Run-Time Type
Information,.

5 With the APT_DECLARE_PERSISTENT macro, which is required, you declare

object persistence for operator objects transmitted from Orchestrate to the
processing nodes. Chapter 11, Enabling Object Persistence, describes the persistence
mechanism.

9-11 You must override the virtual function initializeFromArgs_ and the two pure
virtual functions, describeOperator(), and runLocally(). Overrides are in this
example.

12 See “Argument-List Descriptions” on page 5-3 and the header file,

$APT_ORCHHOME/include/apt_util/argvcheck.h, for documentation on the
ARGS_DESC string.

13 With APT_DEFINE_OSH_NAME, you connect the class name to the name used to
invoke the operator from osh and pass your argument description string to
Orchestrate. Orchestrate’s argument-checking facility is described in Chapter 5,
Orchestrate’s Argument-List Processor,. Also see
$APT_ORCHHOME/include/apt_framework/osh_name.h for documentation on this
macro.

Orchestrate 7.0 Developer Guide 17

Chapter 1 Creating Operators APT_Operator Derivation Examples

Table 2 APT_Operator Derivation With No Arguments (continued)

14-15 APT_IMPLEMENT_RTTI_ONEBASE and APT_IMPLEMENT_PERSISTENT are

required macros that implement run-time type information and persistent object
processing.

16 ExampleOperator::ExampleOperator() defines a default constructor for the

operator. This is necessary for the operator to work. Orchestrate requires that all
operators have a public default constructor, even if it is empty.

18 With your override of initializeFromArgs_(), you transfer information from the

operator's arguments to the class instance, making it osh aware. Because there are
no arguments for this example, initializeFromArgs_() simply returns
APT_StatusOk. See “Property List Encoding of Operator Arguments” on page 5-19
and the header file, $APT_ORCHHOME/include/apt_framework/operator.h for
documentation on this function.

22 The function serialize() defines complex persistence in a derived class. The

serialization operators operator||, operator<<, and operator>> are declared and
defined by macros in terms of APT_Persistent::serialize().
There are no data variables in ExampleOperator; therefore serialize() is empty. When
there are data variables, as in HelloWorldOp in Table 3, each variable has it own line.
For example: archive2 || member_variable5.

24 Your override of describeOperator() describes the configuration information for the

operator. This example implementation specifies:
• That the operator be run in parallel. Parallel mode is the default; therefore it is
not necessary to explicitly set parallel mode with setKind().
• That there is one input dataset and one output dataset
• That the input and output schemas are as defined by setInputInterfaceSchema()
and setOutputInterfaceSchema()
• That the data partitioning method be random

18 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators APT_Operator Derivation Examples

Table 2 APT_Operator Derivation With No Arguments (continued)

33 With your override of runLocally(), you define what the operator does at run time.
In this example implementation, the lines APT_InputCursor inCur and
APT_OutputCursor outCur declare the input and out cursors; and the functions
setupInputCursor() and setupOutputCursor() initialize them.
The APT_InputAccessorTotype() and APT_OutputAccessorTotype() functions set
up the input and output field accessors.
The example while statement loops over all the records, doing the following for each
record:
• Prints out the initial contents of the two fields in the record
• Changes the contents of the fields and writes the new contents to the output
record
• Prints out the new contents of the fields
• Writes the record to output

APT_Operator Derivation With Arguments

This figure illustrates the interface schema for the HelloWorldOp operator:

input data set

inRec:*;

HelloWorldOp

outRec*;

output data set

The definition for HelloWorldOp follows. Table 3 contains the code for the header
file, and Table 4 has the code for the .C file. The operator takes two arguments
which determine how many times the string “hello world” should be printed and
whether it should be printed in upper or lowercase. The operator simply copies
its input to output. Comments follow the code in the tables.

Orchestrate 7.0 Developer Guide 19

Chapter 1 Creating Operators APT_Operator Derivation Examples

Table 3 APT_Operator Derivation With Arguments: Header File

Code

1 #include <apt_framework/orchestrate.h>

2 class HelloWorldOp : public APT_Operator

{
4 APT_DECLARE_PERSISTENT(HelloWorldOp);
5 APT_DECLARE_RTTI(HelloWorldOp);

public:
7 HelloWorldOp();
8 void setNumTimes(APT_Int32 numTimes);
9 void setUppercase(bool uppercase);

protected:
11 virtual APT_Status initializeFromArgs_(const APT_PropertyList &args,
APT_Operator::InitializeContext context);
12 virtual APT_Status describeOperator();
13 virtual APT_Status runLocally();

private:
15 APT_Int32 runTimes_;
16 bool uppercase_;
};

Comments

1 Include the Orchestrate header file.

2 All operators are derived, directly or indirectly, from APT_Operator.

4-5 Declare run-time type information and persistence.

7 Declare the constructor for this class.

8-9 Declare the C++ initialization methods for this operator. These methods are called
from initializeFromArgs_().

11 -13 Declare the three virtual functions that must be overridden. The function
initializeFrom Args_() is the osh initialization function which makes the operator osh
aware. The function describeOperator() specifies the pre-parallel initialization steps,
and runLocally() specifies the parallel execution steps.

15 -16 Declare the two member variables.

1 10 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators APT_Operator Derivation Examples

Table 4 APT_Operator Derivation With Arguments: .C File

Code

1 #include “hello.h”
2 #include <apt_framework/orchestrate.h>

3 #define HELLO_ARGS_DESC
“{uppercase={optional, description='capitalize or not'},
numtimes={value={type={int, min=1, max=10}, usageName='times'},
optional, description='number of times to print message'}}”

4 APT_DEFINE_OSH_NAME(HelloWorldOp, hello, HELLO_ARGS_DESC);

5 APT_IMPLEMENT_RTTI_ONEBASE(HelloWorldOp, APT_Operator);
6 APT_IMPLEMENT_PERSISTENT(HelloWorldOp);

7 HelloWorldOp::HelloWorldOp()
: numTimes_(1),
uppercase_(false)
{}

11 void HelloWorldOp::setNumTimes(APT_Int32 numTimes)

{
numTimes_ = numTimes;
}

15 void HelloWorldOp::setUpperCase(bool uppercase)

{
uppercase_ = uppercase;
18 }

19 APT_Status HelloWorldOp::initializeFromArgs_(const APT_PropertyList &args,

APT_Operator::InitializeContext context)
{
APT_Status status=APT_StatusOk;
if (context == APT_Operator::eRun) return status;

for (int i = 0; i < args.count(); i++)

{
const APT_Property& prop = args[i];
if (prop.name() == “numtimes”)
numTimes_ = (int) prop.valueList().getProperty(“value”, 0).
valueDFloat();
else if (prop.name() == “uppercase”)
uppercase_ = true;
};
return status;
}

Orchestrate 7.0 Developer Guide 1 11

Chapter 1 Creating Operators APT_Operator Derivation Examples

Table 4 APT_Operator Derivation With Arguments: .C File (continued)

33 void HelloWorldOp::serialize(APT_Archive& archive, APT_UInt8)

{
archive || numTimes_;
archive || uppercase_;
}

38 APT_Status HelloWorldOp::describeOperator()
{
setKind(eParallel);
setInputDataSets(1);
setOutputDataSets(1);
setInputInterfaceSchema(“record (in:*)”, 0);
setOutputInterfaceSchema(“record (out:*)”, 0);
declareTransfer(“in”, “out”, 0, 0);
setCheckpointStateHandling(eNoState);
return APT_StatusOk;
}

49 APT_Status HelloWorldOp::runLocally()
{
APT_Status status = APT_StatusOk,
APT_InputCursor inCur;
setupInputCursor(&inCur, 0);

APT_OutputCursor outCur;
setupOutputCursor(&outCur, 0);
while(inCur.getRecord() && status == APT_StatusOk)
{
transfer(0);
outCur.putRecord();
int i;
for (i=0; i,numTimes_; i++)
{
if (uppercase_)
cout << “hello, Word” << endl;
else
cout << “hello, word” << endl;
}
return status;
}

Comments

1 hello.h is the header file for this example operator. It is shown in Table 3.

2 Include the Orchestrate header file.

1 12 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators APT_Operator Derivation Examples

Table 4 APT_Operator Derivation With Arguments: .C File (continued)

3 A formal description of the operator arguments are given in the

HELLO_ARGS_DESC string. This enables the Orchestrate argv checking facility for
an operator. See “Argument-List Descriptions” on page 5-3 and the header file,
$APT_ORCHHOME/include/apt_util/argvcheck.h, for documentation on this string.

4 With APT_DEFINE_OSH_NAME, you connect the class name to the name used to
invoke the operator from osh and pass your argument description string to
Orchestrate. Orchestrate’s argument-checking facility is fully described in Chapter 5,
Orchestrate’s Argument-List Processor,. Also see
$APT_ORCHHOME/include/apt_framework/osh_name.h for documentation on this macro.

5-6 APT_IMPLEMENT_RTTI_ONEBASE and APT_IMPLEMENT_PERSISTENT are

required macros that implement run-time information and persistent object
processing.

7 HelloWorldOp::HelloWordOp() defines a constructor for the operator. This is

necessary for the operator to work. Orchestrate requires that all operators have a
public default constructor, even if it is empty.

11 - The functions setNumTimes() and setUpperCase() are the initialization methods for this
18 operator. The method setNumTimes() sets the number of times to print the string; and
the method setUppercase() sets whether to print in uppercase letters.

19 With your override of initializeFromArgs_(), you can traverse the property list
Orchestrate creates from your argument description string and transfer information
from the operator’s arguments to the class instance, making it osh aware.

33 Your override of APT_Persistent::serialize() defines complex persistence in a derived

class. The method is called before the operator is parallelized to archive the values of
its member variables, and it is called again after parallelization to restore those
variables from the archive in each parallel copy of the operator.

38 Your override of describeOperator() describes the configuration information for the

operator such as whether it should run sequentially or in parallel, how many input
and output datasets there are, what the input and output schemas are, and what data
is transferred out of the operator. Because the example implementation of this method
has no inter-record state, it must be made checkpointable by calling
setCheckpointStateHandling() with eNoState.

Orchestrate 7.0 Developer Guide 1 13

Chapter 1 Creating Operators APT_Operator Derivation Examples

Table 4 APT_Operator Derivation With Arguments: .C File (continued)

49 With your override of runLocally(), you define what the operator does at run time.
In this example implementation, the lines APT_InputCursor inCur and
APT_OutputCursor outCur declare the input and out cursors; and the functions
setupInputCursor() and setupOutputCursor() initialize them.
The while loop copies the input record to the output record and writes it to output.
Finally, the “hello, world“ string is printed the specified number of times by each
node.

Including Additional APT_Operator Member

Functions
You can also include calls to the following APT_Operator member functions in
your APT_Operator derivation. The header file,
$APT_ORCHHOME/incude/apt_framework/operator.h, contains detailed descriptions
of the member functions.
• addNodeConstraint() or addResourceConstraint()
Specify any operator constraints. The contraints are appended to the existing
constraints.
• checkWasSuccessful()
Indicates whether the operator has been successfully checked.
• declareTransfer()
Specify any data transfers.
• getConcreteInputSchema()/getConcreteOutputSchema
Return the concrete schema associated with a dataset.
• isSegmented()
Determine if the operator’s step is segmented.
• lookupCollectionMethod()
Determine the collection method to be used.
• lookupPartitionMethod()
Determine the partitioning method to be used.
• reportError()
Specify error-description strings.
• setAvailableNodes()
Limits the nodes on which this operator can run. It always clears previous
restraints.
• setCheckpointStateHandling()
Specify the management of step segment boundaries.

1 14 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Defining Parallel and Sequential Operators

• setCollectionMethod()
Specify the collection method for a sequential operator.
• setKind(): Specify parallel or sequential execution of the operator.
• setNodeMap()
Specify how many partitions an operator has and which node each partition
runs on.
• setPartitionMethod()
Specify the partitioning method for a parallel operator.
• setPreservePartitioningFlag() or clearPreservePartitioningFlag()
Modify the preserve-partitioning flag in an output data set.
• setRuntimeArgs()
Specifies an initialization property list passed to initializeFromArgs_() at
runtime.
• setWorkingDirectory()
Set the working directory before runLocally() is called.

Defining Parallel and Sequential Operators

Orchestrate supports two execution modes for operators: parallel and
sequential. Parallel operators execute on multiple processing nodes;
sequential operators execute on a single processing node. By default, the
execution mode of an operator is parallel.
Orchestrate provides two member functions you can use to set the execution
mode of an operator:
• setRequestedKind() sets the requested execution mode of an operator to
either parallel or sequential.
Orchestrate calls setRequestedKind() when you specify the execution mode
of the operator as part of configuring the operator. For example, when you
specify sequential execution by specifying [seq] on the osh command line,
Orchestrate calls setRequestedKind() to set the execution mode of the
operator to sequential.
If this function is not called, and setKind() is not called within
describeOperator(), the operator defaults to parallel.
• setKind() sets the execution mode of an operator to either parallel or
sequential. This function can be called only from within describeOperator().
setKind() overrides a call to setRequestedKind(). If a call to setKind()
specifies a different execution mode than setRequestedKind(), Orchestrate
issues a warning.

Orchestrate 7.0 Developer Guide 1 15

Chapter 1 Creating Operators Defining Parallel and Sequential Operators

Creating Parallel Operators

Orchestrate processing power is directly related to its ability to execute operators
in parallel on multiple processing nodes in a system. With few exceptions, the
operators you create will be parallel.
Graphically, a parallel operator can be represented as shown in the following
figure. This figure shows a two-input, one-output operator.

input data set two-input one-output operator input data set

partitioner partitioner

...

output data set

runLocally()
on separate nodes

All steps containing a parallel operator will also contain a partitioner to divide the
input data set into individual partitions for each processing node in the system.
The partitioning method can either be supplied by the Orchestrate framework or
defined by you. See “Specifying a Partitioning Method” on page 1-20 for more
information.

Creating Sequential Operators

Sequential operators execute on a single processing node in your system. An
operator may be sequential because the very nature of some operations, such as
reading or writing a file, is sequential.

1 16 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Specifying Input and Output Interface Schemas

Orchestrate has no restrictions on mixing parallel and sequential operators within

a step. The following sample step uses both sequential and parallel operators:

input data set

Operator 1
(sequential)

Operator 2
(parallel)

All sequential operators define a collection method specifying how a sequential

operator combines the partitions of an input data set for processing by a single
node. The collection method can be either supplied by the Orchestrate or defined
by you. See “Specifying a Collection Method” on page 1-25 for more information.

Creating Operators that can be Either Parallel or

Sequential
If you want the operator to support both execution modes, you should omit the
call to setKind() within describeOperator() and allow it to default to parallel. An
operator user then can specify the execution mode of the operator as either
parallel or sequential.
If you allow the user to set the execution mode of the operator, you should also
use the default partitioning and collecting method of any for the operator. Any is
the default when the preserve-partioning flag is not set. This allows the user to
set both the execution mode of the operator and the operator’s partitioning
method (for parallel execution) or collection method (for sequential execution) at
run time. See “Specifying a Partitioning Method” on page 1-20 or “Specifying a
Collection Method” on page 1-25 for more information.

Specifying Input and Output Interface Schemas

Operator interface schemas define the number and type of data sets that may be
used as input or output for an operator. In order for a data set to be used as input
or output from an operator, the data set’s record schema must be compatible with
the operator’s interface schema.

Orchestrate 7.0 Developer Guide 1 17

Chapter 1 Creating Operators Specifying Input and Output Interface Schemas

You use protected member functions of APT_Operator to define the interface

schema of a derived operator class. The two member functions shown below
define the number of input and output data sets:
• setInputDataSets()
• setOutputDataSets()
You can define any number of input and output data sets for an operator. Records
of an input data set are read-only. An operator can read the records from an input
data set and process them but it cannot write results to an input data set. Records
of an output data set are read/write. When you use a data set as an output, you
write results to the records of the output data set.
For both input and output data sets, you must specify:
• The names for the fields accessed within a record of the data set
• The data type of each accessed field
To make these specifications, you include calls to two member functions from
within APT_Operator::(): setInputInterfaceSchema() and
setOutputInterfaceSchema(). See the chapter “Orchestrate Data Sets” in the
Ochestrate Shell User’s Guide for information on field data types and defining data
sets.
The following figure is an example of an interface schema for an operator:

input data set

field1:int32; field2:int16; field3:sfloat;

input interface schema

output interface schema

field1:int32; field2:int16; field3:sfloat;

output data set

In this figure, the interface schema for both the input and the output data sets is
the same. Both schemas require the data sets to have at least three fields: two
integer fields named field1 and field2 and a floating-point field named field3. Any
extra fields in the input data set are dropped by this particular operator. See the
Orchestrate 7.0 User Guide for more information on dropping fields from an input
data set.
Note that you can use view adapters to convert components of the record schema
of a data set to match components of the interface schema of an operator. See
“Using a View Adapter with a Partitioner” on page 8-16 and “Example Collector
with a View Adapter” on page 9-16 for information on view adapters.

1 18 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Specifying Input and Output Interface Schemas

Two possible variations to consider as you specify interface schemas are:

• Schema variables, which use wildcards to include a potentially large number
of record fields
• A dynamic schema, which lets the user of the operator define some or all of
the schema
These two variations are introduced briefly in this section. Code examples are
provided in “Examples of Operator Derivations With Options” on page 1-33.

Introducing Schema Variables

The following figure shows how to use variables in both the input and output
data set schemas:

input data set

field1:int32; field2:int16; field3:sFloat; inRec:*;

outRec:*;

output data set

By default, a schema variable, denoted by the type *, references an entire record,

regardless of the data type of each field in the record. In this example, field1, field2,
and field3 are included in the schema variable. You can include schema variables
in your specification for either an input or an output interface schema.
In the figure above, the operator can take any data set as input as long as that data
set contains two integers and one floating-point field with the specified names.
The data set can contain more than the required fields. By default, for an operator
using schema variables, all fields of the input data set are transferred to the
output data set.

Introducing Operators with Dynamic Interface

Schemas
An operator with a dynamic interface schema allows the operator user to define
some or all of the operator’s interface schema. With this type of operator, the
number and data type of the fields processed by the operator can vary every time
the operator is used. For example, one type of dynamic operator could take any
number of double-precision floating-point fields as input. Another could take a
single 8-bit integer or string field as input. A third could take fields of any data
type supported by Orchestrate.

Orchestrate 7.0 Developer Guide 1 19

Chapter 1 Creating Operators Specifying a Partitioning Method

The following figure is an example of an operator with a dynamic interface

schema:

input data set

a:type; b:type; inRec:*;

outRec:*;

output data set

In this example, the operator specifies both an input and an output interface
schema containing a single schema variable. This figure shows that the operator
allows the user to specify two fields of its input interface schema when the
operator is instantiated. The output interface schema is fixed.
Chapter 15, Advanced Features, contains an example implementation of dynamic
schema, as well as code that creates generic functions and accessors. The code is in
Table 41.

Specifying a Partitioning Method

The Orchestrate framework performs partitioning on the input data set(s) of a
parallel operator. Partitioning is the process of dividing an input data set into
multiple segments, or partitions. Each processing node in your system then
performs an operation on an individual partition of the data set rather than on the
entire data set. Typically, an operator uses the same partitioning method for each
input data set; however, you can also use a different method for each input.

1 20 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Specifying a Partitioning Method

The following figure shows an example of a two-input, one-output operator. Each

input in this figure uses its own partitioning method.

input data set two-input one-output operator input data set

partitioner partitioner

...

output data set

runLocally()
on separate nodes

Orchestrate provides a number of different partitioning methods, including:

• any
The operator does not care how data is partitioned; therefore, Orchestrate can
partition the data set in any way to optimize the performance of the operator.
Any is the default partitioning method.
Operators that use a partitioning method of any allow the user to explicitly
override the partitioning method. To set an operator’s partitioning method,
the operator user assigns a partitioner to a data set used as an input to the
operator. The framework then partitions the data set accordingly. See the
Orchestrate 7.0 User Guide for more information.
• round robin
This method divides the data set so that the first record goes to the first node,
the second record goes to the second node, and so on. When the last node in
the system is reached, this method starts back at the first node. This method is
useful for resizing the partitions of an input data set that are not equal in size.
The round robin method always creates approximately equal-sized
partitions.
• random
This method randomly distributes records across all nodes. Like round robin,
random partitioning can rebalance the partitions of an input data set to
guarantee that each processing node receives an approximately equal-sized
partition.
• same
This method performs no repartitioning on the data; the partitions of the
previous operator are inherited. This partitioning method is often used within

Orchestrate 7.0 Developer Guide 1 21

Chapter 1 Creating Operators Specifying a Partitioning Method

composite operators. See Chapter 2, “Creating Composite Operators” for

more information.
• entire
Every instance of an operator on every processing node receives the complete
data set as input. This form of partitioning is useful when you want the
benefits of parallel execution, but you also want each instance of the operator
to be able to access the entire input data set.
• hash by field
Uses a field in a record as a hash key and partitions the records based on a
function of this hash key. You use the class APT_HashPartitioner to
implement this partitioning method.
• modulus
Partitioning is based on a key field modulo the number of partitions. This
method is similar to hash by field, but involves simpler computation.
• range
Divides a data set into approximately equal size partitions based on one or
more partitioning keys. You use the APT_RangePartitioner class to
implement this partitioning method.
• DB2
Partitions an input data set in the same way that DB2 partitions it. For
example, if you use this method to partition an input data set containing
update information for an existing DB2 table, records are assigned to the
processing node containing the corresponding DB2 record. Then, during the
execution of the parallel operator, both the input record and the DB2 table
record are local to the processing node. Any reads and writes of the DB2 table
entails no network activity. See the the Orchestrate 7.0 Operators Reference for
more information.
• other
You can define a custom partitioning operator by deriving a class from the
C++ APT_Partitioner class. Other is the partitioning method for operators
that use custom partitioners.
By default, operators use the partitioning method any. The any partitioning
method allows operator users to prefix the operator with a partitioning operator
to control partitioning. For example, a user could insert the hash partitioner
operator in a step before the derived operator.
To set an explicit partitioning method for the operator that cannot be overridden,
you must include a call to APT_Operator::setPartitionMethod() within
APT_Operator::describeOperator().
Another option is to define your own partitioner for each input to the operator. To
do so, you must derive a partitioner class from APT_Partitioner. See Chapter 8,
“Creating Partitioners” for more information on creating custom partitioners.

1 22 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Specifying Partitioning Style and Sorting Requirements

Specifying Partitioning Style and Sorting

Requirements
To make it possible for your operator users to write correct data flows without
having to deal with parallelism issues, the operator and partitioning classes
define functions that allow you to specify partitioning style and sort
requirements. To ensure that your partitioning and sorting requirements are met,
Orchestrate inserts the apppropriate components into the data flow immediately
after the execution of an operator’s describeOperator() function.

Specifying Partitioning Style

Unlike using APT_Operator::setPartitionMethod to stipulate a specific
partitioning method for an operator, specifying partitioning style does not
prevent you from inserting your own custom partitioner in a data flow, and does
not prevent an operator user from inserting a partitioner in a data flow.
Orchestrate issues a warning when an operator user inserts an unnecessary or
inappropriate partitioner, but will not remove the partitioner.
The partitioning styles are:
• eLocalKeys. All keys are placed in the same partition.
• eDistributedKeys. There is no guarantee that keys are in the same partition.
Partitioning style is defined by the partitioningStyle enumeration of the
APT_PartioningStyle class defined in
$APT_ORCHHOME/incude/apt_framework/paritioningStyle.h.

Based on partitioning style, Orchestrate can group partitioners into categories

based on key locality, making it possible for you to determine if an operator’s
inputs and outputs are appropriately partitioned, even if the operator is not
associated with a particular partitioning method.

! To set and get the partitioning style of an operator input data set:
Use the APT_Operator functions below.
void setInputPartitioningStyle
(APT_PartitioningStyle::partitioningStyle, int inputDS)
APT_PartitioningStyle::partitioningStyle
getPartitioningStyle(int inputDS)
void setOutputPartitioningStyle
(APT_PartitioningStyle::partitioningStyle, int outputDS)

These functions allow you to:

• Direct Orchestrate to insert appropriate partitioners in a data flow
• Require that two inputs be identically partitioned

Orchestrate 7.0 Developer Guide 1 23

Chapter 1 Creating Operators Specifying Partitioning Style and Sorting Requirements

• Determine whether two inputs are identically partitioned

! To set and get the partitioning keys of an operator input data set:
Use these APT_Operator functions:
APT_Operator::setPartitioningKeys(APT_PropertyList keys, int input)
APT_PropertyList APT_Operator::getPartitioningKeys(int input)

The syntax for APT_PropertyList is:

{ key = {value=fieldname }, key = {value=fieldname }, ...}

An example is:
{ key={value=lastname}, key={value=balance} }

! To obtain the partitioning style of a custom partitioner:

Use this APT_Partitioner function:
APT_Partitioner::partitioningStyle
APT_Partitioner::partitioningStyle()

! To get the name of the partitioner last used to partition an operator input:
Use this APT_Operator function:
APT_String APT_Operator::getPartitioningName(int input)

By default, Orchestrate performs partitioner insertion.

! To disable partitioner insertion:

Add the global option -nopartinsertion to your osh command line.

Specifying Sort Keys

You can define sort keys on each of the input and output data sets of an operator.
Setting sort information directs Orchestrate to insert sort components in a data
flow so that your sorting requirements are met. Orchestrate records this
information for datasets that have preserve partitioning set.

! To set and get sort keys for an input data set:

Call these APT_Operator functions:
APT_Operator::setInputSortKeys(APT_PropertyList keys, int input)
APT_PropertyList APT_Operator::getInputSortKeys(int input)

! To set and get sort keys for an output data set:

Call these APT_Operator functions:
APT_Operator::setOutputSortKeys(APT_PropertyList keys, int output)
APT_PropertyList APT_Operator::getOutputSortKeys(int output)

1 24 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Specifying a Collection Method

Note An operator which calls setPartitionMethod() on a data set can call

setInputSortKeys() and setOutputSortKeys() but cannot call
setPartitioningStyle()or setPartitioningKeys() on the data set.

By default, Orchestrate performs sort insertion.

! To disable sort insertion:

Add the global option -nosortinsertion to your osh command line.

Specifying a Collection Method

A collector defines how a sequential operator combines the partitions of an input
data set for processing by a single node. A collection method is the algorithm
used by a sequential Orchestrate operator to combine the partitions of an input
data set into a single input stream. The method may be as simple as combining
the partitions on a first-come, first-served basis, where the sequential operator
processes records in the order in which they are received from the previous
operator. Alternatively, information contained within a record can be used to
determine the order of processing.
Orchestrate provides a number of different collection methods, including:
• any
This method reads records on a first-come, first-served basis.
Operators that use a collection method of any allow the operator user to
explicitly override the collection method to set their own method. To set an
operator’s collection method, the operator user assigns a collector to a data set
used as an input to the operator. Orchestrate then collects the partitions of the
data set accordingly. See the Orchestrate 7.0 User Guide for more information.
• round robin
This method reads a record from the first input partition, then from the
second partition, and so on. When the last processing node in the system is
reached, it starts over.
• ordered
This method reads all records from the first partition, then all records from the
second partition, and so on. This collection method preserves any sorted order
in the input data set.
• sorted merge
This method reads records in an order based on one or more fields of the
record. The fields used to define record order are called collecting keys. You
use the sortmerge collection operator to implement this method.

Orchestrate 7.0 Developer Guide 1 25

Chapter 1 Creating Operators Using Cursors

• other
You can define a custom collection method by deriving a class from
APT_Collector. Operators that use custom collectors have a collection method
of other.
By default, sequential operators use the collection method any. The any collection
method allows operator users to prefix the operator with a collection operator to
control the collection method. For example, a user could insert the ordered
collection operator in a step before the derived operator.
To set an explicit collection method for the operator that cannot be overridden,
you must include a call to APT_Operator::setCollectionMethod() within
APT_Operator::describeOperator().
You can also define your own collection method for each operator input. To do so,
you derive a collector class from APT_Collector. See Chapter 9, “Creating
Collectors” for more information.

Using Cursors
In order for your operators to process an input data set and write results to an
output data set, you need a mechanism for accessing the records and record fields
that make up a data set. Orchestrate provides three mechanisms that work
together for accessing the records and record fields of a data set: cursors,
subcursors, and field accessors.
Cursors let you reference specific records in a data set, and field accessors let you
access the individual fields in those records. You use cursors and field accessors
from within your override of the APT_Operator::runLocally() function. You use
subcursors only with vectors of subrecords. A subcursor allows you to identify
the current element of the vector.
This section describes cursors; the next section covers field accessors. See
Chapter 7, “Using Cursors and Accessors” for more information on subcursors.
You use two types of cursors with data sets: input cursors and output cursors.
Input cursors provide read-only access to the records of an input data set; output
cursors provide read/write access to the current record of an output data set.
To process an input data set, you initialize the input cursor to point to the current
input record. The cursor advances through the records until all have been
processed. Note that once a cursor has moved beyond a record of an input data
set, that record can no longer be accessed.
For the output data set, the output cursor initially points to the current output
record and advances through the rest of the records. As with input data sets, once
a cursor has moved beyond a record of an output data set, that record can no
longer be accessed.

1 26 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Using Cursors

The following figure shows an operator with a single input and a single output
data set:

input data set output data set

previously processed previously written

input records input data set
. . output records
(not available) . . (not available)
. .

input cursor to output cursor

current input record to current
operator output record

direction of
cursor movement direction of
. output data set . cursor movement
. .
. .

Orchestrate uses two classes to represent cursors:

• APT_InputCursor defines a cursor for an input data set.
• APT_OutputCursor defines a cursor for an output data set.
When you create an input cursor, it does not reference a record. You must call
APT_InputCursor::getRecord() from within runLocally() to initialize the cursor
to point to a specific record. When you have finished processing that record, you
again call APT_InputCursor::getRecord() to advance the cursor to the next record
in the input data set, making it the current input record. When no more input
records are available, APT_InputCursor::getRecord() returns false. Typically, you
use a while loop to determine when APT_InputCursor::getRecord() returns false.
When you create an output cursor, it references the first record in the output data
set.
The record output fields are set to the following default values:
• Nullable fields are set to null.
• Integers = 0.
• Floats = 0.
• Dates = January 1, 0001.
• Decimals = 0.
• Times = 00:00:00 (midnight).
• Timestamps = 00:00:00 (midnight) on January 1, 0001.
Orchestrate 7.0 Developer Guide 1 27
Chapter 1 Creating Operators Using Cursors

• The length of variable-length string and raw fields is set to 0.

• The characters of a fixed-length string are set to null (0x00) or to the pad
character, if specified.
• The bytes of fixed-length raw fields are set to zero.
• The tag of a tagged aggregate is set to 0 to set the data type to be that of the
first field of the tagged aggregate.
• The length of variable-length vector fields is set to 0.
After you write to that output record, you must call
APT_OutputCursor::putRecord() to advance the cursor to the next record in the
output data set, making it the current output record.
Table 5 shows an example of a while loop that processes the records of a single
input data set and writes its results to a single output data set. You would place
this loop, along with the code to define and initialize the cursors, within the
override of the runLocally() function.

Table 5 Override of APT_Operator::runlocally()

Code

APT_Status AddOperator::runLocally()
{
3 APT_InputCursor inCur;
4 APT_OutputCursor outCur;

5 setupInputCursor(&inCur, 0);
6 setupOutputCursor(&outCur, 0);

7 while (inCur.getRecord())
{
9 // body of the loop

10 outCur.putRecord();
}

return APT_StatusOk;
}

Comments

3 Define inCur, an instance of APT_InputCursor, the input cursor for the first data set
input to this operator.

4 Define outCur, an instance of APT_OutputCursor, the output cursor for the first data
set output by this operator.

1 28 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Using Field Accessors

Table 5 Override of APT_Operator::runlocally() (continued)

5 Use APT_Operator::setupInputCursor() to initialize inCur. Note that input data sets

are numbered starting from 0. This argument is optional for operators that take a
single input data set.

6 Use APT_Operator::setupOutputCursor() to initialize outCur. Note that output data

sets are numbered starting from 0.

7 Use APT_InputCursor::getRecord() to advance the input cursor to the next input

record. You must call this function before attempting to process an input record
because an input cursor does not initially reference a valid record.
APT_InputCursor::getRecord() returns false when there are no more records in the
input data set, terminating the while loop.

9 Process the record. This includes writing any results to the current output record. You
should not call APT_OutputCursor::putRecord() until after you have written to the
first output record (unless you want default values), because the output cursor
initially points to the first empty record in an output data set.

10 Use APT_OutputCursor::putRecord() to update the current output record, then

advance the output cursor to the next output record.

Not all operators produce an output record for each input record. Also, operators
can produce more output records than there are input records. An operator can
process many input records before computing a single output record. You call
putRecord() only when you have completed processing an output record,
regardless of the number of input records you process between calls to
putRecord().
See Chapter 7, “Using Cursors and Accessors” for more information on cursors.

Using Field Accessors

For an input or output data set, field accessors provide named access to the record
fields. Such access is often necessary if an operator is to process data sets. As
noted later in this section, no field accessor is allowed for schema variables, which
have no defined data type.
Operators use field accessors to read the fields of an input record and to write the
fields of an output record. Field accessors do not allow access to the entire data set
at one time; instead, they allow you to access the fields of the current input or
output record as defined by an input or output cursor. For more information on
cursors, see the previous section.

Orchestrate 7.0 Developer Guide 1 29

Chapter 1 Creating Operators Using Field Accessors

Note that within Orchestrate, the fields of an input record are considered read
only. There is no mechanism for you to write into the fields of the records of an
input data set. Because the fields of an output record are considered read/write,
you can modify the records of an output data set.
The following figure shows a sample operator and interface schemas:
input data set

field1:int32; field2:int32;

AddOperator
field1:int32; field2:int32; total:int32;

output data set

This operator adds two fields of a record and stores the sum in a third field. For
each of the components of an operator’s input and output interface schemas, you
define a single field accessor. In this case, therefore, you need two input accessors
for the input interface schema and three output accessors for the output interface
schema.
This example uses field accessors to explicitly copy field1 and field2 from an input
record to the corresponding fields in an output record. If the input data set had a
record schema that defined more than these two fields, all other fields would be
dropped by AddOperator and not copied to the output data set.
Table 6 contains an example override of APT_Operator::describeOperator().

Table 6 Override of APT_Operator::describeOperator()

Code

APT_Status AddOperator::describeOperator()
{
3 setInputDataSets(1);
4 setOutputDataSets(1);

5 setupInputInterfaceSchema(“record (field1:int32; field2:int32;)”, 0);

6 setupOutputInterfaceSchema(“record (field1:int32; field2:int32;
total:int32;)”, 0);

return APT_StatusOk;
}

Comments

1 30 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Using Field Accessors

Table 6 Override of APT_Operator::describeOperator() (continued)

3 Set the number of input data sets.

4 Set the number of output data sets.

5 Because input data sets are numbered starting from 0, specify the interface schema of
input 0 . You can simply pass a string containing the interface schema as an argument
to APT_Operator::setInputInterfaceSchema(). For more information, see “Specifying
Input and Output Interface Schemas” on page 1-18. Note that you can omit the
argument specifying the number of the data set for this function because there is only
one input data set to the operator.

6 Specify the interface schema of output 0, the first output data set.

Field accessors normally are defined as local variables in

APT_Operator::runLocally(). To access the values of fields defined in an interface
schema, you must create input accessors and output accessors in the
APT_Operator::runLocally() function of the operator.
Table 7 shows an example override of APT_Operator::runLocally().

Table 7 Override of APT_Operator::runLocally()

Code

APT_Status AddOperator::runLocally()
{

APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);

7 APT_InputAccessorToInt32 field1InAcc("field1", &inCur);

8 APT_InputAccessorToInt32 field2InAcc("field2", &inCur);

9 APT_OutputAccessorToInt32 field1OutAcc("field1", &outCur);

APT_OutputAccessorToInt32 field2OutAcc("field1", &outCur);
11 APT_OutputAccessorToInt32 totalOutAcc("total", &outCur);

12 while (inCur.getRecord())
{
14 *totalOutAcc = *field1InAcc + *field2InAcc;

*field1OutAcc = *field1InAcc;

Orchestrate 7.0 Developer Guide 1 31

Chapter 1 Creating Operators Using Field Accessors

Table 7 Override of APT_Operator::runLocally() (continued)

16 *field2OutAcc = *field2InAcc;

17 outCur.putRecord();
}

return APT_StatusOk;
{

Comments

7-8 Define read-only field accessors for the fields of the input interface schema.

9 -11 Define read/write field accessors for the fields of the output interface schema.

12 Use APT_InputCursor::getRecord() to advance the input data set to the next input
record.

14-16 Use APT_InputAccessorToInt32::operator* and

APT_OutputAccessorToInt32::operator* to dereference the field accessors, in order
to access the values of the record fields in both the input and the output data sets.
Note that you could also have used the equivalent member functions
APT_InputAccessorToInt32::value() and APT_OutputAccessorToInt32::setValue()
to access the fields of input and output records. Shown below is Line 14 rewritten
using these functions:
*totalOutAcc = field1InAcc.value() + field2InAcc.value();

17 Use APT_OutputCursor::putRecord() to update the current output record, then

advance the output data set to the next output record.

See Chapter 7, “Using Cursors and Accessors” for more information on accessors.

Field Accessors and Schema Variables

Because schema variables reference the entire current input or output record, they
have no type and, therefore, no field accessor. In the following example, you

1 32 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Examples of Operator Derivations With Options

would need to define a field accessor for field1 but not for the schema variable
components inRec and outRec.

field1:int32; inRec:*;

outRec:*;

See “Including Schema Variables in an Interface Schema” on page 1-34 for more
information on schema variables.

Field Accessors and Dynamic Operators

For an operator with a dynamic schema, the user of the operator typically passes
an argument defining some or all of the interface schema of the operator. As part
of the operator’s runLocally() function, you will have to create appropriate field
accessors for an operator with a dynamic schema. The following figure shows an
example of a dynamic operator:

input data set

a:type; b:type; inRec:*;

outRec:*;

output data set

See “Working with a Dynamic Interface Schemas” on page 1-48 for more
information.

Examples of Operator Derivations With Options

The examples in this section are somewhat more complicated than those in
“APT_Operator Derivation Examples” on page 1-5. They are designed to
illustrate various options for deriving operators. Background information which
you need to use these options, such as on the transfer mechanism and the
multifield accessors, is included.

Orchestrate 7.0 Developer Guide 1 33

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Including Schema Variables in an Interface Schema

As part of their action, many Orchestrate operators pass an entire record
unchanged from the input data set to the output data set. These operators may
add new result fields to the record of the output data set, but they also propagate
the original record through to the output data set.
This section provides background information you need to use schema variables,
including a further discussion of the concept of schema variables and an
explanation of the transfer mechanism. Then two examples are provided: one
showing how to use schema variables with a one-input operator and the other
showing how to create a two-input operator.

Background
The following figure shows an operator containing schema variables in both its
input and its output interface schemas:

field1:int32; inRec:*;

outRec:*;

This figure shows the same operator with an input data set:

unsortedData data set

schema: fName:string; lName:string; age:int32;

age:int32; inRec:*; "inRec:*" ≡ "fName:string; lName:string; age:int32;"

outRec:*; "outRec:*" ≡ "fName:string; lName:string; age:int32;"

sortedData data set

By default, a schema variable in an input interface schema corresponds to an

entire record of the input data set, including the record schema. In this example:
"inRec:*" ≡ "fName:string; lName:string; age:int32;"

performing an operation on inRec corresponds to performing an operation on an

entire record and the record schema of the input data set.

1 34 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Examples of Operator Derivations With Options

When a transfer is declared, an output schema variable, by default, assumes the

record schema of the variable in the input interface. As you can see in the
previous figure, outRec assumes the schema of inRec, which corresponds to the
record schema of the input data set. Therefore, the output interface schema of the
operator is:
"outRec:*" = "inRec:*" ≡ "fName:string; lName:string; age:int32;"

By using a variable to reference a record, operators can copy, or transfer, an entire

record from an input to an output data set. The process of copying records from
an input to an output data set then becomes much simpler because you do not
have to define any field accessors.
Application developers using an operator may use transfer adapters to modify a
schema variable at run-time. This allows users to drop fields from a schema
variable, add fields to a schema variable, or rename the fields represented by a
schema variable. See the Orchestrate 7.0 User Guide for more information on
transfer adapters.

The Transfer Mechanism

A transfer performs an optimized field-by-field copy of the current input record
to the current output record. The operator copies all fields transferred from an
input to an output record without modification.
The following figure shows an operator and how it transfers records using
schema variables:

unsortedData data set

a b c d e current input record
schema: a:string; b:string;
c:int32; d:int16; e:int8;

inRec:*; inRec:*
transfer

outRec:*;
outRec:*

a b c d e current output record

Orchestrate 7.0 Developer Guide 1 35

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Many operators have a more complicated interface schema than that shown in the
previous figure. The next figure shows an operator with four elements in its input
interface schema and three elements in its output interface schema:

unsortedData data set a b c d e current input record

schema: a:string; b:string;
c:int32; d:int16; e:int8;

a:string; c:int32; d:int16; inRec:*; inRec:*

transfer
NewOperator
outRec:*
x:int32; y:int32; outRec:*;

x y a b c d e current output record

fields added by operator

In this example, the operator adds two fields to the output data set and calculates
the values of those fields. For the operator to complete this task, you must define
field accessors for the three defined fields in the input data set as well as the two
new fields in the output data set. You cannot define accessors for the fields
represented by an interface variable.
The total output interface schema of this operator is the combination of the
schema variable outRec and the two new fields, as shown below:

"x:int32; y:int32; outRec:*;"

"x:int32; y:int32; a:string; b:string; c:int32; d:int16; e:int8;"

The order of fields in the operator’s interface schema determines the order of
fields in the records of the output data set. In this example, the two new fields are
added to the front of the record. If the output interface schema were defined as:
"outRec:*; x:int32; y:int32;"

the two new fields would be added to the end of each output record.
A schema variable may contain fields with the same name as fields explicitly
stated in an operator’s output interface schema. For example, if the input data set

1 36 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Examples of Operator Derivations With Options

in the previous example contained a field named x, the schema variable would
also contain a field named x, as shown below.

"x:int32; y:int32; outRec:*;"

"x:int32; y:int32; a:string; b:string; x:int16; d:int16; e:int8;"

name conflict

Orchestrate resolves name conflicts by dropping all fields with the same name as
a previous field in the schema and issuing a warning. In this example, Orchestrate
drops the second occurence of field x which is contained in the schema variable.

Performing Transfers
! To perform a transfer, you must:
1 Define the schema variables affected by the transfer using the
APT_Operator::declareTransfer() function within the body of
APT_Operator::describeOperator()
2 Implement the transfer from within the body of APT_Operator::runLocally()
using the function APT_Operator::transfer().
The code in Table 8 below shows the describeOperator() override for
NewOperator, the operator introduced in the previous figure:

Table 8 Override of describeOperator() With a Transfer

Code

APT_Status NewOperator::describeOperator()
{
setInputDataSets(1);
setOutputDataSets(1);

5 setupInputInterfaceSchema(“record (a:string; c:int32; d:int16;

inRec:*;)”, 0);
6 setupOutputInterfaceSchema(“record (x:int32; y:int32; outRec:*;)”, 0);
7 declareTransfer(“inRec”, “outRec”, 0, 0);

return APT_StatusOk;
}

Comments

Orchestrate 7.0 Developer Guide 1 37

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Table 8 Override of describeOperator() With a Transfer (continued)

5 Specify the interface schema of input 0 (the first data set). You can simply pass a
string containing the interface schema as an argument to
APT_Operator::setInputInterfaceSchema().

6 Specify the interface schema of output 0.

7 Use APT_Operator::declareTransfer() to define a transfer from inRec to outRec.

The function declareTransfer() returns an index identifying this transfer. The first
call to declareTransfer() returns 0, the next call returns 1, and so on. You can store
this index value as a data member, then pass the value to
APT_Operator::transfer() in the operator’s runLocally() function to perform the
transfer.
Because this is the only transfer defined for this operator, the index is not stored.
You could, however, define an operator that takes more than one data set as
input, or one that uses multiple schema variables. If you do, you can define
multiple transfers and, therefore, store the returned index.
With the transfer defined, you implement it. Typically, you call transfer() just
before you call putRecord(). The code in Table 9 below is the runLocally()
override for NewOperator:

Table 9 Override of runLocally() With a Transfer

Code

APT_Status NewOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;

// define accessors

// initialize cursors

// initialize accessors

5 while (inCur.getRecord())
{
// access input record using input accessors

// update any fields in the output record using output accessors

6 transfer(0);
7 outCur.putRecord();
}

1 38 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Table 9 Override of runLocally() With a Transfer (continued)

return APT_StatusOk;
}

Comments

5 Use APT_InputCursor::getRecord() to advance the input cursor to the next input

record.

6 After processing the input record and writing any fields to the output record, use
APT_Operator::transfer() to perform the transfer. This function copies the current
input record to the current output record.

7 Use APT_OutputCursor::putRecord() to update the current output record and

advance the output data set to the next output record.

Note For more efficient processing, use APT_Operator::transferAndPutRecord()

which is equivalent to deferredTransfer() followed by a putRecord().

Example: Using Schema Variables in a Filter Operator

Filter operators remove, or filter, records from a data set based on some
characteristic of the records. The following figure shows a sample operator that
filters an input data set by removing all records with a string field named key that
contains the string “REMOVE”:

key:string; inRec:*;

filterOperator

outRec:*;

This operator uses a single field accessor to determine the value of key. Because
the operator does not modify the record during the filter operation, you need not
define any output field accessors. Note that to iterate through the records of the
output data set, you must still define an output cursor.
FilterOperator uses the transfer mechanism to copy a record without modification
from the input data set to the output data set. If key is equal to “REMOVE”, the
operator simply does not transfer that record.
FilterOperator is derived from APT_Operator, in Table 10 below:

Orchestrate 7.0 Developer Guide 1 39

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Table 10 Example of Using Schema Variables in a Filter Operator

Code

#include <apt_framework/orchestrate.h>

class FilterOperator : public APT_Operator

{
APT_DECLARE_RTTI(FilterOperator);
APT_DECLARE_PERSISTENT(FilterOperator);

public:
FilterOperator();

protected:
virtual APT_Status describeOperator();
virtual APT_Status runLocally();
virtual initializeFrom Args_(const APT_PropertyList &args,
APT_Operator::InitializeContext context);

12 #define ARGS_DESC "{. . .}”

13 APT_DEFINE_OSH_NAME(FilterOperator, exampleOp, ARGS_DESC);
14 APT_IMPLEMENT_RTTI_ONEBASE(FilterOperator, APT_Operator);
15 APT_IMPLEMENT_PERSISTENT(FilterOperator);

16 FilterOperator::FilterOperator();
{}

18 APT_Status FilterOperator::initializeFromArgs_(const APT_PropertyList

&args, APT_Operator::InitializeContext context)
{
return APT_StatusOk;
}

22 APT_Status FilterOperator::describeOperator()
{
setInputDataSets(1);;
setOutputDataSets(1);

26 setInputInterfaceSchema(“record (key:string; inRec:*;)”, 0);

27 setOutputInterfaceSchema(“record (outRec:*;),”, 0);

28 declareTransfer(“inRec”, “outRec”, 0, 0);

return APT_StatusOK;
}

APT_Status FilterOperator::runLocally()

1 40 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Table 10 Example of Using Schema Variables in a Filter Operator (continued)

{
33 APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
36 setupOutputCursor(&outCur, 0);

37 APT_InputAccessorToString keyInAcc(“key”, &inCur);

38 while (inCur.getRecord())
{
40 if (*keyInAcc != “REMOVE”)
{
42 transfer(0);
43 outCur.putRecord();
}
}
return APT_StatusOk;
}
48 void FilterOperator::serialize(APT_Archive& archive, APT_UInt8)
{}
}

Comments

12 See “Argument-List Descriptions” on page 5-3 and the header file,

$APT_ORCHHOME/include/apt_util/argvcheck.h, for documentation on the
ARGS_DESC string.

13 With APT_DEFINE_OSH_NAME, you connect the class name to the name used to
invoke the operator from osh and pass your argument description string to
Orchestrate. Orchestrate’s argument-checking facility is fully described in
Chapter 5, “Orchestrate’s Argument-List Processor”. Also see
$APT_ORCHHOME/include/apt_framework/osh_name.h for documentation on this
macro.

14 - APT_IMPLEMENT_RTTI_ONEBASE and APT_IMPLEMENT_PERSISTENT

15 are required macros that implement run-time type information and persistent
object processing.

16 Defines the default constructor for FilterOperator.

Orchestrate 7.0 Developer Guide 1 41

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Table 10 Example of Using Schema Variables in a Filter Operator (continued)

18 With your override of initializeFromArgs_(), you transfer information from the
operator's arguments to the class instance, making it osh aware. Because there are
no arguments for this example, initializeFromArgs_() simply returns
APT_StatusOk. See “Property List Encoding of Operator Arguments” on page 5-19
and the header file, $APT_ORCHHOME/include/apt_framework/operator.h for
documentation on this function.

22 The describeOperator() function defines the input and output interface schema of
the operator, as well as the transfer from inRec to outRec.

26 Specify the interface schema of input 0 (the first input data set). You can simply
pass a string containing the interface schema as an argument to
APT_Operator::setInputInterfaceSchema().

27 Specify the interface schema of output 0.

28 Use APT_Operator::declareTransfer() to define a transfer from inRec to OutRec.

Because this is the only transfer defined for this operator, the index is not stored.

33 - Define and initialize the input and output cursors.

36

37 Define a read-only field accessor for key in the operator’s input interface schema.
Note that because this operator does not access any fields of an output record, it
does not define any output field accessors.

38 Use APT_InputCursor::getRecord() to advance the input cursor to the next input

record. You must call this function before attempting to process an input record,
because an input cursor does not initially reference a valid record.

40 Determine whether key of an input record is equal to “REMOVE”. If it is not,

transfer the record and update the output data set to the next empty record. If key is
equal to “REMOVE”, bypass the transfer and get the next input record.

42 Use APT_Operator::transfer() to perform the transfer.

43 Use APT_OutputCursor::putRecord() to update the current output record and

advance the output cursor to the next output record.

48 FilterOperator does not define any data members; therefore, serialize() is empty.
You must provide serialize(), even if it is an empty function.

1 42 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Example: Creating a Two-Input Operator

This section describes a sample operator that takes two data sets as input and
creates a single data set as output. The input data sets to this operator consist of:
• A data set containing a list of bank customers. One field in each record defines
the type of bank account.
• A 10-record lookup table, in which each record has two fields. The first field
specifies a bank account type; the second specifies the corresponding daily
interest rate for that account type.
This operator uses the lookup table to determine the daily interest rate for each
customer’s account. The operator creates the output data set by prepending the
interest rate to each customer record.
The following figure shows this operator:

interestTable customerData

accountType:int8; interestRate:dfloat; accountType:int8; inRec:*;

CalculateInterestOperator

interestRate:dfloat; outRec:*;

The input interface schema of the lookup table specifies two fields: an account
type and an interest rate. The input interface schema of customerData specifies a
single field containing the account type as well as a schema variable.
The output interface schema of the operator specifies a schema variable and the
new field interest-Rate. This is the field that the operator prepends to each record
of customerData.

Orchestrate 7.0 Developer Guide 1 43

Chapter 1 Creating Operators Examples of Operator Derivations With Options

The following figure shows CalculateInterestOperator with its input and output
data sets:

interestTable data set customerData data set

schema: accountType:int8; schema: fName:string; lName:string;

interestRate:dfloat; address:string; accountType:int8;

accountType:int8; interestRate:dfloat; accountType:int8; inRec:*;

CalculateInterestOperator

interestRate:dfloat; outRec:*;

outData data set

schema: fName:string; lName:string;

address:string; accountType:int8;
interestRate:dfloat;

CalculateInterestOperator is derived from APT_Operator, as shown in the following

code:

Table 11 Example of a Two-Input Operator

Code

#include <apt_framework/orchestrate.h>

class CalculateInterestOperator : public APT_Operator

{
APT_DECLARE_RTTI(CalculateInterestOperator);
APT_DECLARE_PERSISTENT(CalculateInterestOperator);

public:
CalculateInterestOperator();

protected:
virtual APT_Status describeOperator();
virtual APT_Status runLocally();
virtual initializeFrom Args_(const APT_PropertyList &args,
APT_Operator::InitializeContext context);

12 #define ARGS_DESC "{. . .}”

13 APT_DEFINE_OSH_NAME(CalculateInterestOperator, exampleOp, ARGS_DESC);
14 APT_IMPLEMENT_RTTI_ONEBASE(CalculateInterestOperator, APT_Operator);
15 APT_IMPLEMENT_PERSISTENT(CalculateInterestOperator);

1 44 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Table 11 Example of a Two-Input Operator (continued)

16 CalculateInterestOperator::CalculateInterestOperator()
{}

18 APT_Status ExampleOperator::initializeFromArgs_(const APT_PropertyList

&args, APT_Operator::InitializeContext context)
{
return APT_StatusOk;
}

APT_Status CalculateInterestOperator::describeOperator()
{
24 setKind(APT_Operator::eParallel);
25 setInputDataSets(2);
setOutputDataSets(1);

27 setPartitionMethod(APT_Operator::eEntire, 0);
28 setPartitionMethod(APT_Operator::eAny, 1);

29 setInputInterfaceSchema(“record
(accountType:int8; interestRate:dfloat;)”, 0);
30 setInputInterfaceSchema(“record
(accountType:int8; inRec:*;)”, 1);
31 setOutputInterfaceSchema(“record
(interestRate:dfloat; outRec:*;),”, 0);

32 declareTransfer(“inRec”, “outRec”, 1, 0);

return APT_StatusOK;
}

APT_Status CalculateInterestOperator::runLocally()
{
37 APT_InputCursor inCur0;
38 APT_InputCursor inCur1;
39 APT_OutputCursor outCur0;

setupInputCursor(&inCur0, 0);
setupInputCursor(&inCur1, 1);
setupOutputCursor(&outCur0, 0);

APT_InputAccessorToInt8 tableAccountInAcc(“accountType”, &inCur0);

APT_InputAccessorToDFloat tableInterestInAcc(“interestRate”, &inCur0);
APT_InputAccessorToInt8 customerAcctTypeInAcc(“accountType”, &inCur1);

APT_OutputAccessorToDFloat interestOutAcc(“interestRate”, &outCur0);

47 struct interestTableEntry

Orchestrate 7.0 Developer Guide 1 45

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Table 11 Example of a Two-Input Operator (continued)

{
APT_Int8 aType;
APT_DFloat interest;
};

52 interestTableEntry lookupTable[10];
53 for (int i = 0; i < 10; ++i)
{
bool gotRecord = inCur0.getRecord();
APT_ASSERT(gotRecord);
lookupTable[i].aType = *tableAccountInAcc;
lookupTable[i].interest = *tableInterestInAcc;
}

APT_ASSERT(!inCur0.getRecord());

61 while (inCur1.getRecord())
{
int i;
for (i = 0; i < 10; i++)
{
if (loopupTable[i].aType == *customerAcctTypeInAcc)
{
68 *interestOutAcc = lookupTable[i].interest;
69 transfer(0);
70 outCur0.putRecord();
}
}
}
74 void CalculateInterestOperator::serialize(APT_Archive& archive,
APT_UInt8)
{}
}

Comments

12 See “Argument-List Descriptions” on page 5-3 and the header file,

$APT_ORCHHOME/include/apt_util/argvcheck.h, for documentation on the
ARGS_DESC string.

13 With APT_DEFINE_OSH_NAME, you connect the class name to the name used to
invoke the operator from osh and pass your argument description string to
Orchestrate. Orchestrate’s argument-checking facility is fully described in Chapter 5,
“Orchestrate’s Argument-List Processor”. Also see
$APT_ORCHHOME/include/apt_framework/osh_name.h for documentation on this macro.

1 46 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Table 11 Example of a Two-Input Operator (continued)

14 - 15 APT_IMPLEMENT_RTTI_ONEBASE and APT_IMPLEMENT_PERSISTENT are
required macros that implement run-time type information and persistent object
processing.

16 Defines the default constructor for CalculateInterestOperator.

18 With your override of initializeFromArgs_(), you transfer information from the

operator's arguments to the class instance, making it osh aware. Because there are no
arguments for this example, initializeFromArgs_() simply returns APT_StatusOk. See
“Property List Encoding of Operator Arguments” on page 5-19 and the header file,
$APT_ORCHHOME/include/apt_framework/operator.h for documentation on this
function.

24 Use APT_Operator::setKind() to explicitly set the execution mode to parallel. Calling

this function means the operator cannot be run sequentially.

25 Use APT_Operator::setInputDataSets() to set the number of input data sets to 2.

27 Use APT_Operator::setPartitionMethod() to set the partitioning method of the

lookup table to entire. This partitioning method specifies that every instance of this
operator on every processing node receives the complete lookup table as input.
This data set requires entire partitioning because all instances of this operator require
the complete table as input.

28 Set the partitioning method of input 1, the customer data set. Because this is a parallel
operator and the records in the data set are not related to each other, choose a
partitioning method of any.

29 Specify the interface schema of input 0, the lookup table.

30 Specify the interface schema of input 1, the customer data set.

31 Specify the interface schema of output 0.

32 Declare a transfer from input data set 1, the customer data set to the output data set.

37 Define an input cursor for data set 0, the lookup table.

38 Define an input cursor for data set 1, the customer data.

39 Define an output cursor for data set 0.

47 Define a structure representing an entry in the lookup table. Each record in the lookup
table has two fields: an account type and the daily interest rate for that type of
account.

Orchestrate 7.0 Developer Guide 1 47

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Table 11 Example of a Two-Input Operator (continued)

52 Define lookupTable, an array of 10 elements, to hold the lookup table.

53 Use a for loop to read in each record of the table to initialize lookupTable.
Note that for example purposes, the number of elements in the array and the loop
counter are hard coded, but this practice is not recommended.

61 Use a while loop to read in each record of the customer data. Compare the account
type of the input record to each element in the lookup table to determine the daily
interest rate corresponding to the account type.

68 Read the matching interest rate from the table and add it to the current output record.

69 Transfer the entire input record from the customer data set to the output data set.

70 Update the current output record, then advance the output data set to the next output
record.

74 CalcuateInterestOperator does not define any data members; therefore, serialize() is

empty. Note that you must provide serialize(), even it is an empty function.

Working with a Dynamic Interface Schemas

The following figure shows an example of an operator with a dynamic interface
schema:

input data set

a:type; b:type; inRec:*;

ADynamicOperator
outRec:*;

output data set

This operator specifies a schema variable for both the input and the output
interface schema. In addition, when you instantiate the operator, you must
specify two fields of its input interface schema.
To create an operator with such a dynamic interface schema, you typically
provide a constructor that takes an argument defining the schema. This
constructor may take a single argument defining a single interface field, an array
of arguments defining multiple interface fields, or any combination of arguments
specific to your derived operator.

1 48 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Examples of Operator Derivations With Options

The constructor for the operator shown in the previous figure has the following
form:
ADynamicOperator (char * inputSchema);

The constructor for an operator that supports a dynamic interface must make the
interface schema definition available to the APT_Operator::describeOperator()
function, which contains calls to:
• setInputInterfaceSchema() to define the input interface schema
• setOutputInterfaceSchema() to define the output interface schema
For example:
setInputInterfaceSchema(“record(a:string; b:int64; inRec:*;)”, 0);
setOutputInterfaceSchema(“record (outRec”, 0, 0);

These functions must include any schema components specified in the

constructor or by member functions of the operator.
APT_Operator::runLocally() defines the cursors and field accessors used by the
operator to access the records and record fields of the data sets processed by an
operator. Because users of the operator can define field names and data types for
the interface schema of an operator, APT_Operator::runLocally() must be able to
create the corresponding accessors for those fields.

Using Multifield Accessors

Because you do not know the input interface schema of the operator at compile
time, you need another way to define field accessors for components of the
operator’s input interface schema. One way is by passing a schema definition to
the constructor for the field accessor class APT_MultiFieldAccessor. This
technique is described in this section.

Note A more efficient technique for defining run-time multi-field accessors is

demonstrated in Table 41 in Chapter 15, Advanced Features,. The technique
uses the APT_InputAccessorBase class to set up an array of input accessors.

In the following figure, an APT_MultiFieldAccessor object is used to access three

components of an input interface schema:

previousField()

a:type; b:type; c:type; inRec:*;

accessor 1 current accessor

accessor 2 nextField()

outRec:*; accessor 3

APT_MultiFieldAccessor

Orchestrate 7.0 Developer Guide 1 49

Chapter 1 Creating Operators Examples of Operator Derivations With Options

This figure shows an APT_MultiFieldAccessor object containing three accessors,

one for each possible component of the operator’s input interface schema.
An APT_MultiFieldAccessor object is an array of accessors. In this example, the
array is three elements long, with one element for each of the three user-specified
fields in the operator’s input interface schema. There is a current accessor location,
which you can use to read a field of an input record or read and write the field of
an output record. You use two member functions,
APT_MultiFieldAccessor::previousField() and
APT_MultiFieldAccessor::nextField(), to change the location of the current
accessor in the array.
You must initialize the APT_MultiFieldAccessor object before you can use it to
access record fields. The following statements define and initialize an
APT_MultiFieldAccessor object as part of a dynamic operator’s runLocally()
override:
APT_MultiFieldAccessor inAccessor(inputInterfaceSchema(0);
inAccessor.setup(&incur);

The first line defines inAccessor, an instance of APT_MultiFieldAccessor. This

constructor takes a single argument containing a schema definition. The second
line uses APT_MultiFieldAccessor::setup() to initialize the multifield accessor to
all components of the operator’s input interface schema, except schema variables.
You must specify the input cursor used to access the current input record.
APT_MultiFieldAccessor lets you access any field in a record schema by using
these two steps:
1 Determine the data types of the fields you want to access
2 Read the field values
For example, to access the first field of the operator’s input interface schema, you
use the following statements:
switch(inAccessor.type())
{
case APT_AccessorBase::eInt8:
//access field
break;
case APT_AccessorBase::eUInt8:
// access field
break;
case APT_AccessorBase::eInt16:
.
.
.
default;
}

1 50 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Examples of Operator Derivations With Options

To read the field values, you use the following APT_MultiFieldAccessor member
functions:
• getInt8()
• getUInt8()
• getInt16()
• getUInt16()
• getInt32()
• getUInt32()
• getInt64()
• getUInt64()
• getSFloat()
• getDFloat()
• getString
• getUstring
• getStringField()
• getUStringField()
• getRawField()
• getDate()
• getTime()
• getTimeStamp()
Note that getString() and getUString() return a copy of the field, and
getStringField() and getUStringField() return a reference to the field. You use
overloads of APT_MultiFieldAccessor::setValue() to write to the fields of a
record of an output data set.
After processing the first field, you update the APT_MultiFieldAccessor object to
the next field using the statement:
inAccessor.nextField();

Orchestrate 7.0 Developer Guide 1 51

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Example: Creating an Operator with a Dynamic Interface

The following figure shows an example of an operator with a dynamic interface
schema:

input data set

a:type;b:type; inRec:*;

ADynamicOperator
outRec:*;

output data set

Using this operator, a user can specify up to two fields of the operator’s input
interface schema. The constructor for this operator takes as input a string defining
these two fields. This string must be incorporated into the complete schema
definition statement required by setInputInterfaceSchema() in
describeOperator().
For example, if the constructor is called as:
ADynamicOperator myOp("field1:int32; field2:int16; ");

the complete input schema definition for this operator would be:
"record (field1:int32; field2:int16; inRec:*;) "

ADynamicOperator is derived from APT_Operator, as shown in Table 12:

Table 12 Example of an Operator with a Dynamic Interface

Code

#include <apt_framework/orchestrate.h>
#include <strstream.h>

class ADynamicOperator : public APT_Operator

{
APT_DECLARE_RTTI(ADynamicOperator);
APT_DECLARE_PERSISTENT(ADynamicOperator);

public:
8 ADynamicOperator(char * inSchema);
9 ADynamicOperator();

protected:
virtual APT_Status describeOperator();
virtual APT_Status runLocally();
virtual initializeFrom Args_();

1 52 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Table 12 Example of an Operator with a Dynamic Interface (continued)

private:
15 APT_String inputSchema;

16 #define ARGS_DESC "{. . .}”

17 APT_DEFINE_OSH_NAME(ADynamicOperator, exampleOp, ARGS_DESC);
18 APT_IMPLEMENT_RTTI_ONEBASE(ADynamicOperator, APT_Operator);
19 APT_IMPLEMENT_PERSISTENT(ADynamicOperator);

20 ADynamicOperator::ADynamicOperator(char * inSchema));
{
inputSchema = APT_string(“record(“) + inSchema + “inRec:*;)”;
}

24 APT_Status ADynamicOperator::initializeFromArgs_(const APT_PropertyList

&args, APT_Operator::InitializeContext context)
{
return APT_StatusOk;
}

APT_Status ADynamicOperator::describeOperator()
{
setInputDataSets(1);
setOutputDataSets(1);

setInputInterfaceSchema(inputSchema.data(), 0);
setOutputInterfaceSchema(“record (outRec:*;)”, 0);

declareTransfer(“inRec”, “outRec”, 0, 0);

return APT_StatusOK;
}

APT_Status ADynamicOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);

43 APT_MultiFieldAccessor inAccessor(inputInterfaceSchema(0));

44 inAccessor.setup(&inCur);

while (inCur.getRecord())
{
transfer(0);

Orchestrate 7.0 Developer Guide 1 53

Chapter 1 Creating Operators Examples of Operator Derivations With Options

Table 12 Example of an Operator with a Dynamic Interface (continued)

outCur.putRecord();
{

return APT_StatusOk;
}
52 void ADynamicOperator::serialize(APT_Archive& archive, APT_UInt8)
{
archive || inputSchema;
}

Comments

8 Declare a constructor that takes a string defining part of the input interface schema.

9 Declare a default constructor. This constructor is required by the Orchestrate

persistence mechanism.

15 Define inputSchema, an instance of APT_String, to hold the complete input interface

schema of this operator. The complete input interface schema is equal to the schema
passed in to the constructor, plus the definition of the schema variable inRec.

16 See “Argument-List Descriptions” on page 5-3 and the header file,

$APT_ORCHHOME/include/apt_util/argvcheck.h, for documentation on the
ARGS_DESC string.

17 With APT_DEFINE_OSH_NAME, you connect the class name to the name used to
invoke the operator from osh and pass your argument description string to
Orchestrate. Orchestrate’s argument-checking facility is fully described in Chapter 5,
“Orchestrate’s Argument-List Processor”. Also see
$APT_ORCHHOME/include/apt_framework/osh_name.h for documentation on this macro.

18 - APT_IMPLEMENT_RTTI_ONEBASE and APT_IMPLEMENT_PERSISTENT are

19 required macros that implement run-time type information and persistent object
processing.

20 The constructor creates the complete schema definition statement and writes it to the
private data member inputSchema.

24 With your override of initializeFromArgs_(), you transfer information from the

operator's arguments to the class instance, making it osh aware. Because there are no
arguments for this example, initializeFromArgs_() simply returns APT_StatusOk. See
“Property List Encoding of Operator Arguments” on page 5-19 and the header file,
$APT_ORCHHOME/include/apt_framework/operator.h for documentation on this
function.

1 54 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Customizing Job Monitoring Messages

Table 12 Example of an Operator with a Dynamic Interface (continued)

43 Define inAccessor, an instance of APT_MultiFieldAccessor. The constructor for
APT_MultiFieldAccessor takes a single argument containing a schema definition. In
this example, you use APT_Operator::inputInterfaceSchema() to return the schema
of the operator’s input interface as set by setInputInterfaceSchema() in
describeOperator().

44 Use APT_MultiFieldAccessor::setup() to initialize the multifield accessor for all

components of the operator’s input interface schema.

52 ADynamicOperator defines a single data member, inputSchema. You must serialize

inputSchema within serialize()

Customizing Job Monitoring Messages

The Orchestrate Job Monitor reports on the current state of your job and supplies
information about the operators in your data flow. The data it supplies falls into
four categories: job status, monitor, metadata, and summary. The information is
given in XML notation. See the Orchestrate Job Monitoring chapter in the
Orchestrate 7.0 User Guide for more information about this facililty.
For the operators you write, you can customize the Orchestrate Job Monitor
metadata and summary response messages. This facility is described in the section
“Customizing Metadata and Summary Response Messages” on page 1-55.
You can also asynchronously generate custom messages that do not fall into the
response categories of the Job Monitor. The categories for these messages are
custom_report and custom_instance. These messages are described in
“Generating Custom Messages Asynchronously” on page 1-57.

Customizing Metadata and Summary Response

Messages
Use the APT_CustomReportInfo class to add your own custom information to
Orchestrate’s metadata and summary response messages. This class is declared in
$APT_ORCHHOME/include/apt_util/custreport.h.

These are the arguments to APT_CustomReportInfo:

APT_CustomReportInfo(
const APT_STD::string &name,
const APT_STD::string &desc,
const APT_STD::string &value);

Orchestrate 7.0 Developer Guide 1 55

Chapter 1 Creating Operators Customizing Job Monitoring Messages

Specifying custom information through the APT_CustomReportInfo class

ensures that your custom XML information appears in this
name/description/value format:
<custom_info Name=”custName” Desc=”description of custName”>
add any information here,including information in XML notation
</custom_info>

Orchestrate maintains internal storage vectors for Job Monitor metadata and
summary messages. You add your custom information to the storage vectors using
these two functions which are declared in
$APT_ORCHHOME/include/apt_framework/operator.h:
void addCustomMetadata(APT_CustomReportInfo &);
void addCustomSummary(APT_CustomReportInfo &);

The function addCustomMetadata() must be called within describeOperator();

and addCustomSummary() must be executed after the runLocally() processing
loop. You can add any number of entries to the storage vectors. Your entries are
appended to Orchestrate’s Job Monitoring metadata and summary messages.
Here are examples of Orchestrate-generated metadata and summary messages
with appended custom information:
<response type=”metadata" jobID="17518">
<component ident="custom_import">
<componentstats startTime="2002-06-28 22:087:05"/>
<linkstats portNum="0" portType="out"/>
...
<resource type=”in”>
resource description
</resource>
...
<custom_info Name="custName" Desc="description of custName">
add any information here,including data in XML notation
</custom_info>
...
</component>
...
</response>

<response type="summary" jobId="17518">

<component ident="custom_import" pid="17550">
<componentstats currTime="2002-06-28 22:08:09" percentCPU="10.0"/>
<linkstats portNum="0" portType="out" recProcessed="50000"/>
...
<custom_info Name="custName" Desc="description of custName">
add any information here,including data in XML notation
</custom_info>
...
</component>
...
</response>

1 56 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Declaring External Data for MetaStage

Generating Custom Messages Asynchronously

Orchestrate provides two functions that enable you to asynchronously generate
custom messages that don't fall into any of the Job Monitoring categories. The
functions are declared in $APT_ORCHHOME/include/apt_framework/operator.h:
void sendCustomReport(vector<CustomReportInfo>& customInfo);
void sendCustomInstanceReport(vector<CustomReportInfo>& customInfo);

Call sendCustomReport() before the execution of runLocally(), which is before

the parallel execution of an operator, to generate a message of type
custom_report.

Call sendCustomInstanceReport() during the execution of runLocally(), which is

during parallel execution, to generate a message of type custom_instance_report.
This function provides operator instance information.
Here are examples of output generated by sendCustomReport() and
sendCustomInstanceReport():
<response type="custom_report" jobId="17518">
<component ident="operator_ident">
<custom_info Name="custName" Desc="description of custName">
add any information here,including data in XML notation
</custom_info>
...
</component>
</response>

<response type="custom_instance_report" jobId="17518">

<component ident="operator_ident" instance="2">
<custom_info Name="custName" Desc="description of custName">
add any information here,including data in XML notation
</custom_info>
...
</component>
</response>

Declaring External Data for MetaStage

When your source or sink operator uses a resource that has been touched by a
process outside of Orchestrate, use the declareExternalData() method to supply
metadata for that resource. Examples of resources include files, RDBMS tables,
and lookup tables. Orchestrate makes the metadata available to MetaStage which
uses it to describe the data flow.
From your override of describeOperator(), you call declareExternalData() once
for each external resource used by your operator. These are the arguments to
declareExternalData():

Orchestrate 7.0 Developer Guide 1 57

Chapter 1 Creating Operators Declaring External Data for MetaStage

void declareExternalData
(externalDataDirection direction ,
APT_String computer ,
APT_String softwareProduct ,
APT_String storeType ,
APT_String storeValue ,
APT_String schemaType ,
APT_String schemaValue ,
APT_String collectionType ,
APT_String collectionValue );

externalDataDirection is an enum with two values: eDataSource and eDataSink.

Specify eDataSource if the external data is to be used as input, and eDataSink if
the external data is to be used as output. Use the string arguments to to supply
distinctive process metadata that describes the external resources used by your
operator. Orchestrate makes your metadata available to MetaStage in this form:
Computer="$computer"
SoftwareProduct="$softwareProduct"
DataStore($storeType)="$storeValue"
DataSchema($schemaType)="$schemaValue"
DataCollection($collectionType)="$collectionValue"

For example, here is a call to declareExternalData() within describeOperator():

APT_ExampleImportOperator::describeOperator()
{ ...
hostname = getConductorNode().fastName();
dir = "/tmp";
filename = "importfile.txt";
declareExternalData(eDataSource, hostname, "FileSystem",
"Directory", dir, "File", filename,
"File", filename);
}

Here is the output metadata description from the call:

<resource type="in">
Computer="Conductormachine.ascentialsoftware.com”
SoftwareProduct="FileSystem"
DataStore (Directory)="/tmp”
DataSchema(File)="importfile.txt"
DataCollection(File)="importfile.txt"
</resource>

1 58 Orchestrate 7.0 Developer Guide

Chapter 1 Creating Operators Declaring External Data for MetaStage

Table 13 lists the Orchestrate source and sink operators and the values of their
metadata categories:
Table 13 Metadata for Orchestrate Source/Sink Operators
Operator Source Computer Software Data Store Data Data
or Sink Product Schema Collection
datset(in) Source Conductor FileSystem Directory File File
db2loader Source Server Name DB2 DB Name Owner Table
db2lookup Source Server Name DB2 DB Name Owner Table
db2readop Source Server Name DB2 DB Name Owner Table
hlpread Source Server Name Informix DB Name Owner Table
import Source File Host FileSystem Directory File File
infxread Source Server Name Informix DB Name Owner Table
oralookup Source Server Name Oracle Oracle_SID Owner Table
oraread Source Server Name Oracle Oracle_SID Owner Table
sasdriver(in) Source File Host SAS Directory File File
sasimport Source File Host SAS Directory File File
teraread Source Server Name Teradata DB Name Owner Table
xpsread Source Server Name Informix DB Name Owner Table
dataset(out) Sink Conductor FileSystem Directory File File
db2writeop Sink Server Name DB2 DB Name Owner Table
export Sink File Host FileSystem Directory File File
hplwrite Sink Server Name Informix DB Name Owner Table
infxwrite Sink Server Name Informix DB Name Owner Table
oraupsert Sink Server Name Oracle Oracle_SID Owner Table
orawrite Sink Server Name Oracle Oracle_SID Owner Table
sasdriver(out) Sink File Host SAS Directory File File
sasexport Sink File Host SAS Directory File File
terawrite Sink Server Name Teradata DB Name Owner Table
xpswrite Sink Server Name Informix DB Name Owner Table

Orchestrate 7.0 Developer Guide 1 59

Chapter 1 Creating Operators Declaring External Data for MetaStage

1 60 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

2
Creating Composite Operators
Describes the steps required to define composite operators.

Overview 2 1
A Composite Operator with Two Suboperators 2 2
The APT_CompositeOperator Class Interface 2 4
Composite Operator Example Derivation 2 5

Overview
Composite operators consist of one or more operators that collectively perform a
single action. By combining multiple operators into a composite operator, the
operator user is presented with a single interface, making application
development easier. The operators that make up a composite operator are called
suboperators.
This chapter describes the steps required to define composite operators, using
classes you derive from the Orchestrate base class APT_CompositeOperator. You
follow much the same procedure for creating operators derived from
APT_CompositeOperator as you do for operators derived from APT_Operator.
See Chapter 1, “Creating Operators” for more information on the basic procedure
for creating operators.

21
Chapter 2 Creating Composite Operators A Composite Operator with Two Suboperators

A Composite Operator with Two Suboperators

The following figure shows a composite operator made up of two suboperators:

composite operator
input data set step

import

... suboperators

sort
virtual
data set filter
composite
operator

...

export

output data set

This composite operator contains two suboperators. Each suboperator takes a

single input, creates a single output, and defines its own partitioning method.
Node pool constraints placed on a composite operator propagate to the
suboperators; node map constraints do not propagate to the suboperators. See the
Orchestrate 7.0 User Guide for information on using node maps.
The figure above shows a temporary data set between the two suboperators. This
data set is invisible to the user of the operator because it is created solely to be
used as the output of the first suboperator and the input to the second.
Typically, you create composite operator for one of two reasons:
• To hide complexity from a user of the operator
• To allow a second operator to use a data set already partitioned by a previous
operator

22 Orchestrate 7.0 Developer Guide

Chapter 2 Creating Composite Operators A Composite Operator with Two Suboperators

As an example of the first reason, you may want to create a single operator that
removes all duplicate records from a data set. To do this, you need an operator to
sort the data set so that the duplicate records are adjacent. One solution is to select
a sort operator followed by the remdup operator which removes duplicate
records. By creating a single composite operator from these two operators, you
have to instantiate and reference only a single operator.
The following figure illustrates the second reason for creating composite
operators: allowing an operator to use data already partitioned by the previous
operator. Here, you can see that the first suboperator partitions the input data
sets. The second suboperator does not define a partitioning method; instead, it
uses the already partitioned data from the previous operator.

input data sets

composite operator

...
suboperators

...

output data set

You could use a composite operator to create a join and filter operator. In this
case, the first suboperator would join the two data sets and the second
suboperator would filter records from the combined data set. The second
suboperator, filter, simply processes the already partitioned data set. In order for
this to occur, you must specify same as the partitioning method of the second
suboperator.

Orchestrate 7.0 Developer Guide 23

Chapter 2 Creating Composite Operators The APT_CompositeOperator Class Interface

The APT_CompositeOperator Class Interface

The figure below shows the interface to APT_CompositeOperator.
The pure virtual function describeOperator() is included in the protected
interface. You must override this function.
The header file for this class, $APT_ORCHHOME/incude/apt_framework/composite.h,
contains descriptions of the functions. You can also browse the Orchestrate class
hierarchy using HTML-documented header files. Start your search with the
index.html file in $APT_ORCHHOME/doc/html/.

APT_CompositeOperator class interface

protected:
describeOperator() = 0 redirectInput() setSubOpsSeparatelyConstrained()
markSubOperator() redirectOutput()

The setSubOpsSeparatelyConstrained() function can be called by the composite

operator’s describeOperator() function to turn off the automatic propagation of
the composite operator’s availableNodes() to the sub-operators.
You must override APT_Operator::describeOperator() in your derived class.
However, you follow different guidelines than when deriving directly from
APT_Operator. When deriving from APT_CompositeOperator, you must
include calls to the following functions within describeOperator():
• APT_Operator::setInputDataSets() and
APT_Operator::setOutputDataSets(), to specify the number of input and
output data sets
• APT_CompositeOperator::markSubOperator(), to identify each suboperator
of this composite operator
• APT_CompositeOperator::redirectInput(), to specify the suboperator that
processes each data set input to the composite operator
• APT_CompositeOperator::redirectOutput(), to specify the suboperator that
supplies the output data sets of the composite operator
You can include calls to APT_Operator::addNodeConstraint() or
APT_Operator::addResourceConstraint() within describeOperator() to specify
constraints on any suboperator. You use these functions to constrain the
composite operator and, therefore, all suboperators.
You cannot include the following member functions of APT_Operator within the
describeOperator() function of a composite operator:
• setKind()

24 Orchestrate 7.0 Developer Guide

Chapter 2 Creating Composite Operators Composite Operator Example Derivation

• setInputInterfaceSchema() and setOutputInterfaceSchema()

• declareTransfer()
• setPartitionMethod()
• setCollectionMethod()

Composite Operator Example Derivation

The following code shows the class definition for a composite operator derived
from APT_CompositeOperator:

Table 14 APT_CompositeOperator Derivation()

Code

#include <apt_framework/orchestrate.h>

2 class RemoveDuplicatesOperator: public APT_CompositeOperator

{
APT_DECLARE_RTTI(RemoveDuplicatesOperator);
APT_DECLARE_PERSISTENT(RemoveDuplicatesOperator);

public:
RemoveDuplicatesOperator();

protected:
9 virtual APT_Status describeOperator();

private:
11 APT_DataSet * tempDS;
12 SortOperator * sortOp;
13 RemoveOperator * removeOp;
}
14 #define ARGS_DESC "{. . .};
15 APT_DEFINE_OSH_NAME(RemoveDuplicatesOperator, RemoveDups, ARGS_DESC);
16 APT_IMPLEMENT_RTTI_ONEBASE(RemoveDuplicatesOperator,
APT_CompositeOperator);
17 APT_IMPLEMENT_PERSISTENT(RemoveDuplicatesOperator);

18 APT_Status ExampleOperator::initializeFromArgs_(const APT_PropertyList

&args, APT_Operator::InitializeContext context)
{
return APT_StatusOk;
}
22 RemoveDuplicatesOperator::RemoveDuplicatesOperator()
{

Orchestrate 7.0 Developer Guide 25

Chapter 2 Creating Composite Operators Composite Operator Example Derivation

Table 14 APT_CompositeOperator Derivation() (continued)

24 tempDS = new APT_Dataset;
25 sortOp = new SortOperator:
26 removeOp = new RemoveOperator;
27 sortOp->attachOutput(tempDS, 0);
28 removeOp->attachInput(tempDS, 0);
}

void RemoveDuplicatesOperator::serialize(APT_Archive& archive,

APT_UInt8)
{}

APT_Status RemoveDuplicatesOperator::describeOperator()
{
setInputDataSets(1);
setOutputDataSets(1);

36 markSubOperator(sortOp);
37 markSubOperator(removeOp);

38 redirectInput(0, sortOp, 0);

39 redirectOutput(0, removeOp, 0);

return APT_StatusOk;
}
}

Comments

2 Derive this class from APT_CompositeOperator.

9 Override describeOperator(). You cannot override APT_Operator::runLocally for a

composite operator.

11 Define a pointer to a temporary data set that connects the two suboperators. This
data set will be dynamically allocated in the constructor for RemoveDuplicates.

12 Dynamically allocate the suboperators of the composite operator. This line defines
sortOp, a pointer to a SortOperator, as a private data member of the composite
operator. You instantiate SortOperator in the constructor for RemoveDuplicates.

13 Define removeOp, a pointer to a RemoveOperator, as a private data member of the

composite operator.

26 Orchestrate 7.0 Developer Guide

Chapter 2 Creating Composite Operators Composite Operator Example Derivation

Table 14 APT_CompositeOperator Derivation() (continued)

14 See “Argument-List Descriptions” on page 5-3 and the header file,
$APT_ORCHHOME/include/apt_util/argvcheck.h, for documentation on the
ARGS_DESC string.Orchestrate’s argument-checking facility is fully described in
Chapter 5, “Orchestrate’s Argument-List Processor”.

15 With APT_DEFINE_OSH_NAME, you connect the class name to the name used to
invoke the operator from osh and pass your argument description string to
Orchestrate. See $APT_ORCHHOME/include/apt_framework/osh_name.h for
documentation on this macro. Orchestrate’s argument-checking facility is fully
described in Chapter 5, “Orchestrate’s Argument-List Processor”.

16- 17 APT_IMPLEMENT_RTTI_ONEBASE and APT_IMPLEMENT_PERSISTENT are

required macros that implement run-time information and persistent object
processing.

18 With your override of initializeFromArgs_(), you transfer information from the

operator's arguments to the class instance, making it osh aware. Because there are no
arguments for this example, initializeFromArgs_() simply returns APT_StatusOk.
See “Property List Encoding of Operator Arguments” on page 5-19 and the header
file, $APT_ORCHHOME/include/apt_framework/operator.h for documentation on this
function. Orchestrate’s argument-checking facility is fully described in Chapter 5,
“Orchestrate’s Argument-List Processor”.

22 The constructor for RemoveDuplicatesOperator must instantiate the suboperators and

tie together the suboperators and temporary data sets that make up the composite
operator. In this example, you must specify tempDS as the output data set of sortOp
and the input data set of removeOp.

24 Dynamically allocate a data set.

25 -26 Dynamically allocate a SortOperator object and a RemoveOperator object.

Subops must be allocated dynamically. They are deleted by Orchestrate using the
default destructor, and must not be deleted or serialized by the composite operator.

27 Specify tempDS as the output data set of sortOp. Note that you need not specify an
input data set. The function APT_CompositeOperator::redirectInput() in the
override of APT_Operator::describeOperator() specifies the input data set. See the
override of
APT_Operator::describeOperator().

28 Specify tempDS as the input data set of removeOp. Note that you do not have to
specify an output data set. The function
APT_CompositeOperator::redirectOutput() in the override of
APT_Operator::describeOperator() specifies the output data set.

Orchestrate 7.0 Developer Guide 27

 Table 14 APT_CompositeOperator Derivation() (continued)

36 Specify sortOp as a suboperator.

37 Specify removeOp as a suboperator.

38 Use APT_CompositeOperator::redirectInput() to specify that the data set input to

sortOp corresponds to the first data set input to the composite operator.

39 Use APT_CompositeOperator::redirectOutput() to specify that the data set output

by removeOp corresponds to the first data set output by the composite operator.
 Orchestrate 7.0 Developer Guide

3
Creating Subprocess
Operators
Tells you how to incorporate a third-party executable into your
Orchestrate application.

Overview 3 1
The APT_SubProcessOperator Class Interface 3 2
Deriving from APT_SubProcessOperator 3 3
APT_SubProcessOperator::runSource() 3 3
APT_SubProcessOperator::runSink() 3 4
APT_SubProcessOperator::commandLine() 3 5
Subprocess Operator Example Derivation 3 5

Overview
You can incorporate a third-party application package into your Orchestrate
application even if it was not written for execution in a parallel environment. It is
processed in parallel in Orchestrate.
To execute a third-party application, you create a subprocess operator, which is
derived from the Orchestrate abstract base class APT_SubProcessOperator. A
subprocess operator has all the characteristics of a standard Orchestrate operator,
including:
• Taking data sets as input
• Writing to data sets as output
• Using cursors to access records

31
Chapter 3 Creating Subprocess Operators The APT_SubProcessOperator Class Interface

• Using an view or modify adapter

You need not define an override of APT_Operator::runLocally() for a subprocess
operator as you would for other operators. Instead of using runLocally() to
process records of a data set, a subprocess operator:
1 Passes records or record fields of the input data set to the third-party
application for processing
2 Reads back results from the third-party application
3 Writes the results to records of the output data set
The linked boxes on the right-hand side of the the following figure show how
Orchestrate executes a subprocess operator:

runSource() = 0

UNIX pipe

stdin
subprocess operator stderr
for commandLine() = 0
third-party application error log
third-party application

stdout
UNIX pipe

runSink() = 0

The subprocess operator shown in this figure takes a single input data set and
writes its results to a single output data set. The third-party application must be
configured to take its input from stdin, to write any output to stdout, and to write
any error messages to stderr.

The APT_SubProcessOperator Class Interface

The following figure shows the interface to APT_SubProcessOperator. You use
the member functions to copy records or record fields to and from a third-party
application for processing.

32 Orchestrate 7.0 Developer Guide

Chapter 3 Creating Subprocess Operators Deriving from APT_SubProcessOperator

Three pure virtual functions, commandLine(), runSink(), and runSource(), are

included in the protected interface. You must override these functions.
The header file for this class, $APT_ORCHHOME/include/apt_framework/subproc.h,
contains descriptions of the functions. You can also browse the Orchestrate class
hierarchy using HTML-documented header files. Start your search with the
index.html file in $APT_ORCHHOME/doc/html/.

APT_SubProcessOperator class interface

protected:
constructor getReadBuffer() readTerminatedFromSubPro-
destructor getTerminatedReadBuffer() cess()
commandLine() = 0 getWriteBuffer() runSink() = 0
flushToSubProcess() readFromSubProcess() runSource() = 0

Deriving from APT_SubProcessOperator

APT_SubProcessOperator::runSource()
The function runSource() writes a record, a record field, or any other information
to the third-party application. As part of the write operation, runSource() allows
you to perform preprocessing of the records before they are passed to the third-
party application. Possible preprocessing tasks include rearranging record fields
and stripping off unnecessary fields.
Within your override of runSource(), you use the function
APT_SubProcessOperator::writeToSubProcess() to copy a buffer to the third-
party application through stdin.
While APT_SubProcessOperator::writeToSubProcess() can be used to copy any
type of data stored in a buffer, you commonly use the function to copy an entire
record or a portion of a record.
To get the current input record into a buffer, you perform these steps:
1 Call APT_Operator::getTransferBufferSize() to determine the size of the
buffer required to hold the current input record. This function determines the
size using a schema variable in the input interface schema of the operator.
To use APT_Operator::getTransferBufferSize(), you must have declared a
transfer from a schema variable in the input interface schema of the
APT_SubProcessOperator to a schema variable in the output interface
schema of the operator. APT_Operator::getTransferBufferSize() uses the
information from the transfer to determine buffer size.

Orchestrate 7.0 Developer Guide 33

Chapter 3 Creating Subprocess Operators Deriving from APT_SubProcessOperator

2 Create a buffer large enough to hold the current input record and any
additional information that you want to add to the record.
3 Call APT_Operator::transferToBuffer() to transfer the current input record to
the buffer you created in step 2.
4 Perform any preprocessing on the record required by the third-party
application.
5 Call APT_SubProcessOperator::writeToSubProcess() to copy the buffer to
the third-party application. This application must be configured to receive
inputs over stdin.

APT_SubProcessOperator::runSink()
The function runSink() reads a buffer back from a third-party application. The
returned buffer may contain a record, record fields, results calculated from a
record, and any other output information. runSink() allows you to perform any
post-processing on the results after getting them back from
APT_SubProcessOperator.
To read a fixed-length buffer back from the third-party application, you perform
the following steps:
1 Determine the buffer length. Typically, you call
APT_SubProcessOperator::readFromSubProcess() to read a fixed-length
buffer containing the length of the results buffer. The buffer is read from the
stdout of the subprocess.
2 Allocate a buffer equal to the length determined in step 1.
3 Call APT_SubProcessOperator::readFromSubProcess() again, this time
reading the fixed-length buffer from the third-party application.
To read a variable-length buffer, you perform the following steps:
1 Call APT_SubProcessOperator::getReadBuffer() to read a block of data back
from the subprocess. This function returns a pointer to a buffer containing
data read from the subprocess.
2 Parse the buffer to determine field and record boundaries. Process the buffer
as necessary.
To read a character-delimited buffer call
APT_SubProcessOperator::getTerminatedReadBuffer() to read a block of data
up to a specified delimiter back from the subprocess. This function returns a
pointer to a buffer containing data read from the subprocess.
To write the returned buffer to an output data set, you typically would call
APT_Operator::transferFromBuffer(). This function transfers the results buffer to
the output record of the operator.

34 Orchestrate 7.0 Developer Guide

Chapter 3 Creating Subprocess Operators Subprocess Operator Example Derivation

APT_SubProcessOperator::commandLine()
You override the pure virtual function commandLine() to pass a command line
string to the third-party application. This string is used to execute the subprocess.
As part of the command line, you must configure the subprocess to receive all
input over stdin and write all output to stdout. Orchestrate calls this function
once to invoke the third-party application on each processing node of your
system.
For example, the UNIX gzip command takes its input from standard input by
default. You can use the -c option to configure gzip to write its output to standard
output. An example command line for gzip would be:
"/usr/local/bin/gzip -c"

Including any redirection symbols, for either input to or output from gzip, causes
an error.

Subprocess Operator Example Derivation

This example creates a subprocess operator that converts a five-digit zip code into
a nine-digit zip code. The operator uses a fictional UNIX utility, ZIP9, to perform
the conversion. The following figure shows this operator:

input data set

street:string; city:string; state:string; zip5:string;

Zip9Operator

street:string; city:string; state:string; zip9:string;

output data set

Zip9Operator takes as input a data set containing exactly four fields, one of which
contains a five-digit zip code. As output, this operator creates a data set with
exactly four fields, one of which is a nine-digit zip code.
The input strings cannot contain commas because the UNIX utility ZIP9 takes as
input comma-separated strings. Also, the total line length of the strings must be
less than or equal to 80 characters.

Orchestrate 7.0 Developer Guide 35

Chapter 3 Creating Subprocess Operators Subprocess Operator Example Derivation

The code in Table 15 below contains an APT_SubProcessOperator derivation.

Table 15 APT_SubProcessorOperator Derivation()

Code

#include <apt_framework/orchestrate.h>

2 class Zip9Operator: public APT_SubProcessOperator

{
4 APT_DECLARE_RTTI(Zip9Operator);
5 APT_DECLARE_PERSISTENT(Zip9Operator);

public:
Zip9Operator();

protected:
9 virtual APT_Status describeOperator();
10 virtual APT_String commandLine() const;
11 virtual APT_Status runSink();
12 virtual APT_Status runSource();

13 #define ARGS_DESC”{. . .}”

14 APT_DEFINE_OSH_NAME(Zip9Operator, zip9, ARGS_DESC);
15 APT_IMPLEMENT_RTTI_ONEBASE(Zip9Operator, APT_SubProcessOperator);
16 APT_IMPLEMENT_PERSISTENT(Zip9Operator);

17 Zip9Operator::Zip9Operator()
{}

19 APT_Status ExampleOperator::initializeFromArgs_(const APT_PropertyList

&args, APT_Operator::InitializeContext context)
{
return APT_StatusOk;
}

APT_Status Zip9Operator::describeOperator()
{
setKind(APT_Operator::eParallel);
setInputDataSets(1);
setOutputDataSets(1);

28 setInputInterfaceSchema(“record”
“(street:string; city:string; state:string; zip5:string;), 0);

29 setOutputInterfaceSchema(“record”
“(street:string; city:string; state:string; zip5:string;), 0);

36 Orchestrate 7.0 Developer Guide

Chapter 3 Creating Subprocess Operators Subprocess Operator Example Derivation

Table 15 APT_SubProcessorOperator Derivation() (continued)

return APT_StatusOk;
}

32 APT_String Zip9Operator::commandLine() const

{
return “ZIP9”;
}

36 APT_Status Zip9Operator::runSource()
{
APT_InputCursor inCur;
setupInputCursor (&inCur, 0);

APT_InputAccessorToString street(“street”, &inCur);

APT_InputAccessorToString city(“city”, &inCur);
APT_InputAccessorToString state(“state”, &inCur);
APT_InputAccessorToString zip5(“zip5”, &inCur);

char linebuf[80];
while (inCur.getRecord())
{
// This code builds a comma-separated string containing the street,
// city, state, and 5-digit zipcode, and passes it to ZIP9.

char * write = linebuf;

memcpy(write, street->content(), street->length());
write += street->length();
*write++ = ',';

memcpy(write, city->content(), city->length());

write += city->length();
*write++ = ',';

memcpy(write, state->content(), state->length());

write += state->length();
*write++ = ',';

memcpy(write, zip5->content(), zip5->length());

write += zip5->length();
*write++ = '\n';

size_t lineLength = write - linebuf;

APT_ASSERT(lineLength <= 80);

writeToSubProcess(linebuf, lineLength);
}

Orchestrate 7.0 Developer Guide 37

Chapter 3 Creating Subprocess Operators Subprocess Operator Example Derivation

Table 15 APT_SubProcessorOperator Derivation() (continued)

return APT_StatusOk;
}

66 APT_Status Zip9Operator::runSink()
{
APT_OutputCursor outCur;
setupOutputCursor(&outCur, 0);

APT_OutputAccessorToString street(“street”, &outCur);

APT_OutputAccessorToString city(“city”, &outCur);
APT_OutputAccessorToString state(“state”, &outCur);
APT_OutputAccessorToString zip9(“zip9”, &outCur);

char linebuf[80];
while (1)
{
// read a single line of text from the subprocess
size_t lineLength = readTerminatedFromSubProcess(
linebuf, '\n', 80);
if (lineLength == 0) break;

char* scan = linebuf;

char* streetStr - scan;
while (*scan != ',')
{
scan++;
APT_ASSERT(*scan != '\n');
APT_ASSERT(scan - linebuf < 80);
}
size_t streetLen = scan - streetStr;
street->assignFrom(streetStr, streetLen);
scan++
char* cityStr = scan;
while (*scan != ',')
{
scan++;
APT_ASSERT(*scan != '\n');
APT_ASSERT(scan - linebuf < 80);
}
size_t cityLen = scan - cityStr;
city->assignFrom(cityStr, cityLen);
scan++;

char* stateStr = scan;

while (*scan != ',')
{
scan++;

38 Orchestrate 7.0 Developer Guide

Chapter 3 Creating Subprocess Operators Subprocess Operator Example Derivation

Table 15 APT_SubProcessorOperator Derivation() (continued)

APT_ASSERT(*scan != '\n');
APT_ASSERT(scan - linebuf < 80);
}
size_t stateLen = scan - stateStr;
state->assignFrom(stateStr, stateLen);
scan++’

char* zipStr = scan;

while (*scan != '\n')
{
scan++;
APT_ASSERT(scan - linebuf < 80);;
}
size_t zipLen = scan - zipStr;
zip9->assignFrom(zipStr, zipLen);
outCur.putRecord();
}
return APT_StatusOk;
}

112 void Zip9Operator::serialize(APT_Archive& archive, APT_UInt8)

{}

Comments

2 Derive this class from APT_SubProcessOperator.

4 The macro APT_DECLARE_RITTI() is required to support run-time type

information.

5 the macro APT_DECLARE_PERSISTENT() is required for persistence support.

This macro also inserts a declaration for the APT_Persistent::serialize() function
required by the Orchestrate persistence mechanism.

9 Specify the override of APT_Operator::describeOperator().

10 Specify the override of APT_SubProcessOperator::commandLine().

11 Specify the override of APT_SubProcessOperator::runSink().

12 Specify the override of APT_SubProcessOperator::runSource().

13 See “Argument-List Descriptions” on page 5-3 and the header file,

$APT_ORCHHOME/include/apt_util/argvcheck.h, for documentation on the
ARGS_DESC string. Orchestrate’s argument-checking facility is described in
Chapter 5, “Orchestrate’s Argument-List Processor”.

Orchestrate 7.0 Developer Guide 39

Chapter 3 Creating Subprocess Operators Subprocess Operator Example Derivation

Table 15 APT_SubProcessorOperator Derivation() (continued)

14 With APT_DEFINE_OSH_NAME, you connect the class name to the name used
to invoke the operator from osh and pass your argument description string to
Orchestrate. See $APT_ORCHHOME/include/apt_framework/osh_name.h for
documentation on this macro. Orchestrate’s argument-checking facility is
described in Chapter 5, “Orchestrate’s Argument-List Processor”.

15- 16 APT_IMPLEMENT_RTTI_ONEBASE and APT_IMPLEMENT_PERSISTENT

are required macros that implement run-time information and persistent object
processing.

17 Define the default constructor.

19 With your override of initializeFromArgs_(), you transfer information from the

operator's arguments to the class instance, making it osh aware. Because there are
no arguments for this example, initializeFromArgs_() simply returns
APT_StatusOk. See “Property List Encoding of Operator Arguments” on page 5-
19 and the header file, $APT_ORCHHOME/include/apt_framework/operator.h for
documentation on this function. Orchestrate’s argument-checking facility is
described in Chapter 5, “Orchestrate’s Argument-List Processor”.

28 Specify the input interface schema of the first input data set.

29 Specify the output interface schema of the first output data set.

32 The override of APT_SubProcessOperator::commandLine() specifies the

command line for an example UNIX utility ZIP9, a utility to convert a five-digit
zip code to a nine-digit zip code. The function returns the command line for ZIP9.

36 The override of APT_SubProcessOperator::runSource() copies the street, city,

state, and zip5 fields to ZIP9 as a single line of text containing comma-separated
fields.

66 The override of APT_SubProcessOperator::runSink() copies the street, city, state,

and zip9 fields back from F as a single string containing comma-separated fields.
This function then writes the returned fields to the output data set.

112 Zip9Operator does not define any data members; therefore, serialize() is empty.
You must provide serialize() even if it is an empty function.

3 10 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

4
Localizing Messages
Describes the process of changing operator message code so that Orchestrate
messages can be output in multiple languages.

Introduction 4 2
Using the Messaging Macros 4 2
Including Messaging Macros in Your Source File 4 2
Descriptions of the Message Macros 4 3
APT_MSG() 4 3
APT_NLS() 4 4
APT_DECLARE_MSG_LOG() 4 5
Message Environment Variables 4 5
Localizing Message Code 4 5
Using the APT_MSG() Macro 4 5
Example Conversions 4 6
Converting a Message With No Run-Time Variables 4 6
Converting a Message With Run-time Variables 4 6
Converting a Multi-Line Message 4 6
Steps for Converting Your Pre-NLS Messages 4 7
Eliminating Deprecated Interfaces 4 9

41
Chapter 4 Localizing Messages Introduction

Introduction
Orchestrate National Language Support (NLS) makes it possible for you to
process data in international languages using Unicode character sets. Orchestrate
uses International Components for Unicode (ICU) libraries to support NLS
functionality. For information on Orchestrate’s national language support, see
Chapter 7: National Language Support in the Orchestrate 7.0 User Guide; and access
the ICU home page:
http://oss.software.ibm.com/developerworks/opensource/icu/project
A key part of internationalizing Orchestrate is to make it possible for you to
localize its English-language messages. Orchestrate currently provides the
functionality for outputting messages in Japanese. The next section describes the
source code that makes it possible for you to output Orchestrate’s messages in
any language so that Orchestrate can be used thoughout the world.
Orchestrate’s code utilizes the source-code localization functionality of the ICU
library. Orchestrate provides macros that interface with ICU, so that you can
localize Orchestrate without having to deal directly with the ICU substructure.

Using the Messaging Macros

You need to use three macros in the Orchestrate messaging system. They are
declared in $APT_ORCHHOME/include/apt_util/message.h and are described in
“Descriptions of the Message Macros” on page 4-3.

Including Messaging Macros in Your Source File

! To use the macros in a source file:
1 Add this statement to the top of your source file:
#include <apt_util/message.h>

2 After your #include statement, add these two lines:

APT_Error::SourceModule APT_stem Id("CUST");
APT_DECLARE_MSG_LOG(stem , "$Revision:$");

where stem is a string specifying a unique identifier for the source module,
and CUST is the identifier for custom operators. For example:
APT_Error::SourceModule APT_customOpId("CUST");
APT_DECLARE_MSG_LOG(APT_customOp, "$Revision:$")

For more information on this macro, see “APT_DECLARE_MSG_LOG()” on

page 4-5.

42 Orchestrate 7.0 Developer Guide

Chapter 4 Localizing Messages Using the Messaging Macros

Descriptions of the Message Macros

APT_MSG()
This macro issues an Orchestrate message. It can only be called from .C files.
Its syntax is:
APT_MSG(severity , englishString , argumentArray, logOrSourceModule)
Argument descriptions:
1 The severity argument can have one of the following string values:
– "Info" issues an Orchestrate informational message; execution does not
terminate
– "Warning" issues an Orchestrate warning message; execution does not
terminate
– "Error" issues an Orchestrate error message; execution terminates at the
end of the step
– "Fatal" issues an Orchestrate fatal message; execution terminates
immediately
– "Monitor" issues output to the Orchestrate Job Monitor; execution does
not terminate
– "Metadata" issues metadata output; execution does not terminate
2 An example of the englishString argument is:
"The number of output datasets is invalid."

The string cannot contain a right parenthesis followed by a semi-colon ( ); ).

A new-line is automatically appended to the englishString when the message
is issued.
The string may be parameterized to accept run-time values, using {n} syntax.
See the argumentArray description below.
3 The argumentArray argment lists the parameters that are to be resolved at run
time using the APT_Formattable array. When there are no run-time
parameters, argumentArray should be NULL.
An array example is:
APT_Formattable args [] = { numOutDS, opName };
APT_MSG(Error,
"The number of output datasets {0} is invalid for custom "
"operator {1}.", args, errorLog);

The maximum number of run-time elements in the APT_Formattable array is

10. They can be of type char, int, unsigned int, short int, unsigned short int,

Orchestrate 7.0 Developer Guide 43

Chapter 4 Localizing Messages Using the Messaging Macros

long int, unsigned long int, long long, unsigned long long, float, double,
char*, APT_String, APT_Int53, UChar*, or APT_UString.

Note If you include an object in the APT_Formattable array, such as

APT_Identifier, access its member data using the appropriate
methods.
See $APT_ORCHHOME/include/apt_util/fmtable.h for the definition of
APT_Formattable.

An example without run-time parameters is:

APT_MSG(Error, "The number of output datasets is invalid.", NULL,
errorLog);

4 The final argument to APT_MSG() is logOrSourceModule, the object to which

the message is written.
Its value can be an APT_ErrorLog object , an APT_Error::SourceModule
object, or it can be NULL. If the value is NULL, the message is written to the
APT_Error::SourceModule object APT_localErrorSourceModule.
Message indices are generated automatically by the messaging system. See
“Including Messaging Macros in Your Source File” on page 4-2 for how to add a
message to the messaging system.

APT_NLS()
This macro does not issue an Orchestrate message but simply returns an
APT_UString containing the localized version of the englishString. This is
needed to pass localized strings to other functions that output messages. This
macro can be called from .C files only.
Its syntax is:
APT_NLS(englishString, argumentArray )
An example is:
APT_Formattable args [] = { hostname };
error_ = APT_NLS("Invalid hostname: {0}", args);

Here a member variable is set to a localized string that can be output later.
The two arguments to APT_NLS() are identical to the englishString and
argumentArray arguments of APT_MSG(). See “APT_MSG()” on page 4-3 for the
argument descriptions.
If no run-time arguments are needed, the value of the APT_Formattable array
should be NULL.
englishString cannot contain a right parenthesis followed by a semi-colon ( ); ).

44 Orchestrate 7.0 Developer Guide

Chapter 4 Localizing Messages Message Environment Variables

See “Including Messaging Macros in Your Source File” on page 4-2 for how to
add a message to the messaging system.

APT_DECLARE_MSG_LOG()
This macro uniquely identifies a message. It must appear in files that call
APT_MSG() and APT_NLS().
Its syntax is:
APT_DECLARE_MSG_LOG(moduleId , "$Revision:$");

It should always be used in this context:

APT_Error::SourceModule stem Id("CUST");
APT_DECLARE_MSG_LOG(stem , "$Revision:$");

where stem is a string specifying a unique identifier for the source module, and
CUST is the identifier for custom operators. For example:
APT_Error::SourceModule APT_customOpId("CUST");
APT_DECLARE_MSG_LOG(APT_customOp, "$Revision:$")

See “Including Messaging Macros in Your Source File” on page 4-2 for how to
add a message to the messaging system.

Message Environment Variables

The boolean environment variable, APT_ENGLISH_MESSAGES, issues a
duplicate English message for each APT_MSG() macro issued. It can be used to
print English messages next to their corresponding localized messages so that
they can be understood by English-speakers.
When set to 1, the boolean environment variable, APT_MSG_FILELINE, issues
file and line number debug information for all messages issued with the
APT_MSG() macro with severities of Error, Fatal, or Warning.

Localizing Message Code

Using the APT_MSG() Macro
You use the APT_MSG() macro to localize most of the messages in your source
files. “APT_MSG()” on page 4-3 shows its syntax and describes its arguments.
Do not convert APT_MSG_ASSERT() messages.

Orchestrate 7.0 Developer Guide 45

Chapter 4 Localizing Messages Localizing Message Code

Example Conversions
Converting a Message With No Run-Time Variables
This errorLog() message has no runtime variables:
*errorLog() << "There must be at least 1 coordinate in the input
vectors." << endl;
errorLog().logError(APT_CLUSTERQUALITYOP_ERROR_START+2);

Convert it to this APT_MSG() macro code:

APT_MSG(Error,
"There must be at least 1 coordinate in the input vectors.",
NULL, errorLog());

Converting a Message With Run-time Variables

This errorLog() message has three variables that are to be resolved at runtime:
*errorLog() << "Error code " << errorCode << ": Fields " << fieldName1
<< " and " << fieldName2 << " cannot have variable length." <<
endl;
errorLog().logError(APT_CLUSTERQUALITYOP_ERROR_START+4);

Convert this message by declaring the run-time variables in an APT_Formattable

array and designating them in your englishString argument using {} and an index
into the args[] array. Indexing begins at 0.
The converted message is:
APT_Formattable args[] = { errorCode, fieldName1, fieldName2 };
APT_MSG(Error,
"Error code {0}: Fields {1} and {2} cannot have variable length.",
args, errorLog());

Converting a Multi-Line Message

Your source code may have messages that occupy several lines. For example:
*errorLog() << "Output schema has duplicate field names. If the
" << "-flatten key is being used it is likely generating a scalar "
<< "field name that is the same as another input field." << endl;
errorLog().logError(APT_CLUSTERQUALITYOP_ERROR_START+3);

This message uses several stream operators (<<) over three lines; however, it is a
single error message and should be converted using a single APT_MSG() macro:
APT_MSG(Error, "Output schema has duplicate field names. If the "
"-flatten key is being used it is likely generating a scalar "
"field name that is the same as another input field.",
NULL, errorLog());

46 Orchestrate 7.0 Developer Guide

Chapter 4 Localizing Messages Localizing Message Code

Steps for Converting Your Pre-NLS Messages

! Follow these steps to convert your message code:
1 If $APT_ORCHHOME/include/apt_util/message.h is not included in your source
file, add this line to the top of your file:
#include <apt_util/message.h>

2 After your #include statement, add these two lines:

APT_Error::SourceModule APT_stem Id("CUST");
APT_DECLARE_MSG_LOG(stem , "$Revision:$");

where stem is a string specifying a unique identifier for the source module,
and CUST is the identifier for custom operators. For example:
APT_Error::SourceModule APT_customOpId("CUST");
APT_DECLARE_MSG_LOG(APT_customOp, "$Revision:$")

3 Convert all APT_USER_REQUIRE() macros of this form:

APT_USER_REQUIRE(rep_->numInputs_ <= 128,
APT_PERLBUILDOP_ERROR_START+43,
"The number of inputs attached must be no more than 128");

into this:
if (rep_->numInputs_ > 128)
{
APT_MSG(Fatal,
"The number of inputs attached must be no more than 128",
NULL, NULL);
}

The severity for APT_USER_REQUIRE() is always Fatal.

Note that the if statement is the negation of the APT_USER_REQUIRE()
argument. Also note that the log_or_module argument to APT_MSG() is NULL
because no APT_ErrorLog or APT_ErrorLog::SourceModule object is
included in the APT_USER_REQUIRE() macro.
4 Convert all APT_USER_REQUIRE_LONG() macros of this form:
APT_USER_REQUIRE_LONG(rep_->numInputs_ <= 128, localErrorModule,
APT_PERLBUILDOP_ERROR_START+43,
"The number of inputs attached must be no more than 128");

into this:
if (rep_->numInputs_ > 128)
{
APT_MSG(Fatal,
"The number of inputs attached must be no more than 128",
NULL,localErrorModule);
}

The severity for APT_USER_REQUIRE_LONG() is always Fatal.

Orchestrate 7.0 Developer Guide 47

Chapter 4 Localizing Messages Localizing Message Code

Note that the if statement is not identical to the first

APT_USER_REQUIRE_LONG() argument, and that the log_or_module
argument to APT_MSG() is localErrorModule.
5 Convert all APT_DETAIL_LOGMSG() macros of this form:
APT_DETAIL_LOGMSG(APT_Error::eInfo, PM_PROCESSMGR_INDEX_START + 7,
"APT_PMCleanUp::registerFileImpl("
<< name << ", "
<< flagString(dispositionWhenOK) << ", "
<< flagString(dispositionWhenFailed)
<< ") - path registered is " << fullName);

into this:
APT_Formattable args[] = {name, flagString(dispositionWhenOK),
flagString(dispositionWhenFailed), fullName};
APT_MSG(Info,"APT_PMCleanUp::registerFileImpl({0}, {1}, {2})"
" - path registered is {3}", args, NULL);

6 Convert all APT_DETAIL_LOGMSG_LONG() macros of this form:

APT_DETAIL_LOGMSG_LONG(APT_Error::eInfo, localErrorModule,
APT_PERLBUILDOP_ERROR_START+43, "Timestamp message test 1");

into this:
APT_MSG(Info, "Timestamp message test 1", NULL, localErrorModule);

7 Convert all APT_DETAIL_FATAL() macros with this form.

APT_DETAIL_FATAL(APT_PERLBUILDOP_ERROR_START+43,
"Timestamp message test");

into this:
APT_MSG(Fatal, "Timestamp message test", NULL, NULL);

Note that the severity for APT_DETAIL_FATAL() is always Fatal.

8 Convert all APT_DETAIL_FATAL_LONG() macros with this form:
APT_DETAIL_FATAL_LONG(errorModule, APT_PERLBUILDOP_ERROR_START+43,
"Timestamp message test");

into this:
APT_MSG(Fatal, "Timestamp message test", NULL, errorModule);

Note that the severity for APT_DETAIL_FATAL_LONG() is always Fatal.

9 Replace all errorLog() messages with this form:
*errorLog() << "There must be at least" << numCoords <<
"coordinates in the input vectors."<< endl;
errorLog().logError(APT_CLUSTERQUALITYOP_ERROR_START+2);

into this:
APT_Formattable args [] = { numCoords };
APT_MSG(Error,
"There must be at least {0} coordinates in the input vectors.",
args, errorLog());

48 Orchestrate 7.0 Developer Guide

Chapter 4 Localizing Messages Eliminating Deprecated Interfaces

Note that in addition to logError() messages, there can be logWarning() or

logInfo() messages. The corresponding APT_MSG() severities are Error or
Warning.

10 Replace all occurrences of APT_APPEND_LOG() with appendLog() so that

the string passed to the function is an APT_UString.
For example, change:
APT_APPEND_LOG(log, subLog, "Error when checking operator:");

to:
log.appendLog(subLog, APT_NLS("Error when checking operator:");

11 Replace all occurrences of APT_PREPEND_LOG() with prepend() so that the

string passed to the function is an APT_UString.
For example, change:
APT_PREPEND_LOG(*log, "Trouble importing field \"" <<
path_.unparse() << sub(vecLen, vecElt) << "\"" << data << ",
at offset: " << bufferSave-recStart << ": ");

to:
APT_Formattable args [] = { path_.unparse(), sub(vecLen, vecElt),
data, bufferSave-recStart };
*log.prepend(APT_NLS("Trouble importing field \"{0}{1}\"{2}, at
offset: {3}: ", args);

12 Replace all occurrences of APT_DUMP_LOG() with dump() so that the string

passed to the function is an APT_UString.
For example, change:
APT_DUMP_LOG(*logp, "Import warning at record " <<
rep_->goodRecords_+ rep_->badRecords_ << ": ");

to:
APT_Formattable args [] =
{ rep_->goodRecords_+ rep_->badRecords_ };
*logp.dump(APT_NLS("Import warning at record {0}: ", args));

Eliminating Deprecated Interfaces

To discover deprecated class interfaces in your code, compile your code with this
compile flag:
_NLS_NO_DEPRECATED_STRING_INTERFACE_
The compilation hides the interfaces for all non-NLS classes. As a result, you get a
compilation error for any non-NLS interface.

Orchestrate 7.0 Developer Guide 49

Chapter 4 Localizing Messages Eliminating Deprecated Interfaces

The classes and functions that have been changed for NLS are too numerous to
list. Refer to the header file for the function or class that causes the error, and
make one of these two types of changes:
• Change one or more function prototype parameters containing char* or
APT_String to parameters that use APT_UString. This can be often done by
casting.
or
• In cases where the NLS version of a function could not be overloaded, a “U”
has been added to the beginning of a name, changing it from function_name to
Ufunction_name in the header file. An examples is:
#ifndef _NLS_NO_DEPRECATED_STRING_INTERFACE_
APT_String ident() //deprecated;
#endif
APT_UString Uident();

Substituting the function defined on the fourth line of the example for the
function defined on the first line resolves the compilation error in your .C file.

4 10 Orchestrate 7.0 Developer Guide

 5
Orchestrate’s Argument-List
Processor
Tells you how to take advantage of Orchestrate’s argument-processing facility
which provides automatic run-time validation of your custom-operator input
options.

Overview 5 2
The Advantages of Argument-List Processing 5 2
Argument-List Descriptions 5 3
Supplying an Argument-List Description 5 3
Argument-List Description Elements 5 3
Structure of the Orchestrate-Generated Property List 5 4
Argument-List Description Syntax 5 5
Argument-List Description Examples 5 16
Property List Encoding of Operator Arguments 5 19
Example Property List 5 19
Traversing a Property List 5 19
Error Handling 5 20
Argument-Description Syntax Errors 5 20
Command-Line Argument Errors 5 20
Usage Strings 5 21
Converting Your Wrappered Operators 5 22
Converting a Composite Operator 5 23
Passing Arguments to a Suboperator 5 23
Example Code 5 24
Registering Your Operator Library 5 25
Executing Your Custom Operator 5 25

51
Chapter 5 Orchestrate’s Argument-List Processor Overview

Overview
Orchestrate provides a standardized argc/argv argument list processor to parse
the command-line arguments to your custom operators, including your custom
partitioner and collector operators.
By defining an argument description string in your C++ code, you provide a
description of the arguments expected by your custom operator. Orchestrate uses
your description to guide its interpretation of the argument list given to your
custom operator at run time.
Using your argument description string and the current run-time argument list,
Orchestrate detects any invalid-argument errors that prevent execution of your
operator. If there are no errors, Orchestrate provides your operator code with a
property-list of the validated arguments.
With the knowledge that Orchestrate initially and automatically performs a large
set of error checks, you can add operator code that traverses the Orchestrate-
generated property list to get the argument-validation information you need to
decide how to run your operator.
This chapter gives you the syntax you need to write your argument descriptions.
It also supplies examples of argument-description strings along with sample osh
commands for the described operators, as well as examples of Orchestrate-
generated property lists that are based on the the sample argument-description
strings and osh command-line arguments. It also shows you how to traverse a
property-list.

Note Your argument specification string now replaces the separate .op wrapper file
that was required in previous versions of Orchestrate; however, your existing
wrapped operators continue to be supported. To take advantage of
Orchestrate’s argument processing for your existing wrapped operators, convert
them by following the steps outlined in “Converting Your Wrappered
Operators” on page 5-22.

The Advantages of Argument-List Processing

Orchestrate’s argument-list processing facility has the following advantages:
• It provides you with a way to define your options in a standard and more
efficient form than with a wrapper file. Argument description strings require
fewer lines of code than .op files typically require.
• It provides automatic run-time validation of your options.
• It automatically produces a usage string.

52 Orchestrate 7.0 Developer Guide

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

• It creates a property list you can traverse to examine the input arguments and
their values.
• It supplies error messages for arguments that do not conform to their
descriptions.

Argument-List Descriptions
Supplying an Argument-List Description
You supply a description of the arguments your operator accepts to the
APT_DEFINE_OSH_NAME macro in the form of a quoted list of argument
descriptions. Argument-list descriptions can contain Unicode characters. It’s
character set is determined by the -output_charset option, and its locale is the
same as the locale for Orchestrate output messages. Locale is user-specified at
install time. See the Orchestrate 7.0 User Guide for information on character sets
and locales.

Note The terms arguments and subarguments correspond to operator options and
suboptions.

The operator’s implementation may be spread across several files, but one of the
files is usually named with the operator’s name plus a .C extension. The call to
APT_DEFINE_OSH_NAME can go at the beginning of this file.
The macro accepts three arguments:
APT_DEFINE_OSH_NAME(C, O, U)

Cis the class name associated with the operator, O is the osh name for the
operator, and U is the argument description string. For example:
APT_DEFINE_OSH_NAME(APT_TsortOperator, tsort, TSORT_ARGS_DESC)

At run time, Orchestrate uses your argument description and the actual
arguments given to your operator to produce a property-list encoding of the
arguments and their values to your override of initializeFromArgs_().
The APT_DEFINE_OSH_NAME macro is defined in
$APT_ORCHHOME/incude/apt_framework/osh_name.h. The initializeFromArgs_()
function is defined in $APT_ORCHHOME/incude/apt_framework/operator.h,
partitioner.h, and collector.h.

Argument-List Description Elements

Some operators do not have arguments; but for those that do, the Orchestrate
argument-list processor treats the arguments as a list of argument items. Each
argument item can have the following basic components:

Orchestrate 7.0 Developer Guide 53

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

• Name. An argument description must have an argument name that is unique

within its own scope. For example, two top-level arguments cannot have the
same name, but a subargument may have the same name as a top-level
argument. The argument name is usually an osh option tag without its
leading dash character. For example, the key argument for the tsort operator
is written as -key on the osh command line. Argument names are matched in
a case-insensitive manner.
• Argument values types. An argument item can accept zero or more values.
Each value is of a single specified type, and may be optional, default,
deprecated, or constrained in other ways. The key argument for the tsort
operator takes a fieldName type value; for example, -key firstname
string[10]. The tsort operator also takes a deprecated fieldType value.

• Subarguments. An agument can have subarguments which correspond to the

suboptions of an operator. The key argument item for the tsort operator has
several sub-arguments, including ci and sorted; for example, tsort -key
firstname -ci -sorted.

The argument description string for the tsort operator is given in full in
“Argument-List Description Examples” on page 5-16.
For an operator without arguments, you can supply a minimal argument
description. The minimum description required need only include the otherInfo
parameter and its description subparameter.
For example:
{ otherInfo = {description = “this operator has no arguments.”} }

Structure of the Orchestrate-Generated Property

List
From you argument description list, Orchestrate’s argument list processor
produces a property list in which each top-level property name is that of the
argument item it encodes, and whose value is a property list containing more
information about that argument, if any.
The overall form of the property list is:
{ argName1 = { ... },
argName2 = { ... },
...
}

Each argument item is represented by a set of properties as follows:

argName1 =
{
value = value,
subArgs = argItem = { ... },

54 Orchestrate 7.0 Developer Guide

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

...
}

The argument name is always present as the argName property in the list. Value
properties will be present according to whether the argument item has any
values. Subarguments will be present when the argument item has
subarguments. If an argument item does not have a value or subarguments, it just
appears as an empty property in the property list.
The property list presents argument items in the order in which they are
encountered on the command line. For example, given the argument description
for the tsort operator and this osh command line:
tsort -key product -ci -sorted -hash
-key productid int32
-memory 32
-stats

Orchestrate generates the following property list:

{
key = {value = product, subArgs= {ci, sorted, hash}},
key = {value = productid, value = int32},
memory = {value = 32},
stats
}

The argment description string for the tsort operator is given in full in
“Argument-List Description Examples” on page 5-16.

Argument-List Description Syntax

Table 16 on Page 6 contains complete argument-description syntax and
abbreviated comments. Table 17 on Page 11 contains fuller comments keyed to
the numbered syntax lines in Table 2. You can also reference a detailed
description of the argument description string syntax in the header file,
$APT_ORCHHOME/incude/apt_utils/argvcheck.h.

The format of your argument description string must adhere to these rules:
1 Comments are not permitted inside the argument description.
2 Each line of the description string must end in a backslash and be quoted.
For example:
#define HELLO_ARGS_DESC
““{ “\
“ uppercase = {optional, description = 'capitalize?'}, “\
“ numtimes = { value = { type = {int, min=1, max=10 }, “\
“ usageName='times'}, “\
“ optional, “\
“ description='number of times to print' “\
“ } “\
“}” “\

Orchestrate 7.0 Developer Guide 55

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

Note For readability, the argument-description syntax table and the examples in the
header files omit the quotes and backslashes.

Table 16 Argument Description Syntax

Line # Code Short Comments

op = optional; req =
required;
fuller comments follow the
syntax

1{ argName = op; 0 or more

{
3 description = description_string , req
4 value = op; 0 or more
{
6 type = { type , other_properties } 1 req; syntax starts on line
33
7 usageName = string, req
8 optional, op; value is optional
9 default = type_literal_value, op; affects usage string
10 deprecated op; omit from usage string
},
12 subArgDesc = {argName = {...}, ... }, op; 0 or more
13 minOccurrences = int, op; default = 0
14 maxOccurrences = int, op; default = inf
15 optional, op; same as min/max = 0/1
16 oshName = string, op
17 oshAlias = string, op; 0 or more
18 silentOshAlias = string, op; 0 or more; not in usage
19 default, op; goes in usage string
20 hidden, op; not in usage string
21 deprecated = { property, property, ... } op; not in usage string
},

argName = { {...}, ... },

24 otherInfo = req
{
26 exclusive = {name, name, ... }, op; 0 or more
27 exclusive_required = {name, name, ... }, op; 0 or more
28 implies = {name, name, ... }, op; 0 or more
29 description = string, op; goes in usage string
30 inputs = dataset_type_descriptions, req
31 outputs = dataset_type_descriptions, req

56 Orchestrate 7.0 Developer Guide

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

Table 16 Argument Description Syntax (continued)

Line # Code Short Comments

op = optional; req =
required;
fuller comments follow the
syntax
} expanded syntax of
} dataset_type_descriptions
starts on syntax line 102

Type syntax (expanded from line 6 above) a single type is required

34 type =
{
string, must be the first property
list = { string, string, ... }, op; list of legal values
regexp = regexp, op; regexp for legal values
case = sensitive | insensitive op; default: case-insensive
40 }

42 type =
{
ustring, must be the first property
list = { ustring, ustring, ... }, op; list of legal values
regexp = regexp, op; regexp for legal values
case = sensitive | insensitive op; default: case-insensive
48 }

50 type =
{
int, must be the first property
min = int, op; no lower limit by
default
max = int, op; no upper limit by
default
list = {int, int, ... } op; list of legal values;
56 } list exclusive with
min/max

58 type =
{
float, must be the first property

Orchestrate 7.0 Developer Guide 57

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

Table 16 Argument Description Syntax (continued)

Line # Code Short Comments

op = optional; req =
required;
fuller comments follow the
syntax
min = float, op; no lower limit by
default
max = float, op; no upper limit by
default
list = { float, float, ... } op; list of legal values;
64 } list exclusive with
min/max

66 type =
{
fieldType, must be the first property

min = type_literal, op; no lower limit by

default
max = type_literal, op; no upper limit by
default
list = { type_literal, type_literal, ... }, op; list of legal values
compareOptions = { ... }, op; adjusts comparisons
73 } list exclusive with
min/max

75 type =
{
propList, must be first property
elideTopBraces, op
requiredProperties = op
{ property, property, ... }
81 }

83 type =
{
schema, must be the first property
acceptSubField, op; default: top-level only
acceptVector, op; default: no vectors
acceptSchemaVar op; default: no schema vars

58 Orchestrate 7.0 Developer Guide

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

Table 16 Argument Description Syntax (continued)

Line # Code Short Comments

op = optional; req =
required;
fuller comments follow the
syntax
89 }

91 type =
{
fieldName, must be the first property
input | output, req
acceptSubField op; default: top-level only
96 }

98 type =
{
fieldTypeName, must be the first property
list = { name, name, ... }, op; list of legal type names
noParams op; default: params
accepted
103 }

105 type =
{
pathName, must be the first property
canHaveHost, op
defaultExtension = string op
110 }

Input/Output dataset_type_descriptions (expanded from lines 30 and 31)

inputs | outputs =
{
113 portTypeName = req; name of the dataset
{
115 description = string, req

Orchestrate 7.0 Developer Guide 59

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

Table 16 Argument Description Syntax (continued)

Line # Code Short Comments

op = optional; req =
required;
fuller comments follow the
syntax
116 oshName = string, op
117 minOccurrences = int, op; default = 0
118 maxOccurrences = int, op; default = inf
119 optional, op; same as min/max=0/1
120 required, op; same as min/max=1/1
121 once, op; same as min/max=1/1
122 multiple, op; same as
min/max=1/inf
123 any, op; same as
min/max=0/inf
124 constraints =
{
126 ifarg = argName, port type if specified

127 ifnotarg = argName, port type if not specified

128 ifargis = (name = argName, port type if specified value
value = argValue ),
129 argcount = argName, times the port type appears
130 argcountopt = argName, port appears N or N+1
times
131 portcount = portName output appears input times
},
133 incomplete op
},
portTypeName = { {...}, ... }, op; 0 or more
},

5 10 Orchestrate 7.0 Developer Guide

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

Table 17 Argument Description Syntax Comments

Syntax Comment
Line

1 There can be zero or more argument entries. An argName property is required

for an argument item, and it must be unique within an argument list. It is
matched in a case-insensitive manner.
By default, Orchestrate forms the osh tag for an argument by prepending a
hyphen to the argument’s name. You can override this rule with the oshName
property.

3 The description property is required and is used when generating a usage

string.

4 There can be zero or more value properties to describe the values that are
expected to follow an argument’s name on the command line. The order in
which the value entries are listed determines the order in which the values
must be presented on the command line after the argument name.

6 A single type subproperty must be supplied. Its syntax allows you to specify
one of nine data types which are described starting on line 34 below.

7 The required usageName property defines the name of the type as it appears
on the command line.

8 The optional property indicates that the value need not be supplied on the
command line. Only the final value is optional. The optional property itself is
optional.

9 You can use the optional default flag property to specify a value to be used
when no value is given on the command line. This only affects the generated
usage string. The type’s print/scan generic function is used to read the literal
value of the type.

10 The deprecated flag property is optional. When supplied, the value description
is omitted by default from generated usage strings.

12 Using the optional subArgDesc property allows you to provide additional

flags to be associated with an argument.

13-14 By default, an argument vector may have any number of items matching a
argument description. You can restrict the number of times an argument may
occur by using the optional minOccurrences and maxOccurences properties.
The default values are 0 for minOccurrences and any integer for
maxOccurrences.

Orchestrate 7.0 Developer Guide 5 11

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

Table 17 Argument Description Syntax Comments (continued)

Syntax Comment
Line

15 The optional optional parameter allows you to specify minOccurences = 0 and

maxOccurrences = 1.

16 With oshName, you specify a non-default name for the argument on the osh
command line.

17 An argument may optionally have one or more osh alias names, allowing you
to provide abbreviations, variant spellings, and so on. You specify them using
the oshAlias property.

18 By using the optional silentOshAlias property, an argument may have one or

more osh alias names which are not listed in a usage string.

19 You use the optional default flag property to indicate that the argument
represents a default that is in force if this argument (and typically other related
arguments in an exclusion set with this argument) is not present. This
information is put into the generated usage string, and has no other effect.

20 With the hidden flag property you can optionally describe arguments that are
not normally exposed to the operator user. Hidden argument descriptions are,
by default, omitted from generated usage strings.

21 You can optionally use the deprecated flag parameter to indicate that an
argument description exists only for back-compatibility. Its argument
description is omitted by default from generated usage strings.

24 The required otherInfo parameter allows you to specify constraints.

For an operator without arguments, the minimum required argument
description contains the otherInfo parameter and its description
subparameter. For example:
{
otherInfo = { description = “this operator has no arguments.”}
}

26 With the exclusive constraint, you optionally name a set of arguments which
are mutually exclusive. Multiple exclusive sets may be defined.

27 An exclusive_required constraint is similar to an exclusive constraint,

described on line 26 above; however, one of its listed arguments must be
present.

28 An implies constraint specifies that if one given argument occurs, then another
given argument must also be supplied.

5 12 Orchestrate 7.0 Developer Guide

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

Table 17 Argument Description Syntax Comments (continued)

Syntax Comment
Line

29 Your optional description string is added to the generated usage string.

30-31 Both the input and output properties are required. If they are omitted, warning
messages are emitted when the operator is run.

34-102 The type property must be the first subproperty in a value clause. It must be an
Orchestrate type.
For example:
value = { type = int32,
usageName = “mbytes”,
default = 20
}

Comments for each type are given next.

34-40 String type. The list and regexp subproperties optionally specify the legal
values, either in list form or in a regular expression. If neither of these two
subproperties is specified, any string value is accepted for the argument.
When case has its default value of insensitive, list matching is performed in a
case-insensitive manner, and the regexp is evaluated on a copy of the string
value that has been converted to lowercase.

42-48 Ustring type. The list and regexp subproperties optionally specify the legal
values, either in list form or in a regular expression. If neither of these two
subproperties is specified, any ustring value is accepted for the argument.
When case has its default value of insensitive, list matching is performed in a
case-insensitive manner, and the regexp is evaluated on a copy of the string
value that has been converted to lowercase.

50-56 Integer type. The min and max subproperties are optional. By default, there are
no lower or upper limits. The optional list subproperty specifies a list of legal
values. It is exclusive with min and max.
Integer values are 32 bits, signed. The field value is encoded as a dfloat in the
argument’s value = value property.

58-64 Float type. The min and max subproperties are optional. By default, there are
no lower or upper limits. The optional list subproperty specifies a list of legal
values. It is exclusive with min and max.
Floating-point values are double precision. The field value is encoded as a
dfloat in the argument’s value = value property.

Orchestrate 7.0 Developer Guide 5 13

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

Table 17 Argument Description Syntax Comments (continued)

Syntax Comment
Line

66-73 FieldType type. The optional min and max subproperties may be specified if
the field type supports ordered comparison. By default, there are no lower or
upper limits. The optional list subproperty may be provided if the field type
supports equality comparisons. It specifies a list of legal values and is exclusive
with min and max.
The print/scan generic function is used to parse the type_literal values.
The optional compareOptions subproperty adjusts how comparisons are done
with the min, max, and list values.
The field value is encoded as a string in the argument’s value = value property.

75-81 PropList type. The elideTopBraces and requiredProperties subproperties are

optional. The elideTopBraces property notifies Orchestrate that the user omits
the top-level opening and closing property-list braces.
The field value is encoded as a property list in the argument’s value = value
property.

83-89 Schema type. The acceptSubField, acceptVector, and acceptSchemaVar

subproperties are optional, and their default values indicate not to accept
subfields, vectors, and schema variables.

91-96 FieldName type. You must also specify either input or output. The
acceptSubField subproperty is optional. Its default value is top-level field only.

98-103 FieldTypeName type. Using the optional list subproperty, you can specify
acceptable type names. The noParams subproperty is also optional. The default
is to accept type parameters.

105-110 PathName type. The canHaveHost and defaultExtension subproperties are

optional. CanHaveHost specifies that the pathname can begin with a host
computer designation, host:.

113 You use the required portTypeName property to specify a one-word name for
the port. Input and output ports are the same as input and output datasets.

115 With the required description property, you can describe the purpose of the
port.

116 Use the oshName property to specify the port name for the osh command line.

5 14 Orchestrate 7.0 Developer Guide

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

Table 17 Argument Description Syntax Comments (continued)

Syntax Comment
Line

117-118 You can restrict the number of times a portTypeName property may occur by
using the optional minOccurrences and maxOccurences subproperties. The
default values are 0 for minOccurrences and any integer for maxOccurrences.

119 The optional optional subproperty specifies zero or one occurrence of a

portTypeName property.

120 The optional required subproperty specifies one and only one occurrence of a
portTypeName property.

121 The optional once subproperty has the same functionality as the required
subproperty. It specifies one and only one occurrence of a portTypeName
property.

122 The optional multiple subproperty specifies one or more occurences of a

portTypeName property.

123 The optional any subproperty specifies zero or more occurrences of a

portTypeName property.

124 The constraints property is optional. If it is present, it may not be the empty list.
The syntax supplied provides simple constraint types that make it convenient
to describe most simple cases.

126 The ifarg constraint subproperty specifies that the port type does not appear
unless the argName has been specified. This subproperty can appear more than
once, to specify multiple “enabling” options combined by logical OR. An
example is the reject option for import/export.

127 The ifnotarg constraint subproperty indicates that the port type only appears if
the argName has not been specified. This subproperty can appear more than
once to specify multiple “disabling” options, which are combined by logical
OR. An example is the createOnly option for the lookup operator.

128 The ifargis constraint subproperty indicates that the port type appears if the
specified argName has the specified argValue.
This suboption can be specified more than once to specify multiple “enabling”
values. It can be combined with ifarg or ifnotarg. If it is specified alone, it is
effectively equivalent to also specifying ifarg for the same argName.
An example is “ifNotFound = reject” for the lookup operator.

Orchestrate 7.0 Developer Guide 5 15

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

Table 17 Argument Description Syntax Comments (continued)

Syntax Comment
Line

129 The argcount constraint subproperty indicates that the port type appears
exactly as many times as the argName appears. An example is the percent
option for the sample operator.

130 The argcountopt constraint subproperty indicates that if argName appears N

times, the port type appears either N or N+1 times. An example is the table
option for the lookup operator.

131 The portcount constraint subproperty indicates an output port type that
appears as many times as an input port type with the specified portName. An
example is rejects outputs on the merge operator

133 The incomplete flag indicates the provided input/output description is not
completely accurate given the complexity of the operator’s behavior.

Argument-List Description Examples

Two example argument description strings are given in this section. One is a
simple two-argument example, taken from the HelloWorldOp code example
shown in Table 4 in the next chapter. The other example is the argument
description string for the Orchestrate-supplied tsort operator.

Note The example argument description strings in this section are shown without line
quotes and backslashes. See the argument description in “Argument-List
Description Syntax” on page 5-5 for an example with line quotes and
backslashes.

For an additional example, see the operator implementation in the

examples/Custom_Ops/OrchAPI/HoldOp directory of your Orchestrate installation.
This directory contains the files that implement a wrapperless operator called
holdop. The files are holdop.C and holdop.h. The README file documents the
operator implementation.
Here is the argument description string for HelloWorldOp:
#define HELLO_ARGS_DESC
“{
uppercase = {optional, description = 'capitalize or not'},
numtimes = { value = { type = {int32, min=1, max=10},
usageName='times'},
optional,
description='number of times to print message'
}
}”

5 16 Orchestrate 7.0 Developer Guide

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

APT_DEFINE_OSH_NAME(HelloWorldOp, hello, HELLO_ARGS_DESC);

Orchestrate uses the description elements of your argument description to

produce a usage string for an operator. The usage string for Orchestrate’s tsort
operator is shown in “Usage Strings” on page 5-21.
The second argument description example is the argument description string for
Orchestrate’s tsort operator:
“{key={value={type={fieldName, input},
usageName="name"
},
value={type={fieldTypeName},
usageName="type",
optional,
deprecated
},
subArgDesc={ ci={optional,
description="case-insensitive comparison"
},
cs={optional,
default,
description="case-sensitive comparison"
},
ebcdic={optional,
description=
"use EBCDIC collating sequence"
},
nulls={value={type={string,
list={first,last}
},
usageName=”first/last”,
default=first
},
optional,
description=
”where null values should sort”
}
hash={optional,
description="hash partition using this key"
},
asc={oshAlias="-ascending",
optional,
default,
description="ascending sort order"
},
desc={oshAlias="-descending",
silentOshAlias="-des",
optional,
description="descending sort order"
},
sorted={optional,
description=
"records are already sorted by this key"
},

Orchestrate 7.0 Developer Guide 5 17

Chapter 5 Orchestrate’s Argument-List Processor Argument-List Descriptions

clustered={optional,
description=
"records are grouped by this key"
},
param={value={type={propList, elideTopBraces},
usageName="params"
},
optional,
description="extra parameters for sort key"
},
otherInfo={exclusive={ci, cs},
exclusive={asc, desc},
exclusive={sorted, clustered},
description="Sub-options for sort key:"
},
},
description="specifies a sort key"
},
memory={value={type={int32, min=4},
usageName="mbytes",
default=20
},
optional,
description="size of memory allocation"
},
flagCluster={optional,
description="generate flag field identifying
clustered/sorted key value changes
in output"
},
stable={optional,
default,
description="use stable sort algorithm"
},
nonStable={silentOshAlias="-unstable",
optional,
description="use non-stable sort algorithm
(can reorder same-key records)"
},
stats={oshAlias="-statistics",
optional,
description="print execution statistics"
},
unique={oshAlias="-distinct",
optional,
description=
"keep only first record of same-key runs in output"
},
keys={value={type={schema},
usageName="keyschema"
},
deprecated=key,
maxOccurrences=1,
description="schema specifying sort key(s)"
},

5 18 Orchestrate 7.0 Developer Guide

Chapter 5 Orchestrate’s Argument-List Processor Property List Encoding of Operator Arguments

seq={silentOshAlias="-sequential",
deprecated,
optional,
description="select sequential execution mode"
},
otherInfo={exclusive={stable, nonStable},
exclusive_required={key, keys},
description="Orchestrate sort operator:"
inputs={unSorted={description=”sorted dataset”,
required}
}
outputs={sorted={description=sorted dataset”,
required}
}

}“

Property List Encoding of Operator Arguments

When Orchestrate has processed your argument-description string and the
operator argument list without finding faulty description syntax or invalid
arguments, it produces a property list that is passed to your override of
initializeFromArgs_(). Your initializeFromArgs_() code can then traverse the list
and process the checked command-line arguments as you wish. The property list
is an instance of APT_Property class. See the header file,
$APT_ORCHHOME/include/apt_util/proplist.h, for information on this class.

Example Property List

Orchestrate’s tsort operator is used for this example:
For this osh command line:
tsort -key lastname -ci -sorted -hash
-key balance int32 -memory 32 -stats

Orchestrate’s argument list processor generates this property list based on the
tsort argument description string shown in the previous section:
{ key={value=lastname, subArgs={ci, sorted, hash}},
key={value=balance, value=int32},
memory={value=32},
stats
}

Traversing a Property List

The following example override of _initializeFromArgs_() for HelloWorldOp
shows how to traverse a property list:
APT_Status HelloWorldOp::initializeFrom Args_
(const APT_PropertyList & args,

Orchestrate 7.0 Developer Guide 5 19

Chapter 5 Orchestrate’s Argument-List Processor Error Handling

APT_Operator::InitializeContext context)
{
APT_Status status=APT_StatusOk;
if (context == APT_Operator::eRun) return status;
for (int i = 0; i < args.count(); i++)
{
const APT_Property& prop = args[i];
if (prop.name() == “numtimes”)
numTimes_ = (int) prop.valueList().getProperty(“value”, 0)
.valueDFloat();
else if (prop.name() == “uppercase”)
uppercase_ = true;
};
return status;
}

Error Handling
Argument-Description Syntax Errors
Syntax errors in your argument-description string are not noticed at compile time.
Instead, errors are generated at runtime when Orchestrate uses your argument
description string and the current operator arguments to produce a property-list
object. You can use Orchestrate’s error-log facility to capture errors. Chapter 14,
“Using the Error Log” on page 14-1 describes the error-log facility.

Command-Line Argument Errors

When an osh command line is being processed, any tokens that do not
correspond to an argument description are taken to be error tokens. An error
message is generated for each error token, with the arg token number and text
included in the error message. Orchestrate then discards the error token and
attempts to process the next argument token according to the argument
description.
In a sub-argument list, an unrecognized argument token terminates the sub-
argument list. Orchestrate then checks any constraints, and pops processing back
up to the enclosing argument list.
When an argument token is seen that matches one of the argument descriptions,
subsequent argument tokens are processed using the assumption that the
argument has been correctly matched. Any local discrepancies such as a bad
value, a bad sub-argument list, and so on, are reported along with the argument
tokens making up the argument being processed and any usage-style information
relating to the argument.

5 20 Orchestrate 7.0 Developer Guide

Chapter 5 Orchestrate’s Argument-List Processor Usage Strings

When an argument list is terminated, the constraints are checked for such aspects
as occurrence restrictions, exclusivity, and implies relationships; and any
violations are reported with the appropriate contextual information, including the
argument instances involved and appropriate usage-style information describing
the violated constraints.
When one or more errors are reported, only argument tokens and usage
information specific to the errors at hand are included in the error messages.
Whether or not the entire usage string is generated and issued in a summary
message is up to you, as is the responsibility of identifying the operator or other
context in which the argument-processing error occurred.
Chapter 14, “Using the Error Log” on page 14-1 describes the error-log facility.

Note When operator execution successfully reaches your override of

initializeFromArgs_(), you can be confident that Orchestrate has found no
serious argument-description syntax errors or invalid operator arguments.

Usage Strings
Orchestrate generates a usage string from the description elements in your
argument description string. You can access an operator usage string from the
osh command line.
For example:
$ osh -usage tsort

The usage string generated for the tsort operator follows. The example assumes
that both deprecated options and current options have been requested.
Orchestrate sort operator:
-key -- specifies a sort key; 1 or more
name -- input field name
type -- field type; optional; DEPRECATED
Sub-options for sort key:
-ci -- case-insensitive comparison; optional
-cs -- case-sensitive comparison; optional; default
-ebcdic -- use EBCDIC collating sequence; optional
-hash -- hash partition using this key; optional
-asc or -ascending
-- ascending sort order; optional; default
-desc or -descending
-- descending sort order; optional
-sorted -- records are already sorted by this key; optional
-clustered -- records are grouped by this key; optional
-param -- extra parameters for sort key; optional
params -- property=value pairs(s), without curly braces

Orchestrate 7.0 Developer Guide 5 21

Chapter 5 Orchestrate’s Argument-List Processor Converting Your Wrappered Operators

(mutually exclusive: -ci, -cs)

(mutually exclusive: -asc, -desc)
(mutually exclusive: -sorted, -clustered)
-memory -- size of memory allocation; optional
mbytes -- int32; 4 or larger; default=20
-flagCluster -- generate flag field identifying start of same-key
runs in output; optional
-stable -- use stable sort algorithm; optional; default
-nonStable -- use non-stable sort algorithm (can reorder same-key
records); optional
-stats or -statistics
-- print execution statistics; optional
-unique or -distinct
-- keep only first record of same-key runs in output;
optional
-keys -- schema specifying sort key(s); optional;
DEPRECATED: use -key instead
keyschema -- string
-seq -- select sequential execution mode; optional; DEPRECATED
(mutually exclusive: -stable, -nonStable)
(mutually exclusive: -key, -keys; one of these must be provided)

A complete specification of the argument description language is given in the

$APT_ORCHHOME/include/apt_util/argvcheck.h header file, and a detailed
example argument description is given in
$APT_ORCHHOME/examples/Custom_Ops/OrchAPI/HoldOp.

Converting Your Wrappered Operators

Once you have constructed an argument description string for your wrappered
operator, you need only make a few straightforward code changes to your
operator code to enable Orchestrate’s argument-list processing facility.

To enable argument list processing for an existing wrappered operator:

1 Construct an argument description string for the operator. This process is
described in “Argument-List Descriptions” on page 5-3.
2 Remove your call to the APT_REGISTER_OPERATOR macro in your
operator .C file.
3 Add a call to the APT_DEFINE_OSH_NAME macro in your operator .C file.
This macro is described in “Supplying an Argument-List Description” on
page 5-3.
4 Make any necessary changes to initializeFromArgs_(). If necessary, also make
changes to other functions that processed the output of the wrapper.

5 22 Orchestrate 7.0 Developer Guide

Chapter 5 Orchestrate’s Argument-List Processor Converting Your Wrappered Operators

Converting a Composite Operator

Passing Arguments to a Suboperator
Part of the task of unwrapping a composite operator is to pass the parameter
arguments to the suboperators. You accomplish this by using
APT_Operator::lookupAndInitializeFromArgv():
static APT_Operator* lookupAndInitializeFromArgv
(int argc, const APT_String* argv, APT_ErrorLog&,
const char* ident=0);

Its functionality is described in $APT_ORCHHOME/include/apt_framework/operator.

h and is summarized here:

The static method locates, instantiates, and initializes an instance of the

operator defined by the specified argument array.
Argv[0] is assumed to be the osh name of the operator, as defined by a call to
the APT_DEFINE_OSH_NAME() macro. The remaining elements of the
array are the options to the operator.
To obtain the C++ operator class to instantiate and an argv-checker
description of the operator's expected arguments, the name of the operator is
looked up in the osh operator registry, APT_OshNameRegistry. See
$APT_ORCHHOME/include/apt_framework/osh_name.h and APT_ArgvProcessor
in $APT_ORCHHOME/include/apt_util/argvcheck.h.
An instance of the C++ operator class is instantiated.
The argv-checker description is used to initialize an instance of
APT_ArgvProcessor, which takes the argument array and returns an
APT_PropertyList object representing the processed form of the argument
list.
The initializeFromArgs() method of the operator is called with the processed
argument list.
If ident is non-null, setIdent() is called on the instantiated operator object
with the specified identifier.
The instantiated and initialized operator instance is returned.
If the osh name cannot be resolved or the C++ class cannot be instantiated, a
null pointer is returned and information is written to the APT_ErrorLog
object.
Once a C++ class is instantiated for the operator, its error log object will
receive any subsequent error messages from argv processing,
initializeFromArgs(), etc. If errors occur at this stage, a non-null operator
pointer is returned and its errorLog() contains any reported errors.

Orchestrate 7.0 Developer Guide 5 23

Chapter 5 Orchestrate’s Argument-List Processor Converting Your Wrappered Operators

The argc value should be the index of the last argument in the array; there is
no need for a trailing null argument. The argc/argv array is stored in the
operator, and becomes accessible via the accessArgv() method.

Example Code
ExampleCompositeOperator::initializeFromArgs()
{
// ... other processing occurs.

APT_String importProps[5];
importProps[0]="import";
importProps[1]="-schema";
importProps[2]="record (a: int32;)";
importProps[3]="-file";
// the composite's "filename" argument contains the name of the file
// to import.
importProps[4]=args.getProperty("filename").valueList().
getProperty("value").valueString();
// allocate the operator and initialize it with these arguments,
// default ident. Log any errors to our own error log.
// importOp is an APT_Operator* member variable of this class.
importOp = APT_Operator::lookupAndInitializeFromArgv(5, importProps,
errorLog());
if(!importOp || errorLog().hasError())
return APT_StatusFailed;
// allocation/initialization failed. Our errorLog will have any
// errors!
if(importOp->errorLog().hasError())
{
// an error occurred in the operator's initializeFromArgs() that
// wasn't caught in the argvchecker!
errorLog().appendLog(importOp->errorLog());
return APT_StatusFailed;
}
}

...

ExampleCompositeOperator::describeOperator()
{
// possibly other processing here.
markSubOperator(importOp);
}

5 24 Orchestrate 7.0 Developer Guide

Chapter 5 Orchestrate’s Argument-List Processor Registering Your Operator Library

Registering Your Operator Library

You use the $APT_ORCHHOME/bin/register-library utility to register your
operator library. The syntax is:
register-library myoperatorlib .so
The utility creates a file in the current directory called operator .apt, which holds
all the information necessary for Orchestrate to find all the operators in that
shared library. If that file already exists, register-library adds any new
information to the bottom of the file.
When Orchestrate runs a job, your PATH is searched to find all files called operator .
apt. You can change the way Orchestrate searches for your operator .apt files by
defining a library path with the environment variable
APT_OPERATOR_REGISTRY_PATH.

Executing Your Custom Operator

When you invoke your derived operator from osh, its execution is as follows:
1 Your call to APT_DEFINE_OSH_NAME registers the component’s name and
its expected argument list.
2 osh uses the registered information to process the component’s command-line
arguments, producing a property list which is an instance of APT_Property.
See the header file, $APT_ORCHHOME/incude/apt_util/proplist.h, for
information on this class.
3 An instance of the component is created.
4 attachInput() and attachOutput() are called to attach the indicated datasets to
the input/output ports of your operator.
5 The property list created in step 3 is passed to your override of
initializeFromArgs_() where you can traverse the list to the set up operator
member variables, or decide whether the operator execution should continue
or not.
6 Your override of the describeOperator() function is called. In this function,
you define the number of input and output data sets, their schemas, and other
operator characteristics.
7 Schema consistency checks are performed in the checkConfig() function.
8 Your operator’s serialize() method is then called. Serialized, parallel instances
of the operator are created, one per partition, and each instance receives a
copy of the serialized state.
9 The preRun function is called for all operators on the main program node
during APT_Step::run just prior to parallel execution startup.

Orchestrate 7.0 Developer Guide 5 25

Chapter 5 Orchestrate’s Argument-List Processor Executing Your Custom Operator

10 Your runLocally() method is invoked for each parallel instance of the operator
to do the actual work of reading records, processing them, and writing them
to output.
11 The postRun function is called after step completion.

5 26 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

6
Type Conversions
Describes how to use pre-defined conversion functions in your custom operators
and how to define your own conversion functions.

Default Type Conversions 6 2

Converting Between ustring and string Types 6 2
Non-Default Pre-Defined Type Conversions 6 2
Defining Custom Type-Conversion Functions 6 5
The APT_FieldConversionRegistry Class Interface 6 5
The APT_FieldConversion Class Interface 6 5
Example 1: Using Pre-Defined Type Conversions 6 6
Example 2: Creating New Type Conversions 6 9

This chapter tells you how to use pre-defined type-conversion functions in your
custom operators and tells you how to define your own type-conversion
functions.
There are two kinds of type conversions in Orchestrate: default and non-default.
Non-default conversions are also called explicit or named conversions.
Orchestrate automatically performs type conversions when the output fields from
one operator become the input fields of the next operator in the data flow. These
default conversions are possible between Orchestrate’s built-in numeric types
which are listed in the section “Default Type Conversions” on page 6-2. You can
use the modify operator to perform other field-type conversions.

61
Chapter 6 Type Conversions Default Type Conversions

Orchestrate provides two classes for handling type conversion in your custom
operators. The APT_FieldConversionRegistry class keeps track of all the field
type conversions defined in Orchestrate, and the APT_FieldConversion class
provides the functions that perform the type conversions.

Default Type Conversions

Orchestrate automatically performs default conversions among all of the
following types. No names are associated with these type conversions.
• int8, int16, int32, int64, uint8, uint16, uint32, and uint64 integer types
• sfloat and dfloat floating-point types
• decimal types
• strings composed solely of numeric characters

Converting Between ustring and string Types

When Orchestrate automatically converts from string to ustring, there is no loss
of data because all legal characters from the string type are also part of the ustring
character set. When making the conversion from ustring to string, Unicode
characters that are not part of the string character set default to the Ox1A
(Control-Z) character.

Non-Default Pre-Defined Type Conversions

Non-default type conversions apply to the Orchestrate built-in types not listed in
the previous section, such as raw, date, time, and timestamp. For these non-
default types, you must specify an explicit conversion, otherwise an error is
generated.
The format of a non-default type-conversion function is:
conversion_name (source_type ,destination_type )
Here is a complete list of the non-default type-conversion functions provided by
Orchestrate:
date_from_days_since (int32, date)

date_from_julian_day (uint32, date)

date_from_string (string, date)

date_from_timestamp (timestamp, date)

date_from_ustring (ustring, date)
days_since_from_date (date, int32)

62 Orchestrate 7.0 Developer Guide

Chapter 6 Type Conversions Non-Default Pre-Defined Type Conversions

decimal_from_decimal (decimal, decimal)

decimal_from_dfloat (dfloat, decimal)

decimal_from_string (string, decimal)

decimal_from_ustring (ustring, decimal)
dfloat_from_decimal (decimal, dfloat)

hours_from_time (time, int8)

int32_from_decimal (decimal, int32)

int64_from_decimal (decimal, int64)

is_dfloat_inband_null (dfloat, int8)
is_int16_inband_null (int16, int8)

is_int32_inband_null (int32, int8)

is_int64_inband_null (int64, int8)

is_sfloat_inband_null (sfloat, int8)

is_string_inband_null (string, int8)

is_ustring_inband_null (ustring, int8)

julian_day_from_date (date, uint32)
lookup_int16_from_string (string, int16)
lookup_int16_from_ustring (ustring, int16)
lookup_string_from_int16 (int16, string)
lookup_string_from_uint32 (uint32, string)
lookup_uint32_from_string (string, uint32)

lookup_uint32_from_ustring (ustring, uint32)

lookup_ustring_from_int16 (int16, ustring)

lookup_ustring_from_int32 (uint32, ustring)

lowercase_string (string, string)

lowercase_ustring (ustring, ustring)

microseconds_from_time (time, int32)

midnight_seconds_from_time (time, dfloat)
minutes_from_time (time, int8)
month_day_from_date (date, int8)
month_from_date (date, int8)
next_weekday_from_date (date, date)
notnull (any, int8)
null (any, int8)
previous_weekday_from_date (date, date)
raw_length (raw, int32)

Orchestrate 7.0 Developer Guide 63

Chapter 6 Type Conversions Non-Default Pre-Defined Type Conversions

seconds_from_time (time, dfloat)

seconds_since_from_timestamp (timestamp, dfloat)
string_from_date (date, string)
string_from_decimal (decimal, string)
string_from_time (time, string)
string_from_timestamp (timestamp, string)
string_from_ustring (ustring, string)
string_length (string, int32)
substring (string, string)
time_from_midnight_seconds (dfloat, time)
time_from_string (string, time)
time_from_timestamp (timestamp, time)
time_from_ustring (ustring, time)
timestamp_from_date (date, timestamp)
timestamp_from_seconds_since (dfloat, timestamp)
timestamp_from_string (string, timestamp)
timestamp_from_time (time, timestamp)
timestamp_from_timet (int 32, timestamp)
timestamp_from_ustring (ustring, timestamp)
timet_from_timestamp (timestamp, int32)
uint64_from_decimal (decimal, uint64)
uppercase_string (string, string)
uppercase_ustring (ustring, ustring)
ustring_from_date (date, ustring)
ustring_from_decimal (decimal, ustring)
ustring_from_string (string, ustring)
ustring_from_time (time, ustring)
ustring_from_timestamp (timestamp, ustring)
ustring_length (ustring, int32)
u_substring (ustring, ustring)
weekday_from_date (date, int8)
year_day_from_date (date, int16)
year_from_date (date, int16)
year_week_from_date (date, int8)

For more information on explicit conversion, see the modify and transform
operator chapters in the Orchestrate 7.0 Operators Reference.

64 Orchestrate 7.0 Developer Guide

Chapter 6 Type Conversions Defining Custom Type-Conversion Functions

Defining Custom Type-Conversion Functions

The APT_FieldConversionRegistry Class Interface
The following figure shows the interface to APT_FieldConversionRegistry which
provides a lookup facility for default and non-default explicit type-conversion
functions.
$APT_ORCHHOME/incude/apt_framework/type/conversion.h is the header file for
this class. It contains descriptions of the functions. You can also browse the
Orchestrate class hierarchy using HTML-documented header files. Start your
search with the index.html file in $APT_ORCHHOME/doc/html/.

APT_FieldConversionRegistry class interface

public:
addFieldConversion() lookupAndParse() numConversions()
get() lookupDefault()
getConversion() lookupExplicit()

The function addFieldConversion() adds the given field conversion object to the
field-conversion registry.
lookupAndParse() locates and returns the given explicit conversion.
lookupDefault() locates and returns the default conversion for the given source
and destination schema types; and lookupExplicit() locates and returns the
indicated explicit conversion.

The APT_FieldConversion Class Interface

The following figure shows the interface to APT_ FieldConversion which defines
the facility that validates and perform type conversions. The header file for this
class, $APT_ORCHHOME/include/apt_framework/type/conversion.h, contains
descriptions of the functions. You can also browse the Orchestrate class hierarchy

Orchestrate 7.0 Developer Guide 65

Chapter 6 Type Conversions Example 1: Using Pre-Defined Type Conversions

using HTML-documented header files. Start your search with the index.html file
in $APT_ORCHHOME/doc/html/.

APT_FieldConversion class interface

public: protected
allocConversionData() ident() APT_FieldConversion()
~APT_FieldConversion() isDefault() clone()
canParameterize() own() parse_()
conversionName() parse()
convert() sourceTypeName
destTypeName() unparsed()
disown() validate()
freeConversionData()

Example 1: Using Pre-Defined Type

Conversions
This example shows how to use pre-defined type conversions in an operator
which is derived directly from APT_Operator. The input interface schema has a
single date field; and the output interface schema has two fields: the date field
received from input and an added string field with a fixed length of 1. The value
of the string field is derived from the date field and represents the day of the
week. Monday is considered the first day of the week and day 1 of the week.
The explicit conversion needed is weekday_from_date[Monday], but
weekday_from_date returns an int8 value, which cannot be directly assigned to the
output string field. To perform the necessary conversion, the code example
defines a local int8 variable and explicitly converts it to a string. Table 18 contains
the class declaration and example type-conversion code. Comments follow the
code.

Table 18 Example Operator with Pre-Defined Type-Conversion

Code

#include <apt_framework/orchestrate.h>
2 #include <apt_framework/type/conversion.h>

class PreDefinedConversionOperator : public APT_Operator

{
APT_DECLARE_RTTI(PreDefinedConversionOperator);
APT_DECLARE_PERSISTENT(PreDefinedConversionOperator);

66 Orchestrate 7.0 Developer Guide

Chapter 6 Type Conversions Example 1: Using Pre-Defined Type Conversions

Table 18 Example Operator with Pre-Defined Type-Conversion

public:
PreDefinedConversionOperator();

protected:
virtual APT_Status describeOperator();
virtual APT_Status runLocally();
virtual APT_Status initializeFromArgs_
(const APT_PropertyList &args, InitializeContext context);

private:
// other data members
};

APT_Status PreDefinedConversionOperator::describeOperator()
{
setInputDataSets(1);
setOutputDataSets(1);

setInputInterfaceSchema("record(dField:date)", 0);
setOutputInterfaceSchema("record(dField:date; sField:string[1])", 0);

return APT_StatusOk;
}

APT_Status PreDefinedConversionOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);

APT_InputAccessorToDate dFieldInAcc("dField", &inCur);

APT_OutputAccessorToDate dFieldOutAcc("dField", &outCur);
APT_OutputAccessorToString sFieldOutAcc("sField", &outCur);

while (inCur.getRecord())
{
*dFieldOutAcc = *dFieldInAcc;

36 APT_FieldConversion* nameConv = APT_FieldConversionRegistry::

get().lookupAndParse("weekday_from_date[Monday]”, NULL);

37 APT_Assert(nameConv);

38 APT_Int8 weekday;

39 nameConv->convert(&(*dFieldInAcc), &weekday, 0);

Orchestrate 7.0 Developer Guide 67

Chapter 6 Type Conversions Example 1: Using Pre-Defined Type Conversions

Table 18 Example Operator with Pre-Defined Type-Conversion

40 weekday = weekday + 1;

41 if ( nameConv ) nameConv->disOwn();

42 APT_FieldConversion* conv = APT_FieldConversionRegistry::get().

lookupDefault("Int8", "String");

43 APT_ASSERT(conv);

44 conv->convert(&weekday, &(*sFieldOutAcc), 0);

outCur.putRecord();
}

return APT_StatusOK;
}

Comments

2 Include the Orchestrate header file, conversion.h, which defines the type-
conversion interface.

36 Create a object with explicit parameterized conversion.

37 Assert that the conversion object exists.

38 Create an intermediate local variable to store the conversion result.

39 Call the member function convert() to perform the conversion between the input
field dField and the local variable weekday.

40 The default value of weekday is 0. Increment weekday by 1 to mark the first day.

41 Call disown() on the nameConv conversion object since it is returned via its own()
function.

42 Create a conversion object based on the default conversion between int8 and
string.

43 Use assertion to make sure that the conversion object exists.

44 Call the member function convert() to perform the conversion between the local
variable weekday and the output field sField.

68 Orchestrate 7.0 Developer Guide

Chapter 6 Type Conversions Example 2: Creating New Type Conversions

Example 2: Creating New Type Conversions

This section shows you how to create and register a new type conversion,
raw_from_string. Table 19 contains the implementation code for this example.
Comments follow the code.

Table 19 Example Operator with Custom Type Conversion

Code

#include <apt_util/string.h>
2 #include <apt_framework/rawfield.h>
3 #include <apt_framework/type/conversion.h>

4 class RawStringConversion: public APT_FieldConversion

{
APT_DECLARE_RTTI(RawStringConversion);
APT_DECLARE_PERSISTENT(RawStringConversion);

public:
RawStringConversion();
10 virtual APT_Status convert
(const void *STval, void* DTval, void* data) const;
11 static bool registerConversion();

protected:
13 virtual APT_FieldConversion* clone() const;
};

APT_IMPLEMENT_RTTI_ONEBASE(RawStringConversion, APT_FieldConversion);
APT_IMPLEMENT_PERSISTENT(RawStringConversion);

17 RawStringConversion::RawStringConversion()

18 APT_FieldConversion(eImmutable, "string", "raw", "raw_from_string")

{}

20 APT_FieldConversion *RawStringConversion::clone() const

{
return new RawStringConversion(*this);
}

24 APT_Status RawStringConversion::convert
(const void* STval, void* DTval, void *data) const
{
26 const APT_String &s = *(const APT_String *)STval;
APT_RawField &d = *(APT_RawField *)DTval;

Orchestrate 7.0 Developer Guide 69

Chapter 6 Type Conversions Example 2: Creating New Type Conversions

Table 19 Example Operator with Custom Type Conversion

28 d.assignFrom(s.content(), s.length());
return APT_StatusOk;
}

31 void RawStringConversion::serialize(APT_Archive &ar, APT_UInt8)

{}

33 static RawStringConversion* sRawString = new RawStringConversion;

34 bool RawStringConversion::registerConversion()
{
APT_FieldConversionRegistry::get().addFieldConversion(sRawString);
return true;
}

39 static bool sRegistered RawStringConversion =

RawStringConversion::registerConversion();

Comments

2 Include the Orchestrate header file, rawfield.h, which defines APT_RawField.

3 Include the header file that defines the type-conversion interface.

4 All type conversions are derived, directly or indirectly, from

APT_FieldConversion.

10 You must override the virtual function convert().

11 Define the member function which is used to register the newly defined type
conversion.

13 You must override the virtual function clone().

17 Define the default constructor for this conversion.

18 Setup the conversion information on the initialization line. eImmutable indicates

that this conversion doesn’t accept parameters and is not subject to change. The
arguments "string" and "raw" identify the source schema type and destination
schema type of this type conversion. For an explicit conversion, the conversion
name must be specified. In this example, "raw_from_string" is the conversion
name. The schema type names and the conversion name are case-insensitive.

20 The override of the virtual function clone().

6 10 Orchestrate 7.0 Developer Guide

Chapter 6 Type Conversions Example 2: Creating New Type Conversions

Table 19 Example Operator with Custom Type Conversion

24 The override of the virtual function convert() which performs the conversion
from string to raw. The pointer STval points to the address which contains the
value for the source string type. The pointer DTval points to the valid (already
constructed) instance of the destination raw type. This instance is assigned the
value converted from the source string. For information on the data argument, see
the header file $APT_ORCHHOME/include/apt_framework/type/conversion.h.

26-28 The implementation for the conversion raw_from_string. References are used for
the source and the destination, with the pointers STval and DTval being cast into
APT_String and APT_RawField respectively. The member function
assignFrom() of the class APT_RawField is used to complete the conversion.

31 The serialize() function is empty because RawStringConversion has no internal

state.

33 Create a static instance of RawStringConversion, which is used within

registerConversion() to register the newly defined conversion function.

34 The implementation of the member function registerConversion().

39 Call the function registerConversion() to register the new conversion,

raw_from_string.

Orchestrate 7.0 Developer Guide 6 11

Chapter 6 Type Conversions Example 2: Creating New Type Conversions

6 12 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

7
Using Cursors and Accessors
Tells you how to access records and record fields using cursors and field accessors,
gives you the interfaces to the APT_String and APT_Ustring classes, and lists the
Unicode utilities.

Overview 7 2
How Record Fields Are Referenced 7 2
Cursors 7 3
The APT_InputCursor and APT_OutputCursor Class Interfaces 7 3
Example of Accessing Records with Cursors 7 5
Field Accessors 7 7
Defining Field Accessors 7 8
Using Field Accessors 7 9
Using Field Accessors to Numeric Data Types 7 10
Using Field Accessors to Decimal Fields 7 12
The APT_Decimal Class Interface 7 13
Using Field Accessors to Fixed-Length Vector Fields 7 13
Using Field Accessors to Variable-Length Vectors 7 15
Using Field Accessors to Nullable Fields 7 17
Using Field Accessors to string and ustring Fields 7 21
APT_String and APT_UString Class Interface 7 21
Unicode Utilities 7 25
Using Field Accessors to Raw Fields 7 26
The APT_RawField Class Interface 7 26
Using Field Accessors to Date, Time, and Timestamp Fields 7 29
Using Field Accessors to Aggregate Fields 7 31
Accessing Vectors of Subrecord Fields 7 33

71
Chapter 7 Using Cursors and Accessors Overview

Overview
In order for you to process an input or output data set from within an operator,
you need a mechanism for accessing the records and the record fields that make
up a data set. Orchestrates provides three mechanisms that work together to
perform these tasks: cursors, subcursors, and field accessors.
Cursors let you reference the records of a data set. You use two types of cursors
with data sets: input cursors and output cursors. Input cursors provide read-only
access to the records of a data set; output cursors provide read and write access.
Field accessors, on the other hand, provide access to a field within a record of a
data set.
You use subcursors only with vectors of subrecords. A subcursor allows you to
identify the current element of a subrecord vector.
This chapter describes how to use cursors, subcursors, and field accessors, and
includes example operator derivations. Note that you also use field accessors
from within a derived partitioner or collector. See Chapter 8, “Creating
Partitioners” and Chapter 9, “Creating Collectors” for examples using accessors
within derived partitioners and collectors.

How Record Fields Are Referenced

The following figure shows how you use cursors and accessors to reference
record fields:

data set field accessors to fields of current record

input cursor to int16 string[8] int16 uint32

current input record

.
.
direction of .
cursor movement

As you can see in this figure, a cursor defines the current input or output record of
a data set. Field accessors perform relative access to the current record; allowing
you to access the fields of the current record as defined by a cursor.
In order to access a different record, you update a cursor to move it through the
data set, creating a new current record. However, you do not have to update the

72 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Cursors

field accessors; they will automatically reference the record fields of the new
current record.
A record field is characterized by the field name and data type. Correspondingly,
Orchestrate supplies a different field accessor for every Orchestrate data type. In
order to access a record field, you must create an accessor for the field’s data type.
This chapter contains a description of the different accessor types, as well as
examples using them.

Cursors
Cursors let you reference specific records in a data set, while field accessors let
you access the individual fields in those records. You use cursors and field
accessors from within your override of the APT_Operator::runLocally() function.
This section describes cursors; the next section covers field accessors.
Each input and output data set requires its own cursor object. You use two
Orchestrate classes to represent cursors:
• APT_InputCursor: Defines a cursor object providing read access to an input
data set.
• APT_OutputCursor: Defines a cursor object providing read/write access to
an output data set.
The APT_InputCursor class defines the following functions for making input
records available for access:

The APT_InputCursor and APT_OutputCursor Class

Interfaces
The following figures show the interfaces to the APT_InputCursor and
APT_OutputCursor classes.The header file for these classes,
$APT_ORCHHOME/include/apt_framework/cursor.h, contains descriptions of the
functions. You can also browse the Orchestrate class hierarchy using HTML-
documented header files. Start your search with the index.html file in
$APT_ORCHHOME/doc/html/.

APT_InputCursor class interface

public:
getRecord() lookahead() setLookAhead()
isSetup() schema() setLookaheadWindow()

Orchestrate 7.0 Developer Guide 73

Chapter 7 Using Cursors and Accessors Cursors

APT_OutputCursor class interface

public:
done() putRecord
getRecord() putRecordAndFlushPartition()
isSetup()

A cursor defines the current input or output record of a data set. Field accessors
perform relative access to records; they only allow you to access the fields of the
current record as defined by either an input cursor or an output cursor.
To process an input data set, you initialize the input cursor to point to the current
input record. The cursor advances through the records until all have been
processed. Note that once a cursor has moved beyond a record of an input data
set, that record can no longer be accessed.
For the output data set, the output cursor initially points to the current output
record and advances through the rest of the records. As with input data sets, once
a cursor has moved beyond a record of an output data set, that record can no
longer be accessed.
The following figure shows an input and an output data set.

input data set output data set

previously processed previously written

input records . . output records
(not available) . .
. . (not available)

input cursor to output cursor

current input record to current
output record

empty output
direction of
cursor movement records
. .
. . direction of
. . cursor movement

When you first create an input cursor, it is uninitialized and does not reference a
record. Therefore, field accessors to the input data set do not reference valid data.
You must call APT_InputCursor::getRecord() to initialize the cursor and make
the first record in the data set the current record. You can then use field accessors
to access the fields of the input record.

74 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Cursors

When you have finished processing a record in an input data set, you again call
APT_InputCursor::getRecord() to advance the input cursor to the next record in
the data set, making it the current input record. When no more input records are
available, APT_InputCursor::getRecord() returns false. Commonly, you use a
while loop to determine when APT_InputCursor::getRecord() returns false.
When you first create an output cursor, it references the first record in the output
data set. If the record is valid, the record fields are set to the following default
values:
• Nullable fields are set to null.
• Integers = 0.
• Floats = 0.
• Dates = January1, 0001.
• Decimals = 0.
• Times = 00:00:00 (midnight).
• Timestamps = 00:00:00 (midnight) on January 1, 0001.
• The length of variable-length string, ustring, and raw fields is set to 0.
• The characters of a fixed-length string and fixed-length ustring are set to null
(0x00) or to the pad character, if one is specified.
• The bytes of fixed-length raw fields are set to zero.
• The tag of a tagged aggregate is set to 0 to set the data type to be that of the
first field of the tagged aggregate.
• The length of variable-length vector fields are set to 0.
When you have finished writing to an output record, you must call
APT_OutputCursor::putRecord() to advance the output cursor to the next record
in the output data set, making it the current output record.

Example of Accessing Records with Cursors

The following example of a while loop processes the records of a single input
data set and writes its results to a single output data set. You would place this
loop, along with the code to define and initialize the cursors, within the override
of the runLocally() function.

Table 20 Accessing Records with Cursors in runLocally()

Code

APT_Status ExampleOperator::runLocally()
{

Orchestrate 7.0 Developer Guide 75

Chapter 7 Using Cursors and Accessors Cursors

Table 20 Accessing Records with Cursors in runLocally() (continued)

3 APT_InputCursor inCur;
4 APT_OutputCursor outCur;

5 setupInputCursor(&inCur, 0);
6 setupOutputCursor(&outCur, 0);

7 while (inCur.getRecord())
{
9 // body of loop

10 outCur.putRecord();
}

return APT_StatusOk;
}

Comments

3 Define inCur, an instance of APT_InputCursor, the input cursor for the first data
set input to this operator.

4 Define outCur, an instance of APT_OutputCursor, the output cursor for the first data set
output by this operator.

5 Use APT_Operator::setupInputCursor() to initialize inCur. Note that input data

sets are numbered starting from 0.

6 Use APT_Operator::setupOutputCursor() to initialize outCur. Note that output

data sets are numbered starting from 0.

7 Use APT_InputCursor::getRecord() to advance the input cursor to the next input

record. You must call this function before attempting to process an input record,
because an input cursor does not initially reference a valid record.
APT_InputCursor::getRecord() returns false when there are no more records in the
input data set, terminating the while loop.
APT_InputCursor::getRecord() can take as an argument a skip value that allows
you to bypass records of an input data set. By default, the skip value is 0, causing
you to update the input cursor to the next record in the input data set.
The following statement causes the input cursor to skip two records every time it is
called:
inCur.getRecord(2)

Note that there is no way to go back to the skipped records.

76 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

Table 20 Accessing Records with Cursors in runLocally() (continued)

9 Process the record (including writing any results to the current output record). You
need not call APT_OutputCursor::putRecord() until after you have written to the
first output record, because the output cursor initially points to the first empty
record in an output data set.
Fields in the output record are read/write. This allows you to use the output record
as temporary storage within your operator. The output field values are not
permanent until you call APT_OutputCursor::putRecord() to advance the output
cursor to the next output record.

10 Use APT_OutputCursor::putRecord() to update the current output record, then

advance the output cursor to the next output record.
Not all operators produce an output record for each input record. Also, operators
can produce more output records than there are input records. An operator can
process many input records before computing a single output record. You call
putRecord() only when you have completed processing an output record,
regardless of the number of input records you process between calls to
putRecord().

Field Accessors
Once you have defined a cursor to reference the records of a data set, you define
field accessors to reference record fields. You assign field accessors to each
component of a data set’s record schema that you want to access.
For an input or output data set, field accessors provide named access to the record
fields. Such access is necessary if an operator is to process data sets. No field
accessor is allowed for schema variables, which have no defined data type.
Operators use field accessors to read the fields of an input record and to write the
fields of an output record. Field accessors do not allow access to the entire data set
at one time; instead, they allow you to access the fields of the current input or
output record as defined by an input or output cursor.
Field accessors allow you to work with nullable fields. Using accessors, you can
determine if a field contains a null before processing the field, or you can set a
field to null.
Note that within Orchestrate, the fields of an input record are considered read
only. There is no mechanism for you to write into the fields of the records of an
input data set. Because the fields of an output record are considered read/write,
you can modify the records of an output data set.
This section describes both input and output accessors.

Orchestrate 7.0 Developer Guide 77

Chapter 7 Using Cursors and Accessors Field Accessors

Defining Field Accessors

Each component of a data set’s record schema has a name and data type. When
you define a field accessor, you specify a corresponding data type for the field
accessor.
Orchestrate supplies the following accessor classes for its supported data types:

Table 21 Field Accessor Classes

Field Type Input Accessor Class Output Accessor Class

8-bit signed integer APT_InputAccessorToInt8 APT_OutputAccessorToInt8

8-bit unsigned APT_InputAccessorToUInt8 APT_OutputAccessorToUInt8

integer

16-bit signed integer APT_InputAccessorToInt16 APT_OutputAccessorToInt16

16-bit unsigned APT_InputAccessorToUInt16 APT_OutputAccessorToUInt16

integer

32-bit signed integer APT_InputAccessorToInt32 APT_OutputAccessorToInt32

32-bit unsigned APT_InputAccessorToUInt32 APT_OutputAccessorToUInt32

integer

64-bit signed integer APT_InputAccessorToInt64 APT_OutputAccessorToInt64

64-bit unsigned APT_InputAccessorToUInt64 APT_OutputAccessorToUInt64

integer

Single-precision float APT_InputAccessorToSFloat APT_OutputAccessorToSFloat

Double-precision APT_InputAccessorToDFloat APT_OutputAccessorToDFloat

float

String APT_InputAccessorToString APT_OutputAccessorToString

Ustring APT_InputAccessorToUString APT_OutputAccessorToUString

Raw APT_InputAccessorToRawField APT_OutputAccessorToRawField

Date APT_InputAccessorToDate APT_OutputAccessorToDate

Decimal APT_InputAccessorToDecimal APT_OutputAccessorToDecimal

Time APT_InputAccessorToTime APT_OutputAccessorToTime

Timestamp APT_InputAccessorToTimestamp APT_OutputAccessorToTimestamp

78 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

Here is example code that uses three of the field accessor classes:
// Define input accessors
APT_InputAccessorToInt32 aInAccessor;
APT_InputAccessorToSFloat bInAccessor;
APT_InputAccessorToString cInAccessor;

In addition, the classes APT_InputTagAccessor and APT_OutputTagAccessor

provide access to the tag of a tagged aggregate. This is necessary to determine the
current data type of a tagged aggregate. You must also define individual
accessors for each element of a tagged aggregate. The tag of a tagged aggregate in
an input data set is read-only; the tag of a tagged aggregate in an output data set
is read/write. See “Using Field Accessors to Aggregate Fields” on page 7-31 for
more information.

Using Field Accessors

Field accessors work very much like pointers in that they reference a location in
the record of a specific data type. While field accessors operate like pointers, they
are defined by Orchestrate classes. Do not attempt to mix field-accessor objects
with standard C pointers.
When processing the records of a data set, you are typically interested in the value
of a particular field, not its location. As with pointers, therefore, if you want to
access the value of the field referenced by a field accessor, you must dereference
the field accessor using the * operator for scalars, and the [] operator for vectors.
You can also use member functions of the accessor classes to read a field value for
input data sets, and to read or write field values for an output data set.
All of the input accessor classes, except the TagAccessor classes, support the
following member functions:
• value() returns the value of an input field (same as *).
• valueAt() returns the value of an input vector field (same as []).
• vectorLength() returns the length of an input vector field.
All of the output accessor classes support the following member functions:
• value() returns the value of an output field.
• valueAt() returns the value of an output vector field.
• vectorLength() returns the length of an output vector field.
• setValue() sets the value of an output field.
• setValueAt() sets the value of an output vector field.
• setVectorLength() sets the length of an output vector field.

Orchestrate 7.0 Developer Guide 79

Chapter 7 Using Cursors and Accessors Field Accessors

The remaining sections of this chapter describe how to use accessors with
different field types. These sections include:
• “Using Field Accessors to Numeric Data Types”
• “Using Field Accessors to Decimal Fields”
• “Using Field Accessors to Fixed-Length Vector Fields”
• “Using Field Accessors to Variable-Length Vectors”
• “Using Field Accessors to Nullable Fields”
• “Using Field Accessors to string and ustring Fields”
• “Using Field Accessors to Raw Fields”
• “Using Field Accessors to Date, Time, and Timestamp Fields”
• “Using Field Accessors to Aggregate Fields”

Using Field Accessors to Numeric Data Types

This section contains a basic example using both input and output accessors to
numeric data types. The following figure shows a sample operator and interface
schemas:
input data set

field1:int32; field2:int32;

AddOperator
field1:int32; field2:int32;

output data set

This operator adds two fields of an input record and stores the sum in a field of
the output record. In addition, this operator copies the two fields of the input to
corresponding fields of the output.
For each of the components of an operator’s input and output interface schemas,
you define a single field accessor. In this case, therefore, you need two input
accessors for the input interface schema and three output accessors for the output
interface schema.
This example uses field accessors to explicitly copy field1 and field2 from an input
record to the corresponding fields in an output record. If the input data set had a
record schema that defined more than these two fields, all other fields would be
ignored by AddOperator and not copied to the output data set.
The code in Table 22 below is the describeOperator() function for AddOperator:

7 10 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

Table 22 Accessors to Numeric Data Types in describeOperator()

Code

APT_Status AddOperator::describeOperator()
{
3 setInputDataSets(1);
4 setOutputDataSets(1);

5 setInputInterfaceSchema(“record (field1:int32; field2:int32)”,0);

6 setOutputInterfaceSchema(“record (field1:int32; field2:int32)”,0);

return APT_StatusOk;
}

Comments

3 Set the number of input data sets to 1.

4 Set the number of output data sets to 1.

5 Specify the interface schema of input 0 (input data sets are numbered starting from 0).
You can pass a string containing the interface schema as an argument to
APT_Operator::setInputInterfaceSchema().

6 Specify the interface schema of output 0 (the first output data set).

Field accessors are defined as local variables of APT_Operator::runLocally(). To

access the values of fields defined in an interface schema, you must create input
accessors and output accessors in the APT_Operator::runLocally() function of
this operator. The code in Table 23 shows how the runLocally() function for
AddOperator would be written:

Table 23 Creating Accessors in runLocally()

Code

APT_Status AddOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);

7 APT_InputAccessorToInt32 field1InAcc(“field1”, &inCur);

Orchestrate 7.0 Developer Guide 7 11

Chapter 7 Using Cursors and Accessors Field Accessors

Table 23 Creating Accessors in runLocally() (continued)

8 APT_InputAccessorToInt32 field2InAcc(“field2”, &inCur);

9 APT_OutputAccessorToInt32 field1OutAcc(“field1”, &outCur);

10 APT_OutputAccessorToInt32 field2OutAcc(“field2”, &outCur);
11 APT_OutputAccessorToInt32 totalOutAcc(“total”, &outCur);

12 while (inCur.getRecord())
{
14 *totalOutAcc = *field1InAcc + *field2InAcc;

15 *field1OutAcc = *field1InAcc;
16 *field2OutAcc = *field2InAcc;

17 outCur.putRecord();
}
return APT_StatusOk;
}

Comments

7-8 Define read-only accessors for the fields of the operator’s input interface schema.

9-11 Define read/write accessors for the fields of the operator’s output interface
schema.

12 Use APT_InputCursor::getRecord() to advance the input data set to the next input
record.

14-16 Dereference the field accessors to access the values of the record fields in both the
input and the output data sets.

17 Use APT_OutputCursor::putRecord() to update the current output record and

advance the output data set to the next output record.

Using Field Accessors to Decimal Fields

Orchestrate uses the class APT_Decimal to represent decimal record fields. In
order to access a record field of type APT_Decimal, you use the accessors:
APT_InputAccessorToDecimal and APT_OutputAccessorToDecimal.
Once you have defined an accessor to a decimal, you use the dereference
operator, ->, to call member functions of APT_Decimal to process the field.

7 12 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

The APT_Decimal Class Interface

The following figure shows the interface to APT_Decimal. You use the member
functions to define and manipulate a decimal field. See the header file,
$APT_ORCHHOME/include/apt_util/decimal.h, for descriptions of the functions.
You can also browse the Orchestrate class hierarchy using HTML-documented
header files. Start your search with the index.html file in
$APT_ORCHHOME/doc/html/.

APT_Decimal class interface

public:
constructor content() operator==()
destructor effectiveIntegerDigits() operator!=()
assignFromDecimal() effectivePrecision() operator<()
assignFromDFloat() fractionStart() operator<=()
assignFromInt32() hash() operator>()
assignFromSInt64() integerDigits() operator>=()
assignFromString() integerSize() overlapP()
assignFromUInt64() isValid() precision()
asDFloat() leadingZero() releaseStorage()
asInteger() makeInvalid() repSize()
asIntegerS64() negativeP() setScale()
asIntegerU64() operator const void * signNybble()
asString() operator=() signOk()
clear() operator!() stringLength()
compare()

The APT_Decimal class does not provide arithmetic functions. In order to use a
decimal within an arithmetic expression, you must first convert it to an integer or
float. The following code creates an accessor to a decimal field, then uses the
accessor to call member functions of APT_Decimal to convert it to a dfloat:
APT_InputAccessorToDecimal field1InAcc("field1", &inCur);
APT_OutputAccessorToDecimal field1OutAcc("field1", &outCur);
while (inCur.getRecord())
{
APT_DFloat var2 = field1InAcc->asDFloat();
field1OutAcc->assignFromDFloat(var2);
. . .
}

Using Field Accessors to Fixed-Length Vector Fields

You may also want to use vector fields in an operator. Orchestrate does not
supply special accessors for vector fields. Instead, you use the same accessors for
vectors as you do for scalars. However, member functions are built into the
accessor classes are to support vector access.

Orchestrate 7.0 Developer Guide 7 13

Chapter 7 Using Cursors and Accessors Field Accessors

This section describes how to handle fixed-length vectors. See“Using Field

Accessors to Variable-Length Vectors” on page 7-15 for more information on
variable-length vectors.
The following figure shows an operator containing a vector field in its interface
schemas:
input data set

field1[10]:int32;

AddOperator
field1[10]:int32; total:int32;

output data set

This operator adds all elements of the vector in the input record and stores the
sum in a field of the output record. In addition, this operator copies the input
vector to the output.
For a vector, you only need to define a single accessor to access all vector
elements. The runLocally() function for AddOperator would be written as shown
in Table 24:

Table 24 Accessors to Fixed-Length Vector Fields in runLocally()

Code

APT_Status AddOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);

7 APT_InputAccessorToInt32 field1InAcc(“field1”, &inCur);

8 APT_OutputAccessorToInt32 field1OutAcc(“field1”, &outCur);

9 APT_OutputAccessorToInt32 totalOutAcc(“total”, &outCur);

while (inCur.getRecord())
{
12 *totalOutAcc = 0;

13 for (int i = 0; i < 10; i++)

{
15 *totalOutAcc = *totalOutAcc + field1InAcc[i];

7 14 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

Table 24 Accessors to Fixed-Length Vector Fields in runLocally() (continued)

16 field1OutAcc[i] = field1InAcc[i];
}

outCur.putRecord();
}
return APT_StatusOk;
}

Comments

7 Define a read-only accessor for the fields of the operator’s input interface schema.

8-9 Define read/write accessors for the fields of the operator’s output interface schema.

12 Clear the total in the output record. Note that the initial value of all numeric fields
in an output record is already 0 or NULL if the field is nullable. This statement is
included for clarity only.

13 Create a for loop to add the elements of the input vector to the output total field.

15 Dereference the field accessors and to access the values of the vector elements.

16 Copies the value of element i of the input vector to element i of the output vector.
Since the output interface schema defines the length of the vector in the output
record, you do not have to set it, but you must set the vector length of a variable-
lenth vector.
Note that you also could have used the equivalent statement shown below to write
the field value:
bOutAccessor.setValueAt(i, bInAccessor.valueAt(i));

Using Field Accessors to Variable-Length Vectors

An operator can take variable-length vector fields as input, and can create
variable-length vector fields on output. Each input vector field record can contain
a different length vector. To process the input record, the operator must first
determine the length of the vector and must also calculate a length for the output
vector field. The default length of a variable-length vector in an output data set is
0.

Orchestrate 7.0 Developer Guide 7 15

Chapter 7 Using Cursors and Accessors Field Accessors

The following figure shows an operator containing a vector field in its interface
schemas:
input data set

field1[]:int32;

AddOperator
field1[]:int32; total:int32;

output data set

This operator adds all the elements of the vector in the input record and stores the
sum in a field of the output record. In addition, this operator copies the input
vector to the output. Since the input interface schema defines a variable length for
the vector in the output record, the output interface schema contains a
corresponding variable-length vector.
For a vector, you only need to define a single accessor to access all vector
elements. The runLocally() function for AddOperator in Table 25 would be written
as:

Table 25 Accessors to Variable-Length Vector Fields in runLocally()

Code

APT_Status AddOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);

7 APT_InputAccessorToInt32 field1InAcc(“field1”, &inCur);

8 APT_OutputAccessorToInt32 field1OutAcc(“field1”, &outCur);

9 APT_OutputAccessorToInt32 totalOutAcc(“total”, &outCur);

while (inCur.getRecord())
{
12 *totalOutAcc = 0;

13 field1OutAcc.setVectorLength(field1InAcc.vectorLength());

14 for (int i = 0; i < field1InAcc.vectorLength(); i++)

{
16 *totalOutAcc = *totalOutAcc + field1InAcc[i];

7 16 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

Table 25 Accessors to Variable-Length Vector Fields in runLocally() (continued)

17 field1OutAcc[i] = field1InAcc[i];
}

outCur.putRecord();
}
return APT_StatusOk;
}

Comments

7 Define a read-only accessor for the fields of the operator’s input interface schema.

8-9 Define read/write accessors for the fields of the operator’s output interface schema.

12 Clear the total in the output record. Note that the initial value of all numeric fields
in an output record is already 0 or NULL if the field is nullable. This statement is
included for clarity only.

13 Set the length of the variable-length vector field in the output record.

14 Create a for loop to add the elements of the input vector to the output total field.
APT_InputAccessorToInt32::vectorLength() returns the length of a vector field.

16 Use APT_InputAccessorToInt32::operator*, APT_InputAccessorToInt32::operator[],

and APT_OutputAccessorToInt32::operator* to dereference the field accessors and
to access the values of the vector elements.
Note that you can also use the equivalent member functions to access the fields of
input and output records. Shown below is Line 16 rewritten using these functions:
totalOutAcc.setValue(totalOutAcc.value() + field1InAcc.valueAt(i));

17 Copies the value of element i of the input vector to element i of the output vector.
You can also use the equivalent statement shown below to write the field value:
bOutAccessor.setValueAt(i, bInAccessor.valueAt(i));

Using Field Accessors to Nullable Fields

A field’s nullability defines whether or not the field may contain a null. A field
containing a null does not contain a value. For example, you may have a data set
whose record schema contains an age field. If the age field of a particular record is
null, the age is not known for the person corresponding to the record.

Orchestrate 7.0 Developer Guide 7 17

Chapter 7 Using Cursors and Accessors Field Accessors

As part of processing a record field, you can detect a null and take the appropriate
action. For instance, you can omit the null field from a calculation, signal an error
condition, or take some other action.
To recognize a nullable field, the field of the operator’s interface must be defined
to be nullable. You include the keyword nullable in the interface specification of a
field to make it nullable. For example, all fields of the operator shown in the
following figure are nullable:
input data set

field1:nullable int32; field2:nullable int32;

AddOperator
field1:nullable int32; field2:nullable int32; total:nullable int32;

output data set

The code in Table 26 below is the describeOperator() function for AddOperator:

Table 26 Nullable Fields in describeOperator() Schema

Code

APT_Status AddOperator::describeOperator()
{
setKind(APT_Operator::eParallel);

setInputDataSets(1);
setOutputDataSets(1);

6 setInputInterfaceSchema(“record (field1:nullable int32;

field2:nullable int32);
7 setOutInterfaceSchema(“record (field1:nullable int32;
field2:nullable int32);

return APT_StatusOk;
}

Comments

6 Specify that all fields of the interface schema of input 0 are nullable. You can
individually specify any or all fields of the interface schema as nullable.

7 18 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

Table 26 Nullable Fields in describeOperator() Schema (continued)

7 Specify that all fields of the output interface schema are nullable. For the first
record in the output data set, and after each call to
APT_OutputCursor::putRecord(), the null indicator in all nullable output fields is
set, marking the field as containing a null. Writing a value to an output field clears
the null indicator.

If an input field to an operator contains null and the corresponding operator

interface field is not nullable, Orchestrate issues a fatal error and aborts your
application. You can use view adapters to prevent a fatal error in this case. See
“Using a View Adapter with a Partitioner” on page 8-16 and “Example Collector
with a View Adapter” on page 9-16 for information in how to use field adapters.
Both input and output accessors contain member functions for working with
nulls. For input accessors, you use:
• isNull(): Returns true if the accessor references a field containing a null.
• isNullAt(): Returns true if the accessor references a vector element containing
a null.
• isNullable(): Returns true if the accessor references a nullable field.
For output accessors, you use:
• isNull(): Returns true if the accessor references a field containing a null.
• isNullAt(): Returns true if the accessor references a vector element containing
a null.
• isNullable(): Returns true if the accessor references a nullable field.
• clearIsNull(): Clears the null indicator for the field referenced by an accessor
and sets the field to the default value. If isNullable() returns false,
clearIsNull() does nothing.
For the first record in the output data set, and after each call to
APT_OutputCursor::putRecord(), the null indicator in all nullable output
fields of the new output record is set, marking the field as containing a null.
Writing a value to an output field clears the null indicator, so it is usually
unnecessary to call clearIsNull().
• clearIsNullAt(): Clears the null indicator for the vector element referenced by
an accessor, and sets the value of the field to the default value for its type.
• setIsNull(): Sets the null indicator for the field referenced by an accessor,
marking the field to contain a null. setIsNull() requires that isNullable()
returns true.
• Because the null flag for all nullable output fields is initially set, you only need
to call setIsNull() if you have written valid data to an output field and later
decide to set the field to null.

Orchestrate 7.0 Developer Guide 7 19

Chapter 7 Using Cursors and Accessors Field Accessors

• setIsNullAt(): Sets the null indicator for a vector element referenced by an

accessor, marking the field to contain a null.
You typically use the operator* member function of an input and output accessor
to obtain a field’s value. For an input accessor, using operator* on a field
containing a null causes a requirements violation and aborts your application.
Therefore, you should first determine if a nullable input field contains a null,
using isNull(), before attempting to access it.
For an output accessor, calling operator* always clears the null indicator for a
nullable field, marking the field to contain valid data. You can use setIsNull() to
explicitly set the null indicator in an output field, if necessary.
How you choose to handle nulls detected in an input field or what conditions
cause an output field to be set to null is determined by your operator logic. Often,
a null in an input field will be propagated through to the operator’s output. For
example, the code in Table 27 shown below is the APT_Operator::runLocally()
function for AddOperator:

Table 27 Handling Nullable Fields in runLocally()

Code

APT_Status AddOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);

APT_InputAccessorToInt32 field1InAcc(“field1”, &inCur);

APT_InputAccessorToInt32 field2InAcc(“field2”, &inCur);

APT_OutputAccessorToInt32 field1OutAcc(“field1”, &outCur);

APT_OutputAccessorToInt32 field2OutAcc(“field2”, &outCur);
APT_OutputAccessorToInt32 totalOutAcc(“total”, &outCur);

while (inCur.getRecord())
{
14 if (!field1InAcc.isNull())
*field1OutAcc = *field1InAcc;

16 if (!field2InAcc.isNull())
*field2OutAcc = *field2InAcc;

18 if (!field2InAcc.isNull() && !field2InAcc.isNull())

*totalOutAcc = *field1InAcc + *field2InAcc;

7 20 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

Table 27 Handling Nullable Fields in runLocally() (continued)

20 outCur.putRecord();
}

return APT_StatusOk;
}

Comments

14 Use field1InAcc.isNull() to determine if field1 contains a null. If not, copy it to the

output record.

16 Determine if field2 contains a null. If not, copy it to the output record.

18 If both field1 and field2 contain valid data, perform the addition. Writing to total
clears the null indicator for the field.

20 Call APT_OutputCursor::putRecord() to write the output record.

Using Field Accessors to string and ustring Fields

A string data-type field and a ustring data-type field contain fixed-length or
variable-length character strings. Orchestrate uses the class APT_String to define
a string field and the class APT_UString to define a ustring field. Strings of both
classes are not null terminated; that is, they always have an explicit length. They
can contain null characters.

APT_String and APT_UString Class Interface

APT_String represents the value type of the string schema field used for
processing single-byte character data. It stores its contents in a plain char array.
Strings are sorted using byte-wise comparisons. APT_String assumes all streams
are a sequence of 8-bit characters, and no conversion is done.
APT_UString represents the value type of the ustring schema field used for
processing multi-byte Unicode character data. It stores its contents in an UChar
array. UChar is defined in the ICU header file unicode/utf.h, and its
implementation is operating-system dependent. Strings are sorted using the
collation services described in the Orchestrate 7.0 User Guide. APT_UString uses
conversion services for input from and output to streams which are UTF-8
encoded.
The next figure shows the interface to APT_String and APT_UString. Both these
classes derive directly from APT_BasicString.

Orchestrate 7.0 Developer Guide 7 21

Chapter 7 Using Cursors and Accessors Field Accessors

See the header files, basicstring.h, string.h, and ustring.h in

$APT_ORCHHOME/include/apt_util/ for descriptions of these functions. Also see
the header file unicode_utils.h.
You can also browse the Orchestrate class hierarchy using HTML-documented
header files. Start your search with the index.html file in
$APT_ORCHHOME/doc/html/.

APT_String/APT_UString class interface

public:
constructor isBound() operator>>()
destructor getCollationSeq() operator<<()
addPadding() hash() operator<=()
adopt() isBoundedLength() operator>=()
adopt_badarg() isEmpty() operator ||()
allocBuf() isEmpty_fixed() padChar()
append() isEqualCI() prepareForFielding()
append2() isFixedLength() prepend()
asFloat() isLower() removePadding()
asInteger() isUpper() replace()
assignFrom() isTransformNecessary() setBoundedLength()
badIndex() isVariableLength() setCollationSeq()
bind() length() setFixedLength()
clear() nonTerminatedContent() setLength()
compactPadding() offsetOfSubstring() setPadChar()
compare() occurrences() setVariableLength()
content() operator[ ]() substring()
data() operator+() terminatedContent()
data_nonOwn() operator+=() transform()
equal_CI() operator=() transformLength
equals() operator==() trimPadding()
equivalent() operator!=() toLower()
getChar() operator>() toUpper()
initFrom() operator<()

You access a field of type APT_String or APT_UString using the field accessor
APT_InputAccessorToString or APT_InputAccessorToUString and
APT_OutputAccessorToString or APT_OutputAccessorToUString. Once you
have defined and initialized the accessor, you then use indirect addressing, via
the dereferencing operator ->, to call a member function of APT_String or
APT_UString to process the field.
You can see from the class member functions that APT_String and APT_UString
allow you to copy a string field, using operator=, and compare string fields using
operator==, operator!=, isEqualCI()and other functions. These classes also
contain member functions to access the contents and length of a string field.

7 22 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

The following code creates an input accessor to a string field, then uses the
accessor to call member functions of APT_String:
APT_InputAccessorToString field1InAcc(“field1”, &inCur);

while (inCur.getRecord())
{
size_t fieldLen = field1InAcc->length();
const char * buffer = field1InAcc->content();
. . .
}

The following figure shows an operator containing a variable-length string field

and a fixed-length string field in its interface schemas:
input data set

field1:string; field2:string[10];

StringOperator
field1:ustring; field2:string[10];

output data set

The runLocally() function shown in Table 28 below shows how the StringOperator
would be written:

Table 28 Handling String and Ustring Fields in runLocally()

Code

APT_Status StringOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);

7 APT_InputAccessorToUString field1InAcc(“field1”, &inCur);

8 APT_InputAccessorToString field2InAcc(“field2”, &inCur);

9 APT_OutputAccessorToUString field1OutAcc(“field1”, &outCur);

10 APT_OutputAccessorToString field2OutAcc(“field2”, &outCur);

while (inCur.getRecord())
{
13 APT_Int32 fieldLen = field1InAcc->length();
14 const UChar * buffer = field1InAcc->content();

Orchestrate 7.0 Developer Guide 7 23

Chapter 7 Using Cursors and Accessors Field Accessors

Table 28 Handling String and Ustring Fields in runLocally() (continued)

15 field1OutAcc->assignFrom(buffer, fieldLen);
16 *field2OutAcc = *field2InAcc;

outCur.putRecord();
}
return APT_StatusOk;
}

Comments

7-8 Define the input string accessors using APT_InputAccessorToUString and

APT_InputAccessorToString.
Once you have set up an accessor to a field of type APT_UString or APT_String,
you can use the member functions of APT_UString or APT_String to manipulate
the field. Because accessors are a type of pointer, you can use them to call
APT_String or APT_String member functions using the dereferencing operator, -
>.

9-10 Define the output string accessors using APT_OutputAccessorToUString and

APT_OutputAccessorToString.

13 Determine the number of codepoints in the variable-length ustring field.

14 Return a pointer to the contents of the ustring field. Note that because a string field
is not defined to be null terminated, you may need the length of the field as well.

15 Copy the buffer to the output field, including the ustring length. By default, a
variable-length string field in an output data set has a length of 0; you must set the
string length as part of writing to the field.

16 Directly copy the fixed-length input string field to the output string field.
If the input fixed-length string is longer than the output fixed-length string, the
input string is truncated to the length of the output. If the input string is shorter,
the output string is by default padded with zeros to the length of the output string.
You can call setPadChar() to specify a different pad character.

Processing fixed and variable length vectors of string fields is the same as for
vectors of numeric data types. See “Using Field Accessors to Fixed-Length Vector
Fields” on page 7-13 or “Using Field Accessors to Variable-Length Vectors” on
page 7-15 for an example.

7 24 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

Unicode Utilities
The Unicode utility functions are listed in this section. The header file for these
functions is $APT_ORCHHOME/include/apt_util/unicode_utils.h.

Character Set Conversion Functions

• UTF-8 character set conversion functions:
APT_UCharFromUTF8Stream
APT_UBufferFromUTF8Stream
APT_ULineFromUTF8Stream
APT_ReadUStrFromUTF8Stream
APT_ConvertToUTF8String
APT_ConvertFromUTF8String

• OS character set conversion functions:

APT_ConvertToNativeString
APT_ConvertFromNativeString

• Input character set conversion functions:

APT_ConvertFromInputString
APT_ConvertFromInputStream

• Invariant ASCII character set conversion functions:

APT_UcharsToChars
APT_CharsToUChars

• User-specified character set conversion functions:

ConvertToString
ConvertFromString

String Manipulation Functions.

These functions accept both char and UChar arguments unless otherwise noted:
APT_strchr
APT_strstr
APT_strlen
APT_strcmp
APT_strncmp
APT_strcasecmp (UChar only)
APT_strcpy
APT_strncpy
APT_memset
APT_memcpy
APT_memcmp

Ctype Functions
These functions accept both char and Uchar argments unless otherwise noted:
isprint
isspace
isdigit
digit (UChar only)

Orchestrate 7.0 Developer Guide 7 25

Chapter 7 Using Cursors and Accessors Field Accessors

isalpha
isalnum
islower
tolower
isupper
toupper

File-Related Functions
These functions accept both char and UChar arguments:
APT_fopen
APT_Ufprint
APT_Usprintf
APT_Usscanf

APT_UString Comparison Methods

The following methods accept a collation sequence:
CompareResult compare(const APT_BASICString& str,
CaseMatch c=eCaseSensitive, const APT_CollationSeq* seq=0) const;
CompareResult compare(const C* str, APT_Int32 len,
CaseMatch c=eCaseSensitive, const APT_CollationSeq* seq = 0) const;
CompareResult compare(const C* str, CaseMatch c=eCaseSensitive,
const APT_CollationSeq* seq = 0) const;
bool equivalent(const APT_BasicString& str,
CaseMatch c=eCaseSensitive, const APT_CollationSeq *seq = 0) const;
APT_int32 transform(C* newData, APT_Int32 length,
bool caseInsensitive = false, APT_Int32 strLen = -1,
const APT_CollationSeq* seq = NULL) const;
APT_Int32 transformLength(const APT_CollationSeq* seq = NULL) const;
bool istransformNecessary(const APT_CollationSeq* seq = NULL) const;
void setCollationSeq(APT_CollationSeq* col);
const APT_CollationSeq* getCollationSeq
(const APT_CollationSeq* seq = NULL) const;

The <, >, <=, and => operators also use a collation sequence if a non-default
sequence is available. The default collation sequence uses byte-wise comparisons.

Using Field Accessors to Raw Fields

Orchestrate lets you create raw fields which are untyped collections of contiguous
bytes, and aligned raw fields, a special type of raw field where the first byte is
aligned to a specified address boundary. Raw and aligned raw fields may be
either fixed-length or variable-length.

The APT_RawField Class Interface

The following figure shows the interface to APT_RawField. Orchestrate uses the
class APT_RawField to define both raw and aligned raw fields and provides
member functions to manipulate these fields. See the header file,

7 26 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

$APT_ORCHHOME/include/apt_framework/rawfield.h, for descriptions of these

functions. You can also browse the Orchestrate class hierarchy using HTML-
documented header files. Start your search with the index.html file in
$APT_ORCHHOME/doc/html/.

APT_RawField class interface

public:
constructor isFixedLength() operator<<()
destructor isVariableLength() operator>>()
assignFrom() length() setBoundedLength()
bind() operator=() setFixedLength()
clear() operator==() setLength()
content() operator!=() setVariableLength()
isBoundedLength()

You can access a field of type APT_RawField using the field accessors
APT_InputAccessorToRawField and APT_OutputAccessorToRawField. Once
you have defined and initialized the accessor, you then use indirect addressing,
via the dereferencing operator ->, to call the member functions of APT_RawField
to process the field.
As shown in the class member functions shown above, you can assign to an
APT_RawField object using operator=, as well as compare APT_RawField
objects using operator== and operator!= and other functions. APT_RawField
also contains member functions to access the contents of a raw field.
For example, the following code creates an input accessor to a raw field, then uses
the accessor to call member functions of APT_RawField:

APT_InputAccessorToRawField field1InAcc("field1", &inCur);

while (inCur.getRecord())
{
size_t fieldLen = field1InAcc->length();
const void * buffer = field1InAcc->content();
. . .
}

The following figure shows an operator containing a variable-length and a fixed-

length raw field in its interface schemas:
input data set

field1:raw; field2:raw[10];

RawOperator
field1:raw; field2:raw[10];

output data set

Orchestrate 7.0 Developer Guide 7 27

Chapter 7 Using Cursors and Accessors Field Accessors

The runLocally() function for RawOperator would be written as shown in Table 29

below:

Table 29 Handling Raw Fields in runLocally()

Code

APT_Status RawOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);

7 APT_InputAccessorToRawField field1InAcc(“field1”, &inCur);

8 APT_InputAccessorToRawfield field2InAcc(“field2”, &inCur);

9 APT_OutputAccessorToRawField field1OutAcc(“field1”, &outCur);

10 APT_OutputAccessorToRawField field2OutAcc(“field2”, &outCur);

while (inCur.getRecord())
{
13 size_t fieldLen = field1InAcc->length();
14 const void * buffer = field1InAcc->content();

15 field1OutAcc->assignFrom(buffer, fieldLen);
16 *field2OutAcc = *field2InAcc;

outCur.putRecord();
}

return APT_StatusOk;
}

Comments

7-8 Define the input raw accessors using APT_InputAccessorToRawField.

Once you have set up an accessor to a field of type APT_RawField, you can use the
member functions of APT_RawField to manipulate the field. Because accessors are
a type of pointer, you can use them to call APT_RawField member functions using
the dereferencing operator, ->.

9-10 Define the output raw accessors using APT_OutputAccessorToRawField.

13 Use APT_RawField::length() to return the number of bytes in the variable-length

raw field.

7 28 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

Table 29 Handling Raw Fields in runLocally() (continued)

14 Use APT_RawField::content() to return a pointer to the contents of the raw field.

The returned pointer is of type void*. Note that because a raw field is not defined
to be null terminated, you may need the length of the field as well.

15 After processing, use APT_RawField::assignFrom() to copy the results to the

output raw field, including the field length. You must set the length as part of
writing to the field. By default, a variable-length raw field in an output data set has
a length of 0.

16 Use APT_OutputAccessorToRawField::operator* to copy the input fixed-length

raw field directly to the output field. Because the output field is read/write, you
can process the raw field in place.

Processing vectors of raw fields, either fixed or variable length, is the same as for
vectors of numeric data types. See “Using Field Accessors to Fixed-Length Vector
Fields” on page 7-13 or “Using Field Accessors to Variable-Length Vectors” on
page 7-15 for an example.

Using Field Accessors to Date, Time, and Timestamp Fields

Orchestrate allows you to access fields containing a date, time, or timestamp. It
uses the following classes to represent these data types:
• APT_Date
• APT_Time
• APT_TimeStamp
In order to access a record field of these types, you use the following accessors:

Table 30 Date, Time, and Timestamp Type Field Accessor Classes

Field Type Input Accessor Class Output Accessor Class

Date APT_InputAccessorToDate APT_OutputAccessorToDate

Time APT_InputAccessorToTime APT_OutputAccessorToTime

Timestamp APT_InputAccessorToTimestamp APT_OutputAccessorToTimestamp

Once you have defined an accessor to one of these fields, you use the accessor’s
dereference operator, ->, to call member functions of the corresponding class to
process the field.

Orchestrate 7.0 Developer Guide 7 29

Chapter 7 Using Cursors and Accessors Field Accessors

For example, the following figure shows an operator containing a date field in its
interface schemas:
input data set

field1:date;

DateOperator
field1:date;

output data set

The runLocally() function for DateOperator would be written as shown in Table 31

below:

Table 31 Handling a Date Field in runLocally()

Code

APT_Status DateOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);

7 APT_InputAccessorToDate field1InAcc(“field1”, &inCur);

8 APT_OutputAccessorToDate field1OutAcc(“field1”, &outCur);

9 APT_Date cutoffDate(1997, 01, 01);

while (inCur.getRecord())
{
if (*field1InAcc < cutoffDate);
{
int year = field1InAcc->year();
int month = field1InAcc->month();
int day = field1InAcc->day();
. . .
}
outCur.putRecord();
}

return APT_StatusOk;
}

7 30 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

Table 31 Handling a Date Field in runLocally() (continued)

Comments

7 Create an input accessor to the input data field.

8 Create an output accessor to the input data field.

9 Create cutoffDate, an instance of APT_Date, and initialize it to 1/1/97.

Using Field Accessors to Aggregate Fields

Aggregates are collections of fields somewhat similar to C structures and unions.
Aggregates consist of:
• Subrecords
• Tagged aggregates
If you define aggregates as part of the record schema of a data set, you can define
field accessors to reference the fields of the aggregates.
The following figure shows an operator containing a subrecord and a tagged
aggregate field in its interface schemas:

input data set

a:subrec (aSubField1:int32; aSubField2:sfloat;)

b:tagged (bTaggedField1:string;bTaggedField2:int32;)

AggregateOperator

a:subrec (aSubField1:int32; aSubField2:sfloat;)

b:tagged (bTaggedField1:string;bTaggedField2:int32;)

output data set

In order to access the elements of an aggregate field, you must define accessors to:
• Each element of the aggregate
• The tag for a tagged aggregate

Orchestrate 7.0 Developer Guide 7 31

Chapter 7 Using Cursors and Accessors Field Accessors

The runLocally() function for AggregateOperator is shown below in Table 32:

Table 32 Handling Aggregate Fields in runLocally()

Code

APT_Status AggregateOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);

7 APT_InputAccessorToInt32 aSubfield1In(“a.aSubfield1”, &inCur);

8 APT_InputAccessorToSFloat aSubfield2In(“a.aSubfield2”, &inCur);

9 APT_InputTagAccessor bTagIn;
10 inCur.setupTagAccessor(“b”, &bTagIn);
11 APT_InputAccessorToString bTaggedField1In(“b.bTaggedField1”, &inCur);
12 APT_InputAccessorToInt32 bTaggedField2In(“b.bTaggedField2”, &inCur);

13 APT_OutputAccessorToInt32 aSubfield1Out(“a.aSubfield1”, &outCur);

14 APT_OutputAccessorToSFloat aSubfield2Out(“a.aSubfield2”, &outCur);

15 APT_OutputTagAccessor bTagOut;

16 outCur.setupTagAccessor(“b”, &bTagOut);
17 APT_OutputTagAccessorToString bTaggedField1Out(
“b.bTaggedField1”, &outCur);
18 APT_OutputTagAccessorToInt32 bTaggedField2Out(
“b.bTagedField2”, &outCur);

while (inCur.getRecord())
{
*aSubField1Out = *aSubField1In;
*aSubField2Out = *aSubField2In;

23 switch(bTagIn.tag())
{
case 0:
26 bTagOut.setTag(0);
27 *bTaggedField1Out = *bTaggedField1In;
break;
case 1:
bTagOut.setTag(1);
*bTaggedField2Out = *bTaggedField2In;
break;
}

7 32 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

Table 32 Handling Aggregate Fields in runLocally() (continued)

default:
35 APT_ASSERT();
}

outCur.putRecord();
}

return APT_StatusOk;
}

Comments

7-8 Define input accessor elements for the subrecord. Note that you use dot-delimited
referencing to refer to the fields of an aggregate in much the same way that you do
for the elements of a C structure.
Once you have defined accessors for the subrecord aggregate elements, you access
the fields of a subrecord aggregate in the same way you access ordinary fields.

9-10 Define a tag accessor and accessor elements for the tagged aggregate.

11-12 Define output accessors for fields in the output subrecord.

13-18 Define a tag accessor and accessor elements for the output tagged aggregate.

23 Determine the active tag element. For tagged aggregates, only one element of the
tagged aggregate is active at one time. For an input data set, you use a tag accessor
to determine the currently active element.

26 Set the tag in the output tagged field. For an output record, you must set the tag in
order to specify the data type of a tagged field. Though you can change the tag for
every record in an output data set, data may be destroyed. Once you set the tag for a
record, it is good practice not to change it.

27 Copy the input field to the output field.

35 Use the macro APT_ASSERT(0) to generate an assertion failure if the tag value is
not 0 or 1. This means that the tag has an invalid value because field b is defined to
contain only two elements. Note that this condition should never happen; therefore,
you handle it using an assertion. See Chapter 14, “Using the Error Log” for more
information.

Accessing Vectors of Subrecord Fields

Orchestrate allows you to define vectors of subrecord fields. In order to iterate
through the elements of a vector of subrecords, you need to use a type of cursor

Orchestrate 7.0 Developer Guide 7 33

Chapter 7 Using Cursors and Accessors Field Accessors

called a subcursor. Note that you still need to use cursors to identify the input or
output record and accessors to reference the individual fields of the subrecord.
Like cursors and accessors, Orchestrate defines two types of subcursors: one for
input data sets and one for output data sets. You use the following Orchestrate
classes to define subcursor objects:
• APT_InputSubCursor: provides access for iterating through a vector of
subrecords in an input data set
• APT_OutputSubCursor: provides access for iterating through a vector of
subrecords in an output data set
When you initialize a subcursor, it is set to refer to the first element in the
subrecord vector. Both subcursor classes contain the following member functions
for manipulating the location of the subcursor in a vector: next(), prev(),
setPosition(), and vectorLength(). In addition, APT_OutputSubCursor has the
member function setVectorLength() which you can use to modify the length of a
variable-length subrecord vector in an output data set.
Subcursors reference a vector element relative to the location of an input or
output cursor. As part of initializing a subcursor, you bind it to either an input or
output cursor. Updating an input cursor to the next input record, using
APT_InputCursor::getRecord(), or updating an output cursor to the next output
record, using APT_OutputCursor::putRecord(), resets all bound subcursors to
the first element in their associated vectors.
The following figure shows an operator containing a vector of subrecords:

input data set

a[10]:subrec(aSubField1:int32;aSubField2:Sfloat;)

SubrecordVectorOperator

a[10]:subrec(aSubField1:int32;aSubField2:Sfloat;)

output data set

In order to access the elements of an subrecord vector, you must define:

• An accessor to each element of the subrecord
• A subcursor to each subrecord vector
The runLocally() function for SubrecordVectorOperator would be written a shown
below in Table 33:

7 34 Orchestrate 7.0 Developer Guide

Chapter 7 Using Cursors and Accessors Field Accessors

Table 33 Handling Subrecord Vector Fields in runLocally()

Code

APT_Status SubrecordVectorOperator::runLocally()
{
APT_InputCursor inCur;
APT_OutputCursor outCur;

setupInputCursor(&inCur, 0);
setupOutputCursor(&outCur, 0);

7 APT_InputSubCursor inSubCur(“a”, &inCur);

8 APT_OutputSubCursor outSubCur(“a”, &outCur);

9 APT_InputAccessorToInt32 aSubField1In(“a.aSubField1”, &inCur);

APT_InputAccessorToSFloat aSubField2In(“a.2SubField2”, &inCur);

APT_OutputAccessorToInt32 aSubfield1Out(“a.aSubfield1”, &outCur);

12 APT_OutputAccessorToSFloat aSubfield2Out(“a.aSubfield2”, &outCur);

13 while (inCur.getRecord())
{
15 for (int i = 0; i < inSubCur.vectorLength(); i++)
{
17 *aSubField1Out = *aSubField1In;
18 *aSubField2Out = *aSubField2In;
19 inSubCur.next();
20 outSubCur.next();
}

22 outCur.putRecord();
}

return APT_StatusOk;
}

Comments

7 Define inSubCur, an instance of APT_InputSubCursor. In order to access the

elements of the subrecord vector field, you must define a subcursor for the
aggregate vector and a field accessor to each element of the aggregate.
You pass to the constructor the name of the subrecord field accessed by the
subcursor, and the input cursor used by the subcursor.
Note that an input subcursor starts out as valid, unlike an input cursor.

Orchestrate 7.0 Developer Guide 7 35

Chapter 7 Using Cursors and Accessors Field Accessors

Table 33 Handling Subrecord Vector Fields in runLocally() (continued)

8 Define outSubCur, an instance of APT_OutputSubCursor.

9-12 Create input and output accessors.

13 Call APT_InputRecord::getRecord() to initialize the input cursor to the first input

record. This function also resets all subcursors bound to the input cursor to
reference the first element in the subrecord vector.

15 Create a for loop to index through all vector elements. Use

APT_InputSubCursor::vectorLength() to return the length of the input vector.

17 Copy the first input subrecord field to the output.

18 Copy the second input subrecord field to the output.

19 Use APT_InputSubCursor::next() to increment the input subcursor to the next

element in the vector.

20 Use APT_OutputSubCursor::next() to increment the output subcursor to the next

element in the vector.

22 Use APT_OutputCursor::putRecord() to write the output record. This call resets

all output subcursors bound to this output cursor to reference the first vector
element.

You can also define a vector of subrecords and nest it within a subrecord that is
itself either a vector or a scalar. You use the same procedure described above for
nested subrecord vectors.

7 36 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

8
Creating Partitioners
Describes how to specify the partitioning method of a derived operator and how to
create a custom partitioner.

Overview 7 1
Setting the Preserve-Partitioning Flag 7 3
Choosing a Partitioning Method 7 3
Using the Default Partitioning Method 7 3
Using a Keyless Partitioning Method 7 4
Using the Class APT_HashPartitioner 7 5
Using The Class APT_RangePartitioner 7 7
The APT_Partitioner Class Interface 7 8
Creating a Custom Partitioner 7 8
Overriding APT_Partitioner::describePartitioner() 7 9
Overriding APT_Partitioner::setupInputs() 7 9
Overriding APT_Partitioner::partitionInput() 7 10
Example Partitioning Method Definition 7 11
Orchestrate Hashing Functions 7 15
Using a View Adapter with a Partitioner 7 16

Overview
When you create your own operator classes by deriving from APT_Operator, you
can directly control how an operator partitions data. You can use an Orchestrate-
supplied partitioning method as part of your derived parallel operator or define a
partitioning method by deriving from the base class APT_Partitioner. Once you

81
Chapter 8 Creating Partitioners Overview

have derived a new APT_Partitioner class, you simply include the derived
partitioner within a new operator.
You can even build a library of partitioner classes to use when appropriate. With
this approach, you can mix and match operators and partitioners when different
forms of an operator differ only by how the operator partitions data.
This chapter describes how to specify the partitioning method of a derived
operator and also how to create a custom partitioning method.
When you derive a new parallel operator, you may choose the partitioning
method used by the operator. Orchestrate directly supports the keyless
partitioning methods any, round robin, random, same, and entire. For
partitioners, any is implemented as the most efficient partitioner available, and it
is currently either same or round robin. To use one of these methods, include a
call to APT_Operator::setPartitionMethod() within the
APT_Operator::describeOperator() function of your derived operator. See
“Using a Keyless Partitioning Method” on page 8-4 for information.
For the keyed partitioning method hash by field, Orchestrate supplies the
partitioner class APT_HashPartitioner which allows you to hash a record based
on one or more numeric integer or string fields. See “Using the Class
APT_HashPartitioner” on page 8-5 for information on this class.
Orchestrate also supplies the partitioner class APT_ModulusPartitioner for the
keyed partitioning method modulus. Partitioning is based on a numeric key field
modulo the number of partitions. It is similar to hash by field but involves
simpler computation.
For the range partitioning method, Orchestrate supplies the partitioner class
APT_RangePartitioner. See “Using The Class APT_RangePartitioner” on page 8-
7 for information on this class.
If you want to derive a new partitioning method, or a variation of one of the
methods described above, you can derive a partitioner class from
APT_Partitioner. See “The APT_Partitioner Class Interface” on page 8-8 for
information.
To set the partitioning method for a derived operator, you use the member
functions of both APT_Operator and APT_Partitioner. See Chapter 1, “Creating
Operators” for information on deriving from APT_Operator.
Also see the section “Specifying Partitioning Style and Sorting Requirements” on
page 1-23 which discusses an alternative to stipulating a specific partitioning
method for an operator. This facility allows you to specify your partitioning and
sorting requirements. Based on your specifications, Orchestrate inserts the
apppropriate components into the data flow. This functionality makes it possible
for your operator users to write correct data flows without having to deal with
parallelism issues.

82 Orchestrate 7.0 Developer Guide

Chapter 8 Creating Partitioners Setting the Preserve-Partitioning Flag

Setting the Preserve-Partitioning Flag

Some derived operators carefully order the output data sets as they are processed.
If you want the output data set of a derived operator to preserve its partitioning
so that the next operator to process the data set does not repartition it, you set the
preserve-partitioning flag in the output data set. This typically is done for
operators such as APT_PartitionSortOperator, which creates sorted partitions in
its output data set.
You can use the protected member functions setPreservePartitioningFlag() and
clearPreservePartitioningFlag() to set or clear the preserve-partitioning flag in
the output data set of a derived operator as part of the derived operator’s override
of APT_Operator::describeOperator().
These functions have a lower priority than the member functions of APT_DataSet
which manipulate the preserve-partitioning flag. They can only modify the
preserve-partitioning flag of an output data set if the flag has not been explicitly
set or cleared by the APT_DataSet member functions. Any attempt by an
operator to modify partitioning that has been explicitly set or cleared by the
member functions of APT_DataSet is ignored. You include calls to these member
functions as part of the derived operator’s override of
APT_Operator::describeOperator(). You must call the appropriate member
function for each output data set.

Choosing a Partitioning Method

You must choose a partitioning method appropriate for the particular action of
the associated operator. Some partitioning methods, such as any, round robin,
and random, do not rely on information in the records for partitioning. Other
partitioning methods, such as hash by field, use the fields of the record to
determine the partition for a record.
In some cases, you may want to create an operator that allows the user of the
operator to define the partitioning method. To allow operator user choice, specify
a partitioning method of any for the operator.
See the Orchestrate 7.0 User Guide for a detailed comparison of partitioning
methods.

Using the Default Partitioning Method

The default partitioning method of a derived operator is any. An operator with
this partitioning method does not specify a particular method; therefore,
Orchestrate can partition the input data set in any way that optimizes
performance.

Orchestrate 7.0 Developer Guide 83

Chapter 8 Creating Partitioners Choosing a Partitioning Method

Another advantage of using the any partitioning method is that an operator user
can override the partitioning method. For example, you may create an operator
whose partitioning method is tightly coupled to the actual data set processed at
run time. Specifying the any partitioning methods allows the user to change the
partitioning method for each instance of the operator.
The function APT_Operator::partitionMethod() returns APT_Operator::eAny for
an operator using the partitioning method any.

Using a Keyless Partitioning Method

Orchestrate lets you use the keyless partitioning methods round robin, random,
same, or entire in a derived operator. These partitioning methods are built into
Orchestrate. In order to use any of these partitioning methods, you include a call
to APT_Operator::setPartitionMethod() within the
APT_Operator::describeOperator() function of your derived operator.
APT_Operator::describeOperator() is a pure virtual function that you must
override when deriving from APT_Operator.
The function prototype of APT_Operator::setPartitionMethod() takes two
arguments, as shown below:
void setPartitionMethod(APT_Operator::PartitionMethod pType,
int inputDS );

The first argument, pType, specifies the partitioning method as defined using one
of the following values:
• APT_Operator::eAny (default)
• APT_Operator::eRoundRobin
• APT_Operator::eRandom
• APT_Operator::eSame
• APT_Operator::eEntire
The second argument, inputDS, specifies the number of the input data set to the
operator. Note that the input data sets to an operator are numbered starting from
0.
For example, to use round robin partitioning with an operator that takes a single
input data set, include the following statements within the operator’s
describeOperator() function:
setKind(APT_Operator::eParallel);
setPartitionMethod(APT_Operator::eRoundRobin, 0);

If the operator has two input data sets and you want to partition the data sets
using random, you include the lines:

84 Orchestrate 7.0 Developer Guide

Chapter 8 Creating Partitioners Choosing a Partitioning Method

setKind(APT_Operator::eParallel);
setPartitionMethod(APT_Operator::eRandom, 0); // input data set 0
setPartitionMethod(APT_Operator::eRandom, 1); // input data set 1

Using the Class APT_HashPartitioner

Orchestrate supplies the class APT_HashPartitioner, which partitions a data set
by performing a hashing function on one or more fields of a record.
APT_HashPartitioner uses a dynamic interface schema that allows you to specify
one or more numeric or string fields as input to the partitioner.
The following figure shows an operator using APT_HashPartitioner:

myDS data set

schema: a:int32; b:int32;

c:int16; d:sfloat;

field1:int32; field2:string; field3:int16; in:*;

SortOperator
-----;

...

out:*;

output data set

APT_HashPartitioner does not define any interface schema; you use the
APT_HashPartitioner constructor or the member function
APT_HashPartitioner::setKey() to specify the key fields.
The constructor for APT_HashPartitioner has two overloads:
APT_HashPartitioner();
APT_HashPartitioner(const APT_FieldList& fList);

The first overload creates an APT_HashPartitioner object without specifying any

key fields. You must then use setKey() to specify key fields.
The second form of the constructor creates an APT_HashPartitioner object using
a list of key fields from the input interface schema for the operator. These fields

Orchestrate 7.0 Developer Guide 85

Chapter 8 Creating Partitioners Choosing a Partitioning Method

can be any field type, including raw, date, and timestamp. APT_HashPartitioner
determines the data type of each field from the operator’s input interface schema.
SortOperator requires three fields as input: two integer fields and a string field.
You can specify the partitioner’s interface schema within the describeOperator()
function, as as the code in Table 34 shows below:

Table 34 Partitioner Interface Schema in describeOperator()

Code

APT_Status SortOperator::describeOperator()
{
setKind(APT_Operator::eParallel);

setInputDataSets(1);
setOutputDataSets(1);

setInputInterfaceSchema(“record(field1:int32; field2:int32;
field3:string; in:*;)”, 0);
setOutputInterfaceSchema(“record(out:*;)”, 0);
declareTransfer(“in”, “out”, 0, 0);

9 APT_HashPartitioner * hashPart = new APT_HashPartitioner;

10 hashPart->setKey(“field1”, “int32”);
11 hashPart->setKey(“field2”, “int32”);

12 setPartitionMethod(hashPart, APT_ViewAdapter(), 0);

return APT_StatusOk;
}

Comments

9 Use the default constructor to dynamically allocate an APT_HashPartitioner object.

Partitioner objects must be dynamically allocated within describeOperator(). The
framework deletes the partitioner for you when it is no longer needed.
You must call setKey() to specify the key fields for this APT_HashPartitioner object.

10 Use APT_HashPartitioner::setKey() to specify field1 as a key field for this

APT_HashPartitioner object.
Use setKey() to specify both a field name and a data type for the field. The order in
which key fields are listed is unimportant.

86 Orchestrate 7.0 Developer Guide

Chapter 8 Creating Partitioners Choosing a Partitioning Method

Table 34 Partitioner Interface Schema in describeOperator() (continued)

11 Specify field2 as a key field for this APT_HashPartitioner object.

12 Use APT_Operator::setPartitionMethod() to specify hashPart as the partitioner for

this operator. After calling this function, do not delete the partitioner because the
framework has claim to its memory.
Because you need not use a view adapter with this partitioner, this function creates
and passes a default view adapter.

An application developer using this operator may use adapters to translate the
name of a dataset field and its data type in the input data set schema to match the
operator’s input interface schema. In the previous figure, the data set myDS is
input to the sort operator. An application developer could translate field a and
field c of myDS to field1 and field2 of the operator. Therefore, the hash partitioner
would partition the record by fields a and c. See “Using a View Adapter with a
Partitioner” on page 8-16 for more information on adapters.

Using The Class APT_RangePartitioner

A range partitioner divides a data set into approximately equal size partitions
based on one or more partitioning keys. Range partitioning guarantees that all
records with the same partitioning key values are assigned to the same partition
and that the partitions are approximately equal in size. This means that all nodes
perform an equal amount of work when processing the data set.
You use the class APT_RangePartitioner to implement this partitioning method.
See the Orchestrate 7.0 User Guide for more information.
In order for the range partitioner to determine the partition boundaries, you pass
the range partitioner a sorted sample of a data set to be range partitioned. Note
that this sorted sample is stored in a data file, not as a normal Orchestrate data set.
From this sample, the range partitioner can determine the partition boundaries
for the entire data set.
Typically, you use an Orchestrate step to create the sorted sample of the data set.
Once you have the sorted sample, you can either:
• Instantiate an APT_RangePartitioner object and pass it into the derived
operator. The APT_Operator::describeOperator() override then uses
APT_Operator::setPartitionMethod() to configure the operator to use the
APT_RangePartitioner object.
• Pass a reference to the sorted sample to the operator using an operator
constructor or member function. You then dynamically allocate an
APT_RangePartitioner object in your override of
APT_Operator::describeOperator() using the sorted sample. Finally, you use

Orchestrate 7.0 Developer Guide 87

Chapter 8 Creating Partitioners Creating a Custom Partitioner

APT_Operator::setPartitionMethod() to configure the operator to use the

dynamically allocated range partitioner.

The APT_Partitioner Class Interface

The following figure shows the interface to APT_Partitioner. You use the
member functions to create your own partitioning method.
The virtual functions describePartitioner(), setupInputs(), and partitionInput()
are included in the protected interface. You must override these functions.
See the header file $APT_ORCHHOME/include/apt_framework/partitioner.h for
descriptions of these functions. You can also browse the Orchestrate class
hierarchy using HTML-documented header files. Start your search with the
index.html file in $APT_ORCHHOME/doc/html/.

APT_Partitioner class interface

public: protected: protected (continued)
destructor constructor reportError()
errorInformation() describePartitioner() = 0 scInterface()
ident() discardInput() setRuntimeArgs()
initializationArgs() duplicateInput() setupInputAccessor()
initializeFromArgs() errorLog() setupInputs() = 0
setIdent() initializeFromArgs_() setInputInterfaceSchema()
inputConcreteSchema() setOutputPartitions()
inputInterface() startingConsumerPartition()
inputInterfaceSchema() viewAdaptedSchema()
partitionInput() = 0

Creating a Custom Partitioner

Here are short descriptions of the functions you must override:
• describePartitioner() defines the interface schema of the partitioner.
• setupInputs() defines the field accessors used by the partitioner.
• partitionInput() performs the partitioning operation.
Note that APT_Partitioner contains two other functions: the public member
function initializeFromArgs() and the protected function initializeFromArgs_().
You use these functions to enable Orchestrate’s argument-list processing facility
and to make your paritioner osh aware. See “Property List Encoding of Operator
Arguments” on page 5-19 for more information.
As part of deriving a partitioner class, you can include support for detecting error
and warning conditions and for relaying that information back to users. See
Chapter 14, “Using the Error Log” for information.

88 Orchestrate 7.0 Developer Guide

Chapter 8 Creating Partitioners Creating a Custom Partitioner

Overriding APT_Partitioner::describePartitioner()
Many partitioning methods use the fields of a record to determine the partition
for the record. To access those fields, the partitioner must have an interface
schema, defined by overriding the pure virtual function describePartitioner().
The following figure shows a partitioner with a single integer field named
hashField as its interface schema:

input data set

hashField:int32; sortField:string; in:*;

SortOperator

hashField:int32; partitioner and

partitioner interface schema

...

out:*;
output data set

The input dataset’s concrete schema must be compatible with the partitioner’s
interface schema. In this example, both schemas contain an integer field named
hashField. If an operator’s input interface schema is not compatible with the
partitioner’s schema, you can use an adapter to translate components.
viewAdaptedSchema() returns the dataset concrete schema as projected through
the view adapter. See “Using a View Adapter with a Partitioner” on page 8-16 for
information on view adapters.
A partitioner is not required to define an interface schema if it does not use record
fields as part of its method. This type of partitioner is called a keyless partitioner.
You still must provide an override to describePartitioner(), but the function
should just return APT_StatusOK. See “Example Partitioning Method
Definition” on page 8-11 for a sample override of describePartitioner().

Overriding APT_Partitioner::setupInputs()
After you have established the partitioner’s interface schema, you need to define
field accessors. Field accessors provide named access to any type of field within a
record of a data set. See Chapter 7, “Using Cursors and Accessors” for
information on field accessors. A field accessor is normally defined as a private

Orchestrate 7.0 Developer Guide 89

Chapter 8 Creating Partitioners Creating a Custom Partitioner

data member of the derived partitioner class. You override the pure virtual
function setupInputs() to initialize the field accessors.
The following figure shows a partitioner that defines a single integer field named
hashField as its interface schema:

input data set

hashField:int32; sortField:string; in:*;

SortOperator
partitioner and
hashField:int32; partitioner interface schema

...

out:*;
output data set

In this example, you override the pure virtual function setupInputs() to initialize
the single field accessor used by a partitioner to access hashField. If your
partitioning method does not access record fields, you still must override
setupInputs(), but it should simply return APT_StatusOk. See “Example
Partitioning Method Definition” on page 8-11 for a sample override of
setupInputs().

Overriding APT_Partitioner::partitionInput()
You must override the pure virtual function APT_Partitioner::partitionInput() to
perform the actual partitioning operation. This function contains the code
defining your partitioning method. Here is the function prototype of
partitionInput():
virtual int partitionInput(int numPartitions) = 0;

The function partitionInput() assigns a partition number to a record of an input

data set. Orchestrate calls partitionInput() for each record of an input data set;
you do not call it directly. The argument numPartitions specifies the number of
partitions available for the record. Orchestrate passes numPartitions to
partitionInput(), where numPartitions is guaranteed to be positive.
Your override of partitionInput() must return an integer value denoting the
partition for the current input record. This returned value must satisfy the
requirement:
0 <= returnValue < numPartitions

8 10 Orchestrate 7.0 Developer Guide

Chapter 8 Creating Partitioners Example Partitioning Method Definition

Example Partitioning Method Definition

The following figure shows an operator that partitions an input data set based on
an integer field of the records, and sorts the records based on the integer field and
a string field:

input data set

hashField:int32; sortField:string; in:*;

SortOperator
hashField:int32 ; partitioner and
partitioner interface schema

...

out:*;

output data set

To access the record hashField, the partitioner defines one accessor. The partitioner
schema and operator’s input interface schema both contain an integer field named
hashField. Therefore, they are compatible. If they were not compatible, you could
create a view adapter to translate the interface schema.
The code in Table 35 below shows the derivation of SortPartitioner, the partitioner
for this operator:

Table 35 Partitioner Derivation

Code

#include <apt_framework/orchestrate.h>
2 class SortPartitioner : public APT_Partitioner
{
4 APT_DECLARE_RTTI(SortPartitioner);
5 APT_IMPLEMENT_PERSISTENT(SortOperator);

public:
7 SortPartitioner();
protected:
9 virtual APT_Status describePartitioner();
10 virtual APT_Status setupInputs(int numPartitions);

Orchestrate 7.0 Developer Guide 8 11

Chapter 8 Creating Partitioners Example Partitioning Method Definition

Table 35 Partitioner Derivation (continued)

11 virtual int partitionInput(int numPartitions);

12 virtual APT_Status initializeFromArgs_(const APT_PropertyList &args,
InitializeContext context);

private:
14 APT_InputAccessorToInt32 hashFieldAccessor;

15 #define ARGS_DESC “{. . .}”

16 APT_DEFINE_OSH_NAME(SortPartitioner, sortpart, APT_Partitioner);
17 APT_IMPLEMENT_RTTI_ONEBASE(SortPartitioner, APT_Partitioner);
18 APT_IMPLEMENT_PERSISTENT(SortPartitioner);

19 SortPartitioner::SortPartitioner()
{}

21 APT_Status SortPartitioner::initializeFromArgs_(const APT_PropertyList

&args, APT_Partitioner::InitializeContext context)
{
return APT_StatusOk;
}

25 void SortPartitioner::serialize(APT_Archive& archive, APT_UInt8)

{}

27 APT_Status SortPartitioner::describePartitioner()
{
29 setInputInterfaceSchema(”record(hashField:int32;)”);
return APT_StatusOk;
}

32 APT_Status SortPartitioner::setupInputs(int numPartitions)

{
34 setupInputAccessor(“hashField”, &hashFieldAccessor);
return APT_StatusOk;
}

37 int SortPartitioner::partitionInput(int numPartitions)

{
39 APT_UInt32 hashVal = APT_hash(*hashFieldAccessor);
40 return hashVal % numPartitions;
}
}

Comments

2 Derive SortPartitioner from APT_Partitioner.

8 12 Orchestrate 7.0 Developer Guide

Chapter 8 Creating Partitioners Example Partitioning Method Definition

Table 35 Partitioner Derivation (continued)

4 Include the macro APT_DECLARE_RTTI(), which is required to support run-time
type information. See the Chapter 12, “Run-Time Type Information” for more
information.

5 Include the macro APT_DECLARE_PERSISTENT(), which is required for

persistence support. This macro also inserts a declaration for the
APT_Persistent::serialize() function required by the Orchestrate persistence
mechanism. See Chapter 11, “Enabling Object Persistence” for more information.

7 Include the default constructor for SortPartitioner. This constructor is required for
persistent classes. See Chapter 11, “Enabling Object Persistence” for more
information.

9-12 Override the APT_Partitioner virtual functions: describePartitioner(),

setupInputs(), partitionInput(), and initializeFromArgs_().

14 Define hashFieldAccessor, a field accessor to access hashField.

15 If your partitioner has arguments, you supply a description of them in the

ARGS_DESC string. See “Argument-List Descriptions” on page 5-3 and
$APT_ORCHHOME/include/apt_util/argvcheck.h for documentation on this string.
Orchestrate’s argument-checking facility is fully described in Chapter 5,
“Orchestrate’s Argument-List Processor”.

16 With APT_DEFINE_OSH_NAME, you connect the class name to the name used to
invoke the operator from osh and pass your argument description string to
Orchestrate. See $APT_ORCHHOME/include/apt_framework/osh_name.h for
documentation on this macro. Orchestrate’s argument-checking facility is fully
described in Chapter 5, “Orchestrate’s Argument-List Processor”.

17 Include the macro APT_IMPLEMENT_RTTI_ONEBASE(), which is required to

support run-time type information.

18 Include the macro APT_IMPLEMENT_PERSISTENT(), which is required for

persistence support.

19 Declares a default constructor.

21 With your override of initializeFromArgs_(), you transfer information from the

partitioner’s arguments to the class instance, making it osh aware. See “Property
List Encoding of Operator Arguments” on page 5-19 and the header file,
$APT_ORCHHOME/include/apt_framework/operator.h for documentation on this
function.

25 The function APT_Persistent::serialize() defines complex persistence.

Orchestrate 7.0 Developer Guide 8 13

Chapter 8 Creating Partitioners Example Partitioning Method Definition

Table 35 Partitioner Derivation (continued)

27 Override APT_Partitioner::describePartitioner(), a pure virtual function.

29 Use APT_Partitioner::setInputInterfaceSchema() to specify the interface to this

partitioner. This partitioner’s interface schema consists of a single integer field.

32 You must override the pure virtual function APT_Partitioner::setupInputs() to

initialize the field accessors used by SortPartitioner to access the required fields of a
record.

34 Use APT_Partitioner::setupInputAccessor() to initialize hashFieldAccessor.

37 You must override the pure virtual function APT_Partitioner::partitionInput() to

perform the actual partitioning operation.

39 Use APT_hash() to compute a hash value for the integer field. APT_hash() returns
a partition number for a specified hash key. See the section “Function APT_hash()”
on page 6-14 for more information on this function.

40 Return the hash value modulo numPartitions where numPartitions is passed in by

Orchestrate.

Once you have defined your partitioner, you can use it with a derived operator.
Typically, you define the partitioner within the override of
APT_Operator::describeOperator().
To use SortPartitioner with SortOperator, you use
APT_Operator::setPartitionMethod() within the
APT_Operator::describeOperator() function of SortOperator.
setPartitionMethod() allows you to specify a partitioner class for your
partitioning method. Here is the function prototype of setPartitionMethod():
void setPartitionMethod(APT_Partitioner * partitioner,
const APT_ViewAdapter& adapter,
int inputDS);

In this form, setPartitionMethod() takes three arguments:

• partitioner, a pointer to a partitioner object.
• adapter, a view adapter. If no adapter is required, you can simply pass a
default-constructed adapter.
• inputDS, the input data set for this partitioner.
In the example describeOperator() function below, the partitioner is allocated
and is specified as the partitioner for the operator in setPartitionMethod(). In this

8 14 Orchestrate 7.0 Developer Guide

Chapter 8 Creating Partitioners Example Partitioning Method Definition

example, setPartitionMethod() takes a default-constructed adapter because the

interface schema of the operator and the partition are compatiable.
APT_Status SortOperator::describeOperator()
{
setKind(APT_Operator::eParallel);
setInputDataSets(1);
setOutputDataSets(1);
setInputInterfaceSchema(
"record (hashField:int32; sortField:string; in:*;)", 0);
setOutputInterfaceSchema("record (out:*;)", 0);
declareTransfer("in", "out", 0, 0);
SortPartitioner * sortPart = new SortPartitioner;
setPartitionMethod(sortPart, APT_ViewAdapter(), 0);
return APT_StatusOk;
}

Orchestrate Hashing Functions

As part of your partitioning method, you may choose to calculate a hash value
based on fields of a record. These fields are referred to as hash keys. Orchestrate
provides several overloads of the partitioning hash function APT_hash() to
handle most data types that can be used as hash keys.
APT_hash() returns a partition number for a specified hash key. You must
perform modulo division of the value returned by APT_hash() to ensure that it is
between 0 and one less than the maximum partition number for the
corresponding data set.
Normally, you call APT_hash() from within APT_Partitioner::partitionInput()
when you derive your own partitioning method from APT_Partitioner.
Orchestrate supplies overloads of APT_hash() for the most common data types,
as shown below:
#include <apt_framework/orchestrate.h>
extern APT_UInt32 APT_hash(char key);
extern APT_UInt32 APT_hash(int key);
extern APT_UInt32 APT_hash(long key);
extern APT_UInt32 APT_hash(float key);
extern APT_UInt32 APT_hash(double key);
extern APT_UInt32 APT_hash(char * key, bool caseSensitive = true);
extern APT_UInt32 APT_hash(char * key, size_t keyLength,
bool caseSensitive = true );
extern APT_UInt32 APT_hash(const APT_String& key,
bool caseSensitive = true );
extern APT_UInt32 APT_hash(const APT_StringField& key,
bool caseSensitive = true);

You specify the hash key using the key argument. You use the keyLength argument
to specify the length of a character string if the string is not null-terminated. With
the caseSensitive argument you can specify whether the key represents a case-

Orchestrate 7.0 Developer Guide 8 15

Chapter 8 Creating Partitioners Example Partitioning Method Definition

sensitive character string (caseSensitive = true) or a case-insensitive string

(caseSensitive = false).
In addition to APT_hash(), Orchestrate also provides hashing functions for
decimal, date, time, and timestamp data types:
• APT_Decimal::hash()
• APT_Date::hash()
• APT_Time::hash()
• APT_TimeStamp::hash()

Using a View Adapter with a Partitioner

Typically, you do not create a partitioner for a particular operator. Instead, you
create general-purpose partitioners that may be used with any operator. The
interface schema of a partitioner may not always match the input interface
schema of the operator. Therefore, partitioners allow you to use an view adapter,
an instance of APT_ViewAdapter, to translate components of the operator’s
input interface schema to matching components of the partitioner’s interface
schema.
The following figure shows a sort operator that partitions the records of an input
data set based on two integer fields of a record:

input data set

field1:int32; field2:int32; field3:string; in:*;

SortOperator
pField1:int32; pField2:int32; partitioner and
partitioner interface schema

...

out:*;

output data set

The input interface schema of the partitioner defines two integer fields, pField1
and pField2, which it uses to partition the records of an input data set. This schema
is not compatible with the interface schema of the operator, so it is necessary to
define and initialize an APT_ViewAdapter within the describeOperator()

8 16 Orchestrate 7.0 Developer Guide

Chapter 8 Creating Partitioners Example Partitioning Method Definition

function of the derived operator. Here is the describeOperator() function for

SortOperator:
#include <apt_framework/orchestrate.h>
APT_Status SortOperator::describeOperator()
{
setKind(APT_Operator::eParallel);
setInputDataSets(1);
setOutputDataSets(1);
setInputInterfaceSchema(
"record(field1:int32; field2:int32; field3:string; in:*;)", 0);
setOutputInterfaceSchema("record (out:*;)", 0);
APT_ViewAdapter partitionAdapter (
"pField1 = field1;"
"pField2 = field2;" );

SortPartitioner * sortPart = new SortPartitioner;

setPartitionMethod(sortPart, partitionAdapter, 0);
return APT_StatusOk;
}

partitionAdapter is defined within describeOperator() and is destroyed when this

function completes. However, APT_Operator::setPartitionMethod() makes a
copy of the adapter and passes it to the partitioner, so that when
describeOperator() completes, the destruction of the adapter has no effect on the
partitioner.

Orchestrate 7.0 Developer Guide 8 17

Chapter 8 Creating Partitioners Example Partitioning Method Definition

8 18 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

9
Creating Collectors
Describes how to derive your own custom collectors from APT_Collector.

Overview 9 1
Choosing a Collection Method 9 2
Using the Default Collection Method 9 3
Using a Keyless Collection Method 9 3
Using The Sorted-Merge Collector 9 4
An APT_SortedMergeCollector Example 9 4
The APT_Collector Class Interface 9 6
Creating a Custom Collector 9 6
How APT_Collector Operates 9 7
Overriding APT_Collector::describeCollector() 9 8
Overriding APT_Collector::setupInputs() 9 9
Overriding APT_Collector::selectInput() 9 10
Example Collector Derivation 9 11
Example Collector with a View Adapter 9 16

Overview
When you create your own sequential operator classes by deriving from
APT_Operator, you can directly control how an operator performs collection.
You can use an Orchestrate-supplied collection method as part of your derived
operator or define your own collection method by deriving from the base class

91
Chapter 9 Creating Collectors Choosing a Collection Method

APT_Collector. Once you have derived a new APT_Collector class, you include
the derived collector within an operator.
You can even build a library of collector classes to use when appropriate. This lets
you mix and match operators and collectors when different forms of an operator
differ only by how the operator performs collection.
When you derive a sequential operator, you can choose the collection method
used by it. Orchestrate directly supports the keyless collection methods any,
round robin, and ordered. The default collection method is any. With the any
method, an operator reads records on a first-come first-served basis.
To use one of these methods, you include a call to
APT_Operator::setCollectionMethod() within the
APT_Operator::describeOperator() function of your derived operator. See
“Using a Keyless Collection Method” on page 9-3 for information.
Orchestrate also supplies the sorted merge collection method, a keyed collector.
To use the sorted merge collection method, use the class
APT_SortedMergeCollector. See “Using The Sorted-Merge Collector” on page 9-
4 for information.
To derive a new collection method, or a variation on one of the methods described
above, you can derive a collection class from APT_Collector. See “The
APT_Collector Class Interface” on page 9-6.

Choosing a Collection Method

You must choose a collection method appropriate for the particular action of its
associated operator. The built-in collection methods, any, round robin, and
ordered, do not rely on information in the records to define the order of records
read by the operator. You can, however, define a collection method that uses the
fields of the record to determine the partition for a record.
You often use any when the location of the record in the data set is irrelevant. For
example, in an unsorted data set, each record has no relationship to the record
immediately before or after it. An operator that simply performs the same
operation on every record would therefore use any as its collection method.
Another advantage of any is that your sequential operator is not forced to wait for
a record from any one partition.
In some cases, you may want to create an operator that allows the operator user to
define the collection method of the operator. In this case, you choose a collection
method of any for the operator. An operator using the any collection method can
have this method overridden by specifying a different collection method on a
data set input to the operator.

92 Orchestrate 7.0 Developer Guide

Chapter 9 Creating Collectors Choosing a Collection Method

The ordered method requires that all records are read from partition 0 before
beginning to process records from partition 1. However, the records of partition 1
may actually be ready for processing before those from partition 0. In this case,
the sequential operator must wait, possibly creating a processing bottleneck in
your application. Note, though, that the ordered collection method is necessary if
you want to process a totally sorted data set with a sequential operator and
preserve the sort order.
Unless your sequential operator requires a deterministic order for processing
records, you typically will use the any collection method. If you want more
control over the order of records processed by the operator, you can use the
ordered method, the APT_SortedMergeCollector, or a custom collector that you
define.

Using the Default Collection Method

The default collection method of any causes a sequential operator to read records
on a first-come first-served basis. The function
APT_Operator::collectionMethod() returns APT_Operator::eCollectAny for an
operator using the collection method any.
One advantage of using the any collection method is that an operator user can
override this collection method. For example, you may create an operator whose
collection method is tightly coupled to the actual data set processed at run time.
Specifying the any collection method allows the user to change the collection
method for each instance of the operator.

Using a Keyless Collection Method

Orchestrate lets you use the collection methods round robin and ordered without
deriving a new APT_Collector class. These collection methods are built into
Orchestrate. Simply include a call to APT_Operator::setCollectionMethod()
within the APT_Operator::describeOperator() function of your derived operator.
The function prototype of APT_Operator::setCollectionMethod() takes two
arguments, as shown below:
void setCollectionMethod(APT_Operator::CollectionMethod cType,
int inputDS);

The first argument, cType, specifies the collection method as defined by the
following values:
• APT_Operator::eCollectRoundRobin
• APT_Operator::eCollectOrdered
The second argument, inputDS, specifies the number of the input data set to the
operator. Note that these data sets are numbered starting from 0.

Orchestrate 7.0 Developer Guide 93

Chapter 9 Creating Collectors Choosing a Collection Method

For example, to use round robin collection with a sequential operator that takes a
single input data set, include the following statements within the operator
describeOperator() function:
setKind(APT_Operator::eSequential);
setCollectionMethod(APT_Operator::eCollectRoundRobin, 0);

If the operator has two input data sets and you want to use ordered for both,
include the lines:
setKind(APT_Operator::eSequential);
setCollectionMethod(APT_Operator::eCollectOrdered, 0);
setCollectionMethod(APT_Operator::eCollectOrdered, 1);

Using The Sorted-Merge Collector

APT_SortedMergeCollector defines a sorted collection method for a data set
input to a sequential operator. APT_SortedMergeCollector determines the order
of records processed by a sequential operator by examining one or more
collecting key fields in the current record of each partition of an input data set.
Typically, you use APT_SortedMergeCollector with a sequential operator that
processes a partition-sorted data set as created by APT_PartitionSortOperator. In
this case, you specify as the collecting key fields those fields you specified as
sorting key fields to the APT_PartitionSortOperator.

An APT_SortedMergeCollector Example
Orchestrate supplies the class APT_SortedMergeCollector, which orders the
records processed by a sequential operator, based on one or more fields of a
record. APT_SortedMergeCollector uses a dynamic interface schema that allows
you to specify one or more numeric or string fields as input.

94 Orchestrate 7.0 Developer Guide

Chapter 9 Creating Collectors Choosing a Collection Method

The following figure shows a sequential operator using

APT_SortedMergeCollector:

field1:int32; field2:string; field3:int16; in:*;

MyOperator
-----;

out:*;
output data set

APT_SortedMergeCollector does not define any interface schema; you use the
APT_SortedMergeCollector member function
APT_SortedMergeCollector::setKey() to specify the collecting key fields.
MyOperator requires three fields as input: two integer fields and a string field. You
can specify the collector’s interface schema within the operator’s
describeOperator() function, as shown below:
APT_Status MyOperator::describeOperator()
{
setKind(APT_Operator::eSequential); // set mode to sequential
setInputDataSets(1);
setOutputDataSets(1);
setInputInterfaceSchema(
"record(field1:int32; field2:int32; field3:string; in:*;)", 0);
setOutputInterfaceSchema("record (out:*;)", 0);
declareTransfer("in", "out", 0, 0);
// Define the collector
APT_SortedMergeCollector * coll = new APT_SortedMergeCollector;
coll->setKey("field1", "int32");
coll->setKey("field2", "int32");
setCollectionMethod(coll, APT_ViewAdapter(), 0);

return APT_StatusOk;
};

In the example above, the default constructor is used to dynamically allocate an

APT_SortedMergeCollector object. Orchestrate deletes the collector for you.
You must call setKey() to specify the key fields for the
APT_SortedMergeCollector object. In this example, setKey() specifies field1 as

Orchestrate 7.0 Developer Guide 95

Chapter 9 Creating Collectors The APT_Collector Class Interface

the primary collecting key field and field2 as the secondary collecting key field for
the APT_SortedMergeCollector object. The function setKey() is used to to specify
both a field name and a data type for the field.
The function APT_Operator::setCollectionMethod() specifies coll as the collector
for this operator. It is not necessary to use an input field adapter with this
collector; a default view adapter is passed instead.

The APT_Collector Class Interface

The following figure shows the interface to the APT_Collector class. You use this
class to create your own collection methods.
The three pure virtual functions, describeCollector(), selectInput(), and
setupInputs(), are included in the protected interface. You must override these
functions.
See the header file, $APT_ORCHHOME/include/apt_framework/collector.h, for
descriptions of the functions. You can also browse the Orchestrate class hierarchy
using HTML-documented header files. Start your search with the index.html file
in $APT_ORCHHOME/doc/html/.

APT_Collector class interface

public: protected: protected (continued)
destructor constructor reportError()
errorInformation() atEOF() selectInput() = 0
errorLog() describeCollector() = 0 setRuntimeArgs()
ident() discardInput() setupInputAccessor()
initializationArgs() initializeFromArgs_() setInputInterfaceSchema()
initializeFromArgs() inputInterface() setupInputs() = 0
SetIdent() inputConcreteSchema() viewAdaptedSchema()
inputInterfaceSchema()

Creating a Custom Collector

To define your own collection method, you derive a class from APT_Collector.
APT_Collector contains three pure virtual functions that you must override in
your derivation:
• describeCollector() defines the interface schema of the collector.
• selectInput() determines the input partition from which the operator reads
the next record.
• setupInputs() initializes the field accessors used by the collector.

96 Orchestrate 7.0 Developer Guide

Chapter 9 Creating Collectors Creating a Custom Collector

Note that APT_Collector contains two other functions: the public member
function initializeFromArgs() and the protected function initializeFromArgs_().
You use these functions to enable Orchestrate’s argument-list processing facility
and to make your collector osh aware. As part of deriving a collector class, you
can include support for detecting error and warning conditions and for relaying
that information back to users. See Chapter 14, “Using the Error Log” for more
information.

How APT_Collector Operates

To create your own collector, you must be familiar with the way a collector
operates. The following figure shows a sequential operator combining the records
of an input data set:

input data set partitions

p0 p1 p2 pN
...
current record
in each partition

operator collector
(sequential)

output data set partition

In this figure, each input partition has a current record, corresponding to the
record that a sequential operator would read if it consumed a record from that
partition.
When a sequential operator calls APT_InputCursor::getRecord() as part of its
override of APT_Operator::runLocally() to obtain the next record from an input
data set, the collector determines the partition that supplies the record. The
selected partition then updates itself, so the next record in the partition becomes
the current record. When any partition becomes empty because it has supplied its
final record, that partition returns an End Of File (EOF) whenever a record is
requested from it.
The call to APT_InputCursor::getRecord() causes the operator to call
APT_Collector::selectInput(), one of the pure virtual functions that you must
override when defining a collector. This function returns the number of the input
partition supplying the record read by the operator. Your override of

Orchestrate 7.0 Developer Guide 97

Chapter 9 Creating Collectors Creating a Custom Collector

selectInput() implements the algorithm defining the order of records supplied to

the operator.
As part of its algorithm for determining the partition number, your override of
selectInput() may interrogate fields within the current record of each partition.
This allows you to use information in the record to determine the order in which
the sequential operator reads records. To access the fields of a record, you must
define input accessors to each record field in each partition you want to access.
The following figure shows a collector that uses field information to determine
record order:

input data set partitions

p0 p1 p2 pN
...
current record
in each partition

operator record field collector

(sequential)

output data set partition

You use the override of APT_Collector::setupInputs() to define the field

accessors used by the collector. If your collector uses no accessors, this function
should return APT_StatusOK.

Overriding APT_Collector::describeCollector()
Many collection methods use the fields of a record to determine the order of
records processed by a sequential operator. To access those fields, the collector
must have an interface schema, defined by overriding the pure virtual function
describeCollector().

98 Orchestrate 7.0 Developer Guide

Chapter 9 Creating Collectors Creating a Custom Collector

The following figure shows a collector with a single integer field named
collectorField as its interface schema:

input data set

collectorField:int32; aField:string; in:*;

collectorField:int32;
collector and
collector interface schema

out:*;
output data set

An operator’s input interface schema must be compatible with the collector’s

interface schema. In this example, both contain an integer field named
collectorField. If an operator’s input interface schema is not compatible with the
collector’s schema, you can use an view adapter to translate components. See
“Example Collector with a View Adapter” on page 9-16 for more information.
A collector is not required to define an interface schema if it does not use record
fields as part of its collection method. This type of collector is called a keyless
collector. You still must provide an override to describeCollector(), but the
function should just return APT_StatusOk. See “Example Collector Derivation”
on page 9-11 for a example override of describeCollector().

Overriding APT_Collector::setupInputs()
After you have established the collector’s interface schema, you need to define
field accessors for each component of the collector’s interface schema. Field
accessors provide named access to any type of field within a record of a data set.
See the chapter Orchestrate Data Sets in the Orchestrate 7.0 User Guide for more
information. Field accessors normally are defined as a private data member of the
derived collector class. You then override the pure virtual function setupInputs()
to initialize the field accessors.

Orchestrate 7.0 Developer Guide 99

Chapter 9 Creating Collectors Creating a Custom Collector

The following figure shows a collector that defines a single integer field named
collectorField as its interface schema:

input data set

collectorField:int32; aField:string; in:*;

collectorField:int32;
collector and
collector interface schema

out:*;
output data set

In this example, you override the pure virtual function setupInputs() to initialize
one field accessor for each partition of the input data set to access collectorField. If
your collection method does not access record fields, you still must override
setupInputs(), but it should just return APT_StatusOk. See “Example Collector
Derivation” on page 9-11 for a sample override of setupInputs().

Overriding APT_Collector::selectInput()
You must override the pure virtual function APT_Collector::selectInput() to
perform the actual collection operation. Here is the function prototype of
selectInput():
virtual int selectInput(int numPartitions) = 0;

selectInput() returns the number of the input partition supplying the next record
to the operator. Orchestrate calls selectInput() each time the operator reads a
record from the data set; you do not call it directly. The argument numPartitions
specifies the number of input partitions. Orchestrate passes numPartitions to
selectInput(), where numPartitions is guaranteed to be positive.
Your override of selectInput() must return an integer value denoting the partition
supplying the input record. This returned value must satisfy the requirement:
0 <= returnValue < numPartitions

9 10 Orchestrate 7.0 Developer Guide

Chapter 9 Creating Collectors Example Collector Derivation

Example Collector Derivation

The following figure shows an operator that combines the partitions of an input
data set based on an integer field of the records:

input data set

collectorField:int32; aField:string; in:*;

collectorField:int32;
collector and
collector interface schema

out:*;
output data set

This operator uses a collector that determines the next field by inspecting a single
integer field. The partition whose current record has the smallest value for
collectorField supplies the record to the operator. To access the collectorField, the
collector defines one accessor for each partition of the input data set.
The collector’s schema and the operator’s input interface schema both contain an
integer field named collectorField. Therefore, they are compatible. If they were not,
you could create an view adapter to translate the interface schema. See “Example
Collector with a View Adapter” on page 9-16 for more information.
The code in Table 36 below shows the derivation of MyCollector, the collector for
this operator:

Table 36 Example APT_Collector Derivation

Code

#include <apt_framework/orchestrate.h>

2 class MyCollector: public APT_Collector

{
4 APT_DECLARE_RTTI(MyCollector);
5 APT_DECLARE_PERSISTENT(MyCollector);

public:
7 MyCollector();
8 ~MyCollector();

Orchestrate 7.0 Developer Guide 9 11

Chapter 9 Creating Collectors Example Collector Derivation

Table 36 Example APT_Collector Derivation (continued)

protected:
10 virtual APT_Status describeCollector();
virtual APT_Status setupInputs(int numPartitions);
virtual int selectInput(int numPartitions);
13 virtual APT_Status initializeFromArgs_(const APT_PropertyList
&args,
InitializeContext context);

private:
15 APT_InputAccessorToInt32 * collectorFieldAccessors;
16 int numParts;

17 #define ARGS_DESC “{otherInfo=

{description='An example collector with no arguments to
describe”}}
18 APT_DEFINE_OSH_NAME(MyCollector, mycollector, APTCollector);
19 APT_IMPLEMENT_RTTI_ONEBASE(MyCollector, APT_Collector);
20 APT_IMPLEMENT_PERSISTENT(MyCollector);

21 MyCollector::MyCollector()
: collectorFieldAccessors(0), numParts(0)
{}

24 MyCollector::~MyCollector()
{
26 delete[] collectorFieldAccessors;
}

28 APT_Status SortPartitioner::initializeFromArgs_(const
APT_PropertyList
&args, APT_Partitioner::InitializeContext context)
{
return APT_StatusOk;
}

32 APT_Status MyCollector::describeCollector()
{
34 setInputInterfaceSchema(“record(collectorField:int32;)”);
return APT_StatusOk;
}

APT_Status MyCollector::setupInputs(int numPartitions)

{
39 collectorFieldAccessors = new
APT_InputAccessorToInt32[numPartitions];
for (int n = 0; n < numPartitions; n++)
{

9 12 Orchestrate 7.0 Developer Guide

Chapter 9 Creating Collectors Example Collector Derivation

Table 36 Example APT_Collector Derivation (continued)

42 setupInputAccessor(“collectorField”,
&collectorFieldAccessors[0], n);
}
return APT_StatusOk;
}

int My Collector::selectInput(int numPartitions)

{
48 int minVal = INT_MAX;
49 int minPartIndex = -1;

50 for (int n = 0; n < numPartitions; n++)

{
52 if (atEOF(n)) continue;
53 if (*collectorFieldAccessors[n] <= minVal)
{
minVal = *collectorFieldAccessors[n];
minPartIndex = n;
}
}
59 APT_ASSERT(minPartIndex != -1);
60 return minPartIndex;
}
}

Comments

2 Derive MyCollector from APT_Collector.

4 Include the macro APT_DECLARE_RTTI(), which is required to support run-

time type information. See Chapter 12, “Run-Time Type Information” for more
information.

5 Include the macro APT_DECLARE_PERSISTENT(), which is required for

persistence support. This macro also inserts a declaration for the
APT_Persistent::serialize() function required by the Orchestrate persistence
mechanism. See Chapter 11, “Enabling Object Persistence” for more information.

7 Include the default constructor for MyCollector. This constructor is required for
persistent classes.

8 Define the destructor for MyCollector.

10-13 Override the virtual functions.

Orchestrate 7.0 Developer Guide 9 13

Chapter 9 Creating Collectors Example Collector Derivation

Table 36 Example APT_Collector Derivation (continued)

15 Define collectorFieldAccessors, a pointer to an array of field accessors used to read
collectorField in the partitions of the input data set. You need a single accessor for
each field in each partition that you want to access.

16 Define numParts to hold the length of the accessor array referenced by

collectorFieldAccessors.

17 If your collector has arguments, you supply a description of them in the

ARGS_DESC string. See “Argument-List Descriptions” on page 5-3 and
$APT_ORCHHOME/include/apt_util/argvcheck.h for documentation on this string
and Orchestrate’s argv checking facility. Orchestrate’s argument-checking facility
is fully described in Chapter 5, “Orchestrate’s Argument-List Processor”.

18 With APT_DEFINE_OSH_NAME, you connect the class name to the name used
to invoke the operator from osh and pass your argument description string to
Orchestrate. See “Argument-List Descriptions” on page 5-3 and
$APT_ORCHHOME/include/apt_framework/osh_name.h for documentation on this
macro. Orchestrate’s argument-checking facility is fully described in Chapter 5,
“Orchestrate’s Argument-List Processor”.

19 Include the macro APT_IMPLEMENT_RTTI_ONEBASE(), which is required to

support run-time type information. See Chapter 11, “Enabling Object
Persistence” for more information.

20 Include the macro APT_IMPLEMENT_PERSISTENT(), which is required for

persistence support. See Chapter 11, “Enabling Object Persistence” for more
information.

21 The default constructor.

24 Define the destructor for MyCollector.

26 Delete the array of accessors used by the collector.

28 With your override of initializeFromArgs_(), you transfer information from the

collector’s arguments to the class instance, making it osh aware. See “Property
List Encoding of Operator Arguments” on page 5-19 and the header file,
$APT_ORCHHOME/include/apt_framework/operator.h for documentation on this
function. Orchestrate’s argument-checking facility is fully described in Chapter 5,
“Orchestrate’s Argument-List Processor”.

32 Override APT_Collector::describeCollector.

34 Use APT_Collector::setInputInterfaceSchema() to specify the interface to this

collector. This interface schema consists of a single integer field.

9 14 Orchestrate 7.0 Developer Guide

Chapter 9 Creating Collectors Example Collector Derivation

Table 36 Example APT_Collector Derivation (continued)

39 Define an array of field accessors, one for each input partition, and initialize
collectorFieldAccessors, a pointer to the array.

42 Use APT_Collector::setupInputAccessor() to initialize a field accessor to

collectorField in each input partition.

48 Define minVal to hold the current minimum value for collectorField and initialize it
to INT_MAX, the largest supported APT_Int32.

49 Define minPartIndex to hold the number of the partition with the minimum value
for collectorField.

50 Iterate through all partitions of the input data set.

52 Use APT_Collector::atEOF() to determine whether the current partition contains

a valid record. APT_Collector::atEOF() returns true after the final record has
been read from a partition.

53 If a partition contains a record, compare the record’s collectorField value against

the current minimum field value. If the current record’s collectorField is less than
the current minVal, update minVal and minPartIndex accordingly.

59 Your override of APT_Collector::selectInput() should always return a partition

index. This statement issues an assertion failure if you have iterated through all
the partitions and have not found a partition number to return. You can also
return -1 to indicate a fatal error.

60 Return the partition number of the record read by the operator.

Once you have defined your collector, you can use it with a derived operator.
Typically, you define the collector within the override of
APT_Operator::describeOperator().
To use MyCollector with MySequentialOperator, you use
APT_Operator::setCollectionMethod() within the
APT_Operator::describeOperator() function of MySequentialOperator.
setCollectionMethod() allows you to specify a collector object for your operator.
Here is the function prototype of setCollectionMethod():
void setCollectionMethod(APT_Collector * collector,
const APT_ViewAdapter& adapter,
int inputDS);

In this form, setCollectionMethod() takes three arguments:

• collector, a pointer to a collector object.

Orchestrate 7.0 Developer Guide 9 15

Chapter 9 Creating Collectors Example Collector with a View Adapter

• adapter, a view adapter. If no adapter is required, you can simply pass a

default-constructed adapter.
• inputDS, the input data set for this collector.
Here is the describeOperator() function for MySequentialOperator. The collector
MyCollector is dynamically allocated; Orchestrate performs the deletion. The
function APT_Operator::setCollectionMethod() specifies opCollector as the
collector for this operator. This function takes a default-constructed adapter
because the interface schema of the operator and collector are compatible.
APT_Status MySequentialOperator::describeOperator()
{
setKind(APT_Operator::eSequential);
setInputDataSets(1);
setOutputDataSets(1);
setInputInterfaceSchema(
"record (collectorField:int32; aField:string; in:*;)", 0);
setOutputInterfaceSchema("record (out:*;)", 0);
declareTransfer("in", "out", 0, 0);
MyCollector * opCollector = new MyCollector;
setCollectionMethod(opCollector, APT_ViewAdapter(), 0);
return APT_StatusOk;
}

Example Collector with a View Adapter

Typically, you do not create a collector for a particular operator. Instead, you
create general-purpose collectors that may be used with any sequential operator.
The interface schema of a collector may not always match the input interface
schema of the operator. Therefore, collectors allow you to use a view adapter, an
instance of APT_ViewAdapter, to translate components of the operator’s input
interface schema to matching components of the collector’s interface schema. This
section describes how to use an adapter with a collector.

9 16 Orchestrate 7.0 Developer Guide

Chapter 9 Creating Collectors Example Collector with a View Adapter

The following figure shows a sequential operator that combines the records of an
input data set, based on a single integer field of the records:

input data set

field1:int32; field2:int32; field3:string; in:*;

collectorField:int32; collector and

collector interface schem a

out:*;

output data set

The input interface schema of the collector defines the integer field collectorField
that it uses to combine the records of an input data set. This schema is not
compatible with the interface schema of the operator, so an APT_ViewAdapter
object must be defined and initialized within the describeOperator() function of
the derived operator. Here is the describeOperator() function for
MySequentialOperator:
#include <apt_framework/orchestrate.h>
APT_Status MySequentialOperator::describeOperator()
{
setKind(APT_Operator::eSequential);
setInputDataSets(1);
setOutputDataSets(1);
setInputInterfaceSchema(
"record(field1:int32; field2:int32; field3:string; in:*;)", 0);
setOutputInterfaceSchema("record (out:*;)", 0);
APT_ViewAdapter collectorAdapter(
"collectorField = field1;");

MyCollector * opCollector = new MyCollector;

setCollectionMethod(opCollector, collectorAdapter, 0);
return APT_StatusOk;
}

An instance of APT_ViewAdapter, collectorAdapter, is defined to translate field1 of

the operator’s input interface schema to collectorField of the collector’s interface
schema.
This translation does not contain any form of type conversion. You also can use
adapters to perform type conversion during a translation.

Orchestrate 7.0 Developer Guide 9 17

Chapter 9 Creating Collectors Example Collector with a View Adapter

collectorAdapter is defined within describeOperator() and is destroyed when this

function completes. However, APT_Operator::setCollectionMethod() makes a
copy of the adapter and passes it to the collector, so when describeOperator()
completes, the destruction of the adapter has no effect on the collector.
APT_Operator::setCollectionMethod() is used to specify opCollector as the
collector for this operator and collectorAdapter as the input field adapter for the
collector.
Note that you can also create the adapter and pass it to the collector using the
single statement:
setCollectionMethod(opCollector, "collectorField = field1;", 0);

9 18 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

10
Using Field Lists
Describes field lists which let you reference multiple schema components using a
single object.

Overview 10 1
Creating a Field List 10 2
The APT_FieldList Class Interface 10 2
Expanding a Field List 10 3
Using Field Lists with Operators 10 4

Overview
Field lists, represented by the class APT_FieldList, let you reference multiple
schema components using a single object. By doing so, you can pass one or more
field references to a function using a single argument.
You can also use field lists when creating operators with a dynamic interface
schema. This allows a user of the operator to specify some or all of the operator’s
interface schema.
This chapter describes field lists, then explains how to use field lists with
functions and operators.

10 1
Chapter 10 Using Field Lists Creating a Field List

Creating a Field List

To create a field list, you must know the record schema for the data set. Consider
the following example, which defines a record schema containing five fields:
static char schema[] = "record"
"( a:int32; "
" b:int32; "
" c:int16; "
" d:sfloat; "
" e:string; )";

A function that can operate on one or more fields of a data set using this record
schema could have the following prototype:
void processFields(const APT_FieldList& fList);

This function takes as an argument the list of fields to process. The field list you
include can have several forms, as shown below:
processFields("a, b, e"); // comma-separated list of fields
processFields("a - c"); // field range
processFields("a - c, e, f"); // field range and a comma-separated
// list
processFields("*"); // all fields

To create a field list, in other words, you use one or more of these elements:
• Individual field identifiers.
• Field ranges, which are two field identifiers separated by a hyphen. The first
field must appear earlier in the record schema definition than the second (the
fields must be in schema, not alphabetic, order). A field range includes all the
fields whose identifiers fall within the range.
• A wildcard (*), which represents all of the fields in the record schema.

The APT_FieldList Class Interface

The following figure shows the interface to APT_FieldList which allows you to
reference multiple schema components using a single object. . The header file for
this class, $APT_ORCHHOME/include/apt_framework/utils/fieldlist.h, contains
descriptions of the functions. You can also browse the Orchestrate class hierarchy
using HTML-documented header files. Start your search with the index.html file
in $APT_ORCHHOME/doc/html/.

10 2 Orchestrate 7.0 Developer Guide

Chapter 10 Using Field Lists Expanding a Field List

APT_FieldList class interface

public:
addFields() field() keepFields()
dropFields() fieldIndex() numFields()
expand() isExpanded() unparse()

Expanding a Field List

When you create a field list, it is in one of two states:
• Expanded: An expanded field list contains an APT_FieldSelector object for
each field in the list. A field selector contains a complete reference to a field.
Using the selector, you can obtain the information that defines that field. Any
field list that you create which contains only a comma-separated list of fields
is expanded.
• Unexpanded: An unexpanded field list does not contain information about
the field data types. You create an unexpanded field list if you specify a field
range or use the "*" wildcard.
To use any field list, you must expand it, if necessary, by specifying a record
schema against which to expand the list. This record schema is called the context
for the field list expansion. For example, the following statement creates an
unexpanded field list using a field range:
APT_FieldList listObject("a - c");

You then specify the following record schema as the context and expand the list
using APT_FieldList::expand():
static char schema1[] = "record"
"( a:int32; "
" b:int32; "
" c:int16; "
" d:sfloat; "
" e:string; )";
listObject.expand(schema1);

The expanded field list contains three APT_FieldSelector objects: one each for the
fields a, b, and c. If the field list is already expanded, APT_FieldList::expand()
does nothing. Note that you cannot unexpand an expanded field list.
Selecting a different record schema as the context results in a different number of
selectors. Consider, for example, the following schema:
static char schema2[] = "record"
"( a:int32; "
" a1:int32; "
" a2:int16; "
" a3:sfloat; "

Orchestrate 7.0 Developer Guide 10 3

Chapter 10 Using Field Lists Using Field Lists with Operators

" c:string; )";

listObject.expand(schema2);

In this example, the expanded field list contains five field selectors, one for each
field in the record schema.
After you expand a field list, you can access the APT_FieldSelector for each field
in the list. Using the APT_FieldSelector, you can then determine the data type of
the field and process the field accordingly.

Using Field Lists with Operators

You can use field lists when you specify the interface schema to an operator with
a dynamic interface. The following example defines one such operator, called
DynamicOperator. The operator is shown in the figure below:

input data set

inRec:*;

DynamicOperator

outRec:*;

output data set

DynamicOperator:
• Takes a single data set as input
• Writes its results to a single output data set
• Has an input interface schema consisting of a single schema variable inRec
and an output interface schema consisting of a single schema variable outRec
To use this operator, you would specify the field list to its constructor, as shown
in this statement:
static char schema[] = "record" // Schema of input data set
"( a:int32; "
" b:int32; "
" c:int16; "
" d:sfloat; "
" e:string; )";
DynamicOperator aDynamicOp("a - c"); // Specify the input schema

Table 37 below shows the definition of DynamicOperator. Comments follow the

code in the table.

10 4 Orchestrate 7.0 Developer Guide

Chapter 10 Using Field Lists Using Field Lists with Operators

Table 37 Using Field Lists in a Derived Operator

Code

#include <apt_framework/orchestrate.h>

class DynamicOperator : public APT_Operator

{
APT_DECLARE_RTTI(DynamicOperator);
APT_DECLARE_PERSISTENT(DynamicOperator);

public:
DynamicOperator();
8 DynamicOperator(const APT_FieldList& fList)
: inSchema(), inFList(fList)
{}

protected:
virtual APT_Status describeOperator();
virtual APT_Status runLocally();
virtual APT_Status initializeFromArgs_(const APT_PropertyList
&args,
APT_Operator::InitializeContext context);

private:
16 APT_Schema inSchema;
17 APT_FieldList inFList;

. . .

APT_Status DynamicOperator::describeOperator()
{
setKind(APT_Operator::eParallel);
setInputDataSets(1);
setOutputDataSets(1);

23 APT_Schema tempSchema = viewAdaptedSchema(0);

24 APT_FieldList::Error err;
25 inFList.expand(tempSchema, &err);
26 if (err.errorOccurred())
{
reportError(err.description());
return APT_StatusFailed;
}
31 int NumFields = inFList.numFields();
32 for (int n = 0; n < numFields; n++)
{

Orchestrate 7.0 Developer Guide 10 5

Chapter 10 Using Field Lists Using Field Lists with Operators

Table 37 Using Field Lists in a Derived Operator (continued)

34 APT_FieldSelector fs = inFList.field(n);
35 APT_SchemaField f = tempSchema.field(fs);
36 inSchema.addField(f);
}

38 APT_SchemaField inField;
inField.setIdentifier(“inRec”);
inField.setTypeSpec(“*”);
41 inSchema.addField(inField);

42 setInputInterfaceSchema(inSchema, 0);
setOutputInterfaceSchema(“record (outRec:*;), 0);

declareTransfer(“inRec”, “outRec”, 0, 0);

Return APT_StatusOK;
}

Comments

8 Define a constructor that takes a field list as an argument.

16 Define storage for the complete input interface schema to the operator.

17 Define storage for the input field list specifying the input schema.

23 Use APT_Operator::viewAdaptedSchema() to obtain the complete record schema

of the input data set. You will expand the input field list in the context of the input
data set’s record schema.

24 Define err, an instance of APT_FieldList::Error, to hold any errors generated when

expanding the field list.

25 Use APT_FieldList::expand() to expand the field list relative to the record schema
of the input data set.
Note that if the field list does not contain a wildcard or field range, the expand
function does nothing.

26 If any errors occurred during the expansion, print the description of the error and
return APT_StatusFailed from describeOperator().

31 Use APT_FieldList::numFields() to determine the number of fields in the field list.

32 Create a for loop to iterate through the field lists. For each element in the field list,
add a schema field to inSchema, the input interface schema of the operator.

10 6 Orchestrate 7.0 Developer Guide

Chapter 10 Using Field Lists Using Field Lists with Operators

Table 37 Using Field Lists in a Derived Operator (continued)

34 Use APT_FieldList::field() to get a field selector from the expanded field list.

35 Use APT_Schema::field() to return an APT_SchemaField object corresponding to

a record schema field of the input data set.

36 Use APT_Schema::addField() to add the APT_SchemaField object to inSchema.

38 - 41 After adding all the fields of the field list to the input interface schema, you must
create a schema field component for the schema variable "inRec:*;" and add that
component to the input interface schema.
Use APT_Schema::addField() to add the schema variable to allow transfers from
the input data set to the output data set.

42 Use APT_Operator::setInputInterfaceSchema() to set the input interface schema

of the operator to the schema contained in inSchema.

Orchestrate 7.0 Developer Guide 10 7

Chapter 10 Using Field Lists Using Field Lists with Operators

10 8 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

11
Enabling Object Persistence
Tells you how to make your derived components persistent so that your C++
objects will maintain their integrity throughout storage and loading on
multiple nodes.

Overview 11 2
Simple Persistence and Complex Persistence 11 2
Storing and Loading Simple and Complex Persistent Objects 11 3
Creating Archives 11 3
The APT_Archive Class Interface 11 4
Using the Serialization Operators 11 4
Defining Simple-Persistent Classes 11 7
Defining Complex-Persistent Classes 11 8
The APT_Persistent Class Interface 11 9
Implementing APT_Persistent::serialize() 11 10
Serializing Pointers 11 11
Serializing Arrays 11 11
Persistence Macros 11 13

11 1
Chapter 11 Enabling Object Persistence Overview

Overview
As part of the process of executing a parallel application, Orchestrate creates
copies of objects, such as operators, data sets, and partitioners, and then transmits
and loads these objects into the memory of your multiple processing nodes. The
storing and loading of C++ objects to and from files and memory buffers is called
object persistence.
In order for your application to run on multiple nodes, all C++ objects utilized by
your Orchestrate application must support persistence to ensure that your objects
have data portability, structural integrity, valid inheritance, polymorphism, object
versioning, and interoperability with third-party software components.
All classes in the Orchestrate class library already support persistence. You must
add persistence only to those classes that you derive from Orchestrate base classes
or from your own class hierarchy. You add either simple or complex persistence to
a class depending on the functionality of the class.
The Orchestrate persistence mechanism is similar to the C++ stream I/O facility.
Instead of writing objects to streams or reading objects from streams, Orchestrate
stores and loads objects to an archive. An archive has an associated direction, or
mode: either storing for outputting objects or loading for inputting objects.
The process of storing or loading an object is called object serialization. Archives
use overloads of the input and output stream operators, operator>> and
operator<<, to load from and store to an archive. The Orchestrate persistence
mechanism also provides a bidirectional serialization operator, operator||,
which can both load from and store to an archive.
Orchestrate persistence provides two methods for making your classes persistent:
simple-persistence and complex-persistence. Simple-persistence requires less
memory, but it is not adequate for classes with complex functionality. The
following section lists the conditions that require complex-persistence for a class.

Simple Persistence and Complex Persistence

Whether you add simple or complex persistence to a class depends on the
functionality of the class.You must make your class complex-persistent if any of
the following conditions is true:
• You want to serialize pointers to your class. For example, polymorphism is
required when serializing pointers to the class. That is, you want dynamic
typing when serializing a pointer so that the actual, possibly derived, object is
stored and loaded.
• Your class contains complex-persistent sub-structure with the possibility of
structural sharing.

11 2 Orchestrate 7.0 Developer Guide

Chapter 11 Enabling Object Persistence Storing and Loading Simple and Complex Persistent
Objects

• You want data versioning for your class. This is necessary when you change
the definition of a class and then load an object that was stored using an older
class definition. At load time, complex-persistence automatically initializes
the object using the current class definition.
To make a class complex-persistent, you directly or indirectly derive your class
from the APT_Persistent base class.
If your class has none of the complex conditions outlined above, you can make
your class simple-persistent. To do this you simply need to define serialization
operators. Derivation from APT_Persistent is not necessary. The serialized
representation of a simple-persistent object consumes no archive storage other
than that used to serialize the object.
The following section tells you how to create archives and store and load objects
to and from archives. This information applies to both simple and complex
persistence.
These sections tell you how to define persistent classes:
• “Defining Simple-Persistent Classes” on page 11-7
• “Defining Complex-Persistent Classes” on page 11-8

Storing and Loading Simple and Complex

Persistent Objects
! To store an object:
1 Create a storing archive.
2 Use a serialization operator to store the object in the archive.

! To load an object into memory on the processing node:

1 Create a loading archive.
2 Use a serialization operator to load the object from the archive.
The next two subsections describe how to create archives and use serialization
operators.

Creating Archives
The Orchestrate base class APT_Archive defines the base-level functionality of
the archive facility. Orchestrate uses the derived classes APT_FileArchive and
APT_MemoryArchive to perform object serialization. APT_FileArchive is used
to store objects to a file or load objects from a file. APT_MemoryArchive is used
to store objects to a memory buffer or load objects back from a memory buffer.

Orchestrate 7.0 Developer Guide 11 3

Chapter 11 Enabling Object Persistence The APT_Archive Class Interface

Accessing a buffer typically is faster than accessing a file. Objects stored to a

buffer can later be stored to a file.
For example, this line of code creates a storing file archive using the file
input.dat:
APT_FileArchive ar("input.dat", APT_Archive::eStoring);

This line of code creates a loading file archive using the file output.dat:
APT_FileArchive ar("output.dat", APT_Archive::eLoading);

Note that you establish the archive mode when you create the archive object. It’s
mode cannot be subsequently changed. Also, archives do not support a random-
access seeking mechanism. You must load objects back from an archive in the
order in which you store them.

The APT_Archive Class Interface

The following figure shows the interface to the APT_Archive class. See the header
file, $APT_ORCHHOME/include/apt_util/archive.h, for descriptions of the
functions. You can also browse the Orchestrate class hierarchy using HTML-
documented header files. Start your search with the index.html file in
$APT_ORCHHOME/doc/html/.

APT_Archive class interface

public:
checkLoading() ident() isMagicBad()
checkStoring() isEOF() isNull()
deleteWhenDoneSerializing() isFull() isStoring()
failOnBadClass() isLoading() setFailOnBadClass()
flush()

Using the Serialization Operators

Classes that support persistence allow Orchestrate to save an object to an archive
and restore the object from an archive using the following C++ serialization
operators:
• operator<<: the unidirectional store operator for storing objects to an archive
Example: operator<<(APT_Archive&, const my-class&)
• operator>>: the unidirectional load operator for loading objects from an
archive
Example: operator>>(APT_Archive&, const my-class&)

11 4 Orchestrate 7.0 Developer Guide

Chapter 11 Enabling Object Persistence Using the Serialization Operators

• operator||: the bidirectional serialization operator for either storing an object

to an archive or loading an object from an archive, based on the mode of the
referenced archive
Example: operator||(APT_Archive&, const my-class&)
The unidirectional operators, operator>> and operator<<, are used when it is
clear that the function to be performed is either storing or loading. You can use
the bidirectional operator when you are defining general-purpose serialization
functions that allow you to either load or store an object.
Examples of operator<< and operator>> are shown below. The derived class
Catalog supports persistence.
The following example creates a Catalog object and stores it to an archive. A file-
based archive of type APT_FileArchive is created as a storing archive. Archives
can be used either for storing or for loading, but the same archive object cannot be
used for both. The operator<< operator is used to store the Catalog object to the
archive.
Catalog gCat;
void saveCatalogToFile(const Catalog& cat, const char * toFile)
{
APT_FileArchive ar(toFile, APT_Archive::eStoring);
ar << cat;
};

This example loads an object from an archive:

void restoreCatalogFromFile(Catalog& cat, const char * fromFile)
{
APT_FileArchive ar(fromFile, APT_Archive::eLoading);
ar >> cat;
};

Note that any complexity in the internal structure of Catalog does not complicate
the code required to store and load instances of Catalog.
The bidirectional operator, operator||, can perform either storing or loading. It
determines its action based on the mode of the archive supplied to the function.
Here is an example that uses operator||:
APT_FileArchive ar1("input.dat", APT_Archive::eLoading);
APT_FileArchive ar2("output.dat", APT_Archive::eStoring);
Catalog gCat;
ar1 || gCat; // load gCat from input.dat
ar2 || gCat; // store gCat to output.dat

The APT_Archive base class defines bidirectional serialization operators for the
following Orchestrate data types:
• signed or unsigned 8-, 16-, 32- or 64-bit integers
• single-precision (32 bits) or double-precision (64 bits) floats

Orchestrate 7.0 Developer Guide 11 5

Chapter 11 Enabling Object Persistence Using the Serialization Operators

• time of day
• boolean
The one-value serialization operators are shown below:
class APT_Archive
{
public:
...
friend APT_Archive& operator|| (APT_Archive& ar, APT_UInt8& d);
friend APT_Archive& operator|| (APT_Archive& ar, APT_Int8& d);
friend APT_Archive& operator|| (APT_Archive& ar, char& d);

friend APT_Archive& operator|| (APT_Archive& ar, APT_UInt16& d);

friend APT_Archive& operator|| (APT_Archive& ar, APT_Int16& d);

friend APT_Archive& operator|| (APT_Archive& ar, APT_UInt32& d);

friend APT_Archive& operator|| (APT_Archive& ar, APT_Int32& d);

friend APT_Archive& operator|| (APT_Archive& ar, APT_UInt64& d);

friend APT_Archive& operator|| (APT_Archive& ar, APT_Int64& d);

friend APT_Archive& operator|| (APT_Archive& ar, unsigned int& d);

friend APT_Archive& operator|| (APT_Archive& ar, int& d);

friend APT_Archive& operator|| (APT_Archive& ar, time_t& d);

friend APT_Archive& operator|| (APT_Archive& ar, float& d);

friend APT_Archive& operator|| (APT_Archive& ar, double& d);
friend APT_Archive& operator|| (APT_Archive& ar, bool& d);
...
};

In addition, the base class defines the corresponding directional serialization

operators operator>> and operator<<. See
$APT_ORCHHOME/include/apt_util/archive.h for a complete definition of
APT_Archive.
Because derived classes define data in terms of the Orchestrate built-in data types,
you construct serialization operators using the supplied operators defined in
APT_Archive. This means that you need not write serialization operators from
scratch; you can simply build serialization operators from those supplied in
APT_Archive.
If you are creating an operator, partitioner, or collector class compatible with osh,
the Orchestrate shell command facility, you can override initializeFromArgs_() in
the derived class to handle object initialization. In this case, you still must include
an override of APT_Persistent::serialize() in your derived class, but it can be
empty.

11 6 Orchestrate 7.0 Developer Guide

Chapter 11 Enabling Object Persistence Defining Simple-Persistent Classes

Defining Simple-Persistent Classes

You can make a class simple-persistent if it contains no complex functionality.
The section “Simple Persistence and Complex Persistence” on page 11-2 lists the
conditions that give a class complex functionality.
To make a class simple-persistent, you simply need to define serialization
operators for the class. Derivation from APT_Persistent is not necessary.
The following example code shows how simple-persistence can be built into a
class:
#include <apt_util/archive.h>

class FPair
{
float x_, y_;
public:
...
friend APT_Archive& operator|| (APT_Archive& ar, FPair& d)
{
return ar || d.x_ || d.y_; }
};
// define operator<< and operator>>
APT_DIRECTIONAL_SERIALIZATION(FPair);
}

Note that you explicitly provide only the bidirectional serialization operator. By
including the APT_DIRECTIONAL_SERIALIZATION macro, you also provide
both the store and load unidirectional serialization operators. This allows FPair
objects to be serialized in the same manner as built-in types.
For example:
APT_MemoryArchive ar;
FPair fp = ...;
ar << fp;

For simple classes such as FPair, just defining serialization operators suffices to
make a class persistent. Simple-persistent classes have low overhead, since they
don't inherit from a persistence base class, and their serialized representation
consumes no archive storage other than that used to serialize the class members.
When loading, you always call the serialize() function on a default-constructed
object. This is because the loaded object has either just been dynamically
allocated, or it has been explicitly destroyed and default-constructed in place.
This policy simplifies the serialize() function, since it need not worry about the
previous state of an object. With simple persistence, a class's serialization operator
needs to recognize that it might be loading over the pre-load state of an object.
Note that when a simple-persistent object is loaded, the object's state is
overwritten by the serialization operator for the class. It is up to the serialization

Orchestrate 7.0 Developer Guide 11 7

Chapter 11 Enabling Object Persistence Defining Complex-Persistent Classes

operator to properly manage any state that the object may have had prior to
loading.
A crucial limitation of simple-persistent classes is that pointers to class objects
cannot be serialized. If it is necessary to serialize an object pointer, the class must
be complex-persistent.

Defining Complex-Persistent Classes

You must make a class complex-persistent if it contains complex functionality.
The section “Simple Persistence and Complex Persistence” on page 11-2 lists the
conditions that give a class complex functionality.
To make a class complex-persistent and usable within Orchestrate, follow these
rules:
Rule 1 Your class must publicly inherit from APT_Persistent, directly or
indirectly.
Rule 2 Concrete classes must use the APT_DECLARE_PERSISTENT macro in
the .h file and the APT_IMPLEMENT_PERSISTENT macro in the .C file.
Similarly, abstract classes must use the
APT_DECLARE_ABSTRACT_PERSISTENT macro in the .h file and the
APT_IMPLEMENT_ABSTRACT_PERSISTENT macro in the .C file. These
macros generate operator>>, operator<<, and operator|| overloads
automatically to support serialization.
Rule 3 You must create a default constructor. The default constructor need not
be public; it may be protected or private. A compiler-generated default
constructor is acceptable.
Rule 4 You must implement APT_Persistent::serialize() in all derivations,
regardless of whether the derived class is concrete or abstract.
Note that the APT_DECLARE_PERSISTENT macro includes a declaration for
APT_Persistent::serialize() in the .h file.
APT_Persistent::serialize() is the only function that you must define to support
persistence in a derived class. All three serialization operators are declared and
defined by macros in terms of APT_Persistent::serialize().
Rule 5 You must provide Run-Time Type Information (RTTI) support on the
derived class. See Chapter 12, “Run-Time Type Information” for more
information. Note that RTTI currently is not supported for template classes.
The following example shows how to apply the above rules to make your class
complex-persistent:
In a file named ComplexPersistentClass.h, we define a complex persistent class
named ComplexPersistentClass.

11 8 Orchestrate 7.0 Developer Guide

Chapter 11 Enabling Object Persistence The APT_Persistent Class Interface

#include <apt_framework/orchestrate.h>

class ComplexPersistentClass: public APT_Persistent // Rule 1

{
APT_DECLARE_PERSISTENT(ComplexPersistentClass);// Rules 2 & 4
APT_DECLARE_RTTI(ComplexPersistentClass); // Rule 5
public:
ComplexPersistentClass(int, float);
private:
ComplexPersistentClass(); // Rule 3
int i_;
float f_;
};

In the file ComplexPersistentClass.C, the class is implemented as follows:

#include "ComplexPersistentClass.h"

APT_IMPLEMENT_PERSISTENT(ComplexPersistentClass); // Rule 2
APT_IMPLEMENT_RTTI_ONEBASE(ComplexPersistentClass, // Rule 5
APT_Persistent);

ComplexPersistentClass::ComplexPersistentClass() // Rule 3
: i_(0), f_(0.0) {}
CompexPersistentClass::ComplexPersistentClass(int i, float f)
: i_(i), f_(f) {}

void Complex PersistentClass::serialize(APT_Archive& archive,

APT_UInt8) // Rule 4
{
archive || i_;
archive || f_;
}

Note that the definition of APT_Persistent::serialize() uses the built-in forms of

APT_Archive::operator|| to implement persistence for ComplexPersistentClass.
See the next section for more information.

The APT_Persistent Class Interface

The following figure shows the interface to the APT_Persistent class. See the
header file, $APT_ORCHHOME/include/apt_util/persist.h, for descriptions of the
functions. You can also browse the Orchestrate class hierarchy using HTML-
documented header files. Start your search with the index.html file in
$APT_ORCHHOME/doc/html/.

Orchestrate 7.0 Developer Guide 11 9

Chapter 11 Enabling Object Persistence Implementing APT_Persistent::serialize()

APT_Persistent class interface

public: protected: private:
operator||() checkPost() dispatchSerialize()=0
checkPre() instantiate()
cloneBySerializing() reInit()=0
cloneType() serialize()
copyBySerializing() serializeObject()
doThisBase() wrapSerialize()
loadPointer()
registerClass()
storePointer()

Implementing APT_Persistent::serialize()
You use the macro APT_DECLARE_PERSISTENT() to declare
APT_Persistent::serialize() as a private member function in a derived class. The
function prototype of APT_Persistent::serialize() is:
void serialize(APT_Archive& archive, APT_UInt8);

The first argument to APT_Persistent::serialize() specifies an archive. The second

argument is reserved.
Because APT_Persistent::serialize() is called whenever you load or store an
object, you typically implement your definition of APT_Persistent::serialize()
using operator||. The function operator|| determines its operation by the mode
of the archive passed to it.
Note that you can also use the member functions of APT_Archive to determine
the mode of the archive if your class requires special processing for either a load
or a store. This approach lets you use operator<< and operator>> within
APT_Persistent::serialize().
When storing an object to an archive, APT_Persistent::serialize() stores the object.
When storing a pointer to an archive, the object referenced by the pointer is
stored. If the object has already been stored, subsequent stores are not performed.
APT_Persistent::serialize() simply stores a pointer to the previously stored
object.
When an object from an archive is being loaded, the action of
APT_Persistent::serialize() depends on whether you are loading a pointer or a
reference:
• If you are loading a pointer, Orchestrate dynamically allocates the object, calls
the default constructor for the object, and then calls
APT_Persistent::serialize() to load the object from the archive.

11 10 Orchestrate 7.0 Developer Guide

Chapter 11 Enabling Object Persistence Serializing Pointers

• If an object has already been loaded, a second load is not performed. Instead,
operator|| simply initializes the pointer to reference the loaded object.
• If you are loading an object via a reference to a class derived from
APT_Persistent, Orchestrate destroys the object, default constructs the object
in place, and then calls APT_Persistent::serialize() to load the object from the
archive.
• When loading objects of simple data types, such as integers and floats,
APT_Persistent::serialize() simply loads the object.
During loading, an object’s previous state is irrelevant because the object is
always default constructed. Your definition of APT_Persistent::serialize() should
not have any effects other than reading, writing, or otherwise modifying the
object being serialized.

Serializing Pointers
Using classes that support persistence lets you save and load objects directly or
use a pointer to an object. Basic data types such as integers and characters,
however, do not support serialization via a pointer.
You must write your APT_Persistent::serialize() overload to handle serialization
of pointers to data types that do not support the Orchestrate persistence
mechanism. For example, char * pointers can often be replaced by an instance of
the persistent class APT_String. Because APT_String supports persistence, you
can serialize a reference to an APT_String object.

Serializing Arrays
The persistence mechanism does not directly contain support for serializing
arrays. If your classes contain array members, you must build support for array
serialization within APT_Persistent::serialize().
Arrays are relatively simple to serialize. The serialization code depends on
whether the array is fixed-length (and fixed-allocation) or variable-length (and
dynamically allocated).
The example below contains both fixed-length and variable-length arrays. In this
example, ObjClass is a persistent class and Container is a persistent class
containing two ObjClass arrays. When writing the APT_Persistent::serialize()
definition for Container, you would handle the ObjClass arrays as follows:
#include <apt_framework/orchestrate.h>

class ObjClass: public APT_Persistent

{
APT_DECLARE_PERSISTENT(ObjClass);

Orchestrate 7.0 Developer Guide 11 11

Chapter 11 Enabling Object Persistence Serializing Arrays

APT_DECLARE_RTTI(ObjClass);

public:
ObjClass();
. . .
}
class Container: public APT_Persistent
{
APT_DECLARE_PERSISTENT(Container);
APT_DECLARE_RTTI(Container);

public:
Container()
:variable_(0), nVariable_(0)
{}
~Container()
{
delete[] variable_;
}
. ..

private:
ObjClass fixed_[12]; // define a fixed-length array
ObjClass* variable_; // define a variable-length array
int nVariable_; // contains length of array variable_
}

The definition of APT_Persistent::serialize() for Container is shown below in

Table 38. This definition is written using the bidirectional operator|| so that it
can be used for both loading and storing Container objects. Comments follow the
code in the table.

Table 38 Override of APT_Persistent::serialize with Arrays

Code

#include <apt_framework/orchestrate.h>

APT_IMPLEMENT_RTTI_ONEBASE(Container, APT_Persistence);
APT_IMPLEMENT_PERSISTENT(Container);

void Container::serialize(APT_Archive& archive, APT_Persistent);

{
int n;

// A fixed-length array is easy.

8 for (n = 0; n < 12; n++)
{
archive || fixed_[n];
}

11 12 Orchestrate 7.0 Developer Guide

Chapter 11 Enabling Object Persistence Persistence Macros

Table 38 Override of APT_Persistent::serialize with Arrays (continued)

// A variable-length array requires more work.
13 archive || nVariable_;
14 APT_ASSERT(nVariable_ >= 0);
15 if (archive.isLoading())
{
17 if (nVariable_ > 0)
{
variable_ = new ObjClass[nVariable_];
}
}
22 for (n = 0; n < nVariable_; n++)
{
archive || variable_[n];
}
}

Comments

8 Use a simple for loop to serialize the elements of the fixed-length array.

13 Serialize the length of the variable-length array nVariable_.

14 Ensure that the length of the variable-lenth array is greater than 0.

15 Use the APT_Archive::isLoading() member function to determine whether the

operation is a load.

17 If this is a load operation, delete the current contents of variable_..

22 Serialize the elements of the variable-length array.

Persistence Macros
This section describes the macros that you use when declaring and defining
classes that support the Orchestrate persistence mechanism.
• APT_DECLARE_ABSTRACT_PERSISTENT() declares the operator||,
operator<<, operator>>, and serialize() definitions within the definition of an
abstract base class that supports persistence.
#define APT_DECLARE_ABSTRACT_PERSISTENT(className);

className: Specifies the name of the persistent class.

• APT_DECLARE_PERSISTENT() declares the operator||, operator<<,
operator>>, and serialize() definitions within the definition of a class that
supports persistence.

Orchestrate 7.0 Developer Guide 11 13

Chapter 11 Enabling Object Persistence Persistence Macros

#define APT_DECLARE_PERSISTENT(className);

className: Specifies the name of the persistent class.

• APT_IMPLEMENT_ABSTRACT_PERSISTENT() implements operator||,
operator<<, and operator>> in terms of serialize() for an abstract base class
supporting persistence. This macro means that you only have to define
serialize() for the abstract base class; you do not have to define operator||,
operator<<, and operator>>.
#define APT_IMPLEMENT_ABSTRACT_PERSISTENT(className);

className: Specifies the name of the persistent class.

• APT_IMPLEMENT_PERSISTENT() implements operator||, operator<<,
and operator>> in terms of serialize() for a class supporting persistence. This
macro means that you only have to define serialize() for the class; you do not
have to define operator||, operator<<, and operator>>.
#define APT_IMPLEMENT_PERSISTENT(className);

className: Specifies the name of the persistent class.

11 14 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

12
Run-Time Type Information
Describes Orchestrate’s run-time type information facility.

Overview 12 1
Determining the Run-Time Data Type of an Object 12 2
Performing Casts 12 3
Adding RTTI Support to Derived Classes 12 3
RTTI Macros 12 5
Derivation Macros 12 5
Macro APT_IMPLEMENT_RTTI_ONEBASE() 12 6
Application Macros 12 6

Overview
Any derived object class that you want to use in an Orchestrate application must
support the Run-Time Type Information (RTTI) facility. This applies to any class
you derive from an Orchestrate base class, directly or indirectly. RTTI lets you
determine the data type of an object at run-time so that Orchestrate can optimize
functions for some data types, simplify general-purpose functions, and perform
checked casts on pointers.
Support for RTTI is also required by the Orchestrate persistence mechanism. See
Chapter 11, “Enabling Object Persistence” for information.
The Orchestrate RTTI facility is based on the approach described in The C++
Programming Language, Second Edition, by Bjarne Stroustrup.

12 1
Chapter 12 Run-Time Type Information Determining the Run-Time Data Type of an Object

Deriving your classes from a common base class is not necessary to support RTTI.
The RTTI facility supports:
• Multiple inheritance
• Virtual base classes
• Downcasting from a virtual base class
• Checked casts to and from any base pointer or derived pointer
The two primary objectives of the RTTI facility are to determine the run-time data
type of an object and to cast a pointer to a derived type or a base type. These
objectives are described in the next two sections.

Determining the Run-Time Data Type of an

Object
Although modern compilers include RTTI support, Orchestrate classes must
define their own RTTI structures to ensure compatibility with the Orchestrate
framework.
You can determine the run-time data type of any object instantiated from a class
that supports the Orchestrate RTTI facility. Given this capability, you can perform
conditional branching within your application on the data type of an object.
Pointers and references can have both static and dynamic run-time data types.
The static data type, which never changes, is determined when the pointer or a
reference is instantiated.
In the example below, the class DClass is derived from the base class BClass:
DClass dObject; // Static type of dObject is DClass.
BClass * basePtr; // Static type of basePtr is BClass.

The dynamic data type, on the other hand, can change at run-time, as shown here:
DClass dObject; // Static type is DClass.
BClass * basePtr = &dObject; // Static type of basePtr is BClass,
// but its dynamic is DClass.
const char * sType = APT_STATIC_TYPE(*basePtr).name(); // returns
// “BClass"
const char * dType = APT_DYNAMIC_TYPE(*basePtr).name();// returns
// "DClass"

This example uses two RTTI macros:

• APT_STATIC_TYPE(): Returns an APT_TypeInfo object describing the static
data type of an object reference. The static data type is the data type of the
reference, not of the object referenced.

12 2 Orchestrate 7.0 Developer Guide

Chapter 12 Run-Time Type Information Performing Casts

• APT_DYNAMIC_TYPE(): Returns an APT_TypeInfo object describing the

dynamic data type of an object reference. The dynamic data type is the data
type of the object referenced.
The class APT_TypeInfo contains information describing the data type of an
object. APT_TypeInfo::name() returns a string containing the class name of the
data type. See “RTTI Macros” on page 12-5 for more information on these macros.

Performing Casts
Casting lets you assign a pointer to another pointer of a different data type. The
data types of both the pointer and the casted pointer must support the RTTI
facility.
You perform checked casts using the APT_PTR_CAST() macro which converts a
pointer to a pointer of a new data type. If the cast cannot be performed, the
pointer is set to 0. The data types of both the pointer and the casted pointer must
exist in the same inheritance hierarchy.
Shown below is an example using APT_PTR_CAST(). This example uses the
class DClass derived from the base class BClass:
BClass bObject;
BClass * bPtr = &bObject;
// cast bPtr to type DClass
DClass * dPtr = APT_PTR_CAST(DClass, bPtr);

if (dPtr)
// APT_PTR_CAST() returns 0 if the cast cannot be performed
{
...
};

Adding RTTI Support to Derived Classes

You must build RTTI support into any class you derive from one of the
Orchestrate abstract base classes. You also can build the Orchestrate RTTI facility
into your own object classes that are not derived from Orchestrate base classes.
Orchestrate supplies macros that simplify the inclusion of the RTTI facility into a
derived class. The following example shows the definition of the class
ExampleOperator, derived from APT_Operator. In the example.h file, you declare
the class as shown below:
#include <apt_framework/orchestrate.h>
class ExampleOperator: public APT_Operator
{
APT_DECLARE_RTTI(ExampleOperator);
APT_DECLARE_PERSISTENT(ExampleOperator);

Orchestrate 7.0 Developer Guide 12 3

Chapter 12 Run-Time Type Information Adding RTTI Support to Derived Classes

public:
ExampleOperator();
protected:
virtual APT_Status describeOperator();
virtual APT_Status runLocally();

};

The macro APT_DECLARE_RTTI() is used to declare ExampleOperator as a class

that supports the RTTI facility. With the APT_DECLARE_PERSISTENT macro,
which is required, object persistence for operator objects transmitted from
Orchestrate to the processing nodes is declared.
In example.C, you define ExampleOperator as shown below:
#include "example.h"

APT_IMPLEMENT_RTTI_ONEBASE(ExampleOperator, APT_Operator);
APT_IMPLEMENT_PERSISTENT(ExampleOperator);
...

The macro APT_IMPLEMENT_RTTI_ONEBASE() is used to implement the

RTTI facility for a derived class with a single base class, and the macro
APT_IMPLEMENT_PERSISTENT() is used to implement the Orchestrate
persistence mechanism.
You use different macros depending on the derivation hierarchy of a class. For
example, if you define a class using multiple inheritance, you declare and define
the class as shown in the following example. In the .h file:
class MI_Class: public B1, public B2
{
APT_DECLARE_RTTI(MI_Class);
APT_DECLARE_PERSISTENT(MI_Class);
public:
...
};

The class MI_Class is derived from two base classes: B1 and B2. Typically, both
base classes should support the RTTI facility. If a base class does not support
RTTI, that class cannot be used as part of a checked cast.
Note that MI_Class is a completely user-defined class; it is not derived from an
Orchestrate base class.
In the .C file, you must include the macros:
APT_IMPLEMENT_RTTI_BEGIN(MI_Class);
APT_IMPLEMENT_RTTI_BASE(MI_Class, B1);
APT_IMPLEMENT_RTTI_BASE(MI_Class, B2);
APT_IMPLEMENT_RTTI_END(MI_Class);
...

The APT_IMPLEMENT_RTTI_BEGIN() macro starts a block defining the base

classes of MI_Class, the APT_IMPLEMENT_RTTI_BASE() macros specify the

12 4 Orchestrate 7.0 Developer Guide

Chapter 12 Run-Time Type Information RTTI Macros

base classes of MI_Class, and the APT_IMPLEMENT_RTTI_END() macro ends

the definition.
An APT_IMPLEMENT_RTTI_BEGIN() - APT_IMPLEMENT_RTTI_END()
block can contain only a sequence of APT_IMPLEMENT_RTTI_BASE() macros.
A checked cast to an ambiguous base class yields the first base class defined in an
APT_IMPLEMENT_RTTI_BEGIN() - APT_IMPLEMENT_RTTI_END() block.
For cases in which the class being defined has no bases or only one base, the
APT_IMPLEMENT_RTTI_NOBASE() and
APT_IMPLEMENT_RTTI_ONEBASE() macros can be used instead.

RTTI Macros
This section describes the Orchestrate macros that support RTTI. These macros
fall into two categories:
• Derivation macros
• Application macros

Derivation Macros
You use RTTI derivation macros when you are declaring and defining object
classes that support the Orchestrate RTTI.
• APT_DECLARE_RTTI(). You insert this macro in a class definition to
support the RTTI facility.
APT_DECLARE_RTTI(className);

className: Specifies the name of the class.

• APT_IMPLEMENT_RTTI_BASE(). This macro specifies a base class of a
derived class within an APT_IMPLEMENT_RTTI_BEGIN() -
APT_IMPLEMENT_RTTI_END() block in the source code file for the
derived class.
APT_IMPLEMENT_RTTI_BASE(className, baseClass);

className: Specifies the name of the derived class.

baseClass: Specifies a base class of className.
• APT_IMPLEMENT_RTTI_BEGIN(). If a derived class has two or more base
classes, you must specify each base class within an
APT_IMPLEMENT_RTTI_BEGIN() - APT_IMPLEMENT_RTTI_END()
block in the source code file for the derived class. This macro defines the
beginning of this block.
APT_IMPLEMENT_RTTI_BEGIN(className);

Orchestrate 7.0 Developer Guide 12 5

Chapter 12 Run-Time Type Information RTTI Macros

className: Specifies the name of the derived class.

• APT_IMPLEMENT_RTTI_END(). If a derived class has two or more base
classes, you must specify each base class within an
APT_IMPLEMENT_RTTI_BEGIN() - APT_IMPLEMENT_RTTI_END()
block in the source code file for the derived class. This macro defines the end
of this block.
APT_IMPLEMENT_RTTI_END(className);

className: Specifies the name of the derived class.

• APT_IMPLEMENT_RTTI_NOBASE(). If a derived class has no base class,
you must insert this macro in the source code file of the derived class.
APT_IMPLEMENT_RTTI_NOBASE(className);

className: Specifies the name of the derived class.

Macro APT_IMPLEMENT_RTTI_ONEBASE()
If a derived class has a single base class, you use this macro in the source code file
of the derived class to specify the base class name.
APT_IMPLEMENT_RTTI_ONEBASE(className, baseClass);

className: Specifies the name of the derived class.

baseClass: Specifies a base class of className.
This macro is equivalent to:
APT_IMPLEMENT_RTTI_BEGIN(baseClass);
APT_IMPLEMENT_RTTI_BASE(className, baseClass);
APT_IMPLEMENT_RTTI_END(baseClass);

Application Macros
You use the RTTI application macros to determine the data type of objects
instantiated from a class that supports the Orchestrate RTTI facility.
• APT_DYNAMIC_TYPE(). This macro returns an APT_TypeInfo object
describing the dynamic, or run-time, data type of an object. The object must be
instantiated from a class that supports RTTI.
APT_TypeInfo& APT_DYNAMIC_TYPE(object);

object: Specifies an object instantiated from a class that supports the RTTI
facility.
The example below uses APT_DYNAMIC_TYPE(), with the class DClass
derived from the base class BClass:
DClass dObject;
BClass * basePtr = &dObject;

12 6 Orchestrate 7.0 Developer Guide

Chapter 12 Run-Time Type Information RTTI Macros

const char * dType = APT_DYNAMIC_TYPE(*basePtr).name();

// returns "DClass"

• APT_PTR_CAST(). This macro lets you perform a checked cast to assign a

pointer of one data type to a pointer of a different data type. The pointer and
casted pointer both must reference objects instantiated from classes that
support RTTI.
destType * APT_PTR_CAST(destType, sourcePtr);

destType: Specifies the resultant data type of the cast. The data types destType
and sourcePtr must exist in the same inheritance hierarchy; otherwise the
pointer is set to 0.
sourcePtr: A pointer to convert.
APT_PTR_CAST() returns 0 if sourcePtr does not exist in the same inheritance
hierarchy of type destType, or if sourcePtr is equal to 0.
• APT_STATIC_TYPE(). This macro returns an APT_TypeInfo object
describing the static data type of an object reference. The class must support
RTTI.
const APT_TypeInfo& APT_STATIC_TYPE(object);

object: Specifies an object or a reference to an object instantiated from a class

that supports the RTTI facility.
The example below uses APT_STATIC_TYPE(), with the class DClass derived
from the base class BClass:
DClass dObject;
const char * dType = APT_STATIC_TYPE(dObject).name();
// returns "DClass"

• APT_TYPE_INFO(). This macro returns an APT_TypeInfo object describing

the class type. The class must support RTTI.
const APT_TypeInfo& APT_TYPE_INFO(className);

className: Specifies a class that supports the RTTI facility.

Orchestrate 7.0 Developer Guide 12 7

Chapter 12 Run-Time Type Information RTTI Macros

12 8 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

13
Debugging Your Applications
Describes Orchestrate’s debugging facilities.

Overview 13 1
Choosing a Debugger 13 2
Specifying Program Execution Mode 13 2
Modifying the Configuration File 13 3
Debugging With Third Degree on Tru64 13 4
Debugger Options with dbx 13 5
Debugging in Sequential Mode 13 5
Debugging in Parallel Mode 13 9
Profiling Your Application 13 11
The Score Dump 13 12

Overview
Parallel processing implies the simultaneous execution of multiple processes on
multiple nodes. Under these circumstances, tracking program execution with a
debugger can be extremely difficult. For this reason, the Orchestrate development
environment offers several features that simplify the debugging of parallel
applications. Using these features, you can start debugging your application as a
sequential program executing in a single process on a single node, and then
proceed by degrees until you execute the program in full parallel mode.

13 1
Chapter 13 Debugging Your Applications Choosing a Debugger

In order to debug your application, you must set these options:

• Debugger. Orchestrate supports debuggers on all supported platforms.
• Program execution mode. Orchestrate supports debugging in both parallel
and sequential execution modes.
• System configuration. Orchestrate allows you to modify the configuration
file to control the system resources used by your program.
The following sections describe each of these options.

Choosing a Debugger
Orchestrate supports the following debuggers:
• dbx on AIX and Sun Solaris
• gdb on HP-UX
• ladebug on Tru64
Add the path to your debugger to your PATH variable. Because all debugging is X-
based, your DISPLAY environment variable must be set to an explicit host name.
Note that a host name of :0.0 (meaning “this host's X server”) is not an acceptable
setting as it would configure Orchestrate to open debugger windows on the
individual nodes of the your machine.
If you are using dbx, the debugger starts in an xterm window; therefore,
Orchestrate must know where to find the xterm binary. The default location is
/usr/bin/X11/xterm. You can override this default by setting the
APT_PM_XTERM environment variable to the appropriate path.

Specifying Program Execution Mode

The mode of Orchestrate program execution, parallel or sequential, is controlled
by the APT_EXECUTION_MODE environment variable. By default, parallel
execution is enabled. You set the APT_EXECUTION_MODE environment
variable to specify sequential execution.You do not need to change or recompile
your application when changing execution mode.
In sequential mode, the application is executed entirely on a single node, and
multiple instances of operators are executed in sequence rather than
simultaneously. Orchestrate calls operators as subroutines in the main process or
executes them in separate processes, according to the particular value you assign
to APT_EXECUTION_MODE. Possible values for this variable are discussed in
“Debugging in Sequential Mode” on page 13-5.

13 2 Orchestrate 7.0 Developer Guide

Chapter 13 Debugging Your Applications Modifying the Configuration File

Modifying the Configuration File

The information in your Orchestrate configuration file
($APT_ORCHHOME/etc/config.apt) is used to determine the configuration of
your system. When executing in parallel mode, Orchestrate attempts to run your
application on every processing node in the default node pool as defined in the
configuration file. Data sets are partitioned according to the number of executing
nodes, and an appropriate number of processes are spawned on these nodes to
execute the operators in your application.
In sequential mode, the application is always executed on a single node.
However, the number of nodes in the configuration file is still used to determine
the number of partitions into which data sets are divided and, thus, the number of
times each operator is called.
In order to simplify testing and debugging, you should start with a configuration
file that defines a single processing node, meaning each data set has a single
partition. As you gain confidence in your application, you can enable more
processing nodes in the configuration file.
When Orchestrate executes an application, it first searches the current working
directory, then $APT_ORCHHOME/etc for the file config.apt. You should
make a copy of the original configuration file and edit only the copy during
application development.
You can override the default location and name of the Orchestrate configuration
file using the environment variable APT_CONFIG_FILE. Use this variable to
explicitly specify the name and location of your local version of the configuration
file. During testing, you may wish to create several configuration files: one that
defines only a single processing node, one that defines a few nodes, and one that
defines all the nodes in your system. As testing progresses, you can increase the
number of nodes by changing the environment variable APT_CONFIG_FILE to
point to the appropriate configuration file.
More information about the configuration file can be found in the Orchestrate 7.0
Installation and Administration Manual.

Orchestrate 7.0 Developer Guide 13 3

Chapter 13 Debugging Your Applications Debugging With Third Degree on Tru64

Debugging With Third Degree on Tru64

This section describes how to set up the Third Degree debugger with an osh
executable to debug your application. Follow these steps:
1 Compile your source code and create your shared libraries. Include this
compiler option:
-g -pthread"

Name your library file libop.so, and create an op.so link to it, where op is the
name of your operator. The first library is used to rebuild the osh executable;
the second library is used for operator registration.
For example, for the file sum.C, which contains the source code for the sum
operator, there should be two libraries: libsum.so and sum.so.
If the source code is not for operator implementation, it is not necessary to
create the link.
2 Rebuild the osh executable linked with your shared libraries:
/bin/cxx -D__STANDARD_TEMPLATE_SPECIALIZATION__
-D__STANDARD_TEMPLATE_INSTANTIATION__
-stdnew -ieee -msg_display_number -msg_disable 111
-D _XOPEN_SOURCE_EXTENDED -D _OSF_SOURCE -DRW_MULTI_THREAD -D_LP64
-pthread -g -no_inline osh.o -o osh.new
-L/APT_ORCHHOME_dir /lib -L/Custom_Operator_dir -lorchoslOSF1
-lorchOSF1 -lorchmonitorOSF1 -lorchcoreOSF1 -l*** -so_archive -
lpthreads -L/ICU_Lib_dir
-lustdio -licuuc -lm

Where:
• APT_ORCHHOME_dir is the value of your $APT_ORCHHOME variable.
• Custom_Operator_dir is your shared-library directory. Place the
shared libraries built in step 1 in this directory.
• ICU_Lib_dir is the directory where the ICU libraries are installed.
• Your shared libraries replace ***.
To obtain osh.o, the osh executable, send email to:
[email protected]

3 Generate the new executable, osh.new:

third -pthread -incobj ***.so [-incobj ...]
-L/Custom_Operator_dir osh.new

This generates a new version of the osh executable, osh.new.third, in the

current directory, as well as debuggable libraries and the log file,
osh.new.3log.

13 4 Orchestrate 7.0 Developer Guide

Chapter 13 Debugging Your Applications Debugger Options with dbx

4 Set your $LD_LIBRARY_PATH environment variable to your working directory.

5 Run ./osh.new.third [arguments].
6 Analyze the error log.
Steps 3, 4, and 5 can be combined with this command:
third -pthread -incobj ***.so [-incobj ...] -L/Custom_Operator_dir
-display osh.new [osh arguments ]

To obtain more information on how to use Third Degree, access:

http://h30097.www3.hp.com/dtk/misc_docs/prog_gd/index.html
For more Third usage options, refer to the Third man page:
man third

Debugger Options with dbx

With the dbx debugger, you can use the APT_DEBUG_SIGNALS environment
variable to specify signals that cause the debugger to start. If any of these signals
occur within an APT_Operator::runLocally() function, dbx is invoked.
You can set APT_DEBUG_SIGNALS to any one of the following values:
• SIGBUS
• SIGPIPE
• SIGKILL
• SIGSEGV
• SIGABRT
• SIGSYS
• SIGTRAP

Debugging in Sequential Mode

Debugging a parallel Orchestrate application can be difficult when the
application executes in multiple UNIX processes on multiple processing nodes. In
order to simplify application development and debugging, you can execute your
Orchestrate application in sequential execution mode.
Sequential mode executes your application on a single processing node. To run
your application in sequential execution mode, you must set the
APT_EXECUTION_MODE environment variable to one of the following values:

Orchestrate 7.0 Developer Guide 13 5

Chapter 13 Debugging Your Applications Debugging in Sequential Mode

APT_Execution Mode Meaning

ONE_PROCESS one-process mode

MANY_PROCESS many-process mode

NO_SERIALIZE many-process mode, without serialization

In one-process mode:
• The application executes in a single UNIX process. You only need to run a
single debugger session, and you can set breakpoints anywhere in your code.
• Data is partitioned according to the number of nodes defined in the
configuration file.
• Orchestrate calls APT_Operator::runLocally() as a subroutine. Each
operator’s runLocally() function is called the number of times appropriate for
the number of partitions on which it must operate.
• Orchestrate operators are saved using the persistence mechanism after
initialization, and are loaded each time before execution of runLocally().
In many-process mode, rather than calling operators as subroutines, the
framework forks a new process for each instance of each operator and waits for it
to complete. If APT_EXECUTION_MODE is set to NO_SERIALIZE, the
persistence mechanism is not used to load and save objects. Turning off
persistence may be useful for tracking errors in serialization code in your derived
classes. If turning off serialization in a program that previously crashed results in
the program executing correctly, the problem may be located in your serialization
code.
You can also set APT_DEBUG_OPERATOR to specify the operators to start your
debugger on. If not set, no debugger is started. If set to an operator number (as
determined from the output of APT_DUMP_SCORE), your debugger is started
for that single operator. If set to -1, your debugger is started for all operators. See
“The Score Dump” on page 13-12 for more information on APT_DUMP_SCORE.
In addition, setting APT_DSINFO causes the process manager to print a report
showing the operators, processes, and datasets in the step it is running.
When running Orchestrate in parallel mode, you can set the environment variable
APT_DEBUG_PARTITION to control which partitions your debugger starts on.
If set to a single number, your debugger is started for just that partition. If set to
-1, your debugger is started for all partitions.
The following table shows the interaction of the APT_DEBUG_OPERATOR and
APT_DEBUG_PARTITION environment variables:

13 6 Orchestrate 7.0 Developer Guide

Chapter 13 Debugging Your Applications Debugging in Sequential Mode

APT_DEBUG_ APT_DEBUG_ effect

OPERATOR PARTITION

not set any value no debugging

-1 -1 or not set debug all processes

-1 partition number debug a specific partition of all

operators

operator number -1 debug all partitions of a specific

operator

operator number partition number debug a specific partition of a

specific operator

During program execution in sequential mode, virtual data sets are written to
files in the directory defined by the environment variable APT_SEQ_VDS_DIR,
which is, by default, the current working directory. These files are named using
the prefix aptvds. This allows you to examine a virtual data set as part of
debugging your application. Virtual data sets are deleted by the framework after
they are used.
Because your application executes on a single processing node in sequential
execution mode, it has access only to the resources of that node. Therefore, when
using the sequential execution mode, you typically construct a testing data set as
a subset of your complete data set. The testing data set should be small enough
that it can easily be handled on a single processing node using available system
resources.
Most application testing and debugging can be done in sequential one-process
mode.
Here is a typical list of debugging steps:
1 Set APT_EXECUTION_MODE to ONE_PROCESS.
2 Edit the configuration file to define a single processing node. This avoids data
partitioning, allowing you to concentrate on debugging your main program
and operators.
3 Check the use of global and static member variables.
When running in full parallel mode, globals and statics cannot be used to
communicate between operators or between the main program and operators.
In one-process mode, however, there is no separation of the global memory of
the main program and operators; therefore, bidirectional communication
through globals and statics is (temporarily) possible. A bug of this type will
not manifest itself until you run in many-process mode or in parallel mode.

Orchestrate 7.0 Developer Guide 13 7

Chapter 13 Debugging Your Applications Debugging in Sequential Mode

You should guard against this possibility by careful coding and quick
inspection during testing. Test and debug your main program: command line
parsing, logical flow, and so on.
4 Test and debug your overrides of the persistence function
APT_Persistent::serialize().
In parallel execution mode, the serialize() function is used to store copies of
objects, such as operators and data sets, transmit the copies to the processing
nodes in your system, and load the objects into memory on the processing
node. In sequential mode, even though the application executes on a single
node, serialize() is still used to save objects after initialization and to
reinitialize objects before each use.
Because incorrect serialization can cause a variety of problems at different
stages of testing, it is wise to check this code early in the process. Temporarily
setting APT_EXECUTION_MODE to NO_SERIALIZE turns off serialization
and may help to localize a bug in your serialization code.
Test and debug your operator overrides of
APT_Operator::describeOperator() and APT_Operator::runLocally().
APT_Operator::describeOperator() configures an Orchestrate operator for
execution. When you derive your own operators, you override
APT_Operator::describeOperator() to configure the operator.
APT_Operator::runLocally() defines the action of an operator. As the name
implies, this function executes locally on each processing node when you are
running in parallel execution mode. In one-process sequential mode, this
function is called as a subroutine for each input data set partition for an
operator. In many-process mode, this function is executed in a separate
process for each input data set partition.
Because APT_Operator::describeOperator() always executes sequentially,
even when running in parallel execution mode, it is easier to debug than
APT_Operator::runLocally(). Bugs that occur in APT_Operator::runLocally()
may occur in parallel and are therefore harder to debug, particularly in the
maintenance phase of programs. For this reason, you should put as much
setup and initialization as possible in APT_Operator::describeOperator().
For example, it is better to initialize data structures in
APT_Operator::describeOperator() and store them in data members of an
operator class than to build and initialize the data structure on first use within
APT_Operator::runLocally(). Note that this makes serialization more
complicated, because more information must be serialized.
5 Test and debug your custom partitioners. In an earlier step, you edited the
configuration file to define just one processing node, so the partitionInput()
functions of your custom partitioners have so far only been called with an
argument of 1. If this seems to work, edit the configuration file to increase the
number of nodes and test the ability of your partitioners to divide data sets

13 8 Orchestrate 7.0 Developer Guide

Chapter 13 Debugging Your Applications Debugging in Parallel Mode

into a larger number of partitions. Because partitioners act identically in any

execution mode, it is easier to debug your partitioners before attempting to
run in many-process or parallel-execution mode.
Increasing the number of nodes in the configuration file also causes the
runLocally() function of your parallel operators to be called multiple times,
rather than just once. The operator is initialized by the persistence mechanism
each time before runLocally() is called. If your application seems to fail on the
second call to an operator’s runLocally() function, check for faulty or
incomplete serialization.
6 Repeat step 5 for your custom collectors.
Once your application works in one-process mode, you can continue by running
it in many-process mode, and finally in full parallel mode. Many-process mode is
very similar to parallel mode, with three exceptions:
• All processes execute on a single node.
• Operator instances are run in separate processes, with calls to UNIX fork()
instead of execve().
• The main program waits until each forked process is completed before
starting the next.

Note Named pipes can cause processes to hang.

The forking of the new process in many-process mode has two implications for
debugging. First, changes to global variables in the main program will be visible
to operator instances, although the reverse is not true. As stated earlier,
communication using global variables is not possible during parallel execution
and should be considered a bug, even though the application may work as
intended when run in sequential mode.
The second implication of forking is that operator instances in the main process
are never actually used, and remain freshly initialized while the forked processes
are executed. Because the forked processes are exact copies of the main process,
each forked operator instance is automatically initialized as it is forked. Thus, the
persistence mechanism is redundant in the many-process execution mode. To test
for bugs in the serialization code, you can turn off persistence by setting
APT_EXECUTION_MODE to NO_SERIALIZE.

Debugging in Parallel Mode

When your application works properly in sequential execution mode, it is time to
test the program in parallel execution mode. In parallel mode, your application
executes in multiple UNIX processes on multiple processing nodes using the full

Orchestrate 7.0 Developer Guide 13 9

Chapter 13 Debugging Your Applications Debugging in Parallel Mode

capabilities of the Orchestrate framework. Parallel execution is enabled by

unsetting the environment variable APT_EXECUTION_MODE.
Most program testing should already be done by the time you first execute in
parallel mode. Parallel mode is primarily a test of your system configuration and
your application’s ability to handle large amounts of data. The following types of
bugs also may be exposed when you run in parallel mode:
• Communication between operators, or between operators and the main
program, by means of global or static member variables, is impossible in
parallel execution mode. Bugs involving communication through globals may
escape detection in sequential execution mode.
• Initialization of operators and other objects is entirely dependent on the
persistence mechanism. Remaining bugs in serialization, especially
incomplete serialization, will become evident at this time.
You can edit your configuration file to slowly increase the number of processing
nodes. You can start by executing in parallel on just a single node and proceed
from a few nodes to the full parallel system. Try this using the testing data set you
used when debugging in sequential mode. Use the full data set only in the last
stage of testing.
In parallel execution mode, you can start your debugger on any or all processes of
an Orchestrate job. This enables you to debug the code in
APT_Operator::runLocally() on a single processing node or on every node in
your system. Your application halts execution immediately before Orchestrate
calls an operator’s runLocally().
Debugging in parallel execution mode is controlled by setting the
APT_DEBUG_STEP, APT_DEBUG_OPERATOR, and
APT_DEBUG_PARTITION environment variables. These environment variables
configure Orchestrate to start the debugger just before the call to runLocally() for
a specific operator. These environment variables allow you to debug all partitions
of an operator, one partition of every operator, one specific process, or all
processes.
Use the following procedure to start debugging:
1 If the debugger is not already in your path, set the environment variable
DEBUGGER to specify the path of the debugger.
2 Determine the step number, operator number, and partition number of the
process to be debugged. You might want to debug all the processes associated
with a particular operator in a particular step, all the operators containing a
particular partition of data, or just a single partition of a single operator.
Each operator has a number associated with it which is its position in the
collection of operators that make up a step. These numbers run from 0 to
(n - 1), where n is the number of operators in the step. The instances, or

13 10 Orchestrate 7.0 Developer Guide

Chapter 13 Debugging Your Applications Profiling Your Application

partitions, of an operator are numbered from 0 to (m - 1), where m is the

number of processing nodes executing the operator. Steps are numbered
sequentially from 0; the step number is incremented each time a step
completes (successfully or not).
When an application runs, Orchestrate prints a report showing the operators,
processes, and data sets in the running application. For more information, see
“The Score Dump” on page 13-12.
3 Set the environment variables APT_DEBUG_STEP,
APT_DEBUG_OPERATOR, and APT_DEBUG_PARTITION appropriately.
These variables can be set to select an individual step, operator, or partition,
or they can be set to -1 to specify all steps, operators, or partitions.
APT_DEBUG_OPERATOR: Specifies the operators to start debuggers on. If
not set, no debuggers are started. If set to an operator number (as determined
from the output of APT_DUMP_SCORE), debuggers are started for that
single operator. If set to -1, debuggers are started for all operators. See “The
Score Dump” on page 13-12 for more information on APT_DUMP_SCORE.
APT_DEBUG_PARTITION: Specifies the partitions to start debuggers on.
One instance, or partition, of an operator executes on each node executing the
operator. If set to a single number, debuggers are started for just that partition.
If set to -1, debuggers are started for all partitions.
For example, setting APT_DEBUG_STEP to 0, APT_DEBUG_OPERATOR to
1, and APT_DEBUG_PARTITION to -1 starts debuggers for every partition
of the second operator in the first step.
4 Run the program. Orchestrate will analyze the program, start all of the
processes needed to execute the program, establish the necessary data
connections, then stop when it detects a breakpoint condition as determined
by APT_DEBUG_STEP, APT_DEBUG_OPERATOR, and
APT_DEBUG_PARTITION.
5 Wait for the debuggers to start up. This can take several minutes if many
processes are being debugged. Each debugger starts in an xterm window
whose title describes the process controlled by that debugger.
6 Set any breakpoints needed in the processes being debugged. This typically
means a breakpoint in an APT_Operator::runLocally() override using the
debugger command stop in.
7 Use the debugger continue statement to resume your application.

Profiling Your Application

Orchestrate enables you to profile your application with the UNIX gprof utility.
You use a profile-enabled version of osh, called osh.pg, which was compiled and

Orchestrate 7.0 Developer Guide 13 11

Chapter 13 Debugging Your Applications The Score Dump

linked with the -pg option. It is provided in your installation directory in

$APT_ORCHHOME/bin/.

To enable profiling of your operators, compile and link your dynamic libraries
with the -pg option, and profile your application as a sequential program
executing as a single process on one node.
When you run your application with osh.pg, a profile file named gmon.out is
generated. Use this command to produce a profile data report:
gprof [options ] osh.pg gmon.out

See the gprof man page for a detailed description of this utility.

The Score Dump

If you set the environment variable APT_DUMP_SCORE, Orchestrate prints a
report showing the operators, processes, and data sets in a running application.
For Korn and Bourne shells, you can use the following statements to set
APT_DUMP_SCORE from the command line:
$ APT_DUMP_SCORE=1
$ export APT_DUMP_SCORE

You can unset the environment variable using the command:

unset APT_DUMP_SCORE

For example, shown below is the data-flow diagram for a simple application with
one step and two operators:

test_ds.ds

step

TestOperator0

TestOperator1

out_ds.ds

13 12 Orchestrate 7.0 Developer Guide

Chapter 13 Debugging Your Applications The Score Dump

When you execute this application with APT_DUMP_SCORE set, Orchestrate

prints the following report. Comments follow the output in the table.

Table 39 APT_DUMP_SCORE Report

Output

1 ORCHESTRATE Application Framework

Copyright (C) 1995-2001 Torrent Systems, Inc.
All Rights Reserved

ORCHESTRATE VX.Y

5 ##I TFSC00000 09:13:40(000) APT configuration file: ./config/apt

6 This step has 3 datasets:

7 ds0: test_ds.ds -> op0[2p] (parallel TestOperator0)
8 ds1: op0[2p] (parallel TestOperator0) => op1[2p] (parallel TestOperator1)
9 ds2: op1[2p] (parallel TestOperator1) -> out_ds.ds

10 It has 2 operators:
11 op0[2p] (parallel TestOperator0) on node0[op0,p0] node1[op0,p1]
12 op1[2p] (parallel TestOperator1) on node0[op1,p0] node1[op1,p1]

##I TFSC00001 09:13:45(000) Step execution finished with status = OK.

##I TCOS00022 09:13:45(001) Startup Time 0:04
Production Run Time 0:0116:28:44 00

Comments

1 Program identification and version information.

5 Path to the Orchestrate configuration file.

6 Number of data sets used in the step.

7 Information about ds0, the first data set. Data sets are number in the form dsNN,
where NN specifies the data set number.
ds0 is read from the file test_ds.ds and is used as input to operator op0 (an instance
of TestOperator). The number in brackets, [2p], indicates that op0 has two
partitions.

8 Information about ds1. The data set is delivered as output by op0 and is used as
input by op1 (an instance of TestOperator).

9 Information about ds2. The data set is delivered as output by op1 and is written to
the file out_ds.ds.

Orchestrate 7.0 Developer Guide 13 13

Chapter 13 Debugging Your Applications The Score Dump

Table 39 APT_DUMP_SCORE Report (continued)

10 Number of operators in the step.

11 Information about op0, the first operator. Operators are numbered in the form
opNN, where NN is the number of the operator in the step.The operator is an
instance of class TestOperator. It runs on two nodes, called node0 and node1 (the
available nodes were defined in the Orchestrate configuration file). The numbers
in brackets after each node indicate both the operator and the partition numbers,
which are used when debugging in parallel execution mode.

12 Information about op1. The operator is an instance of class TestOperator and runs
on nodes node0 and node1.

Note that the operators, data sets, and nodes are numbered sequentially from 0.
The operator and partition numbers are used to set breakpoints when executing
in parallel, as described in“Debugging in Parallel Mode” on page 13-9.

13 14 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

14
Using the Error Log
Describes the Orchestrate error log facility and the APT_ErrorLog class.

Overview 14 1
The APT_ErrorLog Class Interface 14 2
Base Class Error Log Functions 14 3
The Error Log Compared to the Assertion Failures Facility 14 3
Using the Error Log 14 4
Error Log Structure 14 4
Defining Message Identifiers 14 5
Using the Error Log in a Derived Operator 14 6

Overview
Orchestrate allows you to derive your own classes from its base classes. As part of
deriving a class, you must provide overrides for any pure virtual functions in the
base class and, optionally, overrides for any virtual functions.
Users of your derived class call its member function to create and configure a
class instance. As part of this process, users may cause an error or warning
condition. Errors are often the result of an invalid configuration or missing
configuration information.
In Orchestrate, you can use the error-log facility in your derived class to hold
error and warning messages; Orchestrate then displays the accumulated
information to the user.

14 1
Chapter 14 Using the Error Log The APT_ErrorLog Class Interface

In order to relay error or warning information to the user, you write a message to
the APT_ErrorLog for that object. You may also write an informational message
to the error log that is not an error or a warning, but may be useful to users for
detecting and correcting problems.
If the condition is mild, meaning the object can still be used, you write a warning
message to the error log and return APT_StatusOk from the function override.
Upon return from an overridden function, Orchestrate checks the error log for the
object. If any warning messages are stored in the log, they are written to standard
error. By default, standard error corresponds to the screen on the workstation
used to invoke the application.
If the condition is severe enough that the object is unusable, you write the error
information to the log. Additionally, a function override should return
APT_StatusFailed in the case of an error. When a function returns
APT_StatusFailed, any error or warning information in the log is printed to
standard error and the application is terminated. For this reason, you should
make sure that error messages clearly state any information that will aid the user
in determining the cause of the error.
A function may detect several different error or warning conditions. The error log
allows you to write multiple messages to it, for errors, warnings, and any
additional information. You can use the member functions of APT_ErrorLog to
return a count of the number of errors, warnings, or informational messages
written to it.

The APT_ErrorLog Class Interface

The next figure shows the interface to APT_Error Log which defines the error log
facility. The header file for this class, $APT_ORCHHOME/include/apt_util/errlog.h,
contains descriptions of the functions. You can also browse the Orchestrate class
hierarchy using HTML-documented header files. Start your search with the
index.html file in $APT_ORCHHOME/doc/html/.

APT_ErrorLog class interface

public:
constructor hasInfo() operator*()
appendLog() hasWarning() resetError()
dump() ident() resetInfo()
getWarning() infoCount() resetWarning()
errorCount() log() setIdent()
getError() logError() setModuleId()
getInfo() logInfo() warningCount()
hasError() logWarning()

14 2 Orchestrate 7.0 Developer Guide

Chapter 14 Using the Error Log Base Class Error Log Functions

Each instance of an Orchestrate operator, partitioner, or collector is assigned an

instance of APT_ErrorLog. In your function overrides, you can access the
APT_ErrorLog object for the class instance, then write error and warning
information to the log using APT_ErrorLog member functions.

Base Class Error Log Functions

The Orchestrate base classes provide a member function that returns the error log
object for the class instance. These functions are:
• APT_Operator::errorLog()
• APT_Partitioner::errorLog()
• APT_Collector::errorLog()
Upon return from the override of any pure virtual function, Orchestrate checks
the object’s error log. If the log contains any error, warning, or informational
messages, Orchestrate writes them to standard error on the workstation that
invoked the application. See the Orchestrate 7.0 User Guide for information on
controlling the output format of a message.
Note that Orchestrate performs no other action other than to display the contents
of the error log. If the function override detects an error condition, it should
return APT_StatusFailed as well as writing to the error log. Depending on your
application, when a warning occurs, you can return APT_StatusFailed or
APT_StatusOk.

The Error Log Compared to the Assertion

Failures Facility
In addition to the error log facility, Orchestrate also provides an assertion
mechanism. Assertions are used to immediately signal an error condition that is
too severe for further processing to continue. For example, the error condition
may be the result of falling through all arms of a case statement or a violation of a
consistency check.
An assertion failure causes an application to print a message to standard error on
the node that invoked the application, then abort execution of the application. The
message contains the file name and line number of the assertion failure.
In comparison, writing to the error log causes Orchestrate to take no action other
than to print the accumulated error and warning information upon return from
the function. It is only by a function override returning APT_StatusFailed that
Orchestrate will abort an application.

Orchestrate 7.0 Developer Guide 14 3

Chapter 14 Using the Error Log Using the Error Log

Many function overrides combine both assertions and the error log. See the
Orchestrate 7.0 User Guide for more information.

Using the Error Log

Error Log Structure
An APT_ErrorLog object consists of four buffers: an accumulator, an error log, a
warning log, and an information log. The structure of an APT_ErrorLog object is
shown below:

getError()
resetError()
error log
logError()

getWarning()
operator* resetWarning()
accumulator warning log

logWarning()
getInfo()
logInfo() resetInfo()
information log

APT_ErrorLog

As you can see in this figure, you write a string containing error, warning, or
other information to the accumulator using the operator* function. The operator*
function allows you to use the following syntax to create an error log and write to
the accumulator:
APT_ErrorLog eLog(APT_userErrorSourceModule);
*eLog << "a string";
// Use the appropriate function to copy the accumulator to a log
// buffer

In this example, APT_userErrorSourceModule defines the module identifier for

the error log. See the next section for information about the module identifier.
After writing to the accumulator, you call logError(), logWarning(), or logInfo()
to copy the information from the accumulator to the appropriate log buffer.
Calling these functions appends the information in the accumulator to the current
contents of the appropriate buffer, then clears the accumulator.
In order to access the information in the error log buffer, you use getError(). This
function returns a string containing the buffer contents. You can then call

14 4 Orchestrate 7.0 Developer Guide

Chapter 14 Using the Error Log Using the Error Log

resetError() to clear the buffer. You use similar functions to access the warning
and informational log buffers.
You can also use the member function APT_ErrorLog::dump() to display the
information stored in an error log. This function writes all the information
contained in an APT_ErrorLog to standard error on the workstation that invoked
the application, then resets all the log buffers to the empty state.
One use of dump() is to periodically purge an APT_ErrorLog that may contain a
large number of messages. For example, you may have an error or warning
caused by each record processed by an operator. Because an operator can process
a huge number of records, the amount of memory consumed by an
APT_ErrorLog object can become correspondingly large. Using dump(), you can
purge the APT_ErrorLog object to prevent the object from overflowing memory.

Defining Message Identifiers

The error log contains error, warning, and informational messages that are
relayed to users. In order to better classify messages than simply by the categories
error, warning, and information, Orchestrate allows you to define individual
identifiers for each message written to the log. This will allow you to better
catalog and track your messages.
A message identifier has two parts: the module identifier and the message index
number. The module identifier defines the functional component of Orchestrate
that issued the message. The message index is a unique value for each message
within each module identifier.
Note that you do not have to construct an APT_ErrorLog object for a derived
operator, partitioner, or collector. Orchestrate generates APT_ErrorLog objects for
you, with the module identifier APT_userErrorSourceModule, when it
instantiates an object from a derived class. You will only need to specify a module
identifier if you create your own APT_ErrorLog objects.
The list of available module identifiers is extensible. Contact Ascential Software if
you require your own unique module identifier.
The message index number is a unique value for each error, warning, or
informational message you write to an error log for a given module identifier.
Typically, you define a base value for your message index, then modify this value
for every unique message written to the log.
For example, the following code defines the base value for all messages written to
the log:
#define MESSAGE_ID_BASE 0

When writing an error message to the error log, you would then specify a
message index value as shown below:

Orchestrate 7.0 Developer Guide 14 5

Chapter 14 Using the Error Log Using the Error Log

APT_ErrorLog eLog(APT_userErrorSourceModule);// create an error log

*eLog << "a string"; // write a message to error log
eLog.logError(MESSAGE_ID_BASE + 1); // log the error message and index

Subsequent messages written to the log would have an index of MESSAGE_ID_BASE

+ 2, MESSAGE_ID_BASE + 3, etc. All Orchestrate messages have a unique index
within each module identifier. Creating a unique index for each of your error
messages will allow you to catalog and track your messages.

Using the Error Log in a Derived Operator

This section contains an example using the error log in an operator class derived
from APT_Operator. In this example, the ExampleOperator::describeOperator()
override makes sure that the user has specified at least one key field. If not,
ExampleOperator::describeOperator() write a message to the error log, and
returns APT_StatusFailed.
Shown below in Table 40 is the class declaration for this example. Comments
follow the code.

Table 40 Using the Error Log in a Derived Operator

Code

#include <apt_framework/orchestrate.h>

class ExampleOperator : public APT_Operator

{
APT_DECLARE_RTTI(ExampleOperator);
APT_DECLARE_PERSISTENT(ExampleOperator);

public:
ExampleOperator();
8 void setKey(const APT_String& name);
void setStable();

protected:
virtual APT_Status describeOperator();
virtual APT_Status runLocally();
virtual APT_Status initializeFromArgs_(const APT_PropertyList &args,
APT_Operator::InitializeContext context);

private:
15 int numkeys;
// other data members

. . .

14 6 Orchestrate 7.0 Developer Guide

Chapter 14 Using the Error Log Using the Error Log

Table 40 Using the Error Log in a Derived Operator (continued)

17 #define MESSAGE_ID_BASE 0

APT_Status ExampleOperator::describeOperator()
{
setKind(APT_Operator::eParallel);
setInputDataSets(1);
setOutputDataSets(1);
setInputInterfaceSchema("record (iField:int32; sField:string)”, 0);
setOutputInterfaceSchema(“record (iField:int32; sField:string)”, 0);

25 if (numKeys == 0)
{
26 APT_ErrorLog& eLog = errorLog();
27 *eLog << “no keys specified for operator.”;
28 eLog.logError(MESSAGE_ID_BASE + 1);
29 return APT_StatusFailed;
}

Return APT_StatusOK;
}

Comments

8 Use the member function, setKey(), to set a key field.

15 The variable numKeys is a private variable containing the number of key fields
specified using setKey().

25 If the number of keys is 0, the user has not specified a key field, and an error
condition exists.

26 Use APT_Operator::errorLog() to access the APT_ErrorLog object for the operator

instance. This function returns a reference to the APT_ErrorLog object.
Alternatively, you could use the following statement to replace lines 26 and 273:
*errorLog() << "No keys specified for operator.";

APT_ErrorLog::operator*() returns an ostream to which messages can be written.

27 Use APT_ErrorLog::operator*() to write an error message to the accumulator.

Orchestrate 7.0 Developer Guide 14 7

Chapter 14 Using the Error Log Using the Error Log

Table 40 Using the Error Log in a Derived Operator (continued)

28 Use APT_ErrorLog::logError() to copy the contents of the accumulator to the error
log buffer, including the message index.

29 Return APT_StatusFailed from the override. Orchestrate checks the

APT_ErrorLog object for the class upon a return and displays any contained
warning or error information.
In addition, because the function returned APT_StatusFailed, Orchestrate
terminates the application.

Note that you do not have to return from the function after detecting a single
error or warning. You may want to execute the entire function, logging multiple
error and/or warnings before returning. In this case, you can define a flag to
signal that an error occurred and check that flag before returning from the
function.

14 8 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

15
Advanced Features
Presents the functionality of several advanced features.

Generic Functions, Accessors, and Dynamic Schema 15 1

Unit-of-Work Batch Processing 15 9
Defining Units-of-Work 15 10
Resetting Operator State 15 11
Per-Partition Processing 15 11
Combinable Operators 15 11
When to Use Combinable Operators 15 11
Advantages 15 12
Disadvantages 15 12
Using Combinable Operators 15 12
Methods Used by Combinable Operators 15 13
Virtual Methods 15 13
Non-Virtual Methods 15 14
The Combinable Controller 15 17
Limitations and Risks 15 17
Data Set Buffering 15 18

Generic Functions, Accessors, and Dynamic

Schema
Two code examples, representing the files types.C and types.h, are used to
illustrate the functionality of these three advanced features. The files implement
an operator which displays information about incoming data types.

15 1
Chapter 15 Advanced Features Generic Functions, Accessors, and Dynamic Schema

To implement generic functions, you define a two-level class hierarchy directly

below Orchestrate’s abstract base class, APT_GenericFunction. A first-level class
defines a generic-function interface consisting of one or more pure virtual
member functions; its derived second-level classes provide the concrete
implementations of the generic-function interface.
$APT_ORCHHOME/incude/apt_framework/type/function.h. is the header file for
APT_GenericFunction.
Orchestrate supplies several pre-defined first-level classes. APT_GFEquality,
APT_GFComparison, and APT_GFPrint are examples. You can extend these
classes or you can define your own two-level generic-function class hierarchy
using the code in this section as an example. The example code defines this class
hierarchy:

APT_GenericFunction

TypeOpGF

TypeOpStringGF TypeOpInt32GF

The code also demonstrates how generic accessors can be safely cast into strongly
typed accessors by casting an APT_InputAccessorBase pointer to both a string
pointer and an int32 accessor. Using Orchestrate’s default conversion mechanism,
the code also converts int8 field values to int32 values. In addition, the code
demonstrates how to build a schema variable field and add it to the schema so
that entire records can be transferred without change from input to output.
The following table shows the code of types.C. The code is followed by comments
which are keyed to code line numbers. Table 42 follows. It contains the type.h
header file which defines the TypesOp sample operator. The comments for Table
42 are included directly in the code.

Table 41 types.C

Code

1 #include “types.h”

2 class TypeOpGF : public APT_GenericFunction

{
APT_DECLARE_RTTI(TypeOpGF);
APT_DECLARE_ABSTRACT_PERSISTENT(TypeOpGF);

15 2 Orchestrate 7.0 Developer Guide

Chapter 15 Advanced Features Generic Functions, Accessors, and Dynamic Schema

Table 41 types.C
public:
7 TypeOpGF(char *type) : APT_GenericFunction(type, “typeop”, ““) {};

virtual APT_string display(APT_InputAccessorBase *)=0;

};

APT_IMPLEMENT_RTTI_ONEBASE(TypeOpGF, APT_GenericFunction);
APT_IMPLEMENT_ABSTRACT_PERSISTENT(TypeOpGF);

void TypeOpGF::serialize(APT_Archive & ar, APT_UInt8 version)

{
// TypeOPGF has no internal state to serialize
}

15 class TypeOpStringGF : public TypeOpGF

{
APT_DECLARE_RTTI(TypeOpStringGF);
APT_DECLARE_PERSISTENT(TypeOpStringGF);

public:
TypeOpStringGF() : APT TypeOpGF(“string”)
{}

virtual APT_GenericFunction* clone() const

{
return new TypeOpStringGF(*this);
}

APT_String display(APT_InputAccessorBase *inAcc)

{
return “This is a string; its value is: " +
**(APT_InputAccessorToString *) inAcc;}
}
30 };

void TypeOpStringGF::serialize(APT_Archive & ar, APT_UInt8 version)

{
// TypeOPStringGF has no internal state to serialize.
}

APT_IMPLEMENT_RTTI_ONEBASE(TypeOpStringGF, TypeOpGF);
APT_IMPLEMENT_PERSISTENT(TypeOpStringGF);

36 class TypeOpInt32GF : public TypeOpGF

{

Orchestrate 7.0 Developer Guide 15 3

Chapter 15 Advanced Features Generic Functions, Accessors, and Dynamic Schema

Table 41 types.C
APT_DECLARE_RTTI(TypeOpInt32GF);
APT_DECLARE_PERSISTENT(TypeOpInt32GF);

public:
TypeOpInt32GF() : TypeOpGF(“int32”)
{}

virtual APT_GenericFunction* clone() const

{
return new TypeOpInt32GF(*this);
}

APT_String display(APT_InputAccessorBase *inAcc)

{
char tmp[25];
sprintf(tmp, "%d", **(APT_InputAccessorToString *)inAcc);
return APT_String("This is an int32; its value is: ") +tmp;
}
53 };

void TypeOpInt32GF::serialize(APT_Archive & ar, APT_UInt8 version)

{}

APT_IMPLEMENT_RTTI_ONEBASE(TypeOpInt32GF, TypeOpGF);
APT_IMPLEMENT_PERSISTENT(TypeOpInt32GF);

int registerGFs()
{
APT_GenericFunction *tmp = new TypeOpStringGF();
APT_GenericFunctionRegistry::get().addGenericFunction(tmp);
tmp = new TypeOpInt32GF();
APT_GenericFunctionRegistry::get().addGenericFunction(tmp);
return 0;
}

static int sTypeOp = registerGFs();

67 APT_IMPLEMENT_RTTI_ONEBASE(TypesOp, APT_Operator);
APT_IMPLEMENT_PERSISTENT(TypesOp);
69 APT_REGISTER_OPERATOR(TypesOp, types);

70 define ARG DESC “{}”

APT_DEFINE_OSH_NAME(TypesOp, fieldtypes, ARG_DESC);

72 TypesOp::TypesOp()
{}

#define isNumber(PROP) ( PROP.kind() == APT_Property::eDFloat )

15 4 Orchestrate 7.0 Developer Guide

Chapter 15 Advanced Features Generic Functions, Accessors, and Dynamic Schema

Table 41 types.C
#define getDFloat(PROP) ( PROP.valueDFloat() )
#define getInt32(PROP) ( APT Int32) PROP.valueDFloat() )
#define isInt32(PROP) ( getDFloat (PROP) == getInt32(PROP) )

78 APT_Status TypesOp::initializeFromArgs_(const APT_PropertyList &args,

APT_Operator::InitializeContext
context)
{
APT_Status status=Apt_StatusOk;
return status;
}

83 void TypesOp::serialize(APT_Archive& archive, APT_UInt8)

{}

85 APT_Status TypesOp::describeOperator()
{
setInputDataSets(1);
setOutputDataSets(1);

APT_Schema schema=viewAdaptedSchema(0);

for (int i=0; i < schema.numFields(); i++)

{
APT_SchemaField &field=schema.field(i);
const APT_FieldTypeDescriptor *fd = field.typeDescriptor();

94 if (APT_PTR_CAST(APT_Int8Descriptor, fd))
{
APT_FieldTypeDescriptor *int32fd=APT_FieldTypeRegistry::get().
lookupBySchemaTypeName(“int32”);
field.setTypeDescriptor(int32fd);
}
100 }

APT_SchemaField schemaVar;
schemaVar.setIdentifier(“in”);
schemaVar.setKind(APT_SchemaField::eSchemaVariable);
schema.addField(schemaVar);

setInputInterfaceSchema(schema, 0);
setOutputInterfaceSchema(“record (out:*)”, 0);
declareTransfer(“in”, “out”, 0, 0);

setCheckpointStateHandling(eNoState);
return APT_StatusOk;
}

Orchestrate 7.0 Developer Guide 15 5

Chapter 15 Advanced Features Generic Functions, Accessors, and Dynamic Schema

Table 41 types.C

APT_Status TypesOp::runLocally()
{
APT_Status status = APT_StatusOk;

APT_Schema schema = inputInterfaceSchema(0);

int count=0;
for(; count < schema.numFields();)
{
118 if (schema.field(count).kind() != APT_SchemaField::eValue)
schema.removeField(count);
else
count++;
}

APT_InputAccessorBase *accessors = new APT_InputAccessorBase[count];

TypeOpGF **gfs = new TypeOpGF *[count];

for (int q=0; q < count; q++)

{
gfs[q]=APT_PTR_CAST(TypeOpGF,
APT_GenericFunctionRegistry::get().lookupDefault(schema.field(q).
typeDescriptor()->schemaTypeName(), “typeop”));
130 }

131 APT_InputCursor inCur;

setupInputCursor(&inCur, 0);

for (int p=0; p < schema.numFields(); p++)

inCur.setupAccessor(schema.field(p).path(), &accessors[p]);

APT_OutputCursor outCur;
136 setupOutputCursor(&outCur, 0);

137 while (inCur.getRecord() && status == APT_StatusOk)

{
for (int z=0; z < schema.numFields(); z++)
{
if (gfs[z])
{
*errorLog() << gfs[z] ->display(&accessors[z]);
errorLog().logInfo(1);
}
}

transfer(0);

15 6 Orchestrate 7.0 Developer Guide

Chapter 15 Advanced Features Generic Functions, Accessors, and Dynamic Schema

Table 41 types.C
outCur.putRecord();
149 }

delete[] gfs;
delete[] accessors;
return status
}

Comments

1 Include the header file, types.h. It defines the TypesOP operator which is directly
derived from APT_Operator.

2 Create a first-level generic-function class, TypeOpGF, to provide a public interface

for the two classes generic-function classes that derive from TypeOpGF.

7 The three arguments to APT_GenericFunction are:

type: a schema type name which identifies the schema type
"typeop": an interface name which uniquely identifies the generic function
interface among all the interfaces of the same schema type
"“: You can supply an implementation name or the empty string for the third
argument. If you supply a name, the implementation must be explicitly
selected by name. If you supply the empty string, the generic interface is the
default implementation for its schema type and interface name.
The second and third arguments are used together to determine the generic
function to be used. See the call to lookupDefault() called from the runLocally
method and the call to lookupBySchemaTypeName() in the describeOperator
method for TypesOp.

15-30 & These code lines define the TypeOpStringGF and TypeOpInt32GF classes that
36-53 provide string and int32 implementations of generic functions. Except for data
type, the two implementations are very similar.
Overriding clone() is an APT_GenericFunction base-class requirement.
The display() function displays the data type and value. You can call this type of
function to perform many kinds of operations with any number of arguments. For
example: . . .
The display() function in this case takes a generic accessor base pointer so that the
return value can be safely cast to any Orchestrate data type pointer.

Orchestrate 7.0 Developer Guide 15 7

Chapter 15 Advanced Features Generic Functions, Accessors, and Dynamic Schema

Table 41 types.C
67-69 The APT_IMPLEMENT_RTTI_ONEBASE macro supports run-time type
information. The APT_IMPLEMENT_PERSISTENT macro defines Orchestrate’s
persistence mechanism, and APT_REGISTER_OPERATOR relates an operator to
its osh invocation.

70 The argument description string for TypesOp is empty because the operator takes
no arguments.

72 TypesOp::TypesOp() is the constructor for the operator.

78 Because the operator has no arguments, its osh initialization function,

initializeFromArgs_ simply returns APT_StatusOK.

83 The serialize method is part of the Orchestrate persistence mechanism. Before the
operator is parallelized, this method is called to archive the values of its member
variables; and it is called again after parallelization, to restore those variables from
the archived values in each parallel copy of the operator. In this case, there are no
members to serialize.

85 Just before an operator is parallelized, call the describeOperator() method to set

important operator properties.

94-100 A simple assignment is used to convert int8 types into int32 types because
Orchestrate supplies a default conversion between these types. When there is no
default conversion, an adapter must be used to perform the conversion.

118-130 In the parallel execution method runlocally(), schema variables are removed from
a copy of the schema in order to exclude them from having accessors assigned to
them. This method also skips over subrecs and tagged fields.The runlocally()
method then determines how many accessors are needed, and declares pointers to
generic functions to be used with the accessors.

131-136 The method also defines a cursor for the input data set, sets up accessors to their
corresponding fields, defines a cursor for the output data set.

137-149 The while loop iterates over the available input records, calling the generic
functions. The loop could also have used APT_InputAccessorBase::type() to
return an enum on which to run switch(); however, in many cases that technique is
not as efficient as using generic functions, particularly when there are a lot of fields
in a record. Since a generic function has not been created for every datatype, the
code guards against generic functions that have not been allocated.

15 8 Orchestrate 7.0 Developer Guide

Chapter 15 Advanced Features Unit-of-Work Batch Processing

Table 42 types.h

#include <apt_framework/orchestrate.h>

class TypesOp : public APT_Operator

{
APT_DECLARE_PERSISTENT(typesOp);
APT_DECLARE_RTTI(TypesOp);

public:
// constructor
TypesOp();

// C++ initialization methods for this operator which are

// called from the initializeFromArgs_ method.

protected:

// osh initialization function which makes the operator "osh-aware".

virtual APT_Status initializeFromArgs_(
const APT_PropertyList &args,
APT_Operator::InitializeContext context);

// pre-parallel initialization
virtual APT_Status describeOperator();

// parallel execution method

virtual APT_Status runLocally();

// There are no member variables.

};

Unit-of-Work Batch Processing

Using the unit-of-work functionality, you can perform a minimal set of
transaction-like processes in Orchestrate. The functionality allows you to put
records into distinct groups, called units-of-work, which flow through
Orchestrate for batch-processing by operators performing aggregations, sorts,
transformations, and so on.
Some typical transactional functionality is not currently supported such as
rollbacks when unit-of-work failures occur, or the ability to restart reading from a
particular persistent dataset entry. Also, unit-of-work processing is currently not
applicable to work flows that require parallel-to-sequential partition collection.

Orchestrate 7.0 Developer Guide 15 9

Chapter 15 Advanced Features Unit-of-Work Batch Processing

When an operator encounters an end-of-unit marker, getRecord() returns false,

and each set of records is appended as a batch to the output dataset(s) or database
table(s).
Units-of-work provide functionality that is similar to repeatedly invoking osh,
possibly with different arguments to read different sets of input data. However,
units-of-work avoid the overhead of restarting a job.

Defining Units-of-Work
The producing operator is responsible for dividing its records into units-of-work
and for determining where to pick up on subsequent incoming units-of-work.
Any operator or dataset read which does not have specific unit-of-work semantics
has its entire output dataset treated as part of every unit-of-work for a step.
In the runLocally() or doFinalProcessing() method of your operator, your calls to
the markEndOfUnit() method on output cursors divides the records into units-
of-work.
The header file for this method is:
$APT_ORCHHOME/include/apt_framework/cursor.h

The syntax for the method is:

void APT_OutputCursor::markEndOfUnit(
APT_String a="", APT_String b="",
APT_String c="", APT_String d="");

The four arguments are intended for application-developer debugging.

In the following example, records are sorted by the altitude field. When the value
of this field is over 25000, an end-of-unit marker is attached to the record at
output.
APT_Status UOWOperator::runLocally()
{
// set up input and output data set cursors

// define an accessor to the altitude field

APT_InputAccessorToInt32 altitudeInAcc(“altitude”, &inCursor);

// set a unit-of-work marker on the current output cursor when

// altitude is over 25000.
while (inCursor.getRecord())
if *altitudeInAcc > 25000
outCursor->markEndOfUnit(“FlightTechnologies”, “altitude”,
“UOWOperator”, “runLocally”);
...
}

15 10 Orchestrate 7.0 Developer Guide

Chapter 15 Advanced Features Combinable Operators

An operator that calls the markEndOfUnit() function on any output cursor does
not propagate any further incoming end-of-unit markers from its input datasets.
Instead, the operator may progatate its own end-of-unit markers.

Resetting Operator State

End-of-unit markers cause downstream operators to flush themselves and reset to
an appropriate state. You use the resetForNextUnit() method to specify how the
downstream operator should be reset to a known state where it can receive more
input data.
The resetForNextUnit() method is called automtically by Orchestrate after
runlocally() returns and any input data set has received an end-of-unit marker.
The method is useful for retaining internal state, such as a lookup table or
database connection, that requires time to recreate between units-of-work. If you
do not overload resetForNextUnit(), Orchestrate provides a default
implementation which resets the operator itself.
The header file for this method is:
$APT_ORCHHOME/include/apt_framework/operator.h

Its syntax is:

virtual APT_Status APT_Operator::resetForNextUnit();

Per-Partition Processing
The postFinalRunLocally() function is invoked for each dataset partition after
runLocally() is called. When unit-of-work processing is done, the runLocally()
function may be called multiple times. In this case, the postFinalRunLocally()
function is called after the last invocation of runLocally().

Combinable Operators
A combinable operator is an operator that is managed by the combinable operator
controller. A combinable operator processes an input record and then returns
control to the controller. This frees the Orchestrate framework from waiting for
the operator to consume all of its input. All Orchestrate combinable operators are
based on the abstract base class APT_CombinableOperator.

When to Use Combinable Operators

When you use combinable operators, you reduce the total number of processes in
a step by combining several operators into one. This allows performance gains in
certain situations. To decide whether to use combinable operators, you must
weigh the advantages and the disadvantages of these operators.

Orchestrate 7.0 Developer Guide 15 11

Chapter 15 Advanced Features Combinable Operators

Advantages
Combinable operators can substantially improve performance for certain kinds of
operations by reducing the record transit time between operators. They do this by
eliminating the time that non-combinable operators need to pack records into a
buffer as the data passes between operators.
In addition, combinable operators allow you to process records whose size
exceeds the 32 kilobyte limit imposed by Orchestrate. When data flows between
two non-combinable operators, it is stored in a buffer which limits record size to
32 kilobytes. With combinable operators there is no buffer limit, so record size is
essentially unlimited until the combinable operator outputs to a non-combinable
operator or repartitions.

Disadvantages
Although combinable operators can confer substantial performance advantages
to data flows, combinable operators expose the complexity of the Orchestrate
internal API and as such their use comes with certain risks. These risks may be
minimized, however, by following the guidelines in this chapter. Also,
combinable operators are not always an appropriate choice. For example, using
combinable operators reduces pipeline parallelism and so could actually slow
down the data flow. See “Limitations and Risks” on page 17.

Using Combinable Operators

Warning Before attempting to implement combinable operators, carefully read this

chapter. Improper use of combinable operators can result in difficult-to-
diagnose malfunctions.
Also read:
• The header file $APT_ORCHHOME/include/apt_framework/operator.h.
• the example showing how to use the APT_CombinableOperator class in the
directory $APT_ORCHHOME/examples/Custom_Ops/OrchAPI/Combinable. Look at
the short README file and at the simple and well-documented example,
twiddle.C.

Another example shows how to write a combinable operator that can output
multiple records for each input record, making use of the
requestWriteOutputRecord() and writeOutputRecord() methods which are
described later in this chapter. Fully documented source code is in the file
$APT_ORCHHOME/examples/Custom_Ops/OrchAPI/ComplexCombinable/twiddle.C.

15 12 Orchestrate 7.0 Developer Guide

Chapter 15 Advanced Features Combinable Operators

Note The source code files for this example and the first example have the same
filename, twiddle.C. Make sure you are in the proper directory when looking for
sample code.

When you implement a combinable operator, the operator must:

1 Perform initialization to prepare the operator to process input and output.
2 Contain a method to handle the processing of each record received.
3 Output no more than one record for every record input. If multiple records
need to be output for each input record, the program must request a callback
for each output record required.

Methods Used by Combinable Operators

The following two sections describe methods you will find useful in writing
combinable operators. Within each section, the methods are listed alphabetically
by operator name.

Virtual Methods
A virtual method is a method which you must implement. The virtual methods
listed are called by the framework.
• APT_Status doFinalProcessing()

This method is called once per operator instance. If this method is only
outputting one record, use transferAndPutRecord(). If there are multiple
outputs for a operator instance, use requestWriteOutputRecord() instead,
which returns control to the framework to process the outputs.
• APT_Status doInitialProcessing()

This method lets you generate output records before doing input processing.
This method is called once per operator instance. If this method is only
outputting one record, use transferAndPutRecord(). If there are multiple
outputs for a operator instance, use requestWriteOutputRecord() instead,
which returns control to the framework to process the outputs. For an
example of how this method is used, see twiddle.C, lines 351 - 389.
• void outputAbandoned(int outputDS)

This method sends a notification to the combinable operator that the output is
no longer needed. This method is called by the framework when the operator
following this operator has received all the input it needs.
• void processEOF(int inputDS)

You use this method for processing that needs to be done after the last record
for the input data set has been received. The framework calls this method once
for each input.

Orchestrate 7.0 Developer Guide 15 13

Chapter 15 Advanced Features Combinable Operators

• void processInputRecord(int inputDS)

You use this method to apply processing on a per record basis. Use no more
than one putRecord() or transferAndPutRecord() per output data set for each
input record. Call requestWriteOutputRecord() for each additional record
output in the method. For an example of how this is used, see twiddle.C, line
391 to the end of the file.
• APT_Status writeOutputRecord()

The framework calls this method in response to operator calls to

requestWriteOutputRecord() when the controller is ready for the next output
record. You can output one record for each call to this function. You can call
requestWriteOutputRecord() again if necessary.

Non-Virtual Methods
A non-virtual method is one that you do not need to implement. The combinable
operator that you write can call the following non-virtual methods.
• void abandonInput(int inputDS)

This method notifies Orchestrate to stop sending records from inputDS to the
operator. After this function is called, atEOF() for the input will return true
and processInputRecord() cannot be called with this input as the argument. If
atEOF(inputDS) is true, this function does nothing. Conceptually,
abandonInput() is called automatically for all inputs when the operator
terminates.
Only call this method if inputDS is less than inputDatasets() and
inputConsumptionPattern() equals eSpecificInput.
• int activeInput()

Returns the number of the currently active input. This value remains stable
throughout the entire dynamic scope of processInputRecord() or
processEOF(). Calls to setActiveInput() or advanceToNextInput() do not
affect the value of activeInput().
• void clearCombiningOverrideFlag()

Sets the combiningOverrideFlag() flag for the operator to false and sets the
hasCombiningOverrideFlag() flag to true.

Only call this method if remainingInputs() is greater than 0.

• bool combiningOverrideFlag()

This function returns the combining override flag for the operator. It returns
true if setCombiningOverrideFlag() was called and false if
clearCombiningOverrideFlag() was called. Do not call this function unless
hasCombiningOverrideFlag() is true.
• bool hasCombiningOverrideFlag()

15 14 Orchestrate 7.0 Developer Guide

Chapter 15 Advanced Features Combinable Operators

This function indicates if an operator's combining override flag value has been
set. If this flag is set to true, operator combining will be prevented and
combinable operators will be treated as ordinary operators. See also the
related functions combiningOverrideFlag(), setCombiningOverrideFlag(),
and clearCombiningOverrideFlag().
• APT_InputAccessorInterface* inputAccessorInterface(int inputDS)

This method provides access to the input accessor interface associated with
each input defined for this operator. It is to be used for setting up input
accessors. Call this method in doInitialProcessing() instead of exposing
input cursors.
To call this method, inputDS must be nonnegative and less than
inputDataSets().

For an example, see the call to this method from doInitialProcessing() in

twiddle.C, lines 375–376.
• InputConsumptionPattern inputConsumptionPattern()

This method returns the input consumption pattern set in describeOperator()

by a call to setInputConsumptionPattern(). You can call this any time after the
framework calls describeOperator().
You can only call setInputConsumptionPattern() from within
describeOperator().

setInputConsumptionPattern() can take three values:

– eSpecificInput

This value specifies that the operator exercises direct control over the
consumption pattern by means of the setActiveInput() or
advanceToNextInput() functions. This is the default.
– eBalancedInput

This value requests the framework to drive the consumption pattern in a

balanced manner, consuming one record from each input not at the end of
file marker in a circular fashion. The operator must not call setActiveInput
or advanceToNextInput.
– eAnyInput

The operator is indifferent to the order of record consumption; the

framework can direct the operator to process records from any input. The
operator must not call setActiveInput or advanceToNextInput.
• APT_OutputCursor* outputCursor(int outputDS)

This method provides access to the output cursor associated with each output
defined for this operator. Use it to set up output accessors, and to call
putRecord().

Orchestrate 7.0 Developer Guide 15 15

Chapter 15 Advanced Features Combinable Operators

For performance reasons, it is better to call this function once in

doInitialProcessing() and then store the returned pointer for frequent
access, for example in processInputRecord().
To use this method, outputDS must be nonnegative and less than
outputDataSets().

For an example of how this is used, see the call to this method from
doInitialProcessing() in twiddle.C, lines 385–386.
• int remainingOutputs()

Returns the number of outputs this operator has not counting abandoned
outputs.
• void requestWriteOutputRecord()

Use this method when your combinable operator method needs to output
multiple records for a single input record.
Within the functions doInitialProcessing(), processInputRecord(),
processEOF(), doFinalProcessing(), and writeOutputRecord(), at most one
putRecord() or transferAndPutRecord() operation can be performed per
output port.
If your operator requires additional putRecord() operations, it must call
requestWriteOutputRecord() to schedule a call to writeOutputRecord(). The
requested call to writeOutputRecord() will take place before any other calls
back into this operator.
Multiple calls to this method within a single activation of
doInitialProcessing(), processInputRecord(), processEOF(),
doFinalProcessing(), and writeOutputRecord() have the same effect as a
single call. To write out multiple records, call requestWriteOutputRecord()
from writeOutputRecord() after the previous record has been output.
• void setCombiningOverrideFlag()

Sets to true both the combiningOverrideFlag() flag and the

hasCombiningOverrideFlag() flag for the operator.

Only call this method if remainingInputs() is greater than 0.

• void terminateOperator(APT_Status s)

This method sets the termination status of the operator as specified and
terminates as soon as the current function returns. If all input has not been
consumed, a warning is issued and the remaining input is consumed.
This method must be called only from within the dynamic scope of
doInitialProcessing(), processInputRecord(), processEOF(), or
writeOutputRecord(). It may not be called during or after
doFinalProcessing().

15 16 Orchestrate 7.0 Developer Guide

Chapter 15 Advanced Features Combinable Operators

The Combinable Controller

The combinable controller links the output records of combinable operators to the
inputs of other operators based on the flow specified by the step. It is called by the
Framework which provides it with the combinable operators to be executed. One
combinable operator controller runs per group of combinable operators per node.
When the combinable controller processes records:
1 The first operator activates and outputs a record.
2 In order, subsequent operators activate, take the previous operator output as
input, and output a record.
3 The last operator activates and takes the previous operator output.
4 Starting with the last operator, and proceeding in reverse order, the
combinable controller marks the output of the previous operator as
consumed. If operators have called requestWriteOutputRecord() to handle
multiple write requests, these requests are processed at this time.

Limitations and Risks

• Operators that are combined have to run on the same nodes with the same
degree of parallelism. This makes them unsuitable for some applications.
• Operators that are combined cannot form a multi-process pipeline, which can
make combinable operators slower than non-combinable operators in cases
where pipeline parallelism confers a performance advantage.
• It is the responsibility of the programmer to make sure no records are lost by
the overwriting of output from a previous write, and that the input is not
reused.
• If there is more than one output per input, the other outputs must be queued
for later use using requestWriteOutputRecord().
• A combinable operator should not use any "static" or global variables, because
if two such operators are combined into the same process, they will share the
same globals.
• When writing a combinable operator, do not use runLocally().
Instead of runLocally(), a combinable operator implements the function
processInputRecord() to consume input records, one by one.
• When writing a combinable operator, do not use getRecord().
A combinable operator must never call getRecord(); instead, the framework
calls getRecord() on the combinable operator's behalf and then calls
processInputRecord() for each record.

Orchestrate 7.0 Developer Guide 15 17

Chapter 15 Advanced Features Data Set Buffering

• After calling putRecord() or transferAndPutRecord(), a combinable operator

must not do anything to disturb the output record before returning control to
the framework.
The safest thing to do after calling putRecord() or transferAndPutRecord() is
to return to the framework.

Data Set Buffering

Orchestrate inserts buffering in your data-flow model when needed to prevent
deadlock, a situation which occurs when one operator is unable to read its input
because a previous operator in the step is blocked from writing to its output. A
description of Orchestrate’s buffering mechanism and how you can
programmatically control it is in Append A, "Data Set Buffering" in the Orchestrate
7.0 User Guide.

15 18 Orchestrate 7.0 Developer Guide

 Orchestrate 7.0 Developer Guide

Index
A interface 14 2
accessors member functions 14 2
advanced 15 1 APT_EXECUTION_MODE environment
accessors and cursors variable 13 5
introduction 7 2 APT_FieldConversion class 6 5
Adobe Acrobat Reader APT_FieldConversionRegistry class 6 5
in the Orchestrate install xvii APT_FieldList class 10 1
reading online documentation xvi interface 10 2
advanced features APT_HashPartitioner class
accessors 15 1 deriving a hash-by-field partitioner 8 5
data set buffering 15 18 APT_InputCursor class
dynamic schema 15 1 interface 7 3
generic functions 15 1 APT_Operator class
aggregate fields derivation
field accessors to 7 31 example with arguments 1 9
APT_Archive Class example with no arguments 1 5
interface 11 4 examples with options 1 33
APT_Collector class member functions 1 14
interface 9 6 interface 1 4
operation of 9 7 APT_OutputCursor class
APT_CompositeOperator class interface 7 3
example derivation 2 5 APT_PartioningStyle 1 23
interface 2 4 APT_Partitioner class
APT_CONFIG_FILE environment variable 13 interface 8 8
3 overriding partitionInput() 8 10
APT_DEBUG_OPERATOR environment APT_Persistent class
variable 13 6, 13 11 deriving complex persistent classes 11 8
APT_DEBUG_PARTITION environment interface 11 9
variable 13 11 member functions 11 10
APT_Decimal class APT_PM_XTERM environment variable 13 2
interface 7 13 APT_Property class
APT_DEFINE_OSH_NAME macro 5 3 encoding operator arguments 5 19
APT_DUMP_SCORE environment APT_PTR_CAST() macro 12 7
variable 13 12 APT_RangePartitioner class
APT_ErrorLog class deriving a range partitioner 8 7
example derivation 14 6 APT_RawField class

Index 1
Index

member functions 7 26 APT_InputAccessorToDecimal 7 12

APT_SEQ_VDS_DIR environment variable 13 APT_InputAccessorToRawField 7 27
7 APT_InputAccessorToString 7 22
APT_SortedMergeCollector class APT_InputCursor 7 3
deriving a sorted-merge collection APT_InputSubCursor 7 34
method 9 4 APT_MemoryArchive 11 3
example 9 4 APT_Operator 1 4
APT_StatusFailed 14 2 APT_OutputAccessorToRawField 7 27
APT_StatusOk 14 2 APT_OutputAccessorToString 7 22
APT_SubProcessOperator class APT_OutputCursor 7 3
deriving subprocess operators 3 1 APT_OutputSubCursor 7 34
interface 3 2 APT_Partitioner 8 8
APT_ViewAdapter class APT_PartitioningStyle 1 23
deriving a view adapter 8 16 APT_PartitionSortOperator 9 4
argument-list description APT_Persistent 11 9, 11 10
component elements 5 3 APT_Property 5 19
examples 5 16 APT_Rawfield 7 26
supplying 5 3 APT_SortedMergeCollector 9 4
syntax 5 5 APT_SubProcessOperator 3 2
argument-list processing 5 2–5 26 APT_Time 7 29
advantages 5 2 APT_TimeStamp 7 29
converting wrappered operators 5 22 APT_TypeInfo 12 2
error handling collection methods 9 2
argument-description syntax errors 5 choosing 9 2
20 overriding APT_Collector::selectInput() 9
command-line argument errors 5 20 10
overview 5 2 using the default method 9 3
property lists 5 19 collectors
property-list encoding 5 19 a sorted-merge example 9 4
supplying an argument-list description 5 an example with a view adapter 9 16
3 an integer-field example 9 11
usage strings 5 21 choosing a collection method 9 2
defining an interface schema 9 8
defining field accessors 9 9
C deriving 9 2
casting pointers 12 3, 12 7 overriding APT_Collector::
classes setupInputs() 9 9
APT_Archive 11 4 overriding APT_
APT_Collector 9 6 Collector::describeCollector() 9 8
APT_CompositeOperator 2 4 overriding APT_Collector::selectInput() 9
APT_CustomReportInfo 1 55 10
APT_Date 7 29 performing the collection operation 9 10
APT_Decimal 7 13 using keyless collection methods 9 3
APT_ErrorLog 14 2, 14 4 using the default method 9 3
APT_FieldConversion 6 5 using the sorted merge method 9 4
APT_FieldConversionRegistry 6 5 combinable operators
APT_FieldList 10 2 advantages 15 12
APT_FieldSelector 10 3 disadvantages 15 12
APT_FileArchive 11 3 limitations and risks 15 17

Index 2 Orchestrate 7.0 Developer Guide

Index

methods 15 13 E
the controller 15 17 environment variable
compiler for your platform xii APT_ENGLISH_MESSAGES 4 5
compiler options xii APT_MSG_FILELINE 4 5
composite operators environment variables
an example 2 5 APT_CONFIG_FILE 13 3
diagram with two suboperators 2 2 APT_DEBUG_OPERATOR 13 6, 13 11
configuration file APT_DEBUG_PARTITION 13 11
for debugging 13 3 APT_DUMP_SCORE 13 12
cursors APT_EXECUTION_MODE 13 5
accessing records 7 3 APT_PM_XTERM 13 2
cursors and accessors
APT_SEQ_VDS_DIR 13 7
introduction 7 2
DISPLAY 13 2
custom operator
error log
executing 5 25
an example 14 6
customer support services xviii, 2 xviii
compared to the asseration-failures
facility 14 3
structure 14 4
D error logs
date fields base class member functions 14 3
field accessors to 7 27 execution mode
dbx specifying 13 2
debugger options 13 5 execution of your custom operator 5 25
debugger for your platform xii external data
debugger options xii declaring for MetaStage 1 57
with dbx 13 5
debugging 13 1
in parallel execution mode 13 9 F
in sequential execution mode 13 5 field accessors
modifying the configuration file 13 3 accessing record fields 7 7
score dump 13 12 Orchestrate data type accessor classes 7 8
sequential execution mode to date fields 7 29
steps 13 7 to decimal fields 7 12
setting the execution mode 13 2 to fixed length vector fields 7 13
decimal fields to nullable fields 7 17
field accessors to 7 12
to numeric fields 7 10
defining units-of-work 15 10
to raw fields 7 26
deriving field lists 10 1
to string fields 7 21
DISPLAY environment variable 13 2 to time fields 7 29
documentation
to timestamp fields 7 29
Orchestrate titles xiv
to variable length vector fields 7 15
osh syntax conventions xvi
field lists
reading with the Adobe Acrobat
an example with an operator 10 4
Reader xvi
creating 10 2
searching for text xvii
expanded 10 3
typographic conventions xv
operator interface schema 10 4
dynamic interface schemas 1 19
record schema 10 3
dynamic schema 15 1
unexpanded 10 3

Orchestrate 7.0 Developer Guide Index 3

Index

filter operator APT_IMPLEMENT_RTTI_END() 12 5, 12

an example 1 39 6
APT_IMPLEMENT_RTTI_NOBASE() 12
5, 12 6
G APT_IMPLEMENT_RTTI_
generic functions 15 1 ONEBASE() 12 4, 12 5, 12 6
APT_PTR_CAST() 12 3, 12 7
APT_STATIC_TYPE() 12 2, 12 7
H APT_TYPE_INFO() 12 7
messaging code
hashing functions 8 15
eliminating deprecated interfaces 4 9
header files
example conversions 4 6
HTML-documented xiii
localizing 4 5
textual xiii
steps for converting 4 7
messaging environment variable
APT_ENGLISH_MESSAGES 4 5
I APT_MSG_FILELINE 4 5
interface schemas messaging macro
dynamic 1 19 APT_DECLARE_MSG_LOG 4 5
internationalizing operators 4 2 APT_MSG 4 3
APT_NLS 4 4
messaging macros
J including them in a source file 4 2
Job Monitor messages 1 55 MetaStage
customizing metadata and summary declaring external data for 1 57
response messages 1 55
generating messages asynchronously 1 57
N
National Language Support
M Unicode utilities 7 25
macro nullable fields
APT_DECLARE_MSG_LOG 4 5 field accessors to 7 17
APT_MSG 4 3 numeric fields
APT_NLS 4 4 field accessors to 7 10
macros
APT_ASSERT() 7 33
APT_DECLARE_ABSTRACT_ O
PERSISTENT 11 8 object persistence
APT_DECLARE_PERSISTENT 11 8 defining complex-persistent classes 11 8
APT_DECLARE_RTTI() 12 4, 12 5 macros for declaring and defining
APT_DEFINE_OSH_NAME 5 3 classes 11 13
APT_DYNAMIC_TYPE() 12 3, 12 6 overriding APT_Persistent::serialize() 11
APT_IMPLEMENT_ABSTRACT_ 10
PERSISTENT 11 8 serialization operators 11 4
APT_IMPLEMENT_PERSISTENT 11 8 serializing arrays 11 11
APT_IMPLEMENT_RTTI_BASE() 12 4, serializing pointers 11 11
12 5 operators
APT_IMPLEMENT_RTTI_BEGIN() 12 4, collection methods 9 2
12 5 using keyless methods 9 3

Index 4 Orchestrate 7.0 Developer Guide

Index

using sorted merge 9 4 parallel or sequential operators

using the default method 9 3 creating 1 17
creating operators that can be parallel or partitioner insertion
sequential 1 17 disabling 1 25
creating parallel operators 1 16 partitioning
creating sequential operators 1 16 choosing a partitioning method 8 3
debugging 13 1 example method definition 8 11
defining parallel and sequential hash-by-field method 8 5
operators 1 15 hashing functions 8 15
deriving introduction 8 2
choosing a collection method 9 2 obtaining the partitioner last used on
dynamic interface schemas 1 48 operator input 1 24
example 1 52 overriding APT_
using multifield accessors 1 49 Partitioner::describePartioner() 8 9
field accessors and dynamic overriding APT_
operators 1 33 Partitioner::partitionInput() 8 10
including schema variables 1 34 overriding APT_
Partitioner::setupInputs() 8 10
filter operator example 1 39
preserve-partitioning flag 8 3
performing transfers 1 37 using a view adapter 8 16
the transfer mechanism 1 35 using keyless methods 8 4
two-input operator example 1 using the APT_HashPartitioner class 8 5
43 using the class APT_Range_Partitioner 8
outline of basic requirements 1 3 7
specifying a collection method 1 25 using the default method 8 3
specifying a partitioning method 1 20 partitioning keys
specifying input and output interface setting and getting on an input data set 1
schemas 1 17 24
using cursors 1 26 partitioning style
using field accessors 1 29 obtaining it for a custom partitioner 1 24
dynamic interface schema specifying 1 23
an example 10 4 persistence
dynamic interface schema with defining complex-persistent classes 11 8
field lists 10 4 macros for declaring and defining
examples with options 1 33 classes 11 13
field lists with 10 4 overriding APT_Persistent::serialize() 11
internationalizing 4 2 10
subprocess operators 3 1 serialization operators 11 4
Orchestrate application development serializing arrays 11 11
debugging 13 1 serializing pointers 11 11
osh commands preserve-partitioning flag
syntax xvi derived operators and 8 3
program execution mode
specifying 13 2
P property list of operator arguments
parallel execution mode example 5 19
debugging 13 9 traversing 5 19
parallel operators
creating 1 16

Orchestrate 7.0 Developer Guide Index 5

Index

R S
raw fields schema variables
data type 7 27 introducing 1 19
field accessors to 7 26, 7 27 schemas
record fields specifying 1 17
accessing with field accessors 7 7 score dump 13 12
aggregate fields sequential operators
field accessors to 7 31 creating 1 16
date fields serialization
field accessors to 7 29 of arrays 11 11
decimal fields of pointers 11 11
field accessors to 7 12 serialization operators 11 4
fixed length vector fields sink operators in Orchestrate 1 59
field accessors to 7 13 sort insertion
nullable fields disabling 1 25
field accessors to 7 17 sort keys
numeric fields setting and getting for an input data set 1
field accessors to 7 10 24
raw fields setting and getting for an output data
data type 7 27 set 1 24
field accessors to 7 27 specifying 1 24
string fields source operators in Orchestrate 1 59
data type 7 22 string field
field accessors to 7 22 converting to ustring field 6 2
subrecord fields string fields
field accessors to 7 31 data type 7 22
tagged aggregate fields field accessors to 7 21, 7 22
field accessors to 7 31 string to ustring data type conversions 6 2
time fields subcursors
field accessors to 7 29 accessing subrecord fields 7 33
timestamp fields subprocess operators
field accessors to 7 29 introduction 3 1
variable length vector fields overriding APT_
field accessors to 7 15 SubProcessOperator::runSink() 3 4
records overriding APT_
accessing using cursors 7 3 SubProcessOperator::runSource() 3 3
accessing using cursors and accessors 7 2 passing a command line string 3 5
RTTI reading buffers 3 4
dynamic data types 12 2 writing records 3 3
macros subrecord fields
application macros 12 6 accessing 7 33
derivation macros 12 5 field accessors to 7 31
performing casts 12 3 suprocess operators
static data types 12 2 overriding APT_
support for derived classes 12 3 SubProcessOperator::commandline() 3
run-time data type 5
determining 12 2 syntax conventions for osh commands xvi
run-time type information
support for derived classes 12 3

Index 6 Orchestrate 7.0 Developer Guide

Index

T
tagged aggregate fields
field accessors to 7 31
time fields
field accessors to 7 27
timestamp fields
field accessors to 7 27
transaction-like processes 15 9
type conversions
default 6 2
defining custom functions 6 5
example
creating new conversions 6 9
using pre-defined functions 6 6
non-default predefined 6 2

U
Unicode utilities for National Language
Support 7 25
unit-of-work functionality 15 9
resetting operator state 15 11
unit-of-working functionality 15 10
usage strings
argument-list description derivation 5 21
ustring field
converting to string field 6 2
ustring to string data type conversions 6 2

V
vector fields
field accessors to 7 13, 7 15
view adapters
deriving from APT_ViewAdapter class 8
16

Orchestrate 7.0 Developer Guide Index 7

Index

Index 8 Orchestrate 7.0 Developer Guide