Scribd
Upload
4 views

datastage processing

The document outlines various data processing stages, each with specific purposes and functionalities. Key stages include the Transformer Stage for complex data transformations, the Filter Stage for selective data passing, and the Aggregator Stage for performing aggregate calculations. Additional stages such as Join, Merge, Lookup, and Change Capture enhance data manipulation and management capabilities.

Uploaded by

sai Charan
Copyright
© © All Rights Reserved
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
4 views

datastage processing

The document outlines various data processing stages, each with specific purposes and functionalities. Key stages include the Transformer Stage for complex data transformations, the Filter Stage for selective data passing, and the Aggregator Stage for performing aggregate calculations. Additional stages such as Join, Merge, Lookup, and Change Capture enhance data manipulation and management capabilities.

Uploaded by

sai Charan
Copyright
© © All Rights Reserved
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
You are on page 1/ 2

1.

Transformer Stage
Purpose: This is one of the most powerful and versatile stages.

Functionality: It allows for complex data transformations using custom logic. You
can write expressions to modify, convert, filter, and derive new data fields.

2. Filter Stage
Purpose: To selectively pass rows of data based on specified conditions.

Functionality: Apply conditions to filter out unwanted rows, ensuring only the
relevant data continues in the flow.

3. Aggregator Stage
Purpose: To perform aggregate calculations.

Functionality: Summarize data by calculating totals, averages, counts, and other


aggregate metrics based on specified groupings.

4. Remove Duplicates Stage


Purpose: To eliminate duplicate records.

Functionality: Identify and remove duplicate rows based on specified key columns,
ensuring data uniqueness.

5. Join Stage
Purpose: To combine data from multiple datasets.

Functionality: Perform inner, left, right, or full outer joins based on key
columns, merging related data into a single dataset.

6. Merge Stage
Purpose: To consolidate data from multiple datasets.

Functionality: Specify master and update datasets, merging records while handling
updates and inserts appropriately.

7. Lookup Stage
Purpose: To enhance data by performing lookups.

Functionality: Fetch additional data from reference datasets, using key columns to
match and retrieve the necessary information.

8. Sort Stage
Purpose: To order data.

Functionality: Sort records based on one or more key columns, with options to
handle null values and specify sort order.

9. Modify Stage
Purpose: To change metadata properties of columns.

Functionality: Modify data types, rename columns, and perform type conversions.

10. Pivot Stage


Purpose: To transform row data into columnar data or vice versa.

Functionality: Convert data structures for more flexible analysis and reporting.

11. Copy Stage


Purpose: To create duplicates of data.

Functionality: Duplicate the input data, allowing parallel processing or splitting


the data flow for different operations.

12. Change Capture Stage


Purpose: To capture changes between two datasets.

Functionality: Identify differences (inserts, updates, and deletes) between a


source dataset and a target dataset.

13. Change Apply Stage


Purpose: To apply captured changes.

Functionality: Apply the identified changes from the Change Capture stage to a
target dataset

You might also like