OBIA ETL Guide 11.1.1.8.1
OBIA ETL Guide 11.1.1.8.1
OBIA ETL Guide 11.1.1.8.1
ETL Guide
11g Release 1 (11.1.1.8.1)
E51481-01
March 2014
Explains how to set up, perform, and monitor ETL processes.
Dan Hilldale
Contributors: Oracle Business Intelligence development, product management, and quality assurance teams.
This software and related documentation are provided under a license agreement containing restrictions on
use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your
license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license,
transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse
engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is
prohibited.
The information contained herein is subject to change without notice and is not warranted to be error-free. If
you find any errors, please report them to us in writing.
If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it
on behalf of the U.S. Government, the following notice is applicable:
U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software,
any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users
are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and
agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and
adaptation of the programs, including any operating system, integrated software, any programs installed on
the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to
the programs. No other rights are granted to the U.S. Government.
This software or hardware is developed for general use in a variety of information management
applications. It is not developed or intended for use in any inherently dangerous applications, including
applications that may create a risk of personal injury. If you use this software or hardware in dangerous
applications, then you shall be responsible to take all appropriate failsafe, backup, redundancy, and other
measures to ensure its safe use. Oracle Corporation and its affiliates disclaim any liability for any damages
caused by use of this software or hardware in dangerous applications.
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of
their respective owners.
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks
are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD,
Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced
Micro Devices. UNIX is a registered trademark of The Open Group.
This software or hardware and documentation may provide access to or information on content, products,
and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly
disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle
Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your
access to or use of third-party content, products, or services.
Contents
Preface ................................................................................................................................................................. v
Audience.......................................................................................................................................................
Documentation Accessibility .....................................................................................................................
Related Documents .....................................................................................................................................
Conventions .................................................................................................................................................
v
v
v
v
vii
1 ETL Overview
Before You Begin Running ETL Processes ............................................................................................... 1-1
About ETL Architecture ............................................................................................................................. 1-1
About ETL Phases ....................................................................................................................................... 1-3
About the ODI Repository ......................................................................................................................... 1-3
About Load Plans ........................................................................................................................................ 1-4
About Changed Data Capture................................................................................................................... 1-4
About Knowledge Modules....................................................................................................................... 1-5
About Reverse Knowledge Modules........................................................................................................ 1-5
About Multi-Source Environments........................................................................................................... 1-6
About ETL Roles.......................................................................................................................................... 1-7
iv
Preface
Oracle Business Intelligence Applications is a comprehensive suite of prebuilt
solutions that deliver pervasive intelligence across an organization, empowering users
at all levels - from front line operational users to senior management - with the key
information they need to maximize effectiveness. Intuitive and role-based, these
solutions transform and integrate data from a range of enterprise sources and
corporate data warehouses into actionable insight that enables more effective actions,
decisions, and processes.
Oracle BI Applications is built on Oracle Business Intelligence Suite Enterprise Edition
(Oracle BI EE), a comprehensive set of enterprise business intelligence tools and
infrastructure, including a scalable and efficient query and analysis server, an ad-hoc
query and analysis tool, interactive dashboards, proactive intelligence and alerts, and
an enterprise reporting engine.
Audience
This document is intended for managers and implementers of Oracle BI Applications.
Documentation Accessibility
For information about Oracle's commitment to accessibility, visit the Oracle
Accessibility Program website at
http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc.
Access to Oracle Support
Oracle customers have access to electronic support through My Oracle Support. For
information, visit
http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or visit
http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are
hearing impaired.
Related Documents
See the Oracle Business Intelligence Applications documentation library at
http://docs.oracle.com/cd/E51479_01/index.htm for a list of related
Oracle Business Intelligence Applications documents.
Conventions
The following text conventions are used in this document:
v
vi
Convention
Meaning
boldface
italic
monospace
vii
viii
1
ETL Overview
1
1-1
knowledge module (IKM). In this example, step 1 issues a SQL statement on the
source that joins the GL_SET_OF_BOOKS table with the HR_ORGANIZATION_
INFORMATION table. The join is executed on the source database, and the resulting
data is staged. Then, a second join occurs at the load transform stage between the W_
DOMAIN_G table and the temporary stage table, which results in the loading of the
stage table W_INV_ORG_DS.
Note that Oracle Database is the only database type supported for the Oracle BI
Applications repository schemas and the Business Analytics Warehouse.
Figure 11 On-Premise ETL Architecture
Figure 12 shows the general load plan pattern. There are four main stages: The SDE
(source dependent extract) tasks extract data from the source dimension and fact tables
and load the data into universal dimension and fact staging tables. The SIL tasks are
common and load data from the universal staging tables into the warehouse staging
tables. Figure 12 depicts a dependency between the dimension table and the fact
table. Therefore, the SIL DIM must be executed before the SIL FACT, to resolve the
dimension key. The SIL DIM has a database sequence that generates the key, and then
the SIL FACT looks up that key when loading the fact staging table.
SDE. SDE stands for Source Dependent Extract. In the first phase, SDE tasks
extract data from the source system and stage it in staging tables. SDE tasks are
source specific.
SIL. SIL stands for Source Independent Load. Load tasks transform and port the
data from staging tables to base fact or dimension tables. SIL tasks are source
independent.
PLP. PLP stands Post Load Process. PLP tasks are only executed after the
dimension and fact tables are populated. A typical usage of a PLP task is to
transform data from a base fact table and load it into an aggregate table. PLP tasks
are source independent.
The master and work repositories must reside in the same database schema, and the
database type must be Oracle. Both the master and work repositories are set up during
the Oracle BI Applications installation process.
Note that the default ODI repository ID is 500. This ID is part of the internal ID for
every object that is created within the ODI repository. Having a repository ID greater
than or equal to 500 is critical to ensure that the objects you create do not overlap with
any current or future Oracle BI Applications objects. If you change the default ODI
repository ID, make sure the new value is greater than 500.
ETL Overview
1-3
All records from tables that are sources for dimension tables.
Records created after an "Initial Extract Date" from tables that are sources for fact
tables. The Initial Extract Date defines a cut-off so that not all records are loaded
into the data warehouse. You set the Initial Extract Date value for each data source
in the Oracle BI Applications Configuration Manager.
An ETL process can extract a record from a single table or from multiple tables. When
a record is the result of joining multiple tables, one of these tables is identified as the
base table, which defines the granularity of the record. When extracting fact records,
Oracle BI Applications only compares the "Created Date" of the base table to the Initial
Extract Date.
During an incremental load, Oracle BI Applications extracts records that have changed
or were created after a "Last Extract Date." This is done by comparing the Last Extract
Date value to a "Last Updated Date" (or LUD) type column in the source table. If the
source table does not have such a column, Oracle BI Applications extracts all records
from that table. The Last Extract Date is a value that is calculated based on the last
time data was extracted from that table less a "Prune Days" value. The Prune Days
parameter is used to extend the window of the ETL extract beyond the last time the
ETL actually ran. This is to ensure records that may have somehow been missed in a
prior ETL are picked up in the next ETL. Records can be missed in an ETL process
when a record is being updated while the ETL process is running and was not
committed until after the ETL completed.
You set the Prune Days parameter value in Oracle BI Applications Configuration
Manager. Setting a small value means the ETL will extract fewer records, thus
improving performance; however, this increases the chances that records are not
detected. Setting a large number is useful if ETL runs are infrequent, but this increases
the number of records that are extracted and updated in the data warehouse.
1-4 Oracle Business Intelligence Applications ETL Guide
Therefore, you should not set the Prune Days value to a very large number. A large
Prune Days number can also be used to trigger re-extracting records that were
previously processed but have not changed. The value for Prune Days should never be
set to 0.
As stated above, an ETL process can extract a record from a single table but more
commonly extracts records that are the result of joining multiple tables. When
extracting from multiple tables, one table is identified as the base table, which defines
the granularity of the record. When there is a change in the base table, an extract is
triggered for the record. However, there can be cases where a change occurs in a
non-base table but not in the base table itself. If a change occurs in a non-base table
and this should trigger an extract of the record, these tables are referred to as
"auxiliary" tables. Thus, when determining if a record should be extracted, Oracle BI
Applications compares not only the LUD column in the base table but the LUD
columns in all auxiliary tables. If the LUD column changed in any of these tables, the
record is extracted. If changes can occur in a table that is not relevant to triggering an
extract, this table's LUD column is not compared in the incremental filtering logic.
Reverse-engineering (RKM). Used for reading the table and other object metadata
from source databases and to import tables, columns, and indexes into a model.
For more information about RKMs, see "About Reverse Knowledge Modules".
Loading (LKM). Used for efficient extraction of data from source databases for
loading into a staging area (database-specific bulk unload utilities can be used
where available).
Integration (IKM). Used to load data into a target with different strategies, for
example, slowly changing dimensions and insert/update strategies.
Check (CKM). Used to validate and cleanse data.
Journalizing (JKM). Used to record the new and changed data within either a
single table or view or a consistent set of tables or views.
Service (SKM). Exposes data in the form of Web services.
For a detailed description of the KMs available with Oracle BI Applications, see
Appendix B, "Knowledge Modules Reference."
ETL Overview
1-5
temporary tables, and then the RKM calls the ODI API to read from these tables and
write to the ODI metadata tables of the Work repository in incremental update mode.
Note that the Oracle BI Applications ODI repository contains the relevant source data
models. Therefore, you would need to run an RKM only if you have customized tables
in the source system and want to import these changes to the ODI repository. For more
information about customizing tables and tasks, see Oracle Business Intelligence
Applications Administrator's Guide.
2.
3.
Data from both sources, PeopleSoft 9.0 and Oracle EBS 11.5.10, is being loaded into the
target table (ASTXACT_FG) for the first time, which means a full load will occur; and,
generally, when a full load occurs, the target table is truncated. In this case, the load
plans can run in parallel because the data sources are not dependent on each other;
however, the load from the second source should not truncate the target table if it
already contains data from the first source. This issue is resolved as follows: The
Initialize Target Table step truncates the target table. The Load Target Table step, in
turn, has two child parallel steps. Each of these loads the target from each source
system. Finally, the Finalize Target Table step creates any indexes and analyzes the
table. Thus, the generated load plan ensures that the table is truncated only at the
required time (in this case, only once before any source system loads it).
Note that you can have separate load plans for each source, but load plans should not
run in parallel.
Caution: Oracle BI Applications Configuration Manager and ODI
will not stop load plans from running in parallel; however, it is
recommended that you do not do so because of the following reasons:
All load plans truncate target tables upon the initial load. If you
run load plans in parallel, one load plan can truncate the data
loaded by the preceding load plan.
The mappings from SILOS onwards are common and not based
on the source system. If load plans are run in parallel, you can
have a situation in which only partial data is loaded from the
second source due to the data that was loaded from the first
source. To resolve this issue, you need to make sure that the first
load plan completes successfully before the second load plan runs.
In other words, when loading a fact table, the fact table could be
connecting with multiple dimension and lookup tables to get the
final data. When load plans are running parallel, some of the
dimension, lookup, and staging tables could also have data from a
second source. This could lead to some lookups and dimensions
not returning appropriate value for the second source, since they
have not yet been completely loaded
ETL Overview
1-7
For more information about managing security in ODI, see Oracle Fusion Middleware
Developer's Guide for Oracle Data Integrator.
2
Managing Load Plans
2
This chapter provides information about managing load plans for Oracle BI
Applications using Oracle BI Applications Configuration Manager.
The tasks for managing load plans are usually performed by either ETL developers or
ETL operators. The tasks in this section are grouped by the role that generally
performs the task, as listed below.
ETL Developer Tasks
2-1
load plan instance and a first load plan run. If a run is restarted, a new load plan
run is created under this load plan instance. Each execution attempt of the load
plan instance is preserved as a different load plan run in the log.
2.
3.
On the first page of the Define Load Plan series, specify the following information:
Field
Description
Name
Description
Field
Description
Source Instances
4.
Select the data sources from which the fact groups will be
selected.
Click Next.
The second page of the Define Load Plan series is displayed.
5.
In the Available Selections tab, select the fact groups you want to include in the
load plan definition.
Note that fact groups may belong to a hierarchy of fact groups. You can select only
the top level parent fact group and not a child fact group.
A load plan must contain at least one fact group; multiple fact groups may be
selected from one or more data sources.
6.
Click Save to save the load plan. After a load plan is saved, it is displayed in
the Load Plans master list.
Click Save and Generate Load Plan to save the load plan and immediately
generate it.
2-3
2.
In the Load Plans master list, select the load plan you want to duplicate.
3.
4.
On the first page of the Duplicate Load Plan series, specify the following
information:
Field
Description
Name
Description
Source Instances
5.
(Optional) Select the data sources from which the fact groups
will be selected.
Click Next.
The second page of the Duplicate Load Plan series is displayed.
6.
In the Available Selections tab, select the fact groups you want to include in the
load plan definition.
Note that fact groups may belong to a hierarchy of fact groups. You can select only
the top level parent fact group and not a child fact group.
A load plan must contain at least one fact group, and multiple fact groups may be
selected from one or more data sources.
7.
Click Save to save the load plan. After a load plan is saved, it is displayed in
the Load Plans master list.
Click Save and Generate Load Plan to save the load plan and immediately
generate it.
2.
In the Load Plans master list, select the load plan you want to edit.
3.
4.
Field
Description
Name
Description
Source Instances
5.
(Optional) Select the data sources from which the fact groups
will be selected.
Click Next.
The second page of the Edit Load Plan series is displayed.
6.
In the Available Selections tab, select the fact groups you want to include in the
load plan definition.
Note that fact groups may belong to a hierarchy of fact groups. You can select only
the top level parent fact group and not a child fact group.
A load plan must contain at least one fact group, and multiple fact groups may be
selected from one or more data sources.
7.
Click Save to save the load plan. After a load plan is saved, it is displayed in
the Load Plans master list.
Click Save and Generate Load Plan to save the load plan and immediately
generate it.
2.
In the Load Plans master list, select the load plan you want to generate.
3.
2-5
Description
Starting
In Progress
Succeeded
Failed
You can execute a load plan or schedule it for execution after it has been successfully
generated.
2.
In the Load Plans list, select the load plan you want to schedule.
3.
4.
5.
Field
Description
Context
The ODI context to be used when the load plan is run. Note that
Global is the only supported context.
Local Agent
The ODI local agent to be used when the load plan is run.
Log Level
Status
Active
Inactive
Field
Description
Recurrence
On Agent Startup
Simple
Hourly
Daily
Weekly
Yearly
Click Schedule.
You can have separate load plans for each source, but load plans should not run in
parallel
You can only execute a load plan if it was successfully generated. See "Generating
Load Plans" for instructions.
2.
In the Load Plans list, select the load plan you want to execute.
3.
4.
Field
Description
Context
The ODI context to be used when the load plan is run. Note that
Global is the only supported context.
Local Agent
The ODI local agent to be used when the load plan is run.
5.
Click OK.
The following icons show the possible execution statuses that appear in the
Execution Status column in the Load Plan master list. Click the Refresh icon to
refresh the display.
2-7
Description
Starting
In Progress
Succeeded
Failed
Not Executed
6.
(Optional) Click the Show Execution Status Details icon to display detailed
information about the status of the execution.
2.
In the Load Plans master list, select the load plan whose run you want to view.
3.
In the Load Plans toolbar, click the Show Execution Status Details icon.
The Oracle Data Integrator Console is displayed.
4.
Log into the Oracle Data Integrator Console by entering an appropriate User ID
and Password.
The Load Plan Details page is displayed.
For a description of the information displayed on this page, see "Load Plan Details
Page".
Problem with access either to source or target database due to network failure or
expired or otherwise incorrect user names and passwords.
Failure of ODI agent.
Problem with space or storage. Able to connect to source or target database but the
query fails to run due to lack of temp space, disk space, and so on. For files it
could due to inadequate space where the file needs to be placed.
Problem with data, for example incorrect data with lengths larger than the target
column can hold, or null values in Not Null columns.
After such a failure during ETL, to avoid restarting the entire load plan after a failure,
which would require inefficient re-runs of all ETL tasks, you must restart the load from
the same point in its execution once the cause of failure has been diagnosed and
resolved.
Root steps are set to 'Restart From Failure' if Serial and 'Restart from failed
Children' if Parallel.
Sub steps are set to 'Restart From Failure' if Serial and 'Restart from failed
Children' if Parallel.
2-9
The examples below highlight the implications for each type of load plan step.
Serial Load Plan Step
Serial steps are represented by a horizontal icon in the Steps hierarchy in Load Plan
Manager, and by default have a Restart setting of Restart from Failure. In a case where
the load plan fails when running such a step to load a Dimension Group with multiple
serial sub-steps loading individual dimensions, the load plan on restart would start
from the individual sub-step that failed. Any successfully completed serial sub-steps
would not be run again.
Parallel Load Plan Step
Parallel steps are represented by a vertical icon in the Steps hierarchy in Load Plan
Manager and by default have a Restart setting of Restart from Failed Children. In a
typical run, a parallel step with five parallel sub-steps under it would have all five
sub-steps executed in parallel, subject to free sessions being available. If two of those
five steps completed and then the load plan were to fail, when the load plan was
restarted, all the steps that did not complete or failed would be started again.
Scenario Step
At the lowest order in any load plan are the scenario steps. While the parent steps,
whether serial or parallel, are used to set the dependencies, the scenario steps are those
which load the tables. A scenario step, in turn, could have one or more sub-steps,
corresponding to the number of steps inside the package.
In the case of a scenario step failure during execution, consider that the scenario step
may have multiple steps, all under the same session in the Operator log but identified
with different step numbers: 0, 1, 2, and so on. In the case of restart, the scenario would
execute from the failed parent scenario step, re-running all sub-steps.
If you use the Load Plan Generator to generate a load plan, it
will automatically conform to the standard above. If you are manually
altering a generated load plan or creating a new load plan without
using Load Plan Generator, then you should ensure that you conform
to the standard above.
Note:
In ODI Operator, navigate to the Operator log, and select the last failed run for a
load plan.
2.
Double-click the load plan run, and select the Restart Option. You can also
right-click the last run in the Operator log and select Restart.
2.
Click Restart. The Restart button is displayed only when the selected load plan is
the most recent run of the load plan.
Check the maximum number of sessions set to run against the agent. In ODI
Operator, verify that the number of sessions running is equal to the maximum. If
so, then the other sessions are waiting for the running sessions to complete.
Proceed to the next step.
2.
Clean out stale sessions. Stale sessions are sessions that are incorrectly left in a
running state after an agent or repository crash. If an agent crashes or loses its
connection the repository after it has started a session, it is not be able to update
the status of the session in the repository, and such a session becomes stale. Until
the stale session is cleaned it shows up as running in the repository but actually is
not.
Stale sessions are cleaned in multiple ways. Some examples are listed below:
You can manually request specific agents to clean stale sessions in Operator
Navigator or Topology Navigator.
Stale sessions are cleaned when you restart an agent.
When an agent starts any new session, it checks for and resolves stale sessions.
However, if the agent has lost connection to the repository, then it cannot clean
stale sessions.
3.
Check if the agent is alive. To test the agent to see if it is running and still has a
connection to the repository, open it in the Topology Navigator in ODI Studio and
select the Test tab. If the agent test fails, then restart the agent after fixing the issue.
4.
Verify that the ODI Repository and the server hosting it are running and have not
experienced a failure.
5.
If your load plan is in error and you have verified all of the above, then restart the
Load plan.
A scenario run outside of a load plan by itself invokes the Table Maintenance
process. This could, depending on the setting, truncate the table before the load.
To understand this, consider that when a scenario is run inside a load plan table
maintenance tasks are carried out as explicit steps (the parent step name would be
either Initialize or Finalize). The scenario by itself does not call the Table
Maintenance process when run from within the load plan. Rather, this is controlled
by the EXECUTION_ID variable, which is set to the load plan instance ID. If this
variable has a value greater than 0 when a scenario is run, the Table Maintenance
process is not invoked, as would be the case when a scenario is run from within a
load plan with an instance ID. However, if this variable does not have a value
greater than 0, then the scenario invokes the Table Maintenance process. This is the
case when a scenario is run outside the load plan. If you set a value for the
EXECUTION_ID when invoking the scenario from outside a load plan, the table
maintenance steps would not be called.
A scenario step could have many variable values set, either dynamically in the
case of a refresh variable or explicitly by overriding its value at that scenario step
in the load plan. When running a scenario outside the load plan, all the scenario
variables would have only their default values. For this reason, care should be
taken to set the variables appropriately before calling a scenario from outside the
load plan. You can check the variable values that are present in the load plan by
looking at the Operator log, provided the log level was set to 6 when the load plan
ran. The Oracle BI Applications Configuration Manager uses Oracle Diagnostic
Logging. For information about managing log files and diagnostic data, see Oracle
Fusion Middleware Administrator's Guide.
Note:
Start a new load plan run, accepting the overhead associated with restarting the
load from the beginning.
Run the regenerated scenario as stand-alone outside the load plan, marking that
scenario step as complete in the load plan before restarting the load plan. Refer to
"Using Mark as Complete" and "Running a Scenario Standalone" for implications
of using these options.
session. (Note: C$ and I$ tables are created afresh on restart and hence data to these
tables would be committed in between).
New C$ and I$ tables are created on restart and hence data to
these tables would be committed in between load plan start and
restart.
Note:
Note:
In Operator Navigator, select the running or waiting load plan run to stop from
the Load Plan Executions accordion.
2.
Stop Normal - In normal stop mode, the agent in charge of stopping the load
plan sends a Stop Normal signal to each agent running a session for this load
plan. Each agent will wait for the completion of the current task of the session
and then end the session in error. Exception steps will not be executed by the
load plan and once all exceptions are finished the load plan is moved to an
error state.
Stop Immediate - In immediate stop mode, the agent in charge of stopping the
load plan sends a Stop Immediate signal to each agent running a session for
this load plan. Each agent will immediately end the session in error and not
wait for the completion of the current task of the session. Exception steps will
not be executed by the load plan and once all exceptions are finished the load
plan is moved to an error state.
3.
In the Stop Load Plan dialog, select an agent to stop the load plan.
4.
Click OK.
For more information about stopping load plan runs from ODI Studio, see Oracle
Fusion Middleware Developer's Guide for Oracle Data Integrator.
3
Additional Administration Tasks
3
To create the S_ETL tables, you generate a DDL script, which then must be provided to
the Siebel DBA, who will need to create the necessary tables in the Siebel database.
Note that the DB user used for the ETL connection will require all DML privileges
(SELECT, INSERT, DELETE, UPDATES) and DDL (TRUNCATE) privileges to S_ETL
tables.
To generate a DDL script to create S_ETL tables for Siebel sources:
1.
2.
In the Projects editor, expand the following folders: BIApps Project, Components,
Sources, Siebel, Oracle, Generate Siebel DDL, Packages, Generate Siebel DDL,
Scenarios.
3.
4.
Select the Context, ODI Agent, and Log Level, and then click OK.
5.
Option
Description
CREATE_SCRIPT_FILE
REFRESH_MODE
Enter FULL.
3-1
Option
Description
CHAR_CLAUSE
RUN_DDL
Enter N.
SCRIPT_LOCATION
TABLE_MASK
A
Manage Load Plans User Interface Reference
This section contains reference information about the Manage Load Plans user
interface elements in the Oracle BI Applications Configuration Manager.
This section contains the following topics:
Description
Actions Menu:
Duplicate
Enables you to define a new load plan with the same fact
groups as the selected load plan definition but with a
different name and identifier.
Generate
Starts the process to build the load plan in the ODI
repository. A load plan must build successfully before it
can be executed.
Show Generation Status Details
Shows the generation status for each step included in the
load plan.
A-1
Description
Reset Generation Status
Resets the generation status to Not Generated.
Execute
Executes a load plan that has been successfully generated.
If a listed parameter requires a value but a value has not been assigned, the
respective row in the table is tagged with an error icon. Parameters that do not
require a value (value can be null) are not tagged even if no value has been
assigned.
You can filter the list of parameters to display only the data load parameters that
have no value by using the Show drop-down list in the toolbar.
You can export and save content displayed in the table to a Microsoft Excel
formatted file by clicking the Export icon in toolbar.
You can change a parameter value by selecting the parameter in the list, and then
clicking the Edit icon in the toolbar. The Edit Parameter Value dialog is displayed.
To change a parameter value, the user must have been assigned a role that has the
appropriate privilege.
If a source domain in the list contains members that have not been mapped to an
appropriate warehouse domain member, the row in the table is tagged with an
error icon. Some source domain members are not applicable, and, therefore, are
not tagged even if they are unmapped.
You can filter the list of mappings to display only the domains that have
unmapped source members using the Show drop-down list in the toolbar.
A-3
You can export and save content displayed in the table to a Microsoft Excel
formatted file by clicking the Export icon in toolbar.
You can change a domain mapping by selecting the mapping in the list, and then
clicking the Edit icon in the toolbar. The Edit Domain Member Mappings dialog is
displayed. To change a domain member mapping, the user must have been
assigned a role that has the appropriate privilege.
Context - The ODI context to be used when the load plan is run. Note that Global
is the only supported context.
Logical Agent - The ODI Agent to be used when the load plan is run.
On Agent Startup
Simple
Hourly
Daily
Weekly
Yearly
Active
Inactive
Add - This command invokes the Schedule Load Plan dialog, which enables you
to define a new schedule for the selected load plan.
Edit - Applies to a selected schedule. This command invokes the Schedule Load
Plan dialog, which enables you to make changes to the selected schedule.
Remove - This command removes the schedule from the repository.
Update Schedules - This command adds the schedule to the ODI Agent scheduler
if the agent is already started in ODI. Note that if the ODI Agent is already
running at the time you define the schedule in the Oracle BI Applications
Configuration Manager, you must click Update Schedules to add the schedule to
the ODI Agent scheduler.
Description
The first page of the Define Load Plan series requests the following
information:
The second page of the Define Load Plan series contains the following
tabs:
Description
Source Instance
Fact Group
A-5
Name
Description
Source Instance
Fact Group
Description
Navigation tabs
In the Navigation tabs, you can browse for objects contained in the
repository. When an object or node is selected, the Navigation Tab
toolbar displays icons for the actions available for this object or node.
If an action is not available for this object, the icon is greyed out.
The Load Plan tab displays three panels, containing the following
information:
The Load Plan execution tab displays three panels, containing the
following information:
Context - The ODI context to be used when the load plan is run. Note that Global
is the only supported context.
Logical Agent - The ODI Agent to be used when the load plan is run.
Log Level - The level of logging information to retain. The Oracle BI Applications
Configuration Manager uses Oracle Diagnostic Logging. For information about
managing log files and diagnostic data, see Oracle Fusion Middleware
Administrator's Guide.
Status - The status of the schedule. Possible values are the following:
Active
Inactive
On Agent Startup
Simple
Hourly
Daily
Weekly
Yearly
Depending on the Recurrence option you select, options for selecting the date
and time are dynamically displayed.
A-7
B
Knowledge Modules Reference
This section provides detailed information about the Integration Knowledge Modules
(IKMs) available with Oracle BI Applications.
This section contains the following topic:
If the "Synchronize Deletions from Journal" process is executed, the deleted rows
on the target are committed.
To use FLOW_CONTROL and RECYCLE_ERRORS options, the update ley must
be set on the interface.
Hints - This IKM allows the passing of hints into the generated SQL. For more
information, see the article titled "Oracle Business Intelligence Applications
Version 11.1.1.7.1 Performance Recommendations (Doc ID 1539322.1)" on My
Oracle Support.
Alter Session List - Applies a list of alter session commands to the session used by
the KM. Commands should be separated by a semi-colon and without the "ALTER
SESSION" prefix. Each command should be prefixed SRC or TGT depending on
whether it should be executed on the source connection (relevant if using an LKM)
or the target connection. For example:
SRC set TRACEFILE_IDENTIFIER='ODI_TRACE'; SRC set events '10046 TRACE NAME
CONTEXT FOREVER, LEVEL 8'; TGT set TRACEFILE_IDENTIFIER='ODI_TRACE_TGT';
The event queue table must be defined using the Event Queue Table option.
Join between target table and event queue must be defined using the Event Queue
Join option.
Target table must have SCD behavior set for:
Natural key
Starting/ending timestamp
Interface must only select the source data that is changing, as controlled by the
event queue, which lists the natural keys that are changing and the earliest date of
any change.
Either the data is selected from temporary or staging tables which only contain
incremental data
Or
Use the nested IKM BIAPPS Oracle Event Queue Delete Append to include a
join to the event queue in incremental load
If the "Synchronize Deletions from Journal" process is executed, the deleted rows
on the target are committed.
Event Queue Table. The name of the event queue table that holds the incremental
changes. This option is mandatory.
Prerequisites for this option are the following:
Event Queue Join. Assuming the alias T for Target Table and Q for Event Queue
Table, define the equi-join between the Target Table and the Event Queue Table.
This is used in the Event Queue Update and Event Queue Delete steps and in rare
cases may be omitted if neither of those steps are required. The filter on
EARLIEST_CHANGE_DATE should not be included in the join option.
Event Queue Delete. Whether or not to delete records in the target that are being
processed in incremental load. In most cases this should be enabled, but in rare
cases where more than one interface loads the target table then it only needs to be
enabled on the first one.
Event Queue Update. Whether or not to correct effective dates on records in the
target that are affected by the incremental load. In most cases this should be
enabled, but in rare cases where more than one interface loads the target table then
it only needs to be enabled on the last one.
High Data Value. The default value to use for the maximum ending timestamp. In
most cases the default value of #HI_DATE is fine, but for some persisted staging
tables that reflect the OLTP might use a different value e.g. #ORA_HI_DATE.
Unspecified Record. If the target table is a dimension, set this to TRUE to
automatically insert an "Unspecified" record. This is referenced by facts in case no
other dimension record matches. The default column values are determined by
model naming standards using the user-defined function GET_UNSPEC_VALUE.
Alter Session List - Applies a list of alter session commands to the session used by
the KM. Commands should be separated by a semi-colon and without the "ALTER
SESSION" prefix. Each command should be prefixed SRC or TGT depending on
whether it should be executed on the source connection (relevant if using an LKM)
or the target connection. For example:
SRC set TRACEFILE_IDENTIFIER='ODI_TRACE'; SRC set events '10046 TRACE NAME
CONTEXT FOREVER, LEVEL 8'; TGT set TRACEFILE_IDENTIFIER='ODI_TRACE_TGT';
The update key must be defined in the interface and the key columns should be
indexed.
If the "Synchronize Deletions from Journal" process is executed, the deleted rows
on the target are committed.
Soft Delete. There are several additional steps that you can perform if the soft
delete option is enabled. The variables #SOFT_DELETE_FEATURE_ENABLED
(global) and #SOFT_DELETE_PREPROCESS (can be set for each fact or dimension
group) control exactly which steps are executed.
If you are able to implement triggers to efficiently capture deletes on the source
system, you can disable the expensive pre-process steps (which extract all source
records and compare against all target records) and, instead, directly populate the
delete table.
Step
Action
Control
Soft delete
pre-process
#SOFT_DELETE_FEATURE_
ENABLED
Uses #LAST_ARCHIVE_
DATE to filter target by
CREATED_ON_DT system
column (set variable to NULL
to disable this).
Only data sources that have
implemented the primary
extract are included. If a data
source has no records in the
primary extract table then no
records will be added to the
delete table for that data
source.
#SOFT_DELETE_PREPROCESS
Step
Action
Control
#SOFT_DELETE_FEATURE_
ENABLED
Note that all these steps are committed together, along with any other inserts and
updates to the target table. This keeps the data warehouse consistent.
Prerequisites for using this option are the following:
The target table must have the ETL_PROC_WID and W_UPDATE_DT system
columns.
Date Track. Automatically maintains effective from and to dates in the target table,
similarly to a slowly changing dimension. It can handle dates or numbers (usually
date keys, for example, YYYYMMDD). This can also set a current flag column,
which will be 'Y' for the last record and 'N' for earlier records.
Prerequisites for using this option are the following:
Set the slowly changing dimension behavior for the following table columns
in the model:
- Natural key
- Starting/ending timestamp
- Current flag (optional)
The natural key and starting timestamp columns should be indexed if they are
not covered by the update key index.
Change Capture. Captures target table changes to an image table to streamline the
process of reflecting the changes in aggregates.
Step
Action
Control
Truncate change
image table
Always runs
Capture preload
changes
Always runs
Step
Action
Control
Always runs
All these steps are committed together along with any other inserts and updates to
the target table. This keeps the data warehouse consistent.
Prerequisites for using this option are the following:
For W_FACT_F, the image table W_FACT_CMG must be created with all the
columns of the target table as well as the following columns:
CHANGED_IN_TASK
PHASE_CODE
PHASE_MULTIPLIER
Alter Session List - Applies a list of alter session commands to the session used by
the KM. Commands should be separated by a semi-colon and without the "ALTER
SESSION" prefix. Each command should be prefixed SRC or TGT depending on
whether it should be executed on the source connection (relevant if using an LKM)
or the target connection. For example:
SRC set TRACEFILE_IDENTIFIER='ODI_TRACE'; SRC set events '10046 TRACE NAME
CONTEXT FOREVER, LEVEL 8'; TGT set TRACEFILE_IDENTIFIER='ODI_TRACE_TGT';
Analyze Target - Statistics will be collected on the target table before it is loaded if
the KM Option ANALYZE_TARGET is set to True. By default it is set to False.
Analyze Flow Table - Statistics will be collected on the flow table (I$) after it is
loaded if the KM Option ANALYZE_FLOW_TABLE is set to True. By default it is
set to False. This option also affects how the effective dates are calculated in full
load if the date track option is enabled. If the flow table is not analyzed then an
UPDATE statement is used to set the effective to dates. Otherwise a MERGE
statement is used.
Bulk Mode (variable #ETL_BULK_MODE) - If enabled, bulk mode will use the
direct path write to target (append hint) and bypass the I$ table (if no other KM
options requiring it, for example, flow control, recycle errors, and date track). The
bulk mode variable can be set to:
Y - Enabled
F - Enabled for full load only
N - Disabled
The Delete Period Type option must specify which period type the target table is
based on.
For Calendar Periods, the target table must contain a column for the period end
date key:
- Must be a date wid type (YYYYMMDD).
- Must be named PERIOD_END_DT_WID or otherwise identified in the mapping
using the UD1 check box.
Calendar Periods
Calendar periods of day, week, month, quarter and year are supported. For these
period types, the incremental load assumes the interface is loading data for the current
period only. The incremental update works by deleting any existing data for the
current period. Then, the fresh set of data for the current period is inserted.
Optionally, data older than a specified age can be automatically purged with the
Periods to Keep option. The option Periods To Delete does not apply for calendar
periods.
MCAL Calendar Periods
MCAL calendar periods of MCAL Day and MCAL Period are supported. For these
period types, the incremental load assumes the interface is loading data for the current
period and a given number of previous periods. The incremental update works by
deleting any existing data for these periods. Then, the fresh set of data is inserted.
The option Periods To Delete controls how many previous periods (as well as the
current one) are being incrementally loaded by the interface. For example, a value of 1
would indicate reprocessing the current and previous period every load. The Periods
to Keep option does not apply for MCAL calendar periods.
Options for Functionality
Delete Period Type - The type of calendar periods the target table is based on. It is
used for deleting current periods to reprocess or obsolete periods to purge. Valid
values are the following:
Knowledge Modules Reference B-7
For calendar periods, the target table must contain a column for the period
end date key
- Must be named PERIOD_END_DT_WID or otherwise identified in the
mapping using the UD1 check box.
- Must be a date wid type (YYYYMMDD).
Periods to Delete - This option is for configuring additional functionality for the
MCAL calendar periods only. It must be left at the default value (0 - delete current
period only) for calendar periods. Setting this option to a value N > 0 will delete
exactly N Julian periods of data based on the delete period type. For example,
setting to 1 will mean the current and previous periods are deleted.
Periods to Keep - This option is for configuring additional functionality for the
calendar periods only. It must be left at the default value (0 - keep all periods) for
calendar periods. Setting this option to a value N > 0 will delete data in periods
which are exactly N julian periods older than the current julian period
Hints - This IKM allows the passing of hints into the generated SQL. For more
information, see the article titled "Oracle Business Intelligence Applications
Version 11.1.1.7.1 Performance Recommendations (Doc ID 1539322.1)" on My
Oracle Support.
Alter Session List - Applies a list of alter session commands to the session used by
the KM. Commands should be separated by a semi-colon and without the "ALTER
SESSION" prefix. Each command should be prefixed SRC or TGT depending on
whether it should be executed on the source connection (relevant if using an LKM)
or the target connection. For example:
SRC set TRACEFILE_IDENTIFIER='ODI_TRACE'; SRC set events '10046 TRACE NAME
CONTEXT FOREVER, LEVEL 8'; TGT set TRACEFILE_IDENTIFIER='ODI_TRACE_TGT';
Update key must be defined in the interface and the key columns should be
indexed (usually INTEGRATION_ID, DATASOURCE_NUM_ID, SRC_EFF_
FROM_DT).
Slowly changing dimension behavior must be set for all target table columns (set
in model) and must include the following:
- Surrogate key (usually ROW_WID)
- Natural key (usually INTEGRATION_ID, DATASOURCE_NUM_ID)
- Start and end timestamps (usually EFFECTIVE_FROM_DT and EFFECTIVE_
TO_DT)
- Current flag (usually CURRENT_FLG)
Source from and to dates should be Not Null (default to #LOW_DATE or #HI_
DATE).
ETL_PROC_WID should be indexed.
Column Classification
The following table describes how dimension columns should be categorized and the
different behaviors for each classification. The categorization is done on individual
table columns in the model.
Column
Description
SCD Behavior
Surrogate key
Warehouse generated
primary key for
dimension
Surrogate key
Natural key
Business or source
key, unique in
combination with
time
Natural key
Start timestamp
Time record is
effective from
Start timestamp
End timestamp
Time record is
effective to
End timestamp
Current flag
Current flag
Type 2 columns
Insert on change
Update history
columns
Overwrite on change
Always set to the
value from the current
record
Other Flexfields
Overwrite on change
System column
Change columns
Overwrite on change
Record is rejected if
there is no change to
any of these columns
Column
Description
SCD Behavior
Other Flexfields
SCD1 key
Warehouse generated
key corresponding to
the natural key
Overwrite on change
Type 2 Changes - If the variable #TYPE2_FLG is turned off (set to 'N') then the
dimension behaves as a Type 1 dimension. The natural key must be unique (no
history allowed) because no maintenance of the start/end dates is performed.
With #TYPE2_FLG on (set to 'Y') new records will be inserted; changes that update
at least one Type 2 column will also trigger an insert, subject to some restrictions,
and any other change will update the existing record.
Type 2 changes are triggered as follows:
- The incoming record must have at least one type 2 column different when
compared to the current dimension record.
- The new Type 2 record start timestamp is calculated as follows: If there is a
change column with a non-null date value, then use that (the Oracle BI
Applications standard is to use CHANGED_ON_DT as the change column);
otherwise, use the current timestamp (sysdate)
SCD1 Key - Set this to TRUE to automatically maintain a surrogate natural key or
Type 1 key. This is managed using a Type 1 table named according to a standard
pattern. If the dimension table is W_DIMENSION_D, then the Type 1 table will be
W_DIMENSION_T1_D. The sequence that generates the Type 1 key is also named
according to a standard pattern. Continuing the example, it would be named W_
DIMENSION_D_S1W. Additional columns that are at the same grain (also Type 1)
can be automatically maintained on the Type 1 table if they are marked with the
UD1 flag.
Prerequisites for W_DIMENSION_D are as follows:
Type 1 key column must be identified by the SCD1 WID flexfield in the model.
Fill Gaps - Set this to TRUE to automatically extend the first records to cover any
earlier date.
Soft Delete. There are several additional steps that you can perform if the soft
delete option is enabled. The variables #SOFT_DELETE_FEATURE_ENABLED
(global) and #SOFT_DELETE_PREPROCESS (can be set for each fact or dimension
group) control exactly which steps are executed.
If you are able to implement triggers to efficiently capture deletes on the source
system, you can disable the expensive pre-process steps (which extract all source
records and compare against all target records) and, instead, directly populate the
delete table.
Step
Action
Control
Soft delete
pre-process
#SOFT_DELETE_FEATURE_
ENABLED
#SOFT_DELETE_PREPROCESS
Uses #LAST_ARCHIVE_
DATE to filter target by
CREATED_ON_DT system
column (set variable to NULL
to disable this).
Only data sources that have
implemented the primary
extract are included. If a data
source has no records in the
primary extract table then no
records will be added to the
delete table for that data
source.
#SOFT_DELETE_FEATURE_
ENABLED
Note that all these steps are committed together, along with any other inserts and
updates to the target table. This keeps the data warehouse consistent.
Prerequisites for using this option are the following:
The target table must have the ETL_PROC_WID and W_UPDATE_DT system
columns.
B-11
Detection Strategy - To avoid updating the table if no changes have occurred, the
incoming data is compared with the existing record on a set of change columns.
These are either defined in the model column flexfield (OBI_CHANGE_
COLUMN), or otherwise all columns are compared. To always process the
changes, this option can be disabled.
Hints and Full History - This IKM allows the passing of hints into the generated
SQL. For more information, see the article titled "Oracle Business Intelligence
Applications Version 11.1.1.7.1 Performance Recommendations (Doc ID
1539322.1)" on My Oracle Support.
Alter Session List - Applies a list of alter session commands to the session used by
the KM. Commands should be separated by a semi-colon and without the "ALTER
SESSION" prefix. Each command should be prefixed SRC or TGT depending on
whether it should be executed on the source connection (relevant if using an LKM)
or the target connection. For example:
SRC set TRACEFILE_IDENTIFIER='ODI_TRACE'; SRC set events '10046 TRACE NAME
CONTEXT FOREVER, LEVEL 8'; TGT set TRACEFILE_IDENTIFIER='ODI_TRACE_TGT';
Analyze Target - Statistics will be collected on the target table before it is loaded if
the KM Option ANALYZE_TARGET is set to True. By default it is set to False.
Map the source data you want to reference to one of the target columns on the
interface.
Reference it in the SQL Override using the syntax :TARGET_COLUMN.
Hints - This IKM allows the passing of hints into the generated SQL. For more
information, see the article titled "Oracle Business Intelligence Applications
Version 11.1.1.7.1 Performance Recommendations (Doc ID 1539322.1)" on My
Oracle Support.
Alter Session List - Applies a list of alter session commands to the session used by
the KM. Commands should be separated by a semi-colon and without the "ALTER
SESSION" prefix. Each command should be prefixed SRC or TGT depending on
whether it should be executed on the source connection (relevant if using an LKM)
or the target connection. For example:
SRC set TRACEFILE_IDENTIFIER='ODI_TRACE'; SRC set events '10046 TRACE NAME
CONTEXT FOREVER, LEVEL 8'; TGT set TRACEFILE_IDENTIFIER='ODI_TRACE_TGT';
Aggregate Lookup - Aggregates the SQL block by using the user flexfields to
define a grouping of records to restrict the output to one row per group.
Prerequisites for using this option are the following:
B-13
Source SQL Override - Allows the generated nested SQL block to be overridden.
This is useful if the SQL needs to be more dynamic in nature, for example, using
variables to reference dynamic tables.
Prerequisites for using this option are the following:
For each target interface column there must be an identically aliased column in
the SELECT clause of the SQL override.
Use of this option is discouraged. Make sure the use case is reviewed by the
Oracle BI Applications Standards Group before implementing.
Hints - This IKM allows the passing of hints into the generated SQL. For more
information, see the article titled "Oracle Business Intelligence Applications
Version 11.1.1.7.1 Performance Recommendations (Doc ID 1539322.1)" on My
Oracle Support.
The event queue table must be specified in the Event Queue Table option.
Hints - This IKM allows the passing of hints into the generated SQL. For more
information, see the article titled "Oracle Business Intelligence Applications
Version 11.1.1.7.1 Performance Recommendations (Doc ID 1539322.1)" on My
Oracle Support.