0% found this document useful (0 votes)

462 views

Student Guide

Oracle Exam for ODI 1Z0-448

Uploaded by

Florin Nedelcu

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

462 views

Student Guide

Oracle Exam for ODI 1Z0-448

Uploaded by

Florin Nedelcu

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 430

Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
n o
s a
h a
) deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
l o r in to u
u (f nse
e l c l i ce
e d
n N
F lori
Oracle Data Integrator 12c:
Advanced Integration and
Development
Student Guide
D87557GC10
Edition 1.0 | January 2015 | D89774

Learn more from Oracle University at oracle.com/education/

Steve Friedberg Disclaimer

This document contains proprietary information and is protected by copyright and

Technical Contributors other intellectual property laws. You may copy and print this document solely for your
and Reviewers own use in an Oracle training course. The document may not be modified or altered
in any way. Except where your use constitutes "fair use" under copyright law, you
David Allan may not use, share, download, upload, copy, print, display, perform, reproduce,
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

publish, license, post, transmit, or distribute this document in whole or in part without
Brent Dayley the express authorization of Oracle.
Christophe Dupupet
The information contained in this document is subject to change without notice. If you
Gerry Jurrens find any problems in the document, please report them in writing to: Oracle University,
Alex Kotopoulis 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not
warranted to be error-free.
Julien Testut
Restricted Rights Notice

Editors If this documentation is delivered to the United States Government or anyone using
Smita Kommini
the documentation on behalf of the United States Government, the following notice is
ble
Aju Kumar
applicable:
fe r a
U.S. GOVERNMENT RIGHTS
an s
- t r
The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or
n
Graphic Designer
Rajiv Chandrabhanu a no
disclose these training materials are restricted by the terms of the applicable Oracle
license agreement and/or the applicable U.S. Government contract.

Trademark Notice h a s
m ) deฺ
Publishers o of Oracle iand/or
e ฺ c
Oracle and Java are registered trademarks
G u its affiliates. Other names
Syed Imtiaz Ali may be trademarks of their
r a cl ent
respective owners.

Srividya Rameshkumar
@ o tud
e l c his S
u
ฺ n ed se t
l o r in to u
u (f nse
e l c l i ce
e d
i n N
r
Flo
Contents
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

1 Introduction
Lesson Objectives 1-2
Course Objectives 1-3
Target Audience 1-4
Class Introductions 1-5
Agenda of Lessons 1-6
ble
Course Environment 1-8
fe r a
Course Materials 1-9
ans
Course Practices 1-10
n - t r
How Can I Learn More? 1-11 no
s a
Related Training 1-12
h a
) deฺ
Agenda 1-13
c o m u i
e ฺ G
Oracle Data Integrator: Features 1-14
r a cl ent
o tud
Oracle Data Integrator: Features Knowledge Modules 1-15
@
c his S
u
Conventional Integration Process: ETL 1-16
e l
ETL Versus E-LT 1-17
ฺ n ed se t
E-LT Steps 1-18
l o r in to u
(f nse
ODI Architecture 1-19
u
e l c
Summary 1-20
l i ce
e d
n N Practice 1-1: Exploring Your Environment 1-21

F lori 2 ODI Knowledge Modules: Overview

Objectives 2-2
Agenda 2-3
Knowledge Modules: Overview 2-4
Global KMs 2-5
KMs in Action 2-6
Code Generation 2-7
Types of Knowledge Modules: Overview 2-8
Agenda 2-9
How to Select Knowledge Modules 2-10
Creating New Knowledge Modules 2-11
Developing Knowledge Modules 2-12
Editing Knowledge Modules 2-13
Knowledge Module Options 2-14

iii
Description of Knowledge Module Tasks 2-15
Details of the Tasks Are Generic 2-16
Setting Knowledge Module Options 2-17
Replacing Existing Knowledge Modules 2-18
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Agenda 2-20
Developing Your Own Knowledge Module: General Guidelines 2-21
Substitution API 2-22
Using Substitution Methods 2-23
Various ODI Substitution Methods 2-24
Substitution Methods: Basic Examples 2-25
Using Substitution Methods in Actions 2-26
ble
Working with Object Names 2-28
fe r a
Working with Lists of Tables, Columns, and Expressions 2-30
ans
Example: Using getTargetColList to Create a Table 2-33
n - t r
Example: Using getColList in an Insert Values Statement 2-34n o
Example: Using getSrcTablesList 2-35 s a
Generating the Source Select Statement 2-36 h a
) deฺ
c o m u i
e ฺ
Generating the Source Select Statement: Example 2-37
G
Working with Datasets 2-38
r a cl ent
@ o tud
Obtaining Other Information with the API 2-39
e l c his S
u
ed se t
Quiz 2-40
Summary 2-41 ฺ n
in to u
l o r
(f nse
Practice 2-1 2-42
u
c ice
del
3 eDeveloping lKnowledge Modules
N
in Objectives 3-2
r
Flo Agenda 3-3
Developing KMs: Targeting a Particular Stage of your Integration Process 3-4
Developing KMs: What to Avoid and What to Use 3-6
Developing KMs: Other Recommendations 3-7
Agenda 3-8
Using Java in Knowledge Modules 3-9
Using Code Generation Tags: 3-10
Using Code Generation Tags: Examples 3-11
Code Generation Techniques 3-12
Agenda 3-14
Using ODI Substitution Methods: getJoin() 3-15
Using ODI Substitution Methods: getFilter() 3-16
Using ODI Substitution Methods: getPK() 3-17
Using ODI Substitution Methods in Journalizing Knowledge Modules:
Examples 3-18

iv
Using ODI Substitution Methods in Reverse-Engineering KM: getModel() 3-20
Agenda 3-22
Troubleshooting KMs 3-23
Quiz 3-27
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Summary 3-29
Practice 3-1 Overview: Creating a New Knowledge Module for
Data Generation 3-30

4 Designing ODI Integration Mappings

Objectives 4-2
Agenda 4-3
ble
Integration Process: Overview 4-4
fe r a
Typical Integration Process 4-6
ans
Agenda 4-7
n - t r
Integration Mappings: Overview 4-8 no
Basic Integration Process: Sequence of Operations 4-10 s a
Staging Area 4-11 h a
) deฺ
c o m u i
Placing the Staging Area 4-12
e ฺ G
Agenda 4-13
r a cl ent
@ o tud
Designing Integration Mappings: E-LT- and ETL-Style Mappings 4-14
e l c his S
u
ed se t
Designing an ETL-Style Mapping 4-15
ฺ n
in to u
Designing an ETL-Style Mapping: Using Multi-Connection IKM 4-16
l o r
(f nse
Designing an ETL-Style Mapping: Using an LKM and a Mono-Connection IKM 4-19
u
l c ce
Using an LKM and a Mono-Connection IKM: Steps 4-20
e l i
e d
Agenda 4-21
n N
lori
Enforcing ODI Data Quality 4-22
F Ways to Handle Erroneous Data 4-23
Error Recycling 4-24
Building a Data Quality Framework 4-25
Quiz 4-26
Summary 4-27
Practice 4-1: Creating an ODI Mapping for XML to Database Transformation with
ODI Constraint and Error Recycling 4-28

5 Designing Advanced Integration Mappings

Objectives 5-2
Agenda 5-3
Creating Lookups 5-4
Lookup Goal 5-5
Lookup Match Row Rules: Multiple Match 5-6
Lookup Match Row Rules: No-Match 5-7

v
Using Set-Based Operators with Integration Mappings 5-8
Example: Flow with Multiple Datasets 5-9
Defining a Dataset 5-10
Using Set-Based Operators with Integration Mappings 5-11
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Using Set-Based Operators: Guidelines 5-12

Agenda 5-13
Using Partitioning with ODI Mappings 5-14
Partitioning: Definition in Datastore After Reverse Engineering 5-15
Using Partitioning in a Mapping 5-16
Agenda 5-17
Configuring Reusable Mappings 5-18
ble
Using Reusable Mappings 5-19
fe r a
Derived Select for Reusable Mappings 5-20
ans
Derived Select for Reusable Mappings: Limitations 5-21
n - t r
Quiz 5-22 n o
Summary 5-24 s a
Practice 5-1: Configuring Partition Support 5-25 h a
) deฺ
c o m u i
e ฺ
Practice 5-2: Configuring Lookup Match Rules 5-26
G
r a cl ent
6 Managing Object Versions @ o tud
e l c his S
u
ed se t
Objectives 6-2
Agenda 6-3 ฺ n
in to u 6-4
l o r
u (f nsFlags
What Is Version Management?
e 6-5
c e
del Versions
Working with Object
eCreating lic 6-6
n N
ri Restoring Versions 6-7
Flo Agenda 6-8
Using Version Browser 6-9
Using the Version Comparison Tool 6-10
Color Code 6-11
Opening the Version Comparison Tool 6-12
Working with Solutions 6-13
Working with Solutions: Synchronizing 6-15
Handling Concurrent Changes: Concurrent Editing Check 6-16
Handling Concurrent Changes: Object Locking 6-17
Quiz 6-18
Summary 6-19
Practice 6-1 Overview: Working with Versions 6-20

vi
7 Using Variables in ODI
Objectives 7-2
Agenda 7-3
Using Variables in ODI: Overview 7-4
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Variable Scope 7-5

Referring to a Variable 7-6
Referring to a Variable: Using “:” Instead of “#” 7-7
Agenda 7-8
Using Variables in Packages 7-9
Using Variables in Packages: Example 7-10
Using Variables in Mappings 7-11
ble
Using Variables in Mappings: Examples 7-13
fe r a
Using Variables in Object Properties 7-14
ans
Using Variables in Procedures: with Jython 7-15
n - t r
Using Variables in Procedures: with Options 7-16 n o
Using Variables Within Variables 7-17 s a
h a
) deฺ
Using Variables in the Resource Name of a Datastore 7-18
c o m u i
e ฺ
Using Variables in the Resource Name of a Datastore: Steps 7-19
G
r a cl ent
Using a Variable as a Startup Parameter: Example 7-20
@
Using ODI Variables in Topology 7-23 o tud
e l c his S
u
ed se t
Using ODI Variables in Topology: A Server URL 7-24
ฺ n
in to u
Using Variables in a Server URL: Example 7-25
l o r
Agenda 7-27
u (f nse
l c ce
Tracking Variables 7-28
e l i
e d
Tracking Variables: Notes 7-31
n N
lori
Quiz 7-32
F Summary 7-34
Practice 7-1 Overview: Using an ODI Variable as a Startup Parameter 7-35

8 Accelerating Development with Groovy

Objectives 8-2
Agenda 8-3
ODI Software Development Kit: Overview 8-4
SDK-Supported ODI Operations 8-5
ODI Operations Not Supported by SDK 8-6
Combining Different APIs 8-7
Agenda 8-8
Groovy: Introduction 8-9
Groovy Editor: Introduction 8-10
Executing Script with Groovy Editor 8-11
Agenda 8-12

vii
Performing an SDK Task Using Java with Groovy Editor: Example 8-13
Using Custom Libraries 8-14
Defining Additional Groovy Execution Classpath 8-15
Read Input with the odiInputStream Variable 8-16
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Automating Development Tasks: Example 1 8-17

Automating Development Tasks: Example 2a (createModel) 8-18
Automating Development Tasks: Example 2b (createLogicalSchema) 8-19
Automating Development Tasks: Example 3a (Simple Mapping) 8-20
Automating Development Tasks: Example 3b (Mapping and Filter) 8-21
Automating Development Tasks: Example 4 8-22
Quiz 8-23
ble
Summary 8-24
fe r a
Practice 8-1 Overview: Automating ODI Tasks with Groovy 8-25
ans
n - t r
9 Working with ASCII Files n o
Objectives 9-2 s a
Agenda 9-3 h a
) deฺ
c o m u i
Types of Files in ODI 9-4
e ฺ G
Complex Files Concepts 9-5
r a cl ent
Sample Complex XML File 9-7 @ o tud
e l c his S
u
ed se t
Sample Complex JSON file 9-8
ฺ n
in to u
Knowledge Modules for Complex Files 9-9
l o r
(f nse
Requirements for Working with Complex Files 9-10
u
l c
Agenda 9-11
e l i ce
e d
Setting Up the Topology 9-12
n N
lori
JDBC URL Properties 9-13
F JDBC URL: Example 9-14
Complex File Data Server Definition: Example 9-15
Definition of Physical and Logical Schema: Example 9-17
Agenda 9-18
Setting Up an Integration Project and Creating a Complex File Model:
Example 9-19
Setting Up an Integration project and Creating a Complex File Model:
Reverse Engineer 9-20
Designing a Mapping with Complex File Model 9-21
Creating an nXSD File 9-22
nXSD Schema: Example 9-23
Getting to the Native Format Builder Wizard 9-25
Using the Native Format Builder Wizard 9-26
Reading the Source File 9-27
Using the Native Format Builder Wizard 9-28

viii
Quiz 9-29
Summary 9-30
Practice 9-1 Overview: Configuring ODI Topology and ODI Model with
Complex Files 9-31
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Practice 9-2: Configuring Mappings to Support JSON 9-32

10 Integration of ODI in Enterprise Environment and SOA

Objectives 10-2
Agenda 10-3
Three Types of ODI Agents 10-4
Java EE Agent Deployment Features 10-5
ble
Integration of ODI with Fusion Middleware Control 10-6
fe r a
Using the ODI Console 10-7
ans
Using the ODI Console: Example 10-8
n - t r
Agenda 10-9 n o
Types of Web Services 10-10 s a
Data Services: Overview 10-11 h a
) deฺ
c o m u i
Generation of Data Services 10-12
e ฺ G
Public Web Services: Overview 10-13
r a cl ent
@ o tud
Using Public Web Service OdiInvoke and OdiInvokeWebService 10-14
e l c his S
u
ed se t
Installing Public Web Services 10-15
Agenda 10-16 ฺ n
in to u
l o r
(f nse
OdiInvokeWebService Tool 10-17
u
l c ce
OdiInvokeWebService Tool in Action 10-18
e l i
e d
OdiInvokeWebService Tool in a Package 10-19
n N
lori
Agenda 10-20
F ODI with SOA Integration Scenarios 10-21
Integration of ODI Within SOA in Action 10-22
Example 1: Using Data Services 10-23
Example 2: Exposing ODI Process as a Web Service 10-24
Creating the BPEL Process in JDeveloper 10-25
Creating ODI Mapping and ODI Package 10-26
Creating ODI Scenario and Editing the BPEL Process 10-27
Deploying the BPEL Process to WLS and Invoking from Enterprise Manager 10-28
Example 3: Calling a Web Service from Within ODI for Processing ODI Errors with
BPEL Human Workflow 10-29
Processing ODI Errors with Error Hospital 10-30
Processing ODI Errors with BPEL Human Workflow 10-31
Processing ODI Errors with BPEL Human Workflow 1 10-32
Processing ODI Errors with BPEL Human Workflow 2 10-33
Processing ODI Errors with BPEL Human Workflow 3 10-35

ix
Processing ODI Errors with BPEL Human Workflow 10-36
Quiz 10-37
Summary 10-38
Practice 10-1 Overview: Integrating ODI in the Enterprise Environment 10-39
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Practice 10-2 Overview: Exposing an ODI Scenario as a Web Service 10-40

Practice 10-3 Overview: Integrating ODI with a BPEL Process Within SOA 10-41

11 Enhancing ODI Security

Objectives 11-2
Agenda 11-3
Security Concepts: Overview 11-4
ble
Authorizing by Object Instance 11-6
fe r a
ODI Security Navigator: Overview 11-7
an s
Using Generic and Nongeneric Profiles 11-9
n - t r
Built-in Profiles 11-10 n o
SUPERVISOR: A Checkbox, Not a Profile 11-11 s a
Agenda 11-12 h a
) deฺ
c o m u i
Security Policy Approach 11-13
e ฺ G
r a cl ent
Defining Security Policies: Using Generic Profiles 11-14
@ o tud
Defining Security Policies: Using Nongeneric Profiles 11-15
e l c his S
u
ed se t
Using Nongeneric Profiles: Granting an Authorization by Object Instance 11-16
ฺ n
in to u
Unused Authorizations 11-17
l o r
(f nse
Cleaning Up Unused Authorizations 11-18
u
l c
Agenda 11-19
e l i ce
e d
ODI Security Integration: Overview 11-20
n N
lori
Implementing External Authentication (OPSS) 11-21
F Setting the Authentication Mode 11-22
Configuring External Authentication 11-23
Oracle Unified Directory (OUD) 11-24
Agenda 11-25
Configuring External Authentication with OUD 11-26
1. Creating a New LDAP User with OUD 11-27
2a. Editing jps-config-jse.xml to Point to OUD LDAP Server 11-29
2b. Editing jps-config-jse.xml to Point to OUD LDAP Protocols 11-30
3. Running odi_credtool to Set Up the Credentials for the Identity Store 11-31
4. Creating New ODI Master Repository Referencing a User in the External
LDAP Server 11-32
5. Creating a New ODI Connection Referencing a User in the External
LDAP Server 11-33
6a. Creating New ODI Master Repository Referencing a User in the External
LDAP Server 11-34

x
6b. Creating New ODI Master Repository Referencing a User in the External
LDAP Server 11-35
Switching the Master Repository Authentication Mode 11-36
Reactivating Users After Switching to Internal Authentication 11-37
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

External Password Storage 11-38

Setting Up External Password Storage 11-39
Implementing External Password Storage 11-40
Quiz 11-42
Summary 11-43
Practice 11-1 Overview: Implementing ODI External User Authentication 11-44

ble
12 Choosing Integration Strategies: Best Practices
fe r a
Objectives 12-2
ans
Agenda 12-3
n - t r
Integration Strategies 12-4 n o
Agenda 12-5 s a
Strategies with Staging Area on the Target 12-6 h a
) deฺ
c o m u i
e ฺ
Strategies with Staging Area on the Target: Append 12-7
G
r a cl ent
Strategies with Staging Area on the Target: Control Append 12-8
@ o tud
Strategies with Staging Area on the Target: Incremental Update 12-10
e l c his S
u
ed se t
Strategies with Staging Area on the Target: Incremental Update:
ฺ n
in to u
Optimization 12-12
l o r
(f nse
Working with Slowly Changing Dimensions 12-13
u
l c ce
Working with Type 2 Slowly Changing Dimensions 12-14
e l i
e d
Type 2 SCDs: Example 12-15
n N
lori
Implementing Type 2 SCDs 12-16
F Implementing Type 2 SCDs: Steps 12-17
Agenda 12-19
Strategies with Staging Area Different from the Target 12-20
Strategies with Staging Area Different from the Target: Server to
Server Append 12-21
Strategies with Staging Area Different from the Target: Server to File or
JMS Append 12-22
Agenda 12-23
Using ODI for Bulk Processing 12-24
Using ODI for Bulk Processing: Design Patterns 12-25
Using Cross-Reference (XREF) Table: Overview 12-27
Using the XREF Knowledge Module 12-28
Agenda 12-30
Loading Data Patterns 12-31
Oracle GoldenGate: Overview 12-32

xi
Oracle GoldenGate Solutions: Overview 12-33
Oracle GoldenGate Concepts 12-34
Using ODI and OGG Together 12-35
ODI and OGG Topology: Data Servers 12-36
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ODI and OGG Topology: Capture Physical and Logical 12-37

ODI and OGG Topology: Delivery Physical and Logical 12-38
ODI and OGG Topology: CDC 12-39
Initializing CDC Process and Performing Change Operations with OGG 12-40
OGG Command Line: GGSCI 12-41
Agenda 12-42
1. Use Context-Independent Design 12-43
ble
2. Use Procedures Only When Needed 12-44
fe r a
3. Always Enforce Data Quality 12-45
ans
4. Handle Error Cases in Packages 12-46
n - t r
5. Choose Right Knowledge Module 12-47 no
6. Other Best Practices 12-48 s a
Quiz 12-49 h a
) deฺ
c o m u i
Summary 12-51
e ฺ G
r a cl ent
13 ODI and Big Data @ o tud
e l c his S
u
ed se t
Objectives 13-2
Agenda 13-3 ฺ n
n to13-4u
iPhases
l o r
u (f
Big Data in Four
13-5 ens
e
c
del SourceliProjects
Agenda
c
N eOpen 13-6

ri n Oracle Components 13-7

Flo Agenda 13-8
Big Data: Just Another Technology 13-9
Topology for Big Data 13-10
Designer for Big Data 13-11
KMs for Big Data 13-12
Agenda 13-13
ODI and Big Data Webcasts and VM 13-14
Quiz 13-15
Lesson Summary 13-17
Course Summary 13-18

xii
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Introduction

ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori
Lesson Objectives

After completing this lesson, you should be able to:

• Describe the course objectives

• List the agenda of lessons
• Identify the target audience for the course and the
suggested prerequisites
• Describe the course environment e
r a bl
s fe
- t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cgeneral
e d
This lesson
e
provides l
a i overview of the course objectives and agenda of lessons.
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 2

Course Objectives

After completing this course, you should be able to:

• Develop and enhance Knowledge Modules (KMs)

• Design advanced integration mappings
• Manage versions of mappings, and other objects
• Use complex files and other ASCII flat files
• Use variables in ODI b le
f er a
• Use ODI Groovy editor s
• Use ODI within SOA - t r an
n on
• Enhance ODI security s a
a
) h deฺ strategies
• Use best practices for implementing o integration
m i
e ฺ c G u
• Discuss ODI big data options acl nt
or tude
@
cu his S
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 3

Target Audience

This course is directed at:

• Business analysts
• Process architects
• IT developers
• Data administrators
• Professional services consultants ble
fe r a
• Oracle solution architects s
• Data Warehouse administrators - t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u e
ind
Implied e
el licdefinition
the audience is the prerequisite of the information contained in the previous
N
n ODI 12c: Integration and Administration [D82167GC10] or equivalent.
course,
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 4

Class Introductions

Please introduce yourself:

• Name, title/position
• Company
• Experience using Oracle Data Integrator and other
Fusion Middleware products:
– WLS e
– SOA r a bl
s fe
• Reasons for attending - t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Did youeattend lice class, Oracle Data Integrator 12c: Integration and Administration?
del the prerequisite
i n N
lo r
F

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 5

Agenda of Lessons

• Day 1:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

– Lesson 1: Introduction
– Lesson 2: ODI Knowledge Modules: Overview
– Lesson 3: Developing Knowledge Modules
– Lesson 4: Designing ODI Integration Mappings
• Day 2:
a b le
– Lesson 5: Designing Advanced Integration Mappings e r
– Lesson 6: Managing Object Versions a n sf
o n -tr
– Lesson 7: Using Variables in ODI n
s a
– Lesson 8: Accelerating Development with
h aGroovy
) deฺ
– Lesson 9: Working with ASCII Files
c o m ui
c l eฺ nt G
o ra ude
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cecovered per day is at the discretion of the local instructor.
e d
The actual
e
lessons tol i
be
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 6

Agenda of Lessons

• Day 3:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

– Lesson 10: Integration of ODI in Enterprise Environment and

SOA
– Lesson 11: Enhancing ODI Security
– Lesson 12: Choosing Integration Strategies: Best Practices
(includes GoldenGate)
– Lesson 13: ODI and Big Data ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u cecovered per day is at the discretion of the local instructor.
e d
The actual
e
lessons tol i
be
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 7

Course Environment
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Oracle Data Integrator

12c

Web Browser
and Studio
Oracle Unified
Directory (LDAP) Oracle SOA Suite

ble
fe r a
ans
FUSION MIDDLEWARE - t r
WEBLOGIC SERVER
n on
JDeveloper

s a
a
c om) h ideฺ
12 l e ฺc t Gu
Virtual o ac drunning
rMachine e n
Linux
u @ S tu
d e lc this
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce products are used for lessons and practices:
e d e
In this course, the l i
following
n N
lori
• Oracle Database 12c
F • WebLogic Server 12c
• Oracle Data Integrator 12c
• Oracle SOA Suite (BPEL)
• JDeveloper 12c
• Oracle Unified Directory 11g (there is no 12c version as of the course printing)
• MySQL Database 5.6
• Other supporting products
These layered products together comprise the Oracle Data Integrator runtime environment, all
contained in a virtual machine (VM) running 64-bit Oracle Enterprise Linux.

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 8

Course Materials

• Student Guide
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

• Activity Guide
• Online Videos

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce the lecture slides and notes. The Activity Guide contains the practices
d e l i
e Both are in Adobe Reader .pdf format to allow annotation. The videos are on the
The Student Guide contains
n
for N
the course.
i website (used for Practices 12 and 13).
F lorOracle

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 9

Course Practices

• Core topics will be reinforced with hands-on practices.

• Optional topics and practices will be covered at the

instructor’s discretion and if time permits.

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cVirtual
e
If you are d e
taking a l
Live i Class (LVC), the practices are accessed through NoMachine NX
N
n across a VPN.
client
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 10

How Can I Learn More?

Topic Website
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Education, Training http://education.oracle.com

Documentation http://docs.oracle.com/middleware/1213/odi/
Product Downloads http://www.oracle.com/technology/software
Product Articles http://www.oracle.com/technology/pub/articles
Product Support http://www.oracle.com/support
ble
Product Forums http://forums.oracle.com
fe r a
Product Tutorials http://www.oracle.com/oll ans
n - t r
ODI on OTN a no
http://www.oracle.com/technetwork/middleware/data-
integrator/overview/index.html as
) de h ฺ
A-Team
om
http://www.ateam-oracle.com/data-integration/di-odi/
i ฺc t Gu
l e
rac uden
http://www.oracle.com/technetwork/middleware/data-
WebEx Webcasts
o
integrator/odi-11g-webcast-archive-367128.html
@ St
l c u is
e d e t h
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ce
e d e l i
i n N
r
Flo

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 11

Related Training

Course
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ODI 12c: Integration and Administration (prerequisite)

Oracle WebLogic Server: Administration Essentials

Oracle SOA Suite 12c: Essential Concepts

Oracle Directory Services 12c: Essential Concepts ble

fe r a
ODI 12c New Features (two flavors)
ans
n - t r
no
Oracle GoldenGate curriculum (three courses, many OBEs)
a
a s
OBE: ODI Getting Started (five tutorials)
m ) h deฺ
ฺ c o(three courses)
u i
Oracle Enterprise Data Quality curriculum e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce additional information regarding the availability and contents of the
e d e
Your instructor can l i
provide
n N
courses listed in the slide. See http://education.oracle.com for the latest schedules.
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 12

Agenda

Overview of Oracle Data Integrator

• Features
• ETL
• E-LT
• Architecture
ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 13

Oracle Data Integrator: Features

• E-LT architecture provides high performance.

• Active integration enables real-time data warehousing and

operational data hubs.
– CDC technology for real-time data warehousing
– Data services provided to:
— Oracle SOA Suite
Oracle GoldenGate bl e
—

fe r a
• Declarative design improves developer productivity. ans
• n -tr
Knowledge Modules provide flexibility and extensibility.
o
n a
a s
m ) h deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce uses disparate RDBMS engines to process and transform data
e
• ODI’sd e
E-LT l i
architecture
i n N
faster than traditional ETL. This approach optimizes performance and scalability, and lowers
r
Flo overall solution costs.
• ODI turns the promise of active integration into reality by providing all the key components
that are required to enable real-time data warehousing and operational data hubs. ODI
combines three styles of data integration: data-based, event-based, and service-based. ODI
unifies silos of integration by transforming large volumes of data in batch mode, by
processing events in real time through its advanced Changed Data Capture (CDC),
integrating with Oracle GoldenGate (OGG), and by providing data services to the Oracle
Service-Oriented Architecture (SOA) Suite.
• ODI shortens implementation times with its declarative design approach using a GUI and
drag-and-drop. Designers specify what they want to accomplish with their data, and then the
tool generates the details of how to perform the task. With ODI, the business user or the
developer specifies the rules to apply to the integration processes. The tool automatically
generates data flows and administers correct instructions for the various source and target
systems. With declarative design, the number and complexity of steps is greatly reduced,
which in turn shortens implementation times. Automatic code generation reduces the
learning curve for integration developers and streamlines non-IT professionals to the
definition of integration processes and data formats.

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 14

Oracle Data Integrator: Features
Knowledge Modules
• E-LT architecture provides high performance.
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

• Active integration enables real-time data warehousing and

operational data hubs.
• Declarative design improves developer productivity.
• Knowledge Modules provide flexibility and extensibility.
– Predefined, reusable code templates with built-in e
connectivity to all major datastores: r a bl
s fe
— RDBMS
- t r an
— Flat files n no
Messaging Systems (JMS), and so on s a
—

h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce are at the core of the ODI architecture. They make all ODI processes
e d e
• Knowledge l i
Modules
i n N
modular, flexible, and extensible. Knowledge Modules implement the actual data flows and
r
Flo define the templates for generating code across the multiple systems involved in each
process. ODI provides a comprehensive library of knowledge Modules, which can be tailored
to implement existing best practices (for example, for highest performance, for adhering to
corporate standards, or for specific vertical knowhow). By helping companies capture and
reuse technical expertise and best practices, ODI’s Knowledge Module framework reduces
the cost of ownership. It also enables metadata-driven extensibility of product functionality to
meet the most demanding data integration challenges.
ODI streamlines the high-performance movement and transformation of data between
heterogeneous systems in batch, real-time, synchronous, and asynchronous modes. It
dramatically enhances user productivity with an innovative, modularized design approach and
built-in connectivity to all major databases, data warehouse appliances, analytic applications, and
SOA suites.

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 15

Conventional Integration Process: ETL
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Integration process
Extract - Transform (check) - Load
A machine A machine

Source
Target
ORDERS A machine

Transform
SALES
ble
LINES
fe r a
an s
n t r
Errors
-
a no
CORRECTIONS
File
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
Extract Transformr a
o tud Load
@
cu his S
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ceis also known as an extract, transform, and load (ETL) process.
e d e
This integration l
processi
n N
i first part of an ETL process involves extracting data from the source systems. Most data
F lorThe
warehousing projects consolidate data from different source systems.
The transform stage applies a series of rules or functions to the data extracted from the source to
derive the data for loading into the target. Some data sources will require very little or even no
manipulation of data. In other cases, transformations (such as filtering, joining, sorting, and so on)
may be required to meet the business and technical needs of the target database.
The load phase loads the data into the target, usually the data warehouse.
Note: You can add to this process the checks that ensure the quality of data flow, as shown in the
slide.

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 16

ETL Versus E-LT

1. Extract 1. Extract
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

2. Transform 2. Load
3. Load 3. Transform
Conventional ETL architecture Next-generation E-LT architecture

ble
Transform
fe r a
s
Transform
n
Extract
Transform
Load Extract Load-t
n ra
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u e
Data is e deofl the most
one licimportant assets of any company, and data integration constitutes the
n N
backbone of any enterprise’s IT systems. Choosing the technology for data integration is critical
F lorfori productivity and responsiveness of business divisions within an enterprise.
E-LT stands for extract, load, and transform. It includes the processes that enable companies to
move data from multiple sources, reformat and cleanse the data, and load it into another
database, or a data warehouse for analysis, to support a business process.
ODI provides a strong and reliable integration platform for IT infrastructure. Built on the next-
generation architecture of extract, load, and transform (E-LT), ODI delivers superior performance
and scalability connecting heterogeneous systems at a lower cost than traditional, proprietary ETL
products. Unlike the conventional extract, transform, and load (ETL) design, with ODI, E-LT
architecture extracts data from sources, loads it into an optional Staging Area (work schema), then
a target, and transforms it by using the database power according to business rules. The tool
automatically generates data flows, manages their complexity, and administers correct instructions
for the various source and target systems.

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 17

E-LT Steps

1. Extract: Extracting data from various sources

2. Load: Loading the data into the destination stage/target(s)

3. Transform: Transforming the data according to a set of
business rules

Next-generation E-LT architecture

ble
fe r a
ans
Transform
n - t r
no
Transform

s a
Extract Load
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e
1. Extractd e l i
n N
i first step in the E-LT process is extracting data from various sources. Each of the source
Fl orThe
systems may store its data in a completely different format. The sources are usually flat files or
RDBMS, but almost any data storage can be used as a source for an E-LT process.
2. Load
This step involves loading the data into the Staging Area, which must be an RDBMS; then the
destination target(s), which might be a database or data warehouse.
3. Transform
After the data has been extracted and loaded, the next step is to transform the data according to a
set of business rules. The data transformation may involve various operations including, but not
limited to, filtering data, sorting data, aggregating data, splitting/joining data, cleaning data,
generating calculated data based on existing values, and validating data.

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 18

ODI Architecture
OpenTools for Open Web Services
adding OS for decoupled
functionality to invocation of any
Desktop any E-LT/ETL job WebLogic 12c / Application Server ODI jobs
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

JVM ODI Studio Enterprise Mgr FMW Console

Designer
Cloud Control 12c ODI Plug-in

Java EE Operator
Application Servlet Container Web Service Container
Topology
ODI SDK Java EE Runtime WS
Security Application ODI Public Data
Console Java EE WS Services
ODI SDK Agent

SDK APIs for Java

generation of any ODI
ble
system metadata Data Sources Connection Pool
fe r a
an s
n - t rCommon
Repositories Sources and Targets
a no Admin Model

h a s Runtime WS
ODI Master
Repository ODI Work
m ) deฺ
Repository
ฺ c o u i Stand-alone
Agent
Legacy
e
cl ent
Files / XML DBMS
G Applications
ERP/CRM/PLM/SCM
DW / BI / EPM
Metadata is an open
relational model with r a
o tud
Heterogeneous Big
built-in Flex Fields.
u @
c his S
access Data

e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cecentral component of the ODI architecture. This stores configuration
e d e
The repository forms l i
the
n N
information
i about the IT infrastructure; the metadata for all applications, projects, scenarios, and
F lorexecution logs. Repositories can be installed in an online transaction processing (OLTP) relational
database. The repository also contains information about the ODI infrastructure, defined by the
administrators. The two types of ODI repositories are Master and Work Repositories.
At design time, developers work in a repository to define metadata and business rules. The
resulting processing jobs are executed by the agent, which orchestrates the execution by
leveraging existing systems. The agent connects to available servers and asks them to execute
the code. It then stores all return codes and messages in the repository. The agent also stores
statistics, such as the number of records processed, and the elapsed time. Several repositories
can co-exist in an IT infrastructure. The graphic in this slide shows two repositories: one for the
development environment and the other for the production environment. Developers release their
projects in the form of scenarios that are sent to production.
In production, these scenarios are scheduled and executed on a Scheduler Agent that also stores
all its information in the repository. Operators have access to this information and can monitor the
integration processes in real time.
Business users, as well as developers, administrators, and operators, can gain web-based read
access to the repository by using the ODI Console.

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 19

Summary

In this lesson, you should have learned to:

• Describe the course objectives

• List the agenda of lessons
• Identify the target audience for the course and the
suggested prerequisites
• Describe the course environment e
r a bl
s fe
- t r an
no n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 20

Practice 1-1: Exploring Your Environment

In this practice, you will explore the technical environment for

your practices.

ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 1 - 21

Flori
N n
e d e l c u
l
l
i
o r
ce
ฺ n
(f nse
e l
in to u
u
ed se t
@
r a
c his S
e ฺ c
o tud
o m
cl ent G
h
u
a
i
s a
) deฺ
o n n
- t r an
s
fe r a bl
e
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ODI Knowledge Modules: Overview

ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori
Objectives

After completing this lesson, you should be able to:

• Describe the guidelines for working with Knowledge

Modules (KMs)
• Use ODI substitution API for developing KMs

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 2

Agenda

 ODI Knowledge Modules: Overview

• Working with Knowledge Modules

• Developing Knowledge Modules

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 3

Knowledge Modules: Overview

• A KM is a code template containing the sequence of

commands necessary to implement a data integration task.

• There are different predefined KMs for:
– Loading (LKM)
– Integration (IKM)
– Checking (CKM)
– Reverse engineering (RKM) a b le
s f er
– Journalizing (JKM) for changed data capture (CDC) ran
– Deploying data services (SKM) o n -t
n a
• h a s
All KMs work by generating code to be executed at run
) ฺ
time. om ide ฺc t Gu
l e
o rac uden
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce a template containing the code necessary to implement a
e
Technically,d ethis term l i
describes
n N
particular data integration task for a given technology. These tasks include loading data, checking
F loritifor errors, or setting up triggers necessary to implement journalization. However, all KMs
basically work the same way: ODI uses them to generate code, which is then executed by a
technology at run time. The code should be ANSI standard code, thus portable. And/or, the code
could be technology-specific code (for example, SQL*Loader, which is not portable).
Note: KMs are independent of the structure of the source and target data stores. The same KM
can be used, no matter which source table you have, or how many source tables you have.
Likewise, all target tables can use the same KM.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 4

Global KMs

KMs can be shared across multiple projects in two ways:

• Import as “Global”
• Use the built-in ones (not listed)

Project KMs Global KMs

bl e
fe r a
s
ran
Built-in KMs
n-t
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ceKMs to be shared across multiple projects. In previous versions of ODI,
e d
Global KMs
eallow l i
specific
n
KMs N
were always specific to a project and could be used only within the project into which they
i imported.
F lorwere Global KMs are listed in the Designer Navigator in the Global Objects accordion.
New to 12c is a small collection of built-in KMs. These are the same generic KMs, only they differ
in scope. The list of built-in KMs is not displayed in Studio, it is documented either in the manuals
or by a suffix of .GLOBAL in the name. Even though “global” is in the name, they are not listed in
the Global KMs, shown in the middle of the screen.
Why would you not want all KMs to be global? Perhaps you have modified one or two KMs for
custom code, and they may have the same names as their originals.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 5

KMs in Action

KMs implement the required operations.

DB2/UDB

Source (MySQL) Staging Area Transform and

integrate Target (Oracle)
ORDERS
LKM TEMP_1 IKM SALES

ble
LINES
Extract/Join/
Transform IKM
TEMP_
fe r a
Join/Transform
SALES

an s
n - t r
ERRORS
LKM TEMP_2 CKM
a no
CORRECTIONS Extract/Transform
h a s
Check constraints/
File
) deฺ
Isolate errors
m
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d
KMs define
e
a way of l i
implementing business rules on a given technology. They connect the
N
n logic of your business rules to the concrete reality of your data servers.
abstract
F lori
If your data servers change, you just select a different KM to match. Your business rules remain
unchanged. If your business rules change, you do not have to modify any code—you just update
the SQL expressions that define them in ODI.
In your example, you have Loading KMs (LKMs) that describe how to move data from the MySQL
source server to the DB2/UDB Staging Area. Remember that the Staging Area must be a SQL
RDBMS (for example, not JMS). Integration KMs (IKMs) perform the work of creating temporary
tables and moving data onto the Oracle target server. Lastly, Check KMs (CKMs) implement
constraint checking and isolating errors in a separate table.
Note: KMs are generic because they enable data flows to be generated regardless of the
transformation rules. And they are highly specific because the code they generate and the
integration strategy they implement are finely tuned for a given technology.
Of course the source can be almost anything, it is not restricted to MySQL; and the target can be
almost anything, it is not restricted to Oracle.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 6

Code Generation

Implemented business KM:

rules: Truncate Table <%=odiRef.getTable(« »)%>
Design
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Insert into <%=odiRef.getColList(…)%>

time Inc_net = Round
(Sum(Inc * coeff))
Select <%=odiRef.getFrom()%> …
Begin … If .. Then … Else
ftp ...

plus

Other metadata:
Generated code:
Truncate Table SCOTT.EMP - Topology of your
Insert into SCOTT.EMP servers
ble
Select Round(Sum(Sal*Coeff)) …
r
- Models in servers
fe a
Begin … If .. Then … Else
ftp ... s
- Technology, and
an
so on
n - t r
a no
Sources
h a s
Run
o
Truncate
m Table
i eฺSCOTT.EMP
) intodSCOTT.EMP
ฺc … t Gu
Insert
time e
l
Targets
o racODIuAgentd enorchestrates the
u @ S t of the generated code.
running
el c s
e d e thi
r i n ฺn © 2015,uOracle
s and/or its affiliates. All rights reserved.
o Copyright
t o
u (fl nse
d e lc lice
Business
N e and their implementation on a server are being carefully kept separate. Now you
rules
willnsee how ODI combines the two at run time.
F lori
For example, you have a mapping that the net income of an employee is the sum of the income
components multiplied by their coefficients. This is an expression in SQL, but there is no code to
perform this integration.
You have a KM that can wipe a destination table and fill it with source data. However, it knows
nothing about your business rules.
Then you have all the other metadata that you defined in ODI: the topology of your servers, the
models that exist on them, the technologies used by each server, and so on.
When you put these three things together, ODI generates a code to carry out the integration. This
code is specific to a technology, a layout, and a set of business rules. However, if any of these
three things changes, all you need to do is regenerate the code.
At run time, the ODI Agent orchestrates the running of the generated code. Based on execution
locations, various parts of the generated code will be executed either on the Source, Staging, or
Target.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 7

Types of Knowledge Modules: Overview

There are six types of KMs that you may import:

KM Type Description

LKM Loading Assembles data from source data stores to the Staging
Mappings

Area
IKM Integration Uses a given strategy to populate the target data store from
the Staging Area
bl e
CKM Check Checks data in a data store for errors statically or during an
fe r a
integration process
t r a ns
RKM Reverse Retrieves the structure of a data model from a database.
n o n- It
engineering is needed only for customized reverse engineering. a
Models

JKM Journalizing Sets up a system for Changed )Data a s

h Capture (CDC) to
reduce the amount of dataothat needsid m e ฺ
to be processed
e ฺ c G u
SKM Data a
Deploys data services
r cl thateprovide n t access to data in data
Services stores @ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u e
del of KMslicis critical for doing any work with mappings.
The firstegroup
i N
•n Loading Knowledge Modules (LKMs) extract data from the source of mappings. So, if your
r
lo data is stored in flat files, you will need the “File to SQL” LKM.
F
• Integration Knowledge Modules (IKMs) implement a particular strategy for loading the target
of a mapping. Thus, to load an Oracle table while taking into account the slowly changing
dimensions (SCD) properties, a particular IKM can be used.
• Check Knowledge Modules (CKMs) are selected in mappings and constraints can be
individually enforced by ODI at the mapping level. Note the CKM straddles the two groups.
The second group is used for setting up, checking, and configuring models.
• Check Knowledge Modules (CKMs) enforce constraints defined on the target data store.
CKMs are used by models to perform static checks outside of mappings.
• Reverse Engineering Knowledge Modules (RKMs) are needed only to perform customized
reverse engineering. They are used to recover the structure of a data model and are used
when standard reverse engineering cannot be performed.
• Journalizing Knowledge Modules (JKMs) are used to set up Changed Data Capture. This
makes mappings react only to changes in data and can vastly reduce the amount of data
that needs to be transferred.
• Service Knowledge Module (SKM) is the code template for generating data services.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 8

Agenda

• ODI Knowledge Modules: Overview

 Working with Knowledge Modules

• Developing Knowledge Modules

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 9

How to Select Knowledge Modules

• Which KMs would you need?

– Start with basic SQL KMs.

– Add technology-specific KMs as needed.
– For your mappings, you will need LKMs, IKMs, and CKMs
only.
• Become familiar with the library of available KMs, and what
they can do for you. Examine the ODI documentation: a b le
r e
– Developing KMs with ODI
a n sf
tr
– Connectivity and KMs for ODI
n on-
– Developing Integration Projects with ODI as a
m ) h deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cevarious types of KMs are. But how do you know which ones need to be
You nowe d e
know what l
thei
n N
imported into your specific project?
l o ri
F The most important thing to remember is to import all KMs that may be used in a project. Each
time you create a project, you must import the KMs that will be used by that project.
The basic strategy for importing KMs is as follows:
• First, import the most basic SQL KMs. These work on almost every database management
system (DBMS) with acceptable (but not optimized) performance.
• Then, consider importing more specific KMs for the particular technologies involved in your
project. If you are transferring data to an Oracle server, consider adding the Oracle-based
IKMs.
• Technology-specific KMs can take advantage of certain characteristics or the special tools
provided by the technologies. However, it is often best to begin with the most generic KMs to
start your mapping, then choose specific KMs later to increase performance. Refer to the
following documentation on: http://docs.oracle.com/middleware/1213/odi/docs.htm
- Developing Knowledge Modules with Oracle Data Integrator
- Connectivity and Knowledge Modules Guide for Oracle Data Integrator
(especially Substitution API Reference for methods)
- Developing Integration Projects with Oracle Data Integrator

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 10

Creating New Knowledge Modules

In real practice, there are few cases when you create a KM

from the beginning. To create a new KM:

1. Add new KM of appropriate type.
2. Specify the name and add the details to create the
functionality of your KM based on your project needs.

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u cefew cases when you create a KM from the beginning. Most often you
e d e
In real practice, therel i
are
n
wouldN just edit an existing KM (as shown on the next page)
F lori
To create a new KM:
1. Add a new KM of the appropriate type (LKM, IKM, CKM, and so on).
2. Specify the name, technology (technologies if multi-connection), and add the details (code
template) to create the functionality of your KM based on your project needs and available
technology.
In this case, the new KM is associated with only one project (myBigData), as indicated in the panel
title. The Type is disabled because you are creating a new LKM. The reason you have an option
for both Source and Target technology is because you have multi-connections selected. If that had
not been selected, then only Target technology would have been an option. Tasks (used to be
called Details in 11g) is where the actual code commands go. The Tasks tab is shown in the next
slide.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 11

Developing Knowledge Modules
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

1 2

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
l e t G
3rac e n
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u cecreate a KM from the beginning, but rather develop a KM starting from
In most e del you dolinot
cases,
some
i n Nexisting KM. To develop/edit a KM:
o r
Fl 1. Import a KM of appropriate type, or duplicate an existing one. This creates a “Copy of (old
name)” KM. It does not matter if you do the import from either the main Knowledge Module
node or any of the specific KM nodes below it; any kind of KM can be imported from any kind
of node (for example, you can import a RKM from the CKM node).
2. On the Tasks tab, modify the KM by adding a new functionality according to your project
requirements. The green plus is the Add line icon. The light blue arrows move your new task
line up or down, top or bottom. The individual command line highlighted in dark blue is
shown in detail in the Properties panel below.
3. The commands themselves are for either the Source or Target technology. When your
mouse pointer is near that command field, a gear appears on the right side. Clicking the gear
opens the Expression Editor, which gives you color-coded assistance with entering
commands. The Expression Editor is shown in the next slide.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 12

Editing Knowledge Modules
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
lice right of the slide. Clicking that icon opens the Editor you see here.
delin the bottom
Note theegear
i n N
lo r
Use the Knowledge Module Editor to create customized KMs. Keywords for this technology (in the
F example, Oracle SQL), are shown in one color (orange), literals as indicated by “quotes” in blue,
numbers in green, and so on. Delimiters, such as parentheses ( ), brackets [ ], and APIs <% %>,
highlight matching pairs. The colors change depending on the technology.
As you can see, the KM code can be complex.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 13

Knowledge Module Options

General
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Options

bl e
fe r a
ans
n - t r
a no Line-
Specific
a s
m ) h deฺ Options

ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cEditor
e d
The Knowledge
e l
Modulei also has a section to edit the KM’s options. The top example shows
the N
indefault values for the options for this KM. The bottom example shows the options for a specific
F lortask, in this case the “drop check table” task. For this task, the KM default is to not drop the check
table (shown above), but for this task, the override is to drop the check table (shown below). You
can add or remove options.
KMs are used later in this course. For additional detailed information about KMs, see the ODI
Knowledge Modules Reference Guide.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 14

Description of Knowledge Module Tasks
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce Step, Task. A KM is made of tasks.
• Thee d e
hierarchy isl i
Session,
i•n N
Each task has a name and a template for the code to be generated.
r
Flo • Tasks are listed on the Tasks tab.
• After the execution, the task names will be suffixed with the KM name, and viewable in ODI
Operator for a given session.
• Click the task to view the executed code from the KM.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 15

Details of the Tasks Are Generic
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Source and Target

are not known. Only
Technologies are
known.

Substitution
ble
methods
fe r a
ans
Parameters of the
n - t r
substitution
methods a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce on source and command on target. In the Command window, you can
e
Each taskd e
has a l i
command
n
view N
and modify each command manually.
F lori• Details of the tasks are generic:
The source and target tables are not known, only the technologies are known.
• Substitution methods are the placeholders for the table names and column names.
• Parameters of the substitution methods: Tables or columns

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 16

Setting Knowledge Module Options

1. Open mapping
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

2. Physical tab
3. Select object
4. Select KM of
interest
5. Modify Options e
r a bl
6. Save and Run s fe
- t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u coreOff and is defined on the Options tab of each task of the KM. KMs have
e d
Each option
ecan be l
On i
n N
options that enable users to:
F lori• Override or accept the default
• Turn options on or off (Boolean)
• Specify or modify values used by the KM
Options are defined in the KM editor itself (new with 12c).
Options are used in the KM code with the substitution methods such as:
<%=odiRef.getOption "OptionName")%>
KM options are set on the Options tab of each KM task.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 17

Replacing Existing Knowledge Modules

• “Import replace” mode:

– Updates all mappings that use the KM

– Preserves existing options
– Does not update generated scenarios
• Why replace an existing KM?
– Because a newer version is released
ble
– To undo any undesirable changes made
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ca eKM with the same name as an existing KM, you end up with two copies
e
Normally, d
if
e
you l
import i
inin Nproject.
your This can be useful if you want to make changes to the KM without breaking your
F lorexisting mappings.
However, another mode of importing is available: the “import replace” mode. In this mode, the
existing mapping is replaced with the version imported from the disk. All mappings that used the
old KM are automatically updated to use the new version. Also, values that are set for the KM
options in these mappings are transferred to the new version. However, any existing scenarios are
not regenerated. If you want to incorporate changes to the KM into your scenarios, you must
regenerate them.
There are various reasons why you would want to replace a KM. The most common reason is that
a newer version of the KM is released, perhaps by someone on your team. You import the KM
again and all your existing mappings will still work.
Similarly, you may have made some undesirable changes to your KM. You can quickly undo these
changes by re-importing from a saved version.
Note: Any mapping that uses the replaced KM also will be impacted.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 18

Replacing Existing Knowledge Modules

1. Right-click the KM.

2. Select Import Replace.

3. Select the replacement file.

ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce mode, perform the following:
e
To import d
a
e
KM in l i
replacement
n N
Fl ori1. Right-click the name of the KM that is to be replaced.
2. Select Import Replace from the context menu.
3. Enter the name of the XML file that contains the new version of the KM. You can use the
browse button to select a file. Note that you can replace only one KM at a time.
Consider regenerating any scenarios that use the KM to take the changes into account. This
includes both the mappings and packages. Regenerating scenarios is discussed later in the
course.
Note: You should regenerate any scenarios that use the KM. “Regenerate” keeps the version
number; do you want that or do you want a new version number? This will be discussed in the
lesson titled “Managing Object Versions.”

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 19

Agenda

• ODI Knowledge Modules: Overview

• Working with Knowledge Modules

 Developing Knowledge Modules

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 20

Developing Your Own Knowledge Module:
General Guidelines
• Very few KMs are ever created from a blank start. They
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

are usually extensions or modifications of existing KMs.

• Duplicate existing tasks and modify them. This minimizes
typos in the syntax of the odiRef methods.
• All mappings using the KM inherit the new behavior
(unless compiled into a scenario). Remember to make a
copy of the KM if you do not want to alter existing a b le
mappings. Modify the copy, not the original. s f er
- t r an
• on way
Modifying a KM that is already used is a very efficient
n
to implement modifications in the data flow s a affect all
and
h a
the existing developments. m) deฺ c o u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce when developing your own KM is to avoid starting from the beginning.
e
One of the d e
main l i
guidelines
ODI n N
provides more than 100 KMs. Take a look at these existing KMs, even if they are not written
F lorfori your technology. The more examples you have, the faster you develop your own code. You
can, for example, duplicate an existing KM and start enhancing it by changing its technology, or
copying lines of code from another KM.
To speed up development, duplicate existing tasks and modify them. This reduces typos in the
syntax of the odiRef methods.
When developing your own KM, remember that it is targeted to a particular stage of the integration
process. As a reminder:
• LKMs are designed to load remote source data sets to the Staging Area (into “C$” tables).
• IKMs apply the source flow from the Staging Area to the target. They start from the “C$”
tables, may transform and join them into a single “I$” table, may call a CKM to perform data
quality checks on this “I$” table, and finally write the flow data to the target.
• CKMs check data quality in a data store or a flow table (“I$”) against data quality rules
expressed as constraints. The rejected records are stored in the error table (“E$”).

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 21

Substitution API

• Code entered in a KM:

CREATE TABLE<%=odiRef.getTable("L", "INT_NAME",

"A")%> (<%=odiRef.getColList("", "\t[COL_NAME]
[DEST_CRE_DT]", ",\n", "", "")%>)

• Generated code for the table: bl e

fe r a
tra ns
CREATE TABLE db_staging.I$_PRODUCT
n -
( P RODUCT_ID numeric(10),
a no
PRODUCT_NAME varchar(250),
h a s
FAMILY_ID numeric(4), m ) deฺ
varchar(13),leฺc
o
G ui
SKU
r a c ent
LAST_DATE o
timestamp ) ud u @ St
e l c h is
e d e t
i n ฺ n u s
l o r
Copyright ©
t o
2015, Oracle and/or its affiliates. All rights reserved.

u (f nse
c lice by using the ODI substitution API. The API methods are Java
del as templates
KMs areewritten
n N that return a string value. They all belong to a single object instance named “odiRef”.
methods
i
r
loThe same method may return different values depending on the type of KM that invokes it. The
F following example illustrates how you would write a CREATE TABLE statement in a KM and what it
would generate. The following code is entered in a KM:
CREATE TABLE <%=odiRef.getTable("L", "INT_NAME",
"A")%>(<%=odiRef.getColList("", "\t[COL_NAME] [DEST_CRE_DT]", ",\n",
"", "")%>)
The generated code for the PRODUCT table is:
CREATE TABLE db_staging.I$_PRODUCT( P RODUCT_ID numeric(10),
PRODUCT_NAME varchar(250), FAMILY_ID numeric(4), SKU varchar(13),
LAST_DATE timestamp)
The generated code for the CUSTOMER table is:
CREATE TABLE db_staging.I$_CUSTOMER( CUST_ID numeric(10), CUST_NAME
varchar(250), ADDRESS varchar(250), CITY varchar(50), ZIP_CODE
varchar(12), COUNTRY_ID varchar(3))
After being executed with appropriate metadata, the KM has generated a different code for the
PRODUCT and CUSTOMER tables.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 22

Using Substitution Methods

• These methods are used to write generic code, such as:

– Table names in a context-dependent format

– Information about the session
– Source and target metadata information
• Use the method with the following syntax:

<%=odiRef.method_name(parameters)%> bl e
fe r a
• tra
Refer to the Knowledge Module Developer’s Guide-for
ns
o n
Oracle Data Integrator. an s
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cedirect calls to ODI methods implemented in Java. These methods are
e d
Substitution
e
methods l i
are
used
i n N
to generate text that corresponds to the metadata stored in the ODI repository or session
F lorinformation.
If you write an expression to express a business rule, you do not generally know in which context
this expression is run (not hard-coding schema names, host names, and so on). For instance, if
you refer to a specific schema by its name in your test setup, it may not work in your production
environment. But you want to be able to specify your table in a generic way. Substitution methods
enable you to do this. You can specify the table name with a substitution method. At run time, ODI
automatically adds the name of the appropriate physical schema for the context. You can also use
substitution methods to retrieve information about the current session. For example, you can
include the time when the session was launched, its name, or the code for the current context.
Similarly, substitution methods give you access to the metadata about the source and target of
your mapping. The general syntax to use a substitution method is as follows:
<%=odiRef.method_name(parameters)%>
Angle brackets with percentage signs enclose all substitution method calls. The equal sign tells
ODI to replace the tags and everything inside with the result of the method call.
odiRef is a special ODI Java object, which contains all the methods for performing substitution.
You can obtain more information about these methods in Knowledge Module Developer’s Guide
for ODI.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 23

Various ODI Substitution Methods

• Global methods: These methods can be used in any

situation.
– They can be used in any KM or in actions.
• Methods that can be used specifically in KMs of a certain
type:
– Journalizing KM methods
– Loading KM methods ble
fe r a
– Integration KM methods ans
n - t r
– Check KM methods no
a
– Reverse-Engineering KM methods
) has eฺ
– Service KM methods om ฺ c u id
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cebe categorized depending on the type of KM in which they can be used.
e d
Substitution
emethods l i
can
i n N
Global methods are the methods that can be used in any situation (in all KMs and in actions). In
l o raddition to the methods from the “Global Methods” list, some methods can be used specifically in:
F
• Journalizing Knowledge Modules (JKMs)
• Loading Knowledge Modules (LKMs)
• Integration Knowledge Modules (IKMs)
• Check Knowledge Modules (CKMs)
• Reverse-Engineering Knowledge Modules (RKMs)
• Service Knowledge Modules (SKMs)
For details on using each substitution method, refer to Appendix A, “Substitution API Reference” in
Developing Knowledge Modules with Oracle Data Integrator.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 24

Substitution Methods: Basic Examples

• Mapping a target column to its default value defined in

ODI:
• '<%=odiRef.getColDefaultValue()%>'

• Mapping a column by using the system date:

'The year is: <%=odiRef.getSysDate("yyyy")%>'
bl e
fe r a
• Writing a generic SELECT subquery in a filter: ans
n - t r
nFROM o
ORDER_DATE >=(SELECT MAX(ORDER_DATE)-7
s a
) h a ฺ
<%=odiRef.getObjectName("SRC_ORDERS")%>)
ฺ c om uide
a c le nt G
@ or tude
e l cu his S
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
chow
e d e
These examples l
show i to use substitution methods. First, you look at how to access the
N
in value,
default value of a column. In ODI, you can specify a default value for each column. To access this
F lordefault use the getColDefaultValue substitution method. If your column is a string, you
must enclose the entire method call within single quotation marks. If the default value for this
column is unknown, the result of this expression is the “unknown” string with single quotation
marks.
You might want to record the system date into a “last updated” column. Normally, you use a
function available in your database engine. What if your target is not a database, but a flat file? In
this case, you can use the getSysDate substitution method. The system date on the machine
where the agent is running is used. To call this function, pass one argument, which is the date
format to be used. By passing “yyyy,” a four-digit date is returned. Thus, the final string is
something like “The year is: 2014.”
The next example shows how to use a SELECT statement in a filter. For example, you want to
import orders from the ORDERS table. You are retrieving only those orders processed in the last
week. To do this, you create a subquery that retrieves the maximum order date minus seven days
from the ORDERS table. However, the schema containing the ORDERS table may change on
different servers. Therefore, use the getObjectName substitution method to refer to the table
name. The generated code will contain the qualified name of the table. Thus, the final filter
expression refers to something like MSSQL_ORDERS_PROD.SRC_ORDERS.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 25

Using Substitution Methods in Actions

Action corresponds to a DDL operation to generate a procedure for

implementing database changes performed in the ODI model.

• Action Lines Code: Statements valid for the technology
– Example: Dropping and checking a constraint on a table:

ALTER TABLE <%=odiRef.getTable("L", "TARG_NAME",

"A") %> DROP CONSTRAINT
a b le
<%=odiRef.getCK("COND_NAME") %> “L”=Local, fer s
- t r an
“R”=Remote

• Actions Calls Method: n no

s a
<% odiRef.method_name(); %> h
) deฺ a
c o m u i
e ฺ G
r a cl entNote the
Note no
equal sign. @ o tud semi-colon.
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ctoea DDL operation (CREATE TABLE, DROP REFERENCE, and so on) used
d e l i
Ne a procedure to implement in a database the changes performed in a data integrator
An action corresponds
toin
generate
Fl ormodel (GENERATE DDL operation). Each action contains several action lines, corresponding to the
commands required to perform the DDL operation (for example, dropping a table requires
dropping all its constraints first).
Action lines contain statements valid for the technology of the action group. Unlike procedures or
KM commands, these statements use a single connection (SELECT ... INSERT statements are
not possible).
In the style of the KMs, actions make use of the substitution methods to make their DDL code
generic. Action call methods are usable in the action lines only. Unlike other substitution methods,
they are not used to generate text, but to generate actions appropriate for the context. For
example, to perform a DROP TABLE DDL operation, you must first drop all foreign keys referring to
the table. In the DROP TABLE action, the first action line will use the dropReferringFKs()
action call method to automatically generate a Drop Foreign Key action for each foreign key of the
current table. This call is performed by creating an action line with the following code: <%
odiRef.dropReferringFKs(); %>. The syntax for calling the action call methods is: <%
odiRef.method_name(); %>.
Note: The action call methods must be alone in an action line, must be called without a preceding
“=” sign, and require a trailing semi-colon.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 26

The following action call methods are available for Actions:
• addAKs(): Call the Add Alternate Key action for all alternate keys of the current table.
• dropAKs(): Call the Drop Alternate Key action for all alternate keys of the current table.
• addPK(): Call the Add Primary Key for the primary key of the current table.
• dropPK(): Call the Drop Primary Key for the primary key of the current table.
• createTable(): Call the Create Table action for the current table.
• dropTable(): Call the Drop Table action for the current table.
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

• addFKs(): Call the Add Foreign Key action for all the foreign keys of the current table.
• dropFKs(): Call the Drop Foreign Key action for all the foreign keys of the current table.
• enableFKs(): Call the Enable Foreign Key action for all the foreign keys of the current
table.
• disableFKs(): Call the Disable Foreign Key action for all the foreign keys of the current
table.
• addReferringFKs(): Call the Add Foreign Key action for all the foreign keys pointing to
the current table.
• dropReferringFKs(): Call the Drop Foreign Key action for all the foreign keys pointing to ble
the current table. fe r a
ans
n - t r
• enableReferringFKs(): Call the Enable Foreign Key action for all the foreign keys
pointing to the current table.
no
a
• disableReferringFKs(): Call the Disable Foreign Key action for all the foreign keys
s
pointing to the current table. h a
) deฺ
c o m
• addChecks(): Call the Add Check Constraint action for all check constraints of the current
u i
table. e ฺ G
a cl ent
• dropChecks(): Call the Drop Check Constraint action for all check constraints of the
r
current table. @ o tud
l c his S
u
• addIndexes(): Call the Add Index action for all the indexes of the current table.
e
ฺ n ed se t
• dropIndexes(): Call the Drop Index action for all the indexes of the current table.
r in to u
• modifyTableComment(): Call the Modify Table Comment for the current table.
l o
(f nse
• AddColumnsComment(): Call the Modify Column Comment for all the columns of the
u
l c
current table.
e l i ce
e d
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 27

Working with Object Names

When working in Designer, you do not specify physical

information like the database name or schema name because

they may change depending on the execution context. The
physical information will be provided by ODI at execution time.
• The correct physical information will be provided by Oracle
at execution time.
• The substitution API has methods that calculate the fully a b le
qualified name of an object or data store taking into s f er
account the context at run time. - t r an
n on
• For example, for object name: sa ha eฺ
)
getObjectName("L", "OBJ_NAME",
ฺ c om "D") u id
e
cl ent G
r a
o tud
“L”=Local, “W”=Work,

c u @
“R”=Remote S “D”=Data
l i s
n e de e th
i nฺ © t2015, s and/or its affiliates. All rights reserved.
uOracle
o r
Copyright
o
u (fl nse
d e lc lice
e in Designer, you should avoid specifying physical information such as the database
When working
N
in orphysical
name
lorcorrect
schema name because they may change depending on the execution context. The
information will be provided by ODI at execution time by using substitution
F methods.
This API is overloaded in that there are half a dozen possible parameter syntaxes you could use,
each with its own set of parameter types and number of parameters. For example, should it expect
and return a partition name as part of the full name? That depends on the context.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 28

Working with Object Names

Qualified name
Method Usable In
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

required
Any object named
OBJ_NAME getObjectName("L", "OBJ_NAME", "D") Anywhere
The target data LKM, CKM, IKM,
store of the current getTable("L", "TARG_NAME", "A")
mapping JKM
The integration (I$)
table of the current getTable("L", "INT_NAME", "A") LKM, IKM
mapping.
bl e
The loading table
(C$) for the current LKM fe r a
loading phase.
getTable("L", "COLL_NAME", "A")
an s
The error table (E$) n - t r
for the data store
being checked.
getTable("L", "ERR_NAME", "A")
a noCKM, IKM
LKM,

h a s
The data store
being checked getTable("L", "CT_NAME", "A")
m ) deCKM ฺ
ฺ c o u i
The data store
l e t G
referenced by a
foreign key rac uden "A") CKM
getTable("L", "FK_PK_TABLE_NAME",
o
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cemethods that calculate the fully qualified name of an object or data store
e d e
The substitution API l i
has
n N
taking into account the context at run time. These methods are listed in the table in this slide.
F lori
Option choices:
• “L”=Local (for example, schema.table) versus “R”=Remote (for example,
schema.table@hostname)
• “D”=Data versus “W”=Work schema versus “A”=Automatic. It enables ODI to determine
which physical schema to use (the Data schema [“D”] or the Staging (work) schema [“W”]).
• The auto-generated names are C$_0name, C$_1name, C$_2name, and so on, depending
on how many sources there are, where “name” is the target table name.
• In 12c, GUIDs also assigned to keep names unique.
• “CT_NAME” is Constraint Name.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 29

Working with Lists of Tables, Columns,
and Expressions
Generating code from a list of items often requires a “while” or
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

“for” loop. ODI provides methods that help you generate code
based on lists. These methods return a string and accept at
least four parameters:
• Start
• Pattern
• Separator a b le
s f er
• End
- t r an
• Additional optional parameter: n on
s a
Selector h a ) deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
clist
e d
Generating
e
code froml ia of items often requires a “while” or “for” loop. Oracle Data Integrator
N
in methods
addresses this issue by providing powerful methods that help you generate code based on lists.
F lorThese act as “iterators” to which you provide a substitution mask or pattern and a
separator, and they return a single string with all patterns resolved separated by the separator.
All of them return a string and accept at least these four parameters:
• Start: A string used to start the resulting string, for example: “CREATE TABLE”
• Pattern: A substitution mask with attributes that will be bound to the values of each item of
the list
• Separator: A string used to separate each substituted pattern from the following one, for
example, a comma
• End: A string appended to the end of the resulting string, for example, a closing parenthesis
Some of them accept an additional parameter (the Selector) that acts as a filter to retrieve only
part of the items of the list. For example, list only the mapped column of the target data store of a
mapping.
Some of these methods are summarized in the tables on the following pages.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 30

Working with Lists of Tables, Columns,
and Expressions
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Method Description Usable In

Returns a list of columns and expressions that
LKM, CKM, IKM, JKM,
getColList() need to be executed in the context; accepts a
SKM
"selector" as a fifth parameter for filtering

getTargetColList() Returns the list of columns in the target data LKM, CKM, IKM, JKM,
store SKM
Returns the list of columns defined for an
CKM, SKM
getAKColList()
alternate key
ble
getPKColList() Returns the list of columns in a primary key CKM, SKM fe r a
ans
getFKColList() Returns the list of referencing columns and
referenced columns of the current foreign key
CKM, SKM
n - t r
getSrcTablesList() Returns the list of source tables of a mapping a no LKM, IKM
Returns the list of filter expressions in a) h
as ฺ
getFilterList()
mapping o m uidLKM, e IKM
l c
eฺ nt G
c
ra ude
@ St o
l c u is
e d e t h
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ce
Note e d e l i
n N
ri• Instead of the getPKColList() method, you can alternatively use getColList with the
Flo selector parameter set to “PK.”
• Instead of getSrcTablesList(), whenever possible use the getFrom method. The
getFrom method is discussed later in this lesson.
• Instead of getFilterList(), use the getFilter() method, which is more appropriate
in most cases.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 31

Working with Lists of Tables, Columns,
and Expressions
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Method Description Usable In

getJoinList() Returns the list of join expressions in a mapping LKM, IKM
Returns the list of expressions that should appear
getGrpByList() in the group by clause when aggregate functions LKM, IKM
are detected in the mappings of a mapping
Returns the list of expressions that should appear
getHavingList() in the having clause when aggregate functions LKM, IKM
are detected in the filters of a mapping ble
Returns a list of subscribers JKM fe r a
getSubscriberList()
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
Note e d e l i
•n
NInstead of the getJoinList() method, you can alternatively use the getJoin method,
F lori which is usually more appropriate.
• Instead of getGrpByList(), whenever possible use the getGrpBy method.
• Instead of getHavingList(), use the getHaving method, which is more appropriate in
most cases.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 32

Example: Using getTargetColList
to Create a Table
How to use a column list to create a table:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

• KM code:
Create table MYTABLE
<%=odiRef.getTargetColList("(\n",
"\t[COL_NAME] [DEST_WRI_DT]", ",\n", "\n)")%>
• Generates the following formatted statement: e
r a bl
Create table MYTABLE (
ns fe
CUST_ID numeric(10),
t r a
CUST_NAME varchar(250),
n on-
ADDRESS varchar(250), NoteNote
thethe
s
carriage a
carriage
returns
returns
froma
) h"\n"dparms.
from
the the
CITY varchar(50),
m
optional
“\n”
o
parms.
i eฺ
ZIP_CODE varchar(12),
l e ฺc t Gu
COUNTRY_ID varchar(3)
o rac uden
) u@ St lc this
e d e
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ce
e d e
In this example: l i
n N
lori return (\n).
• Start is set to "(\n": The generated code will start with a parenthesis followed by a carriage
F
• Pattern is set to "\t[COL_NAME] [DEST_WRI_DT]": The generated code will loop over
every target column and generate a tab character (\t) followed by the column name
([COL_NAME]), a white space, and the destination writable data type ([DEST_WRI_DT]).
DEST_WRI_DT is a variable, not a keyword, for example, NUMERIC, VARCHAR, CLOB, and so
on.
• The Separator is set to ",\n": Each generated pattern will be separated from the next one
with a comma (,) and a carriage return (\n).
• End is set to "\n)": The generated code will end with a carriage return (\n) followed by a
parenthesis.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 33

Example: Using getColList
in an Insert Values Statement
The following example shows how to use column listing to
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

insert values into a table:

• KM code:
insert into MYTABLE ( <%=odiRef.getColList("",
"[COL_NAME]", ", ", "\n", "INS AND NOT TARG")%>
<%=odiRef.getColList(",", "[COL_NAME]", ", ", "", "INS
AND TARG")%> ) Values ( <%=odiRef.getColList("",
a b le
Note
":[COL_NAME]", ", ", "\n", "INS AND NOT TARG")%>
s f er
the
colon.
<%=odiRef.getColList(", ", "[EXPRESSION]", ", ", "",
- t r an
"INS AND TARG")%> ) on n
• Generates the following statement: s a
h a
) deADDRESS, ฺ
insert into MYTABLE ( CUST_ID, CUST_NAME,
c o m u )i
CITY, COUNTRY_ID , ZIP_CODE,leฺLAST_UPDATE G
Note
the r ac d:ADDRESS, e n t
Values ( :CUST_ID, :CUST_NAME, o t u :CITY,
u@ current_timestamp
colon.
:COUNTRY_ID, 'ZZ2345',
l c i s S )

n e de e th
i nฺ © t2015, s and/or its affiliates. All rights reserved.
uOracle
o r
Copyright
o
u (fl nse
d e lc the values
l i ce that need to be inserted into MYTABLE are either bind variables with
e
In this example,
N
then same name as the target columns or constant expressions if they are executed on the target.
F lorToi obtain these two distinct set of items, the list is split using the Selector parameter:
• "INS AND NOT TARG": First, generate a comma-separated list of columns ([COL_NAME])
mapped to bind variables in the “value” part of the statement (:[COL_NAME]). Filter them to
get only the ones that are flagged to be part of the INSERT statement and that are not
executed on the target.
• "INS AND TARG": Then generate a comma-separated list of columns ([COL_NAME])
corresponding to expression ([EXPRESSION]) that are flagged to be part of the INSERT
statement and that are executed on the target. The list should start with a comma if any
items are found.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 34

Example: Using getSrcTablesList

The following example concatenates the list of the source

tables of a mapping for logging purposes:

• KM code:
insert into MYLOGTABLE ( INTERFACE_NAME, DATE_LOADED,
SOURCE_TABLES ) values ( '<%=odiRef.getPop("POP_NAME")%>',
current_date, '' <%=odiRef.getSrcTablesList("|| ",
"'[RES_NAME]'", " || ',' || ", "")%> )
bl e
fe r a
ans
• Generates the following statement: n - t r
a no
insert into MYLOGTABLE ( INTERFACE_NAME,
h a sDATE_LOADED,
SOURCE_TABLES ) values ( 'Int. CUSTOMER', m ) deฺ
ฺ c o u i
current_date, '' || 'SRC_CUST' e || ','
cl'C$0_CUSTOMER'n t G ||
'AGE_RANGE_FILE' || ',' r||
o a d e )
u@ is S t u
l c
n e de e th
i nฺ © t2015, s and/or its affiliates. All rights reserved.
uOracle
o r
Copyright
o
u (fl nse
d e lc getSrcTablesList
l i ce
N e
In this example, generates a message containing the list of resource names
in as sourcesoperator
used
lorconcatenation
in the mapping to append to MYLOGTABLE. The separator used is composed of a
(||) followed by a comma enclosed by quotation marks (',') followed by
F the same operator again. When the table list is empty, the SOURCE_TABLES column of
MYLOGTABLE will be mapped to an empty string ('').

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 35

Generating the Source Select Statement

Method Description Usable In

Returns the FROM clause of a SELECT statement with the

getFrom() LKM, IKM
appropriate source tables, left, right, and full outer joins
getFilter() Returns filter expressions separated by an “AND” operator LKM, IKM
Returns the special journal filter expressions for the
getJrnFilter() journalized source data store. This method should be LKM, IKM
used with the Changed Data Capture (CDC) framework.
Returns the GROUP BY clause when aggregation
ble
getGrpBy()
functions are detected in the mappings
LKM, IKM
fe r a
Returns the HAVING clause when aggregation functions
t r a ns
- IKM
non
getHaving() are detected in filters. The having clause includes all filterLKM,
expressions containing aggregation functions.
s a
a
) h deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e
getFrom()d emethod l i
n N
lori
• The FROM clause is built accordingly with the appropriate keywords (INNER, LEFT, and so
F on) and parentheses when supported by the technology.
• When used in an LKM, this method returns the FROM clause because it should be executed
by the source server.
• When used in an IKM, it returns the FROM clause because it should be executed by the
Staging Area server.
getFilter method
• When used in an LKM, it returns the filter clause because it should be executed by the
source server.
• When used in an IKM, it returns the filter clause because it should be executed by the
Staging Area server.
getGrpBy() method
• The GROUP BY clause includes all mapping expressions referencing columns that do not
contain aggregation functions.
getHaving() method
• SQL HAVING clause as opposed to WHERE clause

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 36

Generating the Source Select Statement: Example

• To obtain the result set from any SQL source server, use
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

the following SELECT statement in your LKM:

select <%=odiRef.getPop("DISTINCT_ROWS")%> <%=odiRef.getColList("",
"[EXPRESSION]\t[ALIAS_SEP] [CX_COL_NAME]", ",\n\t", "", "")%> from
<%=odiRef.getFrom()%> where (1=1) <%=odiRef.getFilter()%>
<%=odiRef.getJrnFilter()%> <%=odiRef.getJoin()%>
<%=odiRef.getGrpBy()%> <%=odiRef.getHaving()%

ble
• To obtain the result set from any SQL Staging Area server fe r a
n s
to build your final flow data, use the following SELECT
n - tra
statement in your IKM:
a no
h a s
)REW")%> eฺfrom
select <%=odiRef.getPop("DISTINCT_ROWS")%> <%=odiRef.getColList("",
"[EXPRESSION]", ",\n\t", "", "(not TRG) oand m i d
<%=odiRef.getFrom()%> where (1=1) <%=odiRef.getJoin()%> l e ฺc t Gu
r
<%=odiRef.getFilter()%> <%=odiRef.getJrnFilter()%>
o ac den
c u @ Stu
<%=odiRef.getGrpBy()%> <%=odiRef.getHaving()%>

d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce of code you use to obtain the result set from any SQL RDBMS
e
This slide d e
shows you l i
examples
n N
source server and any SQL RDBMS Staging Area server to build your final flow data.
F lori
Note that the getColList is filtered to retrieve only expressions that are not executed on the
target and that are mapped to writable columns. Because all filters and joins start with an AND, the
WHERE clause of the SELECT statement starts with a condition that is always true (1=1).

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 37

Working with Datasets

• Each dataset represents a group of tables and mappings.

• During loading, the LKM always works on one dataset.

• Datasets are merged into the target data store using set-
based operators (UNION, INTERSECT, and so on) at the
integration phase, when certain odiRef APIs are called.
• Example of dataset merging code:
ble
fe r a
<%for (int i=0; i < odiRef.getDataSetCount(); i++){%>
ans
<%=odiRef.getDataSet(i, "Operator")%> select
n - t r
no
<%=odiRef.getUserExit("OPTIMIZER_HINT")%>
<%=odiRef.getPop("DISTINCT_ROWS")%> <%=odiRef.getColList(i,"",
s a
"[EXPRESSION]", ",\n\t", "", "(((INS or UPD) and !TRG) and REW)")%>,
<% if (odiRef.getDataSet(i, "HAS_JRN").equals("1")) { %> JRN_FLAG h
) deฺ a
IND_UPDATE <%} else {%> 'I' IND_UPDATE <%}%> from
c o m u i
e ฺ G
r a cl ent
<%=odiRef.getFrom(i)%> where (1=1) <%=odiRef.getJoin(i)%>

@ o tud
<%=odiRef.getFilter(i)%> <%=odiRef.getJrnFilter(i)%>
<%=odiRef.getGrpBy(i)%> <%=odiRef.getHaving(i)%> <%}%>

e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cEarlier
e d e
ODI supports l
datasets.i versions, such as 11g, required the use of datasets, 12c does not.
EachN
in dataset represents a group of joined and filtered sources tables, with their mappings.
F lorDatasets are merged into the target data store by using set-based operators (UNION, INTERSECT,
and so on) at the integration phase.
During the loading phase, the LKM always works on one dataset. During the integration phase,
when all datasets need to merged, certain odiRef APIs that support working on a specific dataset
are called using an index that identifies the dataset. The example in this slide explains how this
dataset merging is done. A Java For loop iterates over the datasets. The number of datasets is
retrieved by using the getDataSetCount method.
For each dataset, a SELECT statement is issued, each statement separated from the previous one
by the dataset’s set-based operator retrieved by using the getDataSet method. The SELECT
statement is built as in Generating the Source Select Statement (see previous slide), except that
each method call is parameterized with i, the index of the dataset being processed. For example,
getFrom(i) generates the FROM statement for the dataset identified by the value of i. All the
methods that support a parameter for the dataset index also support a syntax without this index
value. Outside an IKM, these methods should be used without the dataset index. Within an IKM, if
used without the dataset index, these methods address the first dataset.
All the methods that support a parameter for the dataset index also support mappings and KMs.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 38

Obtaining Other Information with the API

The following methods provide additional information:

Method Description Usable In

Returns information about the
getPop() LKM, CKM, IKM, JKM
current mapping
Returns information about the
getInfo() Any procedure or KM
current running session
Returns the value of a particular
getOption()
option
Any procedure or KM
ble
fe r a
getFlexFieldValue() Returns information about a
Any procedure or KM
ans
flexfield value
n - t r
getJrnInfo() Returns information about the CDC
framework
LKM, IKM, CKM
a no
h a s
getTargetTable() Returns information about the
target table of a mapping
o m ) LKM, IKM,
i d eฺCKM
Returns information aboutl e ฺc t G
the
u
raacreverse- n RKM
getModel() current model during
o d e
engineering@
u process
S tu
el c s
e d e thi
r i n ฺn © 2015,uOracle
s and/or its affiliates. All rights reserved.
o Copyright
t o
u (fl nse
lc lice
de“Populated.”
“Pop” isefor
i n N
lo r
This slide shows methods that provide additional information, which may be useful. Note that for
F getFlexFieldValue() with the “List” methods, flexfield values can be specified as part of the
pattern parameter.
Flexfields are not modified by Repository upgrades.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 39

Quiz

You should never duplicate an existing KM for developing a

new one.
a. True
b. False

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
Answer:ebd
el lice
i n N
lo r
Explanation: Duplicating an existing KM and enhancing it by changing its technology, or copying
F lines of code from another KM is a regular practice for developing a KM. Note that some of the
built-in KMs cannot be duplicated.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 40

Summary

In this lesson, you should have learned to:

• Describe the guidelines for working with Knowledge

Modules (KMs)
• Use ODI substitution odiRef API for developing KMs

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 41

Practice 2-1

There is no practice for Lesson 2.

Oracle Data Integrator 12c: Advanced Integration and Development 2 - 42

Developing Knowledge Modules

After completing this lesson, you should be able to:

• Describe best practices and techniques of developing and

enhancing Knowledge Modules
• Use ODI substitution methods with Java in KMs
• Use various ODI tags in KMs
• Troubleshoot KMs e
r a bl
s fe
- t r an
no n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 2

Agenda

 Developing Knowledge Modules: Best Practices

• Using Java in KMs

• Using Substitution Methods: Code Examples
• Troubleshooting Knowledge Modules

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 3

Developing KMs: Targeting a Particular Stage
of your Integration Process
1. When developing your own KM, remember that it is
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

targeted at a particular stage of the integration process:

– LKMs load remote source data sets to the Staging Area.
– IKMs apply the source flow from Staging Area to target.
– CKMs, RKMs, JKMs
Optional Host
Source
Transform and
ble
Staging Area integrate
Target
fe r a
ORDERS
LKM TEMP_1 IKM SALES
an s
n - t r
LINES
Extract/Join/
Transform IKM
TEMP_
SALES a no
Join/Transform
h a s
m ) CKMdeฺ ERRORS
LKM TEMP_2
ฺ c o u i
e
cl ent G
CORRECTIONS
File
Extract/Transform
r a
o tud
Check constraints/
Isolate errors

@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce of KMs out of the box. It is recommended that you start by reviewing
e d
ODI provides
e a large l i
number
n
the N
existing
i customize KMs and start from an existing KM that is close to your use case. Duplicate this KM
F lorand it by editing the code.
When developing your own KM, remember that it is targeted at a particular stage of the integration
process. As a reminder:
• LKMs are designed to load remote source data sets to the Staging Area into Loading (“C$”)
tables. LKMs is plural because it always implies multiple data servers.
• IKMs apply the source flow from the Staging Area to the target. They:
- Start from the Loading tables
- May transform and join them into a single integration (“I$”) table
- May call a CKM to perform data quality checks on this integration table
- Finally write the flow data to the target

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 4

Developing KMs: Targeting a Particular Stage
of your Integration Process
– LKMs, IKMs
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

– CKMs check data quality against data quality rules.

– RKMs extract metadata from a provider to ODI repository.
– JKMs are in charge of creating and managing CDC.

Optional Host
Source
Transform and
ble
Staging Area integrate
Target
fe r a
ORDERS
LKM TEMP_1 IKM SALES
an s
n - t r
LINES
Extract/Join/
Transform IKM
TEMP_
SALES a no
Join/Transform
h a s
m ) CKMdeฺ ERRORS
LKM TEMP_2
ฺ c o u i
e
cl ent G
CORRECTIONS
File
Extract/Transform
r a
o tud
Check constraints/
Isolate errors

@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
l data e
cquality
e
• CKMsd echeck l i in a data store or an integration table (“I$”) against data quality
N
in rules expressed as constraints. ODI will not override active DB constraints. You can,
r
Flo however, relax DB constraints. The rejected records are stored in the error table (“E$”).
• If Static=True and Flow=False, then CKM is triggered AFTER the IKM loads the data,
so you will potentially insert/update bad data.
• If Static=True and Flow=True, then it does NOT insert/update bad data, but rather it
diverts those rows into an error table named E$_targ_name.
• So Static does not remove rows, only flags them. Flow DOES remove (really “diverts”)
the rows.
• RKMs are in charge of extracting metadata (reverse-engineer) from a metadata provider to
the ODI repository by using the SNP_REV_xx temporary tables.
• JKMs are in charge of creating and managing the Change Data Capture (CDC)
infrastructure via journaling. JKMs are used with Oracle GoldenGate (OGG) integration. This
is covered in more detail in the lesson titled Choosing Integration Strategies: Best Practices.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 5

Developing KMs: What to Avoid and What to Use

2. Try to avoid: Instead, you should use:

– Creating too many KMs – Fewer KMs, with parameters

– Using hard-coded values – Substitution methods
– Using too many variables – Options or flexfields to gather
in KMs information from the designer
– Writing the KM entirely in – SQL (easier to read, maintain,
a b le
r
Jython, Groovy, or Java and debug) sfe an
– Using <%if%> -tr for
– Boolean check boxooptions
n
statements for conditional conditional codea n
code generation h a s
) ฺ
ฺ c om uide
a c le nt G
@ or tude
e l cu his S
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d
Avoid these
e
common l i
pitfalls:
n N
lori CKM, RKM, and optionally JKM). Do not confuse KMs with procedures. Do not create one
• Creating too many KMs: A typical project requires less than five KMs (one each of LKM, IKM,
F
KM for each specific use case. Similar KMs can be merged into a single one and can be
parameterized using options.
• Hard-coding values, including catalog or schema names in KMs: You should instead use the
substitution methods getTable(), getTargetTable(), getObjectName(), KM options
or others as appropriate.
• Using too many variables in KMs: You should instead use options or flexfields to gather
information from the designer.
• Writing the KM entirely in Jython, Groovy, or Java: You should do this only if it is the
appropriate solution (for example, when sourcing from a technology that has only a Java
API). SQL is easier to read, maintain, and debug than Java, Groovy, or Jython code.
• Using <%if%> statements instead of check box options for conditional code generation

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 6

Developing KMs: Other Recommendations

3. Other code-writing recommendations:

– Keywords should be in consistent case for better readability.

– The code should be correctly indented.
<%
String myOptionValue=odiRef.getOption("OPT001");
if (myOption.equals("TRUE"))
{
out.print("/* Option OPT001 is set to TRUE */"); rab
le
} n s fe
else
n - tra
{
a no
%>
h a s
/* The OPT001 option is not m ) deฺset */
properly
ฺ c o u i
<% e
cl ent G
}
o r a d
%> u@ Stu lc this
e d e
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ce recommendations that apply to KMs:
e d e
Other common l i
code-writing
n N
lori case (for example, all lowercase) for better readability. ODI and Java keywords are usually
• SQL keywords such as “select”, “insert”, “print”, and so on should be in a consistent
F
mixedCase (for example, odiRef.getOption).
• The code should be correctly indented. The generated code should also be indented in order
to be more readable. Many integrated development environment (IDE) code editors can
assist with this.
• Use /* Comments */ liberally to make the code internally self-documenting.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 7

Agenda

• Developing Knowledge Modules: Best Practices

 Using Java in KMs

• Using Substitution Methods: Code Examples
• Troubleshooting Knowledge Modules

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 8

Using Java in Knowledge Modules

• The code of the KMs is partially written in Java.

• <%= … %> tags are used to enclose the fragments of Java

code.
• More Java can be called by using these tags.
Example:

bl e
<%
fe r a
myVar = 'VAR'; n s
%> n - tra
... a no
h a s
select ...
m ) deฺ
from ... ฺ c o u i
l e G
rac ent
where COL1 = <%=myVar%>
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cODI
The codee d e
generation l
ini is able to interpret any Java code enclosed between “<%” and “%>”
n
tags. N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 9

Using Code Generation Tags:
create table <%=odiRef.getTable("L", "TARG_NAME", "A")%> ?if tag
(
<%=odiRef.getColList("", "\t[COL_NAME] [DEST_CRE_DT] <?if ([COL_MANDATORY] == 1)
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

{out.print(\u0022NOT\u0022);}?> NULL", ",\n", "", "")%>

)
%= tag
Compiled by ODI Designer

create table <?=odiRef.getObjectName("L", "CUSTOMER", "A")?>

(
CUST_ID number(10) <?if (1 == 1){out.print("NOT");}?> NULL,
CUSNAME varchar(50) <?if (1 == 1){out.print("NOT");}?> NULL,
ADDRESS varchar(80) <?if (0 == 1){out.print("NOT");}?> NULL,
COUNTRY varchar(50) <?if (0 == 1){out.print("NOT");}?> NULL,
ble
...
fe r a
)
ans
Compiled by the Agent
n - t r
no
Conditional
create table DEMO_PRODUCTION.CUSTOMER
(
s a
CUST_ID number(10) NOT NULL, h
) deฺ a
CUSNAME varchar(50) NOT NULL,
c o m u i
e ฺ G
cl ent
ADDRESS varchar(80) NULL,
COUNTRY varchar(50) NULL,
r a
o tud
)
...
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce of using code generation tags.
e
This slide d e
shows the l i
examples
n N
lori
The % tag is used with most substitution methods, for example to generate a list of columns.
F The ? tag is used with “if” statements, for example to check if the Mandatory attribute of the
columns is flagged or not. In this example, if the column is mandatory, then ODI generates NOT
NULL; otherwise it is NULL.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 10

Using Code Generation Tags: Examples

The substitution methods used in compilation steps:

connect my_user/<%=odiRef.getInfo("DEST_ENCODED_PASS")%>
insert into <%=odiRef.getTable("L","TARG_NAME", "D")%>
select <%=odiRef.getColList("", "[COL_NAME]", ", ", "", "")%> 1
...
Compiled by ODI Designer Generation of a session or a scenario
connect my_user/<?=odiRef.getInfo("DEST_ENCODED_PASS")?>
insert into <?=odiRef.getObjectName("L","TARG_NAME", "MY_LSCHEMA", "D")?>
select EMP_ID, EMP_NAME, EMP_CODE 2
...
ble
Compiled by the Agent Written in the Logs
fe r a
ans
connect my_user/<@=odiRef.getInfo("DEST_ENCODED_PASS")@>
n - t r
no 3
insert into SCOTT.EMP
select EMP_ID, EMP_NAME, EMP_CODE
...
s a
Compiled by the Agent Execution h
) deฺ a
c o m u i
e ฺ G
cl ent
connect my_user/M4678GHT
insert into SCOTT.EMP
o r a d 4
select EMP_ID, EMP_NAME, EMP_CODE
...
u@ is S t u
l c
n e de e th
i nฺ © t2015, s and/or its affiliates. All rights reserved.
uOracle
o r
Copyright
o
u (fl nse
d e lc lice
N e
This slide shows examples of code generation tags used in compilation steps:
n Procedure / KM as it appears in ODI Designer. The “connect” is to log in to SQL*Plus
lori (sqlplus).
1.
F
2. Generated Code:
- Independent from the context, schema different for each physical context. Schema still
a variable.
- Resolution of the attribute (column list) names
- Scenario or session ready to be executed
3. Generated code:
- Resolution of the context (schema.tablename). Password still a variable.
- Sensitive information
- Visible in the ODI logs
4. Executed code, includes the encoded password

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 11

Code Generation Techniques

Using Java Variables and String Functions:

• The following KM code:

<%
String myTableName;
myTableName = "ABCDEF";
%>
a b le
er
drop table
n s
<%=odiRef.getObjectName(myTableName.toLowerCase())%> f
-tra
n o n
• Generates the following: s a
a
) h deฺ
drop table SCOTT.abcdef
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce of how you can use these advanced techniques. The following KM
e
This slide d e
shows an l i
example
n
code N
creates a string variable and uses it in a substitution method:
F lori <%
String myTableName;
myTableName = "ABCDEF";
%>
drop table <%=odiRef.getObjectName(myTableName.toLowerCase())%>
This code generates the following:
drop table SCOTT.abcdef
The assumption is that it is pointing to the SCOTT schema.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 12

Code Generation Techniques

Using a KM option to generate code conditionally:

• The following KM code generates code depending on the

OPT001 option value:
<%
String myOptionValue=odiRef.getOption("OPT001");
if (myOption.equals("TRUE"))
{ out.print("/* Option OPT001 is set to TRUE */"); }
else ble
{ %> /* Option OPT001 is not set to TRUE, e.g. FALSE */ <% } fe r a
ans
%>
n - t r
• no
If OPT001 is set to TRUE, then the following is generated:
a
/* The OPT001 option is set to TRUE )*/h a s
o m i d eฺ
– Otherwise, the following is generated: l e ฺc t Gu
o r ac den
/* The OPT001 option is
u @not S settu to TRUE, e.g. FALSE */
el c s
e d e thi
r i n ฺn © 2015,uOracle
s and/or its affiliates. All rights reserved.
o Copyright
t o
u (fl nse
lc lice
deconditional
You caneuse branching and advanced programming techniques to generate code. The
N
n example illustrates how you can use conditional branching.
following
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 13

Agenda

• Developing Knowledge Modules: Best Practices

• Using Java in KMs

 Using Substitution Methods: Code Examples
• Troubleshooting Knowledge Modules

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 14

Using ODI Substitution Methods: getJoin()

• getJoin(): Returns the entire WHERE clause section

generated for the joins of a mapping

– Usage: public java.lang.String
getJoin(java.lang.Int pDSIndex)
– Parameter (Int):
— pDSIndex: Index identifying which dataset is used
• Example: ble
fe r a
tra ns
insert into <%=odiRef.getTable("L", "COLL_NAME",
n -
"W")%> select <%=odiRef.getColList("", no
s afrom
"[EXPRESSION]", ", ", "", "INS=1")%> a
) h deฺ
<%=odiRef.getFrom()%> where (1=1)
c o m u i
e ฺ G
r a cl ent
<%=odiRef.getJoin()%> <%=odiRef.getFilter()%>
o
<%=odiRef.getGrpBy()%> ud
<%=odiRef.getHaving()%>
u @ St
e l c h is
e d e t
i n ฺ n u s
l o r
Copyright ©
t o
2015, Oracle and/or its affiliates. All rights reserved.

u (f nse
e c
l method
l i ceretrieves the SQL join string (on the source during the loading, on the
e d
The getJoin()
n N
Staging Area during the integration) for a given result of a mapping.
F lori
Parameter
pDSIndex (Int): Index identifying which of the datasets is taken into account by this command
Note: The pDSIndex parameter can be omitted when this method is used in an LKM. It can also
be omitted for IKMs. In this case, the dataset taken into account is the first one.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 15

Using ODI Substitution Methods: getFilter()

• getFilter(): Returns the entire WHERE clause section

generated for the filters of a mapping

– Usage: public java.lang.String
getFilter(java.lang.Int pDSIndex)
– Parameter (Int):
— pDSIndex: Index identifying which dataset is used
• Example: bl e
fe r a
tra ns
insert into <%=odiRef.getTable("L", "COLL_NAME",
n -
"W")%> select <%=odiRef.getColList("", no
s afrom
"[EXPRESSION]", ", ", "", "INS=1")%> a
) h deฺ
<%=odiRef.getFrom()%> where (1=1)
c o m u i
e ฺ G
r a cl ent
<%=odiRef.getJoin()%> <%=odiRef.getFilter()%>
o
<%=odiRef.getGrpBy()%> ud
<%=odiRef.getHaving()%>
u @ St
e l c h is
e d e t
i n ฺ n u s
l o r
Copyright ©
t o
2015, Oracle and/or its affiliates. All rights reserved.

u (f nse
e c
l method l i ce returns filter expressions separated by an AND operator. When used
e d
The getFilter()
an N
inin LKM, it returns the filter clause because it should be executed by the source server. When
F lorused in an IKM, it returns the filter clause because it should be executed by the Staging Area
server.
Notice the WHERE (1=1). This is where a filter would go regardless of whether you do or do not
have a filter. Your clause is added via an AND operator to the always-present 1=1.
The join clauses are considered part of the WHERE.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 16

Using ODI Substitution Methods: getPK()

• getPK(): Returns information about primary key

– Usage: public java.lang.String

getPK(java.lang.String pPropertyName)
– Parameter (String):
— pPropertyName: String that contains the name of the
requested property
• Example: bl e
fe r a
ans
<%=odiRef.getPK("KEY_NAME")%> n - t r
a no
h a s
m ) deฺ
ฺ c
One ofo several
u i
e
cl ent parameters G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u e
l methodlicreturns
e d
The getPK()e information relative to the primary key of a datastore during a check
N
in by IntheanDDL
procedure. action, this method returns information related to the primary key currently
F lorhandled command.
Parameters
• ID: Internal number of the PK constraint
• KEY_NAME: Name of the primary key
• MESS: Error message relative to the primary key constraint
• FULL_NAME: Full name of the PK generated with the local object mask
• <flexfield code>: Flexfield value for the primary key

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 17

Using ODI Substitution Methods in Journalizing
Knowledge Modules: Examples
• getJrnFilter(): Returns the SQL Journalizing filter for
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

a given dataset. Example:

<%=odiRef.getJrnFilter()%>
– Usage: public java.lang.String
getJrnFilter(java.lang.Int pDSIndex)
– Parameter: pDSIndex – Identifies which dataset is used
• getJrnInfo(): Returns journalizing information about a able
datastore. Example: s f er
<%=odiRef.getJrnInfo("FULL_TABLE_NAME")%> - t r an
n n o
– Usage: public java.lang.String a
has eฺ
getJrnInfo(java.lang.String )pPropertyName
c
– Parameter: pPropertyName – ฺString om containing
u id the name
l e G
of the requested propertyrac ent
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce of substitution methods that can be used specifically in
e
This slide d e
shows you l i
examples
n N
Journalizing Knowledge Modules (JKMs).
F lori
The getJrnFilter() method returns the SQL Journalizing filter for a given dataset in the
current mapping. If the journalized table is in the source, this method can be used during the
loading phase. If the journalized table is in the Staging Area, this method can be used while
integrating.
Parameter (Int): pDSIndex – Index identifying which of the datasets is taken into account by this
command.
The getJrnInfo() method returns journalizing information about a datastore.
Parameter (String): pPropertyName – The name of the requested property, for example:
• FULL_TABLE_NAME: Full name of the journalized datastore
• JRN_NAME: Name of the journal datastore
• JRN_SUBSCRIBER: Name of the subscriber
• JRN_METHOD: Journalizing Mode (consistent or simple)
Note: This method returns information about a datastore’s journalizing for a JKM while journalizing
a model/datastore, or for a LKM/IKM in a mapping.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 18

Using ODI Substitution Methods in Journalizing
Knowledge Modules: Examples
• getSubscriberList(): Returns a list of subscribers for
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

a journalized table
– Usage: public java.lang.String
getSubscriberList( java.lang.String pStart,
java.lang.String pPattern, java.lang.String
pSeparator, java.lang.String pEnd)
– Parameters (String): e
r a bl
• Example: s fe
- t r an
no
<%=odiRef.getSubscriberList("\nBegin List\n", n
s a
"- [SUBSCRIBER]", "\n", "\nEnd of aList\n")%>
m ) h deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce method returns a list of subscribers for a journalized table. The
d e l i
e parameter is interpreted and then repeated for each element of the list, and separated
The getSubscriberList()
n N
i its predecessor with the pSeparator parameter. The generated string begins with pStart
pPattern
F lorfrom
and ends with pEnd. If no element is selected, pStart and pEnd are omitted and an empty string
is returned.
Parameters (String):
• pStart: This sequence marks the beginning of the string to generate.
• pPattern: The pattern is repeated for each occurrence in the list. The list of the attributes
usable in a pattern is detailed in the following Pattern Attributes List. Each occurrence of the
attributes in the pattern string is replaced by its value. Attributes must be between square
brackets ([ and ]). Example:
My name is [SUBSCRIBER]
• pSeparator: This parameter separates each pattern from its predecessor. Example: new
line "\n".
• pEnd: This sequence marks the end of the string to generate.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 19

Using ODI Substitution Methods
in Reverse-Engineering KM: getModel()
• getModel(): Returns information about the current data
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

model during the processing of a personalized reverse

engineering
– Usage: public java.lang.String
getModel(java.lang.String pPropertyName))
– Parameter: pPropertyName ─ String that contains the
name of the requested property ble
fe r a
tra ns
select TABLE_NAME, RES_NAME, replace(TABLE_NAME,
n -
'<%=odiRef.getModel("REV_ALIAS_LTRIM")%>'
a no, '')
ALIAS, TABLE_DESC h a s
m ) deฺ
from MY_TABLES
ฺ c o u i
where TABLE_NAME like e
cl ent G
r a
o tud
'<%=odiRef.getModel("REV_OBJ_PATT")%>'
l c u@ is S
n e de e th
i nฺ © t2015, s and/or its affiliates. All rights reserved.
uOracle
o r
Copyright
o
u (fl nse
d e lc lice
N e
This slide shows you examples of substitution methods that can be used specifically in RKM.
n getModel() method returns information about the current data model during the processing
lori
The
F of a personalized reverse engineering. The list of available data is described in the
pPropertyName values table.
Note: This method may be used on the source connection (data server being reverse-engineered)
as well as on the target connection (repository). On the target connection, only the properties
independent from the context can be specified (for example, the schema and catalog names
cannot be used).
Parameters:
• ID: Internal identifier of the current model
• MOD_NAME: Name of the current model
• LSCHEMA_NAME: Name of the logical schema of the current model
• MOD_TEXT: Description of the current model
• REV_TYPE: Reverse engineering type—S for standard reverse, C for customize
• REV_UPDATE: Update flag of the model
• REV_INSERT: Insert flag for the model

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 20

• REV_OBJ_PATT: Mask for the objects to reverse
• REV_OBJ_TYPE: List of object types to reverse-engineer for this model. This is a semicolon-
separated list of object type codes. Valid codes are:
- T: Table
- V: View
- Q: Queue
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

- SY: System table

- AT: Table alias
- SY: Synonym
• TECH_INT_NAME: Internal name of the technology of the current model
• LAGENT_NAME: Name of the logical execution agent for the reverse engineering
• REV_CONTEXT: Execution context of the reverse
• REV_ALIAS_LTRIM: Characters to be suppressed for alias generation
ble
• CKM: Check Knowledge Module fe r a
an s
• RKM: Reverse-engineering Knowledge Module
n - t r
•
a no
SCHEMA_NAME: Physical Name of the data schema in the current reverse context
WSCHEMA_NAME: Physical Name of the work schema in the current reverse context
h a s
CATALOG_NAME: Physical Name of the data catalog in the current reverse context
m ) deฺ
ฺ c o u i
WCATALOG_NAME: Physical Name of the work catalog in the current reverse context
• e
cl ent G
<flexfield code>: Value of the flexfield for the current model
r a
o part oftuthed mask of objects to reverse.
The example retrieves the list of tables that are
@
c his S
u
e l
ฺ n ed se t
l o r in to u
u (f nse
e l c l i ce
e d
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 21

Agenda

• Developing Knowledge Modules: Best Practices

• Using Java in KMs

• Using Substitution Methods: Code Examples
 Troubleshooting Knowledge Modules

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 22

Troubleshooting KMs

Identifying database errors:

• Errors appear often in the following way in ODI:

java.sql.SQLException: ORA-01017: invalid
username/password; logon denied at ...
at ...
at ...
bl e
• java.sql.SQLExceptioncode indicates that a query s fe r a
was made to the database through the JDBC driver, - t r an
which
n on
has returned an error (a database or driver error).
a s
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce about how to troubleshoot problems that you might encounter
e
This topicd e
provides l i
information
n
when Nusing ODI KMs.
F lori
Errors appear often in ODI indicating java.sql.SQLExceptioncode. This error is frequently a
database or driver error, and must be interpreted in this direction.
Only the part of text in bold must first be taken into account (for example, ORA-01017). It must be
searched in the Oracle documentation. If it contains an error code specific to Oracle, the error can
be immediately identified.
If such an error is identified in the execution log, it is necessary to analyze the SQL code sent to
the database to find the source of the error. The code is displayed on the Operator Code tab of the
erroneous task. Often you can copy that code and paste it into SQL Developer to get more error
information.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 23

Troubleshooting KMs

Error Code Common Problem Solution

ORA-12154 TNS: Could not resolve Check the configuration of the

service name TNS aliases on hosts.
ORA-02019 Connection description for Check the KM options for
remote database not found creating the database links.
ORA-00900 Invalid SQL statement The typical case is a missing
quote or an unclosed bracket.
ble
ORA-00923 FROM Keyword not found The typical case is a missing era
where expected quote or an unclosed bracket. a n sf
o n -tr
a n
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u cesolutions:
Common e d e
problems l i
and
n N
lori
ORA-12154 TNS: could not resolve service name
F • TNS alias resolution. This problem may occur when using the OCI driver, or a KM using
database links.
• Check the configuration of the TNS aliases on the machines.
ORA-02019 connection description for remote database not found
• You used a KM using nonexistent database links.
• Check the KM options for creating the database links.
ORA-00900 invalid SQL statement
ORA-00923 FROM Keyword not found where expected
• The code generated by the mapping or typed in a procedure is invalid for the technology, in
this case, Oracle SQL. This is usually related to an input error in the mapping, filter, or join.
• The typical case is a typo: a missing quotation mark, an unclosed bracket, or an unmatched
parenthesis.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 24

Troubleshooting KMs

Syntax error example:

ORA – 00900 Invalid SQL Statement

• Wrong statement:
EXECUTE SCHEMA.PACKAGE.MYPROC(PARAM1, PARAM2)

• Correct statement:
bl e
BEGIN fe r a
SCHEMA.PACKAGE.MYPROC(PARAM1, PARAM2); -tra
ns
n on
END; a s
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cethe call made to a non-SQL syntax, such as the call to an Oracle-stored
e d
A frequent
e
cause is l i
also
n N
procedure using the inappropriate syntax:
Fl ori EXECUTE SCHEMA.PACKAGE.MYPROC(PARAM1, PARAM2)
The valid SQL call for a stored procedure in this environment is:
BEGIN
SCHEMA.PACKAGE.MYPROC(PARAM1, PARAM2);
END;
You can minimize syntax errors by using the Expression Editor (the gear), which dynamically
checks syntax against the selected technology as you type.
Note: The syntax EXECUTE SCHEMA.PACKAGE.PROC(PARAM1, PARAM2) is specific to
SQL*Plus, and does not work with JDBC.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 25

Troubleshooting KMs

Error Code Common Problem Solution

ORA-00904 Invalid column name Keying error. A string that is not a

column name is interpreted as a
column name.
ORA-00903 Invalid table name The table used (source or target)
does not exist in the Oracle schema.
ORA-00972 Identifier is too long. Use aliases or views or shorter
ble
names.
fe r a
ORA-01790 Expression must have You tried to connect two different
t r a ns
the same data type. n-
values that cannot be implicitly
n o
converted (in a mapping
a or join).
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e
Other errors: l i
n N
lori
ORA-00904 invalid column name:
F • Keying error in a mapping/join/filter. A string that is not a column name is interpreted as a
column name, or a column name is misspelled. This error may also appear when accessing
an error table linked to a data store with a recently modified structure. Modify or drop the
error tables and let Oracle Data Integrator re-create it in the next execution.
ORA-00903 invalid table name:
• The table used (source or target) does not exist in the Oracle schema.
• Check the mapping logical/physical schema for the context, and check that the table
physically exists on the schema accessed for this context.
ORA-00972 Identifier is too Long:
• There is a limit in the object identifier in Oracle (usually 30 characters). Remember to
account for prefixes E$, C$, and suffixes. Use aliases, views, or shorter names.
ORA-01790 expression must have the same data type as the corresponding
expression
• You tried to connect (map or join) two different values that cannot be implicitly converted.
• Use the explicit conversion functions on these values.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 26

Quiz

Which is the preferred method of doing conditional code

generation?
a. ODI <%if%> statements
b. Check box options
c. Substitution parameters

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Answer:ebd
el lice
i n N
lo r
Explanation: You should avoid using <%if%> statements instead of check box options for
F conditional code generation. Substitution is used for portability, not conditional code. See the
section titled “Developing KMs: What to Avoid and What to Use” for more information.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 27

Quiz

What is the best way to avoid making typos in writing KM

code?
a. Write the KM entirely in Java.
b. Use numeric variable names such as VAR01 and VAR02
instead of descriptive names such as Cust_ID and
Last_Name.
ble
c. Use the built-in Expression Editor.
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Answer:ecd
el lice
i n N
lo r
Explanation: The Expression Editor does syntax checking as you type, as well as highlighting
F matched and unmatched brackets and parentheses.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 28

Summary

In this lesson, you should have learned to:

• Describe best practices and techniques of developing and

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 29

Practice 3-1 Overview: Creating a New
Knowledge Module for Data Generation
In this practice, you create a new Knowledge Module that can
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

be used with ODI integration mappings for generating

thousands of records with random data in any table in the
Oracle Database.

Oracle Data Integrator 12c: Advanced Integration and Development 3 - 30

Designing ODI Integration Mappings

After completing this lesson, you should be able to:

• Describe the different styles (E-LT, ETL) of integration

mappings
• Describe a typical integration process
• Use best practices for creating integration mappings of
different styles
ble
• Maintain quality of data fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 2

Agenda

 Integration Process: Overview

• Integration Mappings: Overview

• Designing Integration Mappings: Best Practices
• Maintaining Integrity of Data

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 3

Integration Process: Overview

The following elements are used in the integration process:

• An integration table I$_targetname

• The source and/or loading tables

@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cisealways needed in a mapping. This process integrates data from the
e d e
An integration l
process i
n N
source or loading tables into the target datastores by using a temporary integration table.
Fl ori
An integration process uses an integration strategy, which defines the steps required in the
integration process. The following elements may be used in the integration process:
• An integration table (also known as the flow table) is sometimes needed to stage data after
all Staging Area transformations are made. This loading table is named after the target table,
prefixed with I$_. This integration table is the image of the target table with extra fields
required for the strategy to be implemented. The data in this table is flagged, transformed, or
checked before being integrated into the target table. In this example, TEMP_SALES would
really be named I$_SALES.
• The source (C$_name) and/or loading tables (created by the LKM). The integration process
loads data from these tables into the integration table or directly into the target tables. In this
example, TEMP_1 would really be named C$_0, and TEMP_2 would really be named C$_1.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 4

Integration Process: Overview

• Check Knowledge Module.

• Mapping metadata

@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ceModule. The IKM may initiate a flow check phase to check the data in the
e
• Checkd eKnowledge l i
i n N
integration table against some of the constraints of the target table.
r
Flo -Invalid data is removed from the integration table (removed from the flow). It is possible
to re-inject the “bad” rows later.
• Mapping metadata, such as Insert, Update, UD1, and so on, or model metadata such as the
Slowly Changing Dimension (SCD) behavior. They are used at the integration phase to
parameterize column-level behavior in the integration strategies.
- Insert: You can make a SYSDATE inserted as a CREATION_DATE column on the
target, but that is a rare thing to happen on the target.
- UD1: “Unique iDentifier 1”: used in flat files for “key” fields. They could be column
positions, or an XML tag. ODI will enforce “unique and not null” (that is, act like a
primary key) on a flat file. So numbers UD1, UD2, and UD3 can be thought of as the
“order of the composite keys.”
- SCD: You may want to add old/new values, or DATE_CHANGED, and so on.
- UDn and SCD flags can only be used in an IKM.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 5

Typical Integration Process
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Source

1 Loading Data from

2 3
Create Temporary the Source and Perform
Integration Table Loading Table into Transformations
Integration Table
ble
4 fe r a
Load Data from
ans
-
Integration Table to
n t r
Target Table
a no
h a s
m ) deฺ
ฺ c o u i
c l e n t G Target
a
or tude
@
cu his S
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
del
A typicaleintegration lice works in the following way:
process
i n
1.
NCreate a temporary integration table if needed. For example, an update flag taking value I or
o r
Fl U to identify which of the rows are to be inserted or updated.
2. Load data from the source and loading tables into this integration table with the LKM,
executing those of the transformations (joins, filters, mapping) specified on the Staging Area.
3. Perform some transformation on the integration table to implement the integration strategy
using the IKM. For example, compare the content of the integration table with the target table
to set the update flag.
4. Modify the content. Load data from the integration table into the target table.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 6

Agenda

• Integration Process: Overview

 Integration Mappings: Overview

• Designing Integration Mappings: Best Practices
• Maintaining Integrity of Data

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cthee three factors that structure the flow. The most important is choosing
e
Now look d
at
ethe first l
of i
n
whereNto put the Staging Area.
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 7

Integration Mappings: Overview

• An integration mapping consists of a set of rules that

define the loading of a datastore or a temporary target

structure from one or more source datastores.
• A mapping is made up of and defined by the following
components:
– Target datastore
– Datasets ble
fe r a
– Diagram of source datastores s
- t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ca eset of rules that define the loading of a datastore or a temporary target
e
A mapping d e
consists l
of i
n N
structure from one or more source datastores into one or more targets.
Fl ori
An integration mapping is made up of and defined by the following components:
• Target datastore(s): The target datastore (or multiple datastores) is the element that will be
loaded by the mapping. This datastore may be permanent (defined in a model), or temporary
(created by the mapping), or reusable (created in another mapping).
• Datasets: One target is loaded with data coming from several datasets. Set-based operators
(UNION, INTERSECT, and so on) are used to merge the different datasets into the target
datastore. Each dataset corresponds to one diagram of source datastores and the mappings
used to load the target datastore from these source datastores.
• Diagram of source datastores: A diagram of sources is made of source datastores—
possibly filtered—related using joins. The source diagram also includes lookups to fetch
additional information for loading the target. Two types of objects can be used as a source of
a mapping: datastores from the models and reusable mappings. If a mapping is used, its
target datastore—temporary or not—will be taken as a source. The source datastores of a
mapping can be filtered during the loading process, and must be put in relation through joins.
Joins and filters are either copied from the models or defined for the mapping. Joins and
filters are implemented in the form of SQL expressions. A lookup is just a special case of a
join.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 8

Integration Mappings: Overview

Other components of a mapping:

• Expression: This defines the transformations performed

on one or several source attributes to load one target
attribute.
• Staging Area: A logical schema where some of the
transformations (joins, filters, and expressions) take place
• Physical tab(s): New to 12c: one Logical tab and one or erab
le
more Physical implementations/deployments (used toabe n sf
Flow in 11g) o n -tr
a n
• Control: Flow Control checks the flow of adata
h s before it is
integrated into the target. Post-Integration
o m ) control
i d eฺ performs
a static check on the target tablel e ฺcaftert the
G umapping.
c n
ra ude
@ St o
l c u is
e d e t h
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ce the transformations performed on one or several source columns to
e d e
• Expression: l i
Defines
i n N
load one target column. These transformations are implemented in the form of SQL
r
Flo expressions. Each target column has zero or one expressions per dataset. If an expression
is executed on the target, the same mapping applies for all datasets.
• Staging Area: The Staging Area is a logical schema into which some of the transformations
(joins, filters, and mappings) take place. It is by default the same schema as the target’s
logical schema. It is possible to locate the Staging Area on a different location (including one
of the sources). That is necessary if the target’s logical schema is not suitable for staging, for
example, if the target is a file datastore, because file technology has no transformation
capability.
Note: Expressions can be executed either on the source or target/Staging Area. Filters and
joins can be executed either on the source or Staging Area.
• Physical tab(s): Describes how the data flows between the sources, the Staging Area (if it is
different from the target), and the target as well as where joins and filters take place. The
flow also includes the loading and integration methods used by this mapping. These are
selected by choosing Loading (LKMs) and Integration Knowledge Modules (IKMs).
• Control: A mapping implements two points of control. Flow Control checks the flow of data
before it is integrated into the target. Post-Integration control performs a static check on the
target table at the end of the mapping. The check strategy for Flow and Post-Integration
control is defined by a Check Knowledge Module (CKM) and mapping option check boxes.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 9

Basic Integration Process:
Sequence of Operations
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Source Target
Transform and
(MySQL) integrate (Oracle)
ORDERS 4
1 C$_0
4 SALES

Extract/Join/
LINES Transform 33 I$_SALES
bl e
Join/Transform
fe r a
2 C$_1
an s
CORRECTIONS
n - t r
no
Extract/Transform
File

s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cofe operations is more or less unavoidable to implement this integration.
e d e
The following l
sequence i
n N
However, you have some latitude in controlling how and where they are performed.
F lori1. The two source orders tables must be extracted and joined. The constraint that orders must
be closed is applied here.
2. Data from the corrections file must be extracted and transformed into the correct format. This
happens to be a “File to SQL LKM.”
3. Now, data from these two sources must be joined and transformed into a temporary table,
I$_SALES. This temporary table looks identical to the target SALES table.
4. The data from this table must be copied to the target Oracle server by using an IKM.
5. (Not shown) You can also optionally do a data check, or Flow Control, by using a CKM.
6. (Not shown) You can also optionally do journalizing to track changed data by using a JKM.
But the big question is WHERE do you do those things that are shown in the middle of the slide?
The short answer is, “It depends.” Answering that question more thoroughly is the topic for the
next section of this lesson.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 10

Staging Area

• A Staging Area is a separate, often dedicated, area in an

RDBMS where ODI creates its temporary objects and

executes some of your transformation rules.
• Usually, place the Staging Area on the target (or bigger)
data server.
Target (RDBMS)
Source
(MySQL) Staging Area Transform and
ble
The language is SQL integrate
fe r a
ORDERS
1 55
an s
TEMP_1

n - t r
SALES

LINES
Extract/Join/
Transform
3 TEMP_
a no
Join/Transform SALES
h a s
m ) d4 eฺ ERRORS
2
2 TEMP_2
ฺ c o u i
e
cl ent G
CORRECTIONS
File
Extract/Transform
r a
o tud
Check constraints/
Isolate errors (CKM)
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce most of the transformation and error checking is usually performed.
e d
The Staging
e Area is l i
where
n
After N
those tasks are performed, you copy or load the data into the target server.
F lori
The Staging Area is a special area that you create or assign when you set up a database in ODI.
ODI creates temporary tables in the Staging Area, and uses them to perform data transformation.
To prevent temporary tables from cluttering production catalogs, at the very least the Staging Area
should be a separate, dedicated schema (but it does not have to be). To simplify backup
considerations, often the Staging Area is a separate, dedicated tablespace (on technologies
where that is appropriate).
You can place the Staging Area on your source server, your target server, or another server
altogether. However, the best place for the Staging Area is usually on the target (or bigger) server.
This gives you the greatest scope for data consistency checking, and minimizes network traffic.
You now look at some of the consequences of different choices.
But what happens if you decide (or politics dictates) that the Staging Area is not on the target?

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 11

Placing the Staging Area

The Staging Area:

• May be located on:

– The target database (default)
– Another third RDBMS database server
– The source database
– Any schema, but is usually a dedicated schema
ble
• Cannot be placed on the file source data server: fe r a
– This data server is not an RDBMS. ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
oOther tud
Source
c u @ S Target
l i s
n e de e th
i nฺ © t2015, s and/or its affiliates. All rights reserved.
uOracle
o r
Copyright
o
u (fl nse
d e lc lice
N e
The location of the Staging Area can have a large impact on performance. This is due to network
n of data, as well as the horsepower of the staging engine.
transfer
F lori
The Staging Area can be a shared schema (bad practice), or a dedicated schema (best practice).
It can be a shared tablespace or a dedicated tablespace.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 12

Agenda

• Integration Process: Overview

• Integration Mappings: Overview

 Designing Integration Mappings: Best Practices
• Maintaining Integrity of Data

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 13

Designing Integration Mappings:
E-LT- and ETL-Style Mappings
• E-LT-style integration mapping:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

– ODI processes the data in a Staging Area, located on the

target.
– Staging Area and target are located on the same DB
instance.
– The data is loaded from the sources to the targets.
• ETL-style mapping: bl e
fe r a
– ODI processes the data in a Staging Area, which is different
t r a ns
from the target.
n on-
– The data is first extracted from the sourcessand a then loaded
to the Staging Area. a
) h deฺ
– Data transformations take place c m u i Area and
inothe Staging
e ฺ G
r a
intermediate results are stored cl in temp
e n t tables in the Staging
o d
Area. u@ Stu
lc this
e d e
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
c u ce
• Ine
el
andE-LT-styleliintegration mapping, ODI processes the data in a Staging Area, which is
rin Nlocated on the target. Staging Area and target are located on the same database instance (it
Flo is not enough to merely be in the same host). The data is loaded from the sources to the
target(s). To create an E-LT-style integration mapping, follow the standard procedure of
creating an ODI mapping. See “Working with Integration Mapping” in the Oracle Fusion
Middleware Developer’s Guide for Oracle Data Integrator for generic information about how
to design integration mappings.
• In an ETL-style mapping, ODI processes the data in a Staging Area, which is different from
the target. The data is first extracted from the sources and then loaded to the Staging Area.
The data transformations take place in the Staging Area and the intermediate results are
stored in temporary tables in the Staging Area. The data loading and transformation tasks
are performed with the standard ELT KMs.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 14

Designing an ETL-Style Mapping

In an ETL-style mapping, ODI processes the data in a Staging

Area, which is different from the target. ODI provides two ways
for loading the data from a Staging Area to a Target:

Multi-Connection IKM LKM + Mono-Connection IKM

ble
IKM LKM IKM fe r a
ans
n - t r
Staging Target Staging
a noTarget
h a s
m ) deฺ
ฺ c o u i
TEMP_1 SALES TEMP_1
e
cl ent G SALES

r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce to design an ETL-style mapping where the Staging Area is an RDBMS
e d e
In this topic, you l
learn ihow
n N
database or any ANSI-92–compliant database and the target on an RDBMS database.
F lori
In an ETL-style mapping, ODI processes the data in a Staging Area, which is different from the
target (perhaps, due to politics, you are not allowed to make temporary objects on target).
“Different” here could mean on the source, or a third server, or even just a different instance in the
same host server.
ODI provides two ways for loading the data from a Staging Area to a Target:
• Using a Multi-Connection IKM: A Multi-Connection IKM allows updating a target where the
Staging Area and sources are on different data servers.
• Using an LKM and a Mono-Connection IKM: If there is no dedicated Multi-Connection
IKM, use a standard exporting LKM in combination with a standard Mono-Connection IKM.
Note: Depending on the KM strategy that is used, flow and static control are supported.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 15

Designing an ETL-Style Mapping:
Using Multi-Connection IKM
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Source Staging Target

Supported Supported Supported
RDBMS RDBMS RDBMS
Multi-
connection CKM
ble
LKM IKM
fe r a
ans
JDBC, n - t r
JNDI, or a no
Staging DB*Link
h a s
transformations m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
callows
e d e
A Multi-Connection l
IKM i updating a target where the Staging Area and sources are on
N
orin
different (possibly heterogeneous) data servers.
Fl

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 16

Designing an ETL-Style Mapping:
Using Multi-Connection IKM
• Multi-Connection IKM for handling Oracle data:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

– IKM Oracle to Oracle Control Append (DBLINK)

• You can also use the generic SQL multi-connection IKMs:
– IKM SQL to SQL Incremental Update: Integrates data
from any ANSI-SQL92–compliant database into any ANSI-
SQL92–compliant database target table in incremental
update mode.
a b le
Note: Use this KM if it is not possible to create temporary sfe
r
objects in the target server. - t r an
– IKM SQL to SQL Control Append: Source anda target non tables
are on different databases and the mapping’s
h a s Staging Area
is set to the logical schema of theco m) utables
source i d eฺ or a third
ฺ
schema. cle nt G ra ude
@ St o
l c u is
e d e t h
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ce Multi-Connection IKM for handling Oracle data:
e d e
ODI provides the l i
following
n N
lori
IKM Oracle to Oracle Control Append (DBLINK)
F You can also use the generic SQL Multi-Connection IKMs for ANSI SQL-92–compliant
technologies:
IKM SQL to SQL Incremental Update, and IKM SQL to SQL Control Append
IKM SQL to SQL Incremental Update KM integrates data from any ANSI-SQL92–compliant
database into any ANSI-SQL92–compliant database target table in incremental update mode. This
IKM is typically used for ETL configurations: source and target tables are on different databases
and the mapping’s Staging Area is set to the logical schema of the source tables or a third
schema. It allows an incremental update strategy with no temporary target-side objects. Use this
KM if it is not possible to create temporary objects in the target server. Because the application
updates are made without temporary objects on the target, the updates are made directly from
source to target. The configuration where the flow table is created on the Staging Area—and not in
the target—should be used only for small volumes of data. This KM supports flow and static
control.
IKM SQL to SQL Control Append is typically used for ETL configurations: source and target tables
are on different databases and the mapping’s Staging Area is set to the logical schema of the
source tables or a third schema. Use this KM strategy to perform Control Append.
Note: This KM supports flow and static control.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 17

Designing an ETL-Style Mapping:
Using Multi-Connection IKM
To use a Multi-Connection IKM in an ETL-style mapping,
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

perform the following steps:

1. Create a standard integration mapping.
2. On the Logical tab, select the Staging Area different
from Target hint and select the logical schema.
3. On the Physical tab, select one of the Source Sets. In the
Property Inspector, select an LKM to load from the sourceserab
le
to the Staging Area. a n sf
4. Select Target. On the Property Inspector, select o n
ETL
-trMulti-
a n
h s
Connection IKM to load the data from theaStaging Area to
the Target. ) de ฺ
om i ฺc t Gu
l e
o rac uden
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ceIKM in an ETL-style mapping, perform the following steps:
e
To use a d e l i
multi-connection
n N
Fl ori1. Create an integration mapping using the standard procedure as described in the section
“Working with Integration Mapping” in the Oracle Fusion Middleware Developer’s Guide for
Oracle Data Integrator.
2. On the Logical tab of the Mapping Editor, select the Staging Area different from Target hint
and select the logical schema of the source tables or another logical schema that is not a
source or the target. This schema will be used as the Staging Area.
3. On the Physical tab, select one of the Source Sets. In the Property Inspector, select an LKM
from the LKM Selector list to load from the sources to the Staging Area. See the chapter in
the Oracle Fusion Middleware Connectivity and Knowledge Modules Guide for Oracle Data
Integrator that corresponds to the technology of your Staging Area to determine the LKM you
can use. Optionally, modify the KM options.
4. Select the Target by clicking its title. On the Property Inspector, select an ETL Multi-
Connection IKM from the IKM Selector list to load the data from the Staging Area to the
target. Optionally, modify the KM options.
Note: See the chapter in Oracle Fusion Middleware Connectivity and Knowledge Modules
Guide for Oracle Data Integrator that corresponds to the technology of your Staging Area to
determine the IKM you can use.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 18

Designing an ETL-Style Mapping:
Using an LKM and a Mono-Connection IKM
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Source Staging Target

Supported Supported Supported
RDBMS RDBMS RDBMS

Exporting CKM
ble
LKM LKM
fe r a
ans
JDBC or n - t r
DB*Link
a no
Staging h a s Mono-
transformations
o m ) Connection
i d eฺ IKM
e ฺ c G u
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e
Configuration: l i
n N
F lorIfithere is no dedicated Multi-Connection IKM, use a standard exporting LKM in combination with a
standard Mono-Connection IKM. This slide shows the configuration of an integration mapping
using an exporting LKM and a Mono-Connection IKM to update the target data. The exporting
LKM is used to load the flow table from the Staging Area to the target. The Mono-Connection IKM
is used to integrate the data flow into the target table.
Limitations:
Note that this configuration (LKM + exporting LKM + Mono-Connection IKM) has the following
restrictions:
• Changed Data Capture (CDC) of any kind is not supported: neither simple CDC nor
consistent CDC are supported when the source is on the same data server as the Staging
Area (explicitly chosen in the Mapping Editor).
• Temporary Indexes are not supported.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 19

Using an LKM and a Mono-Connection IKM: Steps

To use an LKM and a Mono-Connection IKM in an ETL-style

mapping, perform the following steps:

1. Create an integration mapping.
2. Select the Staging Area different from Target hint.
3. Select an LKM to load the sources to the Staging Area.
4. Select an LKM to load the Staging Area to the target. le
a b
5. Select a standard Mono-Connection IKM to update the sfer
n
target. -tra on
a n
a s
m ) h deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e
To use and e
LKM and al i
Mono-Connection IKM in an ETL-style mapping, perform the following
N
orin
steps:
Fl 1. Create an integration mapping by using the standard procedure as described in the section
“Working with Integration Mapping” in the Oracle Fusion Middleware Developer’s Guide for
Oracle Data Integrator.
2. On the Logical tab of the Mapping Editor, select Staging Area different from Target and
select the logical schema of the source tables or a third schema.
3. On the Physical tab, select one of the Source Sets. On the Property Inspector, select an
LKM from the LKM Selector list to load from the sources to the Staging Area. See the
chapter in the Oracle Fusion Middleware Connectivity and Knowledge Modules Guide for
Oracle Data Integrator that corresponds to the technology of your Staging Area to determine
the LKM you can use. Optionally, modify the KM options.
4. Select the Staging Area. In the Property Inspector, select an LKM from the LKM Selector list
to load from the Staging Area to the target. Optionally, modify the options.
5. Select the Target by clicking its title. In the Property Inspector, select a standard Mono-
Connection IKM from the IKM Selector list to update the target. Optionally, modify the KM
options.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 20

Agenda

• Integration Process: Overview

• Integration Mappings: Overview

• Designing Integration Mappings: Best Practices
 Maintaining Integrity of Data

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 21

Enforcing ODI Data Quality

• Audits provide statistics on the integrity of application data.

• They also isolate data that is detected as erroneous by

applying the business rules:
– Uniqueness rules
– Referential integrity rules
– Validation rules that enforce consistency at the record level
le
• After erroneous records have been identified and isolated erab
in error tables, they can be accessed from: a n sf
r - t
– ODI Studio non a
– Any other front-end application a s
m ) h deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
con
d e l i
e by applying business
Audits provide statistics the integrity of application data. They also isolate data that is detected
asi n N
erroneous rules. For example, if you have an AGE attribute, you can
r
lodecide that no manager should be under 10 years old. An age of 5 is “valid” from a database
F perspective, but not reasonable from a business perspective.
“Uniqueness” (sometimes known as “deduplication”) can be trickier than you might expect.
Consider an EMAIL attribute: should it be not null (required)? Plenty of people do not have email.
If that is the case, then person1 and person2 both may have the same (null) email. What about
sharing email? Could a spouse share an email account? If so, then person3 and person4 both
may have the same not-null email. What about phone numbers where there is a main
switchboard? It can be trickier to decide what you want than it might first appear.
After erroneous records have been identified and isolated in E$_ error tables, they can be
accessed from ODI Studio, or from any other front-end application such as:
• TOAD
• SQL Developer

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 22

Ways to Handle Erroneous Data

• Automatically correct data:

– Create Data Cleansing mappings.

• Accept erroneous data (for the current project):
– Developers need precise rules for filtering.
• Correct the invalid records:
– Invalid data is sent to application e
end users. r a bl
ns fe
• Recycle data: t r a
– Recycle errors back into the n- SOA no BPEL
integration process. s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce erroneous data might be handled:
e d e
The following are l
four iways
n N
lori cleansing mappings that can be scheduled to run at predetermined intervals. Could be
• Automatically correct data: ODI offers a set of tools to simplify the creation of data
F
source > OEDQ or equivalent > ODI.
• Accept erroneous data (for the current project): In this case, mapping developers need
precise rules for filtering out erroneous data later, using ODI filters.
• Correct the invalid records: In this situation, the invalid data is sent to application end
users via various text formats or distribution modes, such as human workflow (the green box
is BPEL human workflow), email, HTML, XML, flat text files, and so on, using ODI packages.
SOA and BPEL integration with ODI will be discussed more in a later practice.
• Recycle data: Erroneous data from an audit can be optionally recycled back into the
integration process. This will be discussed more in the next slide.
Using Check Knowledge Modules (CKMs), you simply have to define the control business rules,
and inconsistent data is automatically isolated for you in error tables. However, isolating erroneous
data is not the only issue in Data Quality. Even if ODI automatically detects duplicated keys,
mandatory fields, missing references, and more complex constraints, you will need to involve
business users in the process of qualifying data discrepancies and make the appropriate
decisions.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 23

Error Recycling
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Source Target
Mapping
with IKM

Errors bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
liceto recycle errors from previous runs so that they are added to the flow
del it is useful
In someecases,
and
i n N again to the target. This method can be useful, for example, when receiving daily
applied
o r
Fl sales transactions that reference product IDs that may not exist (yet). Suppose that a sales record
is rejected in the error table because the referenced product ID does not exist in the product table.
This happens during the first run of the mapping. In the meantime, the missing product ID is
created by the data administrator. Therefore, the rejected record becomes valid and should be re-
applied to the target during the next execution of the mapping.
This mechanism is implemented by IKMs with an extra task that inserts all the rejected records of
the previous executions of this mapping from the error table into the integration table. This
operation is made before calling the CKM to check the data quality, and is conditioned by a KM
option, usually called RECYCLE_ERRORS. This is used to optionally insert the corrected rows the
next time.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 24

Building a Data Quality Framework

You should answer the following questions:

• What level of data quality is required for the Data

Warehouse?
• Who are the business owners of source data?
• What should we do with rejected records?
• Do we need to define an error recycling strategy? le
a b
• Do we need to involve business owners of the data and sfer
report rejected records? - t r an
• How would business users modify erroneousa n
on
source data? h a s
m ) deฺ
• Do we provide a GUI to have them ฺ c omodify u i
l e t G
erroneous records in the o rac tables?
error d e n
c u @ Stu
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
You (or e
l lice should have an answer to each of the questions in the slide before
demanagement)
your
n N your data quality strategy.
defining
i
lo r
F

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 25

Quiz

Which of the following is not true about ETL-style mappings?

a. ODI processes the data in a Staging Area, which is on the

target.
b. The data is first extracted from the sources and then
loaded to the Staging Area.
c. The intermediate results are stored in temporary tables in
the Staging Area. a b le
s f er
d. The data loading and transformation tasks are performed
- t r an
with the standard E-LT KMs. non a
a s
m ) h deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Answer:ead
el lice
i n N
lo r
Explanation: In an ETL-style mapping, ODI processes the data in a Staging Area, which is
F different from the target. The data is first extracted from the sources and then loaded to the
Staging Area. The data transformations take place in the Staging Area and the intermediate
results are stored in temporary tables in the Staging Area. The data loading and transformation
tasks are performed with the standard E-LT KMs.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 26

Summary

In this lesson, you should have learned to:

• Describe the different styles (E-LT, ETL) of integration

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 27

Practice 4-1: Creating an ODI Mapping for XML to
Database Transformation with ODI Constraint and
Error Recycling
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

In this practice, you perform the steps to build an ODI mapping

that will load an XML file with a constraint to the database table.
To maintain data quality, you enable error recycling. Any rows
that do not pass the constraint will be loaded to the error table
on the target database: E$_CLIENT.

Oracle Data Integrator 12c: Advanced Integration and Development 4 - 28

Designing Advanced Integration Mappings

After completing this lesson, you should be able to:

• Implement lookups in ODI mappings

• Create ODI mappings with multiple datasets
• Use set-based operators in ODI mappings
• Use partitioning with ODI mappings
• Create ODI reusable mappings ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 2

Agenda

 ODI Mappings: Advanced Functionality

– Lookups
– Datasets
– Sets
• Partitioning
• Reusable Mappings e
r a bl
s fe
- t r an
no n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 3

Creating Lookups
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Component
Palette

bl e
More on this fe r a
later…
ans
n - t r
a no
h a s
The order
m ) deฺ
matters!
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cavailable
A Lookupe d e
component l i
is in the Component Palette to create lookups by using a source
N
orin
as the driving table and a lookup datastore or mapping. These lookups now appear as compact
Fl graphical objects in the mapping sources diagram.
This feature simplifies the design and readability of mappings that use lookups, and enables
optimized code for execution.
Ensure that the technology you have chosen supports lookups. There are three possible levels of
Lookup Support: None | Join | Select. In this case, the lookup is just a special case of a join. Drag
the Lookup Component from the Palette on the right and drop it in between the source and target
tables. The Lookup Condition is:
SRC_CUSTOMER.SALES_PERS_ID = SRC_SALES_PERSON.SALES_PERS_ID
You can hook up the two inputs by dragging the connectors, or by selecting them from the
Connected From pull-downs. Be careful which you have as the driving table (usually the bigger
one) and which you have as the lookup table (usually the smaller one) because that affects the
cost-based optimizer in composing the SQL.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 4

Lookup Goal
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Source
Sales Name
Target Sales
Name

bl e
fe r a
an
Attribute s
- t r
Expression
n
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ceusing a lookup is to denormalize a table. The source datastores have the
e d
The business
e reason l i
for
n N
salesperson information in a separate table from the customers, and they are linked by the join
i of SALES_PERS_ID.
F lorcriteria The goal is to add the salesperson’s LAST_NAME and FIRST_NAME
to the target customer datastore.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 5

Lookup Match Row Rules: Multiple Match

What happens if there are three salespersons with that ID?

(Data quality enforcement should prevent this, but just

suppose…)
Which person should be selected?

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ceis an RDBMS, and the tables all have primary keys, and the foreign key
e d e
If your source l
technologyi
n N
relationships
i technologyare set up properly, then you should never get this situation. But what happens if the
F lorsource is a file-based datastore? What happens if the “unique and not null” (primary
key) constraint was never set on the SALES_PERS_ID in SRC_SALES_PERSON? This Lookup
Match Rows Multiple Match option can accommodate bad source data and let you transform it into
good target data. You might have a business rule that declares that the last one in the file “wins.”
If your data was set up properly in the first place, then it does not matter what you put here, the
default is fine.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 6

Lookup Match Row Rules: No-Match

What happens if there is no salesperson with that ID?

Only one
option

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u ce
is d
If there e
el
no match inlithe lookup table, then you want to at least see what was the value that did
not
i n N (like an outer join). To do that, rather than just error out, insert the unmatched row with
match
r
lothe driving table attribute value where the missing lookup value should have been. You can also
F insert a string value, such as “Missing person!”, or another attribute from the driving table, or a
calculated value, such as SYSDATE.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 7

Using Set-Based Operators
with Integration Mappings
• Datasets
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

• Set operators
• Merging several data flows in target(s)
• Mappings and diagrams are per data
flow
• Benefits: e
– Faster design r a bl
s fe
– Better performance - t r an
n no
– 11g compatibility a
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cwith,
e
A dataset dis
eoften l
used i but is unrelated to, a set operator.
N
Fl orAindataset represents the data flow that comes from a group of joined and filtered source
datastores. One target can be loaded with data coming from several datasets.
Set-based operators (UNION, UNION ALL, INTERSECT, MINUS, and so on) are used to merge the
different datastores or datasets into the target datastore(s). DISTINCT (a new component with
12c) is not really a set operator, but acts like one.
Each dataset corresponds to one diagram of source datastores and the mappings used to load the
target datastore from these source datastores.
• Benefits of using datasets:
- Accelerates the mapping design
- Reduces the number of mappings needed to merge several data flows into the same
target datastore
- Allows you to “think” in an 11g-style interface
• Drawbacks of datasets:
- They are no longer required, so they are extra overhead (a performance hit).
- They may make the execution location an all-or-nothing decision.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 8

Example: Flow with Multiple Datasets
SS_0 (1): MSSQL_CRM -

COUNTRY
-
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Staging: MEMORY_ENGINE

CUSTOMER
LKM SQL to DataSet0 - Orders
-
REGION Oracle LOAD)

ORDERS C$ (SS0)
Target: ORACLE_SERVER1

SS_1 (2): DB2_HSCM

- EMPLOYEE

DataSet1 –Lines - SALES

ORDER_LINES LKM DB2 to
ble
CITY
Oracle (LOAD) PERSON

fe r a
C$ (SS1) UNION ALL
an s
CURRENCY

-
IKM SQL
n - t r
no
DataSet2 - Bookings
to SQL
PRODUCTS
s a Append

-
SS_2 (3): DB2_HSCM
C$ (SS0)
h
) deฺ a
LKM DB2
c o m u i
to Oracle
EMPLOYEE
e ฺ MINUS
G
BOOKINGS (LOAD)
r a cl ent
PRODUCTS
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l
(f nseo
c u ce
The useeof
el
ddatasets islioptional, but it may help you to think in terms of a “black box of data”
n N to a reusable mapping, covered later in this lesson).
(similar
i
lo r
F

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 9

Defining a Dataset
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Only one
port: output

ble
fe r a
ans
n - t r
a no
h a s
Connection
m ) deฺ
can be
Lookup or ฺ c o u i
Join.
Join e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e
To create d
a
enew l i
dataset:
n N
Fl ori1. Either create an empty dataset when making a new mapping, or drag a Dataset icon from
the Component Palette into the workspace.
2. Drag datastores into the dataset.
3. Drag an attribute (for example, SRC_ORDERS.CUST_ID) from one datastore onto its
matching attribute in another datastore (for example, SRC_CUSTOMER.CUSTID). Note that
different spelling is allowed for attributes/columns.
4. Decide if you want the connection to be a Join or a Lookup.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 10

Using Set-Based Operators
with Integration Mappings
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ORD_05
ORDERS2

ORDERS1
CORRECT
UNION
LINES
SALES
ble
LINES INTERSECT
fe r a
MINUS
ans
n - t r
MODIF_File1
a no
h a s
MODIF_File1
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce are shown merged into the mapping target datastore using set-
e d
In this slide,
eseveral l i
datasets
i n
based N operators such as UNION and INTERSECT.
lo r
F Note that the support for datasets as well as the set-based operators supported depend on the
capabilities of the Staging Area’s technology.
The set-based operators are always executed on the Staging Area.
When designing the integration mapping, the mappings for each dataset must be consistent. This
means that each dataset must have the same number and type of target columns mapped.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 11

Using Set-Based Operators: Guidelines
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
This icon…
ans
…makes
n - t r
this set.
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce for using set-based operators successfully.
e
This slide d e
offers l i
guidelines
n N
lori
• Make sure that your technology supports set-based operators.
F - Check Support Set Operators.
• In your mapping, add several datasets or datastores:
- Save before adding datasets.
- Share the same target structure.
- Order counts for precedence.
• Select an IKM that supports datasets (though datasets are optional with sets).
- Datasets are merged from the Staging Area.
The yellow (unmatched) and gray (matched) arrows in the UNION set are based on the default
auto-mapping. You will probably have to do some more transformations to get the columns to be
the same number and type. For example, the DEAR in SRC_CUSTOMER is numeric (0, 1, 2), and
the DEAR in the two TRG_ tables is VARCHAR (‘Mr.’, ‘Mrs.’, ‘Dr.’), so a CASE needs to happen. Also
the TRG_ tables are expecting a single CUST_NAME whereas the SRC_ has separate LAST_NAME
and FIRST_NAME that will need to be concatenated. Lastly, you may choose to ignore
SALES_PERS_ID in the SET because there is no match in the TRG_ tables.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 12

Agenda

• ODI Mappings: Advanced Functionality

 Partitioning
• Reusable Mappings

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 13

Using Partitioning with ODI Mappings

Partitioning features:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

• Support (sub)partitioning on datastores.

• Reverse-engineer partition information.
• Enable partition selection in mappings.
• Auto-generate SQL statements:
– SELECT * FROM MY_TABLE PARTITION e
r a bl
(MY_PARTITION)
s fe
• Defined by technology: - t r an
no n
– Partitioning Support: No | Partitions | Sub-Partition
a
– Partition Mask: %OBJECT_NAME h a s
m ) deฺ
PARTITION(%PARTITION) c o ui l eฺ nt G
ra udec
@ St o
l c u is
e d e t h
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ce
e d
ODI is able
e
to use l i
database-defined partitions when processing data in partitioned tables used as
N
in or targetsto the
source of integration mappings. These partitions are created in the datastore
F lorcorresponding table, either through the reverse-engineering process or manually. For
example with the Oracle technology, partitions are reverse-engineered by using the RKM Oracle.
The partitioning methods supported depend on the technology of the datastore. For example, for
the Oracle technology, the following partitioning methods are supported: Range, Hash, List.
After they are defined on a datastore, partitions can be selected when this datastore is used as a
source or a target of a mapping.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 14

Partitioning: Definition in Datastore
After Reverse Engineering
SQL> CREATE TABLE trg_customer
( customer_id NUMBER(6)
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

, cust_first_name VARCHAR2(20)
, cust_last_name VARCHAR2(20)
Partition
, cust_address VARCHAR2(20) key
, nls_territory VARCHAR2(30)
, cust_email VARCHAR2(30))
PARTITION BY LIST (nls_territory) (
PARTITION asia VALUES ('CHINA', 'THAILAND'),
PARTITION europe VALUES ('GERMANY', 'ITALY', 'SWITZERLAND'),
PARTITION west VALUES ('AMERICA', 'MEXICO', 'BRAZIL' ),
PARTITION east VALUES ('INDIA', 'PAKISTAN'));
ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
del
Partitioneinformation licebe reverse-engineered by using Customized (not Standard) Reverse
can
n N Knowledge Modules (RKMs), or can be manually created into models.
Engineering
i
lo r
F

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 15

Using Partitioning in a Mapping
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Blank (null)
means that you
ble
will ignore
fe r a
partitions in
an s
t r
SELECT.
n -
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cspecify
e d e
In a mapping, you l
can i the use of partitions on a target datastore.
N
nUPDATE and INSERT, this may not apply. For SELECT statements, this is a performance
lori
For
F issue, not a “requirement.” But what happens if you do not do this? DML INSERT still works
correctly anyway. This is really metadata, more of a hint than a directive.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 16

Agenda

• ODI Mappings: Advanced Functionality

• Partitioning
 Reusable Mappings

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 17

Configuring Reusable Mappings
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
an s
n - t r
a no
Either this way…
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud …or this way.
@
cu his S
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e
Two typesd e
of ODI l i
mappings:
n N
lori
• Target datastore is permanent (predefined in the model; icon shows a straight arrow).
F • Target datastore is reusable, or temporary (defined in the mapping; icon shows circular
arrows).
You can create a mapping that would not have a predefined, permanent target datastore. In this
case, all rows in the target are derived from the rows of one or more source datastores. These
mappings are referred to as “reusable mappings.” Such mappings are designed to be used by
other mappings as sources or targets for further transformation. In some ways, this is similar to a
dataset.
For purposes of configuring the signature, the input signature uses only the output port, and the
output signature uses only the input port. Even though there is an output port on the reusable icon
in this mapping, it is not intended for use in this context. On the other hand, when it comes time to
use the signature in another mapping (as shown in the next slide), then the input signature uses
only the input port, and the output signature uses only the output port (as their names suggest).
Note: This is similar to an Oracle Database inline view.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 18

Using Reusable Mappings
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Reusable
mapping is used
as a source
datastore in the
permanent
mapping.

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce mapping would show up in lots of other mappings, similar to a
e d
The intention
e is that al i
reusable
n N
database view. (If you only used it once, then why bother defining it as a separate reusable thing?)
F lorIni this example, the reusable mapping myReusable, with its target datastore TRG_SALES, is used
in place of a source datastore for the mapping myMap. You can have other datastores and other
permanent mappings in the integration with the reusable mapping at the same time.
In Mapping Properties, you can see that the rows of the reusable mapping are mapped to the rows
of the target datastore (more has to be done, this is just a start).

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 19

Derived Select for Reusable Mappings
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
If this is selected… a no
h a s
…and it is a reusable
m ) deฺ
mapping…
ฺ c o u i
e
cl ent G
…then this option
may appear. r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l
(f nseo
l c u cedatastore that is the target in reusable mapping as a source or as a
e
When usingd ea temporaryl i
n N
lookup table for another mapping, you can choose:
F lori• To use a persistent temporary datastore: You will run a first mapping creating and loading
the temporary datastore, and then a second mapping sourcing from it. In this case, you
would typically sequence the two mappings in a package.
• To use a sub-select: The second mapping generates a sub-select corresponding to the
loading of the temporary datastore. This option is not always available because it requires all
datastores of the source mapping to belong to the same data server (for example, the source
mapping must not have any source sets). You activate this option by selecting Subselect
Enabled on the source.
Note the following when using a reusable mapping as a derived table:
• The generated sub-select syntax can be either a standard sub-select syntax (default
behavior) or the customized syntax from the IKM used in the first mapping.
• All IKM commands except the one that defines the derived-table statement option “Use
current command for Derived Table sub-select statement” are ignored. This limitation
causes, for example, the disabling of support for temporary index management.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 20

Derived Select for Reusable Mappings:
Limitations
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

REU_1

REU_2

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce to using derived select for reusable mappings. A reusable mapping
There aree d e
some l i
limitations
i n
REU_1 N becomes eligible to be a source derived table in another parent mapping REU_2 if, and
r
loonly if:
F
• All source datastores and submappings of REU_1 are on the same physical server as the
target table of REU_2
• The technology of this physical server supports derived tables in the FROM clause
• No datastore within REU_1 is used as journalized

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 21

Quiz

Which of the following is required for using set-based operators

successfully?
a. Your technology must support set-based operators.
b. You must use a file-based technology.
c. The source and target structures must be different.
d. You must use datasets to support sets. e
r a bl
s fe
- t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Answer:ead
el lice
i n N
lo r
Explanation: Make sure that your technology supports set-based operators. Check Support Set
F Operators.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 22

Quiz

What must you do to support database partitions in datastores?

a. Each partition must be its own datastore.

b. Each partition must be its own logical schema.
c. If your database table uses partitions, you must indicate it.
d. Nothing. If you optionally tell ODI about partitions, then it
will use that information. e
r a bl
s fe
- t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Answer:edd
el lice
i n N
lo r
Explanation: Of course you should tell ODI about the underlying structure for optimization, but
F you do not have to. Reverse-engineering may discover it, but even that is optional.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 23

Summary

In this lesson, you should have learned to:

• Implement lookups in ODI mappings

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 24

Practice 5-1: Configuring Partition Support

In this practice, you configure a mapping to take advantage of

partitioning in its query. Reverse-engineering will discover the

metadata.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 25

Practice 5-2: Configuring Lookup Match Rules

In this practice, you will use datastores and a Lookup with

matching rows rules configured.

Oracle Data Integrator 12c: Advanced Integration and Development 5 - 26

Managing Object Versions

After completing this lesson, you should be able to:

• Describe ODI object version management techniques

• Compare objects by using Version Browser for version
management
• Use Version Comparison Tool to monitor modification
between versions
ble
• Describe methods of handling concurrent changes fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 2

Agenda

 ODI Object Version Management: Overview

– Flags
– Creating
– Restoring
• Version Browser

ble
fe r a
an s
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u celesson examines how ODI supports version management.
e
The first d e
section of l i
this
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 3

What Is Version Management?

• Many different kinds of ODI objects

can be versioned.
• Creating a version generates the
XML definition of an object.
• Versions can be restored into any
Work Repository.
• ODI provides a tool (Version a b le
s f er
Browser) to display versions. an
- t r
• Versions of ODI objects can be n on
compared by using the Version s a
a
h eฺ
Comparison Tool, highlighting the om) id
differences in colors. e ฺ c G u
cl nt ra ude
@ St o
l c u is
e d e t h
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ce can be versioned:
e d e
The following ODI l i
objects
n N
lori
• Projects • Scenarios
F • Folders • Model Folders
• Packages • Models
• Mappings • User Functions
• Procedures • Variables
• Sequences
A version is a backup copy of an object. It is checked in at a given time and may be restored
later. Versions are saved in the Master Repository. They are displayed on the Versions tab of
the object window.
Creating a version creates the XML definition of the object, and then stores it in compressed
form in the Master Repository version tables.

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 4

Working with Object Flags

When an object in Designer is created or modified, an icon flag

is displayed in the tree view:

bl e
fe r a
Object Statusniss
a
n-tr
“U” = Updated
no
s a
h a
o m ) Object
i d eฺ status is
l e ฺc t Gu“I” = Inserted
o rac uden
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce is created or modified, a flag is displayed in the tree view on the
When ane d e
object in l i
Designer
n N
lori
object icon to indicate its status:
F • I = Object status is inserted.
• U = Object status is updated.
When an object is inserted, updated, or deleted, its parent objects are recursively flagged as
updated. For example, when a step is inserted into a package, it is flagged as inserted, and
the package, folder(s), and project containing this step are flagged as updated. When an
object version is created, the flags on this object are reset.

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 5

Creating Versions
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

bl e
fe r a
ans
- t r
no3n a
a s
m ) h deฺ
1 ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d
To create
e
a version:l i
n N
lori
1. Select the object for which you want to check in a version. Right-click and select Version
F > Create Version.
2. A window appears. In this window, the Previous Versions (>) button in the top-right
corner expands or collapses the list of versions already checked in. An incremented
version number is automatically generated in the Version field. Modify this version
number if necessary.
3. Enter the details for this version in the Description field. Click OK.
Note: When editing the object, the Versions tab provides a list of versions checked in, with
the checkin date, description, and the name of the user who performed the checkin operation.

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 6

Restoring Versions

Restoring a version cannot be undone!

2 e
r a bl
s fe
- t r an
n no 3
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
1 @o
u S tud
d e lc this
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce cannot be undone. It permanently erases the current object and
e d e
Note: Restoring a l i
version
n N
lori
replaces it with the selected version.
F To restore a version:
1. Select the object for which you want to restore a version. Right-click and select Version
> Restore. A window appears with the list of existing versions.
2. Select the version you want to restore and then click OK.
3. Click Yes to confirm the restore operation.

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 7

Agenda

• ODI Object Version Management: Overview

 Version Browser
– Comparisons
– Solutions
– Locking for concurrent changes

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u celesson examines how the Version Browser works.
e
The nextd e
section of l i
this
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 8

Using Version Browser
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Export
version
as an
XML file.

1
2 Restore the
version.
ble
fe r a
an s
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl e3nt G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce Browser to browse the versions stored in the repository.
e
You can d e
use the l i
Version
n N
lori
1. In the ODI menu, select Version Browser.
F 2. Use the object type and object name drop-down boxes to filter the objects for which you
want to display the list of versions.
3. From the Version Browser, you can restore, export to XML file, or delete existing
versions.
Note: The Version Browser displays the versions that existed when you opened it. Click
Refresh to view all the new versions created since then.

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 9

Using the Version Comparison Tool

• The Version Comparison Tool is a graphical tool used to

view and compare two different versions of an object.

• The comparison results are displayed in two panes, and
the differences between the two compared versions are
color coded.
• Comparison results are organized in a tree view.
• The Version Comparison Tool can generate and print a erab
le
PDF report listing the differences between two particular
a n sf
tr
versions of an object. on- n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce Tool provides the following features:
e d e
The Version l i
Comparison
n N
lori
• Color-coded, side-by-side display of comparison results: The comparison results
F are displayed in two panes, side-by-side, and the differences between the two
compared versions are color coded.
• Comparison results organized in tree view: The tree view of the comparison tool
displays the comparison results in a hierarchical list of node objects in which expanding
and collapsing the nodes is synchronized.
• Report creation and printing in PDF format: The Version Comparison Tool is able to
generate and print a PDF report listing the differences between two particular versions
of an object.
• Supported objects: The Version Comparison Tool supports the following objects:
Project, Folder, Package, Scenario, Interface, Procedure, Knowledge Module,
Sequence, User Function, Variable, Model, Model folder, and Solution.
• Difference viewer functionality: This Version Comparison Tool is a difference viewer
and is provided only for consultation purposes. Editing or merging object versions is not
supported. If you want to edit the object or merge the changes between two versions,
you have to manually make the changes directly in the relevant objects.

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 10

Color Code
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce highlighted. The following color code is applied:
e d e
The differences are l i
color
n N
lori
• White (default): Unchanged
F • Red: Deleted
• Green: Added/new
• Yellow/Brown: Modified (the value inside this field has changed)
Note: If one object does not exist in one of the versions (for example, when it has been
deleted), it is represented as an empty object (with empty values).

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 11

Opening the Version Comparison Tool

…or…
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

1
ble
fe r a
ans
n - t r
a no
h a s2
m ) deฺ
Modified
ฺ c o u i
entries e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cine the Projects tree view. From the Projects tree view in Designer,
e d
• Select
e the l i
object
i n Nselect the object whose versions you want to compare.
r
Flo - Right-click the object.
- Select Version > Compare with version. The Compare with window appears.
- Select the version with which you want to compare the current version.
- Click OK. The Version Comparison Tool opens.
• Using the Versions tab of the object:
- In Designer, open the object whose versions you want to compare.
- Open the Versions tab. The Versions tab provides the list of all versions created
for this object. Select the two versions that you want to compare by keeping the
Ctrl key pressed.
- Right-click and select Compare. The Version Comparison Tool opens.
• Using the Version Browser:
- In Designer, select File > Version Browser.
- Select the two versions that you want to compare.
- Right-click and select Compare. The Version Comparison Tool opens.
The comparison window shows the differences between two versions: on the left pane, the
newer version and on the right pane, the older version of your selected object.
Oracle Data Integrator 12c: Advanced Integration and Development 6 - 12
Working with Solutions

2
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

3 e
r a bl
s fe
- t r an
n no
s a
h
) deฺ a
1 c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
e d
A solution
e
is a l i
comprehensive and consistent set of interdependent versions of objects. Like
N
orin
other objects, it can be checked in at a given time as a version, and may be restored at a later
Fl date. Solutions are saved into the Master Repository. A solution assembles a group of
versions called the solution’s elements.
A solution is automatically assembled by using cross-references. By scanning cross-
references, a solution automatically includes all dependent objects required for a particular
object. For example, when adding a project to a solution, versions for all the models used in
this project’s interfaces are automatically checked in and added to the solution. You can also
manually add or remove elements into and from the solution. Solutions are displayed in the
Solutions tree in Designer Navigator.
These objects may be added to solutions: Projects, Models, Model Folders, Load Plans,
Scenarios, Global Variables, Global User Functions, Global KMs, and Global Sequences.
To Create a Solution:
1. Select Designer tab > Solutions > New Solution.
2. In the Solution editor, enter the name of your solution and a description, and then save
the solution. The resulting solution is an empty shell into which elements may then be
added.

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 13

3. To add an element, drag the object from the tree into the Principal Elements list in the
Solution editor. ODI scans the cross-references and adds any Required Elements
needed for this element to work correctly.
4. You will be asked if you want to check in all versions automatically. Click Yes.
Note: If the objects being added have been inserted or updated since their last checked-in
version, you will be prompted to create new versions for these objects.
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Removing Elements
To remove an element from a solution, select the element that you want to remove in the
Elements list and then click the Delete button. This element is removed from the list. Existing
checked-in versions of the object are not affected.
Comparing Solutions with Smart Import/Export
Solutions were meant to be used for versioning, whereas smart import/export is really meant
to help with import/export dependencies management. It does not manage versions at all. But
ble
r
it is capable of handling missing dependencies in particular for imports. It will also let you
fe a
s
choose what action to take if you are trying to import an object that already exists (overwrite,
an
keep existing object).
n - t r
n o
s a
h a
) deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
l o r in to u
u (f nse
e l c l i ce
e d
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 14

Working with Solutions: Synchronizing
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

1 2

ble
fe r a
3
an s
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce automatically adds required elements that have not yet been
e d e
Synchronizing a l i
solution
n N
lori
included in the solution, creates new versions of modified elements, and automatically
F removes unnecessary elements. The synchronization process brings the content of the
solution up to date with the elements (projects, models, and so on) stored in the repository.
To Synchronize a Solution:
1. Open the solution that you want to synchronize.
2. Click Synchronize in the toolbar menu of the Principal Elements section. ODI scans
the cross-references.
3. If the cross-reference indicates that the solution is up to date, a message appears.
Otherwise, a list of elements to add or remove from the solution is shown. These
elements are grouped into Principal Elements (added manually), Required Elements
(directly or indirectly referenced by the principal elements), and Unused Elements (no
longer referenced by the principal elements). Select the Accept boxes to version, and
include the required elements or delete the unused ones. Then click OK to synchronize
the solution.
Note
• Version creation windows may appear for elements that require a new version to be
created.
• You must synchronize your solutions regularly to keep the solution contents up to date.
You must also synchronize before checking in a solution version.

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 15

Handling Concurrent Changes:
Concurrent Editing Check
ODI provides two methods for
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

handling concurrent changes:

• Concurrent Editing Check:
– Select this parameter to
avoid erasing the work
performed by another user.
– If another user has made ble
changes, the object cannot fe r a
ans
be saved.
n - t r
• Object Locking a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
licesimultaneously in the same ODI project or model. They may all be
del can work
Severaleusers
i n N to the same repository and so the changes they perform are considered
connected
lo r
concurrent.
F
ODI provides two methods for handling these concurrent changes: Concurrent Editing Check
and Object Locking. These two methods can be used simultaneously or separately.
• Concurrent Editing Check: The “Check for concurrent editing” preference can be
selected to prevent you from erasing the work performed by another user on the object
that you try to save. If this parameter is set to true, when you save changes to any
object, ODI checks whether other changes have been made to the same object by
another user since you opened it. If another user has made changes, the object cannot
be saved, and you must cancel your changes.

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 16

Handling Concurrent Changes:
Object Locking
ODI provides two methods for
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

handling concurrent changes:

• Concurrent Editing Check
• Object Locking:
– Automatically activated. A
pop-up window appears to
ask if you want to lock the ble
fe r a
object. s
– Manual locking - t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d
• Object
e Lockingl i
i n N
r - Automatic Object Locking – This mechanism is automatically activated. When an
Flo object is opened in a user interface, a pop-up window appears to ask if you want to
lock the object. As long as an object is locked, only the user who owns the lock
can modify the object, such as editing or deleting. Other operations, such as
executing, can be performed by other users, but with a warning. An object locked
by you appears with a yellow lock icon. An object locked by another object
appears with a red lock icon. When the editing window is closed, a pop-up window
appears to ask if you want to unlock the object.
- You can also manually manage locks on objects. To manually lock an object,
select the object in the tree view, right-click, and select Locks > Lock.

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 17

Quiz

How would you undo the restoring of a version of an object?

a. Import an “original” version that you had earlier exported.

b. Press Ctrl + Z “Undo”.
c. Disconnect Studio from the Master Repository and
reconnect again.

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u c e
Answer: d
ea e l i
i n N
F lorExplanation: Statement A is true. In one sense, restoring a version cannot be undone. It
permanently erases the current object and replaces it with the selected version.
However, you can restore any versions of an object you exported earlier, by importing that
earlier version. See page 6-7.

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 18

Summary

In this lesson, you should have learned how to:

• Describe ODI object version management techniques

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 19

Practice 6-1 Overview:
Working with Versions
1. Create a mapping.
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

2. Create a new version 1.0.0.0 of the mapping.

3. Change some mapping details of the mapping, and then
create another version of the mapping, version 1.0.0.1.
4. Use Version Browser to compare versions of the mapping.

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cBrowser
e
Use the d e
ODI Versionl i and Version Comparison Tool to compare these two versions of
N
n mapping, and detect the differences between them.
lori
the
F Generate a PDF report of the versions, and then restore the original version.

Oracle Data Integrator 12c: Advanced Integration and Development 6 - 20

Using Variables in ODI

After completing this lesson, you should be able to:

• Describe variables’ scope

• Use variables in object properties and in the resource
name of a datastore
• Use variables within variables
• Use variables as startup parameters le
a b
• Use variables in topology s f er
• Perform tracking of variables - t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 2

Agenda

 Using Variables: Overview

• Using Variables in ODI Objects

• Tracking Variables

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 3

Using Variables in ODI: Overview

• Using variables is highly recommended for:

– Creating reusable packages or packages with complex

conditional logic
– Loops and counters
– Mappings
– Procedures
• Variables can be used everywhere within ODI. ble
fe r a
– Project scope ans
n - t r
– Global scope no
• Variables’ values can be stored: s a
h
) deฺ a
– Persistently in the ODI Repositoryom i
e ฺ c G u
– Transiently in the RAM of the
r a clagentent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u cerecommended to create reusable packages or packages with a complex
e d e
Using variables is l i
highly
n N
conditional logic, or in mappings and procedures.
F lori
Variables can be used everywhere within ODI. Be mindful of whether they are in the Project node
or the Global node for scope.
Variables’ values can be stored persistently in the ODI Repository if their Keep History parameter
is set to All values or Latest value. Otherwise, if their Keep History parameter is set to No
History, their value will be kept in the memory of the agent only during the execution of the current
session.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 4

Variable Scope

• Use the Expression Editor to refer to your variables in

packages, integration mappings, and procedures.

• When Expression Editor is not available, manually prefix
variable names with GLOBAL or the PROJECT_CODE.

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce Editor (invoked by clicking the Target Expression gear), the
When youe d e
use the l i
Expression
n N
variables are retrieved directly from the repository.
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 5

Referring to a Variable

Refer to the MY_VAR variable in your objects as follows:

• #MY_VAR
• #MY_PROJECT_CODE.MY_VAR
• #GLOBAL.MY_VAR

Objects
referring to
Variable X
ble
variables
fe r a
an s
Variable X
- t r
Project B
n
a no
Project A
h a s
m ) deฺ
ฺ c o u i
Global
e
cl ent G
Variable X
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
l this syntax,ce the variable must be in the same project as the object referring to it. Its
e
#MY_VAR: d e
With l i
n
valueNwill be substituted.
F lori • To avoid ambiguity, consider using fully qualified syntax by prefixing the variable name with
the project code
#MY_PROJECT_CODE.MY_VAR: Using this syntax allows you to use variables by explicitly stating
the project that contains the variable.
• It prevents ambiguity when two variables with the same name exist, for example, at global
and project level. The value of the variable will be substituted at run time.
#GLOBAL.MY_VAR: This syntax allows you to refer to a global variable. Its value will be
substituted in your code.
• You reference an ODI variable prefixed with the # character. ODI substitutes the name of the
variable by the value before the code is executed (recommended).
• The call of the variable works for OS commands, SQL, and ODI API calls.
• Example:
TOWN = '#GLOBAL.VAR_TOWN‘
Case sensitive: My_Var, my_var, MY_VAR are three different variable names.
Types: numbers (scalar, not array), text, alphanumeric (mix of numbers and text), and dates

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 6

Referring to a Variable: Using “:” Instead of “#”

You can use the variable as a SQL bind variable by prefixing it

with a : rather than a # (substitution).

• The variable is substituted later, when the RDBMS
executes the request (binding).
• Execution plans are reused by CBO.
• It is not necessary to enclose in quotation marks the
bl e
variables that store strings. Example:
fe r a
TOWN_NAME = :GLOBAL.VAR_TOWN_NAME ans
n - t r
• Restrictions: This syntax applies only to some SQL
a no
statements, not for OS commands nor ODI h a sAPI calls.
m ) deฺ
Note: Using the bind variable may resultฺ c o in u i
l e G
performance loss.
o tud rac ent
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
You caneuse dethe liceas a SQL bind variable by prefixing it with a colon “:” rather than a hash
l variable
“#”
i nforNsimple substitution. However, this syntax is subject to restrictions because it applies only to
r
FloSQL SELECT and DML statements, not for OS commands or ODI API calls, and using the bind
variable may result in performance loss. It is advised to use variables prefixed with the # character
to ensure optimal performance at run time. Every SQL statement is compiled fresh every time.
When you reference an ODI variable prefixed with the : character, the name of the variable is
NOT substituted when the RDBMS engine determines the execution plan. The variable is
substituted when the RDBMS executes the request. This mechanism is called binding. If you use
the binding mechanism, it is not necessary to enclose in delimiters (such as quotation marks) the
variables that store strings, because the RDBMS is expecting the same type of data as specified
by the definition of the column for which the variable is used.
For example, if you use the variable TOWN_NAME = :GLOBAL.VAR_TOWN_NAME, the VARCHAR
type is expected, so quotation marks are not needed.
Note: When you reference an ODI variable prefixed with the # character, ODI substitutes the
name of the variable with the value before the code is executed by the technology. In expressions,
the variable reference needs to be enclosed in single quotation marks, for example:
TOWN = '#GLOBAL.VAR_TOWN'.
The call of the variable works for OS commands, SQL, and ODI API calls.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 7

Agenda

• Using Variables: Overview

 Using Variables in ODI Objects

• Tracking Variables

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 8

Using Variables in Packages

The four types of variable steps in packages:

• Declare Variable explicit (or implicit)

• Set Variable value or increment

ble
fe r a
ans
• Refresh Variable via SQL n - t r
a no
h a s
m ) deฺ
• Evaluate Variable for loops ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u cein packages. The four types of variable steps are as follows:
e
Variables d e
are often l i
used
n N
lori
• The Declare Variable step is often placed at the start of a package. It explicitly defines the
F variable that is used in a mapping, procedure, or other package step. In general, variables
are implicitly declared when they are referred to in a package. However, in certain complex
situations, this cannot be guaranteed. Using this kind of step also improves readability,
because you can visually display all the variables that are used in the package.
• The Set Variable step is straightforward and can be used to perform two different actions:
- You can assign a specific value to a variable. For example, you can set an error
counter to zero.
- You can increment the value of a variable.
• The Refresh Variable step updates the variable value by executing the SQL expression of
the variable. Note that variables are not automatically updated. For example, you have a
variable with an expression that retrieves the number of rows in a table.
• The Evaluate Variable step compares the value of the variable against a constant value or
another variable. This comparison is used to create branches and loops. You can use any
mathematical comparison, such as equals, greater than, less than, and so on. You can also
use the SQL membership operator “IN.” The execution path of the package then splits
according to the result of the comparison. Using the “ok” and “ko” tools, you define the “true”
and “false” comparison paths.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 9

Using Variables in Packages: Example

Using variables to control the workflow:

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o stepGu
i
The Refresh Variable step The Evaluate e
clthe valueeofnthe
Variable
t
updates the variable value
r a
compares
by executing the SQL
expression of the variable. u @ o against
variable
value toS tudbranches.
create
a constant

d e lc this
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce of using variable steps in a package to control the workflow. The
e
This slide d e
shows an l i
example
n N
Refresh
i Variable step (implicitly declares, then) updates the variable value by executing the SQL
F lorexpression of the variable. The Evaluate Variable step compares the value of the variable against
a constant value to create true/false branches.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 10

Using Variables in Mappings

Variables can be used in mappings in two different ways:

• As a value for a textual option of a KM

• In all ODI expressions such as attribute expressions,
filters, joins, and constraints
– To substitute the value of the variable into the text of an
expression, precede its name with the # character:
ble
update CLIENT set LASTDATE = sysdate where DATE_YEAR =
fe r a
ns
'#GLOBAL.YEAR' /* DATE_YEAR is CHAR type */
t r a
– You can also use bind variable mechanism (less efficient):
n o n-
update CLIENT set LASTDATE = sysdate where DATE_YEAR
s a =
:GLOBAL.YEAR a
) h deฺ
c m
o“date” utype i variables that
– Bind mechanism must be usedefor ฺ G
a
are used in a filter or join rexpressions: cl ent
@ o tud
SRC.END_DATE > :SYSDATE_VAR
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cofethe variable into the text of an expression, precede its name with the
d e l i
echaracter. The agent or GUI will substitute the value of the variable in the command
To substitute the value
n
hash N
“#”
Fl ori
before executing it.
The following examples show the use of a global variable named “YEAR”:
Update CLIENT set LASTDATE = sysdate where DATE_YEAR = '#GLOBAL.YEAR'
/* DATE_YEAR is CHAR type, with quotes */
Update CLIENT set LASTDATE = sysdate where DATE_YEAR = #GLOBAL.YEAR
/* DATE_YEAR is NUMERIC type */
The “bind variable” mechanism of the SQL language can also be used. However, this is less
efficient, because the relational database engine does not know the value of the variable when it
constructs the execution plan for the query. To use this mechanism, precede the variable with the
colon “:” character, and make sure that the data type being searched is compatible with that of the
variable. Example:
update CLIENT set LASTDATE = sysdate where DATE_YEAR = :GLOBAL.YEAR

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 11

The “bind variable” mechanism must be used for date type variables that are used in a filter or join
expression. The following example shows a filter:
SRC.END_DATE > :SYSDATE_VAR where the variable SYSDATE_VAR is a “date” type variable
with the refresh query:
select sysdate into SYSDATE_VAR from dual
If the substitution method is used for a date variable, you need to convert the string into a date
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

format by using the RDBMS-specific conversion function.

Note: You can drag a variable into most expressions with the Expression Editor.

ble
fe r a
ans
n - t r
n o
s a
h a
) deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
l o r in to u
u (f nse
e l c l i ce
e d
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 12

Using Variables in Mappings: Examples

• Expression:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

'#PRODUCT_PREFIX' || PR.PRODUCT_CODE

• Join:
CUS.CUST_ID = #DEMO.UID * 1000 + FF.CUST_NO

• Filter: bl e
fe r a
ORDERS.QTY between #MIN_QTY and #MAX_QTY
ans
n - t r
a no
• Option value: a s
TEMP_FILE_NAME: #DEMO.FILE_NAME h
) deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u ce
l '#PRODUCT_PREFIX'
e d e
• Expression: l i || PR.PRODUCT_CODE
N
in - Concatenates the current project’s product prefix variable with the product code. As the
r
Flo value of the variable is substituted, you need to enclose the variable with single straight
quotation marks because it returns a string.
• Join: CUS.CUST_ID = #DEMO.UID * 1000 + FF.CUST_NO
- Multiply the value of the UID variable of the DEMO project by 1000 and add the
CUST_NO column before joining it with the CUST_ID column.
• Filter: ORDERS.QTY between #MIN_QTY and #MAX_QTY
- Filter orders according to the MIN_QTY and MAX_QTY thresholds.
• Option Value: TEMP_FILE_NAME: #DEMO.FILE_NAME
- Use the FILE_NAME variable as the value for the TEMP_FILE_NAME option.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 13

Using Variables in Object Properties

It is also possible to use variables as substitution variables in

graphical module fields such as resource names or schema

names.

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
del to use
It is alsoepossible licevariables as substitution variables in graphical module fields such as
n N names or schema names in the topology. You must use the fully qualified name of the
resource
i
r
lovariable (Example: #GLOBAL.MYTABLENAME) directly in the ODI graphical module’s field.
F
Using this method, you can parameterize elements for execution, such as:
• Physical names of files and tables (Resource field in the datastore) or their location (Physical
Schema’s schema [data] in the topology)
• Physical Schema
• Data Server URL

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 14

Using Variables in Procedures: with Jython

You can use variables anywhere within your procedures’ code.

Examples:
• Step 1: SQL
Insert into #DWH.LOG_TABLE_NAME
Values (1, 'Loading Step Started', current_date)
– Add a row to a log table that has a name only known at run e
time. r a bl
s fe
• Step 2: Jython
- t r an
f = open('#DWH.LOG_FILE_NAME', 'w') no n
f.write('Inserted a row in table %s' a %s
a
h
) deฺ
('#DWH.LOG_TABLE_NAME') )
c o m u i
f.close() l e ฺ t G
ra ude c n
– Open file defined by theoLOG_FILE_NAME variable and write
@ t
s Swhich you have inserted a row.
lcutable iinto
the name of the log
n e de e th
i nฺ © t2015, s and/or its affiliates. All rights reserved.
uOracle
o r
Copyright
o
u (fl nse
lc lice
devariables
You caneuse anywhere within your procedures’ code. Code examples are shown in this
i n
slide.N“DWH” is the project name. See the next slide for alternatives with Options.
lo r
F

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 15

Using Variables in Procedures: with Options

• Consider using options rather than variables whenever

possible in procedures.
• Example: Adding a row to a log table that has a name only
known at run time
– Using parameters:
Insert into #DWH.LOG_TABLE_NAME
ble
Values (1, 'Loading Step Started', current_date)
fe r a
ans
n - t r
– Using options: no
s a
h a
eฺ
Insert into <%=snpRef.getOption("LogTableName")%>
m )current_date)
d
Values (1, 'Loading Step Started', o
ฺc t Gu i
l e
o rac uden
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce options rather than variables whenever possible in procedures.
e d
You should
e
consider l i
using
n N
Options
i setactyour
like input parameters. Therefore, when executing your procedure in a package, you
F lorwould option values to the appropriate values.
In this example, you would write Step 1’s code as follows:
Insert into <%=snpRef.getOption("LogTableName")%>
Values (1, 'Loading Step Started', current_date)
Then, when using your procedure as a package step, you would set the value of option
LogTableName to #DWH.LOG_TABLE_NAME.
Note that when using Groovy scripting (discussed in the lesson titled “Using ODI Groovy Editor”),
you need to enclose the variable name in straight double quotation marks ("), for example
"#varname" and "#GLOBAL.varname"; otherwise, the variables are not substituted with the
ODI variable value.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 16

Using Variables Within Variables

You may have variables that depend on other variable values,

for example:
• STORE_ID: Alphanumeric variable. It is passed as a
parameter to a scenario.
– Gives the ID of a store
• STORE_NAME: Alphanumeric variable
– The name of the current store is derived from the Stores a b le
e r
table filtered by the value returned by the concatenationaof
n sf
the STORE_ID and STORE_CODE variables. on-t
r
a n
– The code: s
h
) deฺ a
select STORE_NAME from
c o m u i
e ฺ G
r a cl ent
<%=odiRef.getObjectName("L","STORES","D")%>
o tud
Where id='#DWH.STORE_ID'||'#DWH.STORE_CODE'
@
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
ed el lice
It is sometimes useful to have variables depend on other variable values as shown in this slide.
N
in The “bind variable” mechanism must be used to define the refresh query for a “date” type
F lorNote:
variable that references another “date” type variable. Example:
MY_NOW date type variable has the refresh query:
select sysdate from dual
MY_TODAY date type variable must have the bind-based refresh query:
select :MY_NOW from dual
So you see that MY_TODAY has a “nested” variable inside of it; it is dependent on MY_NOW.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 17

Using Variables in the Resource Name
of a Datastore
Sometimes the names of your source or target datastores are
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

dynamic.
• Example: You need to load flat files into your Data
Warehouse (DWH) with a file name composed of a prefix
and a dynamic suffix such as the current date:
– ORD2014.03.26.dat
• Method of implementation: bl e
fe r a
– Create the FILE_SUFFIX variable in your DWH projectaand
t r ns
set its SQL SELECT statement to select current_date
n on- (or
any appropriate date transformation to match
s athe actual file
suffix format). )h a ฺ
Note: Variables here must have been ฺ c o m
u i declared.
previously
de
l e
c ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce where the names of your source or target datastores are dynamic.
d e l i
You may
Ain Neexample of this is when you need to load flat files into your Data Warehouse with a file
typical
face some situations
r composed of a prefix and a dynamic suffix such as the current date. For example, the order
Floname
file for March 26 would be named ORD2014.03.26.dat.
Note that you can use variables in the resource name of a datastore only in a scenario when the
variable has been previously declared. This is one of the exceptions to the implicit declare rule.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 18

Using Variables in the Resource Name
of a Datastore: Steps
To develop your loading mappings:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

1. Create the FILE_SUFFIX variable in your DWH project

(for example, ORD2014.03.26.dat).
2. Define your ORDERS file datastore in your model and set
its resource name to ORD#DWH.FILE_SUFFIX.dat.
3. Use your file datastore in your mappings.
a b le
4. Design a package as follows: fer s
tran
a. Drag the FILE_SUFFIX variable to the package diagram.
n-
no into the
b. Drag all mappings that use the ORDERS datastore
a
package diagram. has ) deฺ
At run time, the source file name willฺcbe substituted o m u i with the
l e t G
appropriate value.
o rac uden
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cedatastore resource name must be fully qualified with its project code.
e
Note: Thed e
variable inl i
the
i n
When Nusing this mechanism, it is not possible to view the data of your datastore from within
F lorDesigner.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 19

Using a Variable as a Startup Parameter: Example

Stores the value of the

dynamic name of the
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

flat file to be loaded

into the RDBMS table.

#myDemoVar is
used as a resource
ble
parameter for the
fe r a
2 datastore.
an s
n - t r
a no
Create the
h a s
integration mapping
m ) deฺ
for flat file-to-RDBMS
ฺ c o u i
transformation. e
cl ent G
r3a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l
(f nseo
c u
del projects,
In someeintegration lice a flat file needs to be exported into a relational table. In this example:
i n
1.
NThe variable myDemoVar is created and used as a dynamic name of a flat file to be
o r
Fl exported to a relational database table.
2. To reference the flat file dynamically, you edit the source datastore to point to the variable
rather than having the resource file name hard-coded: #myDemoVar.
3. You create a mapping to export a flat file to a relational table. Nowhere in the mapping is
there any indication that you are using variable file names.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 20

Using a Variable as a Startup Parameter: Example
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Create a package with the

variable FileName and the
mapping.

From the package,

4 generate the scenario.
The variable is used as
ble
the startup parameter.
fe r a
5 an s
n - t r
no
s a
h
) deฺa
c o m u i
e ฺ G
6 racl e n t
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cwith
e d
4. Create
ea l
package i the variable and the mapping.
N
i5.n From the package, generate a scenario.
r
Flo 6. Specify that the project variable be used as a startup parameter. You can just accept the
default if there is only one choice.
Note about the GUI: The two unlabeled check boxes to the left of the up/down arrowheads
are actually buttons. They are “Check All” and “Uncheck All.” They are unlabeled, but the
hover tool tip shows their purposes.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 21

Using a Variable as a Startup Parameter: Example
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

The name of the file to

7 be used in the
integration process is
defined as a startup
parameter when you
execute the scenario.

ble
fe r a
t r a ns8
n- no
s a
h
) deฺ a
c o m u i
e ฺ
l deselectnt G
r a cMust e
@ o “Latest t u dValue”
e l cu his S
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e
7. Execute the l i
scenario.
i n
8.
N
ODI prompts for the actual name of the flat file (case-sensitive in Linux) to be loaded into the
r
Flo relational table that will be set as a startup parameter.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 22

Using ODI Variables in Topology

• If your processes run on a large number (hundreds) of

systems, the exact code will be executed on different

physical locations.
• In this case, you have to either:
– Create and maintain hundreds of contexts
(harder to do)
Or a b le
– Create a dynamic environment by using topology variables s f er
(easier to do) - t r an
n n o
• In case of using variables in topology, it is s a
recommended
to create at least two contexts: h
) deฺ a
c o m u i to actual servers
– Development context where all e ฺ
URLs point
G
r a cl ent
– Dynamic context that will
u @ S tud names
o use variable
d e lc this
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
e del lice
Each server and each schema on the server have to be defined in the ODI topology. Now imagine
n N
that you design processes that have to be run on a large number of systems. The exact same
i
o r
Fl code will be executed; only the physical location will be different. In this case, you have to either:
• Create and maintain thousands of contexts (bad)
• Create an environment that will be more dynamic by using variables in topology (good)
It is recommended to create at least two contexts:
• A development context where all URLs in topology point to an actual (hard-coded) server,
not using the variables. This will ensure that data can be viewed, and interfaces can be
tested without any concerns regarding the variables’ resolution.
• The dynamic context will use the variables in topology to name the servers, port numbers,
usernames for the connections, and so on. The package will assign the appropriate values to
the variable and run the interfaces on the appropriate servers. This context will be used only
to validate that the processes defined in the Development context work properly when you
use the variables.
Independently from ODI, you will need a table to store the values that will be used for the topology
variables.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 23

Using ODI Variables in Topology: A Server URL

In some cases, you can refer to a variable in the URL of a

server’s definition (rather than using contexts for different

locations):
• When the number of sources is high (>100)
• When topology is defined externally in a separate table
Example:
ble
jdbc:oracle:thin:@#EDW.STORE_URL
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ceserver and set its JDBC URL, as shown in the slide.
e d
Define one
e
physical l i
data
n N
i the package for loading data for your store:
F lorDefine
• The input variable STORE_ID will be used to refresh the values for STORE_URL and
STORE_ACTIVE variables from the StoresLocation table.
• If STORE_ACTIVE is set to “YES”, then the next steps will be triggered. The mappings refer
to source datastores located according to the value of the STORE_URL variable.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 24

Using Variables in a Server URL: Example

1. Create a StoresLocation table as follows:

Store ID Store Name Store URL Is Active

101 Denver 10.21.32.198:1521:ORA1 NO
102 San Francisco 10.21.34.119:1525:SANF YES
… … … …
199 New York City 10.21.34.11:1521:NYC NO

2. Create three variables in your Enterprise Data Warehouse le

a b
(EDW) project: fer ns
– STORE_ID: Takes the current store ID as an input parameter
t r a
on- ID with
– STORE_URL: Refreshes the URL for the current store
n
the SELECT statement: select StoreUrl from a s
ha
StoresLocation where StoreId = #EDW.STORE_ID
) activity eฺ indicator for
– STORE_ACTIVE: Refreshes the current o m u i d
the current store ID with SELECT l e ฺcstatement:t G
select IsActive fromorStoresLocation ac den
where StoreId =cu @ Stu
d e l t h is
#EDW.STORE_ID

ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ceby using a variable in the URL of your data server’s definition:
e d
To connect
e
to your l i
stores
n N
Fl ori1. Create a StoresLocation table, such as the one shown in the slide. Only one location
should be active at a time.
2. Create three variables in your EDW project:
- STORE_ID: Takes the current store ID as an input parameter
- STORE_URL: Refreshes the current URL for the current store ID with the statement:
select StoreUrl from StoresLocation
where StoreId = #EDW.STORE_ID
- STORE_ACTIVE: Refreshes the current activity indicator for the current store ID with
the SELECT statement:
select IsActive from StoresLocation
where StoreId = #EDW.STORE_ID
Only one location should be active at a time.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 25

Using Variables in a Server URL: Example

3. Define one physical data server for all your datastores and
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

set its JDBC URL to:

– jdbc:oracle:thin:@#EDW.STORE_URL
4. Define your package for loading data from your store.
– The input variable STORE_ID will be used to refresh the
values for STORE_URL and STORE_ACTIVE variables from
the StoresLocation table.
a b le
– If STORE_ACTIVE is set to “YES,” then the next steps willnsbefe
r
-
triggered. Mappings refer to source datastores located
n tra
o
an
according to the value of the STORE_URL variable.
s
ha eyou
To start such a scenario for the New York)store, ฺ would
issue the following OS command: ฺco m uid
a c le nt G
./startscen.sh LOAD_STORE 001
@ or PRODUCTION
t u de "EDW.STORE_ID=199"

e l cu his S
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cwhere
e
There are d e
some l
cases i using contexts for different locations is less appropriate than using
N
in (>100)in the
variables URL definition of your data servers, for example, when the number of sources is
F lorhigh or when the topology is defined externally in a separate table. In these cases, you can
refer to a variable in the URL of a server’s definition.
Suppose you want to load your warehouse from 250 source applications—hosted in Oracle
databases—used within your stores. Of course, one way to do it would be to define one context for
every store. However, doing so would lead to a complex topology that would be difficult to
maintain. Alternatively, you could define a table that references all the physical information to
connect to your stores and use a variable in the URL of your data server’s definition.
Syntax of the startscen, the 001 is the scenario version. You can use a version of -1, and that
will always use the latest version.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 26

Agenda

• Using Variables: Overview

• Using Variables in ODI Objects

 Tracking Variables

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 27

Tracking Variables
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Choices from 0-6,

default is 5

ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u cewith Log Level 6.
e
Tracking d e
requires l i
running
i n N
F lorTracking
•
variables enables you to determine:
Actual values of ODI user variables that were used during an executed session
• Current only or historical values
Variable and sequence tracking record a history of variables and sequences participating in a
session. The values of all variables are displayed in the Operator step in the section “Variable and
Sequence Values.”
With the variable tracking feature, you can also determine whether the variable was used in a
source/target operation or an internal operation such as an Evaluate step. You get a slightly
different display when looking at an incrementing counter step.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 28

Tracking Variables

Variables’ tracking is configured at several levels:

• Select Secure Value if you do not want tracking.

• Operator can view tracked variables in the Variables and
Sequence Values section of the Session Task:

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
lice at several levels:
del is configured
Variableetracking
i N
•n When defining a variable, you can select Secure Value if you do not want the variable to be
lo r
F recorded (similar to a browser’s incognito mode). This is useful when the variable contains
passwords or other sensitive data. If Secure Value is selected, the variable will never be
tracked. It will be displayed as unresolved in the source or target code, not be tracked in the
repository, and it will not be historized.
• When executing or restarting a session, select Log Level 6 in the Execution or Restart
Session dialog box to enable variable tracking. Log Level 6 has the same behavior as Log
Level 5, but with the addition of variable tracking.
When reviewing the execution results in Operator Navigator, you can:
• View tracked variables and sequences in the Variables and Sequence Values section of the
Session Step or Session Task Editor
• Review the source/target operations of an execution in the Session Task Editor

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 29

On the Code tab of the Session Task Editor, click Show/Hide Values to display the code with
resolved variable values. Note that only variables in substitution mode (#VARIABLE) can be
displayed with resolved variable values and that if the variable values are shown, the code
becomes read-only.
Tracking variables is useful for debugging purposes. See Section 22.2.3, “Handling Failed
Sessions” in Oracle Fusion Middleware Developer’s Guide for Oracle Data Integrator for more
information on how to analyze errors in Operator Navigator and activate variable tracking.
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
n o
s a
h a
) deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
l o r in to u
u (f nse
e l c l i ce
e d
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 30

Tracking Variables: Notes

Note the following when tracking variables:

• Each variable value taken in a session can be tracked.

• The values of all tracked variables can be displayed at
step and task levels.
• The source and target code for a step or task can be
viewed or hidden.
• Variables defined as Secure Value are never displayed. erab
le
• When a session is purged, all variable values attached a n
to
sf
o n -tr
that session are purged from RAM along with the n session.
s a
• Bind variables (:VAR_NAME)cannot behtracked; a only
) e ฺ
substituted variables (#VAR_NAME)
ฺ c om
can be u idtracked.
c le imported
n t G
• Tracked values are exported a and
r ude as part of a
o t
cu@ is exported
session when the session
e
S
l h
or imported.
is
e d e t
i n ฺ n u s
l o r
Copyright ©
t o
2015, Oracle and/or its affiliates. All rights reserved.

u (f nse
c lice in ODI Studio and ODI Console sessions.
del is available
Variableetracking
i n N
r
lo the following when tracking variables in ODI:
Note
F • Each value taken by a variable in a session can be tracked. This is not the default behavior,
you must run with log level=6, which incurs extra overhead.
• The values of all tracked variables can be displayed at step and task levels. This includes
when a variable is modified by a step or a task. The Step or Task Editor displays the name
and the new value of the variable.
• The source and target code for a step or task can be viewed with either resolved variable
and sequence values, or hidden variable values that display the variable and sequence
names. Note that if the variable values are shown, the code becomes read-only.
• Variables, such as passwords, that are defined as Secure Value are never displayed in the
resolved code or variable list. A secure variable does not persist any value in the repository,
even if it is refreshed. Note also that the refresh of a secure variable does not work across
two sessions.
• Session purging only purges variable values from memory. If you keep history, then the
values are kept in the repository (on disk) indefinitely.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 31

Quiz

What is the variable name prefix character that substitutes the

name of the variable with the value before the ODI code is
executed?
a. #
b. :
c. &
ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Answer:ead
el lice
i n N
lo r
Explanation: When you reference an ODI variable prefixed with the # character, ODI substitutes
F the name of the variable by the value before the code is executed by the technology. See
page 7-6.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 32

Quiz

What is the variable name prefix character that does a SQL

bind substitution?
a. #
b. :
c. &

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 33

Summary

In this lesson, you should have learned to:

• Describe variables’ scope

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 34

Practice 7-1 Overview:
Using an ODI Variable as a Startup Parameter
This practice covers creating a mapping to export a flat file to a
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

relational table. You create a variable for the name of the flat
file. The actual name of the flat file to be loaded into the
relational table will be defined as a startup parameter at run
time.

Oracle Data Integrator 12c: Advanced Integration and Development 7 - 35

Accelerating Development with Groovy

After completing this lesson, you should be able to:

• Describe ODI SDK

• Use Groovy editor
• Automate ODI development tasks with Groovy

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
e d e
More information: l i
n N
lori
• http://groovy.codehaus.org/ for more information about the Groovy language
F • https://java.net/projects/oracledi/downloads/download/ODI/SDK%20Samples/ODI%2012c/od
i12c_mapping_sdk.zip for sample code
• https://blogs.oracle.com/dataintegration/entry/odi_12c_mapping_sdk_the for discussions

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 2

Agenda

 Interacting Programmatically with ODI

• Using ODI Groovy Editor

• Performing SDK Tasks Using Java with Groovy Editor

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 3

ODI Software Development Kit: Overview

Software Development Kit (SDK) enables developers to:

• Leverage ODI concepts programmatically and dynamically

• Execute ODI methods virtually through Java
• Embed ODI in other products via other IDEs

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
Does the same thing ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u e
The ODIeSDKdel provideslican interface to enable developers to leverage the ODI concepts through an
SDK
i n N
instead of using a GUI such as ODI Studio or Console.
r
lo • ODI can now be embedded in other products that can drive ODI process creation and
F
execution from their own GUI via their own IDE.
• Dynamic Mappings: The structure of the source and/or target systems is very dynamic.
That structure would not be easily implemented through a GUI that leverages fixed metadata
definitions.
An enhanced SDK enables the developer to execute virtually every ODI method through a Java
program that enables developers to leverage ODI concepts through an SDK instead of using ODI
Studio.
Note: “ODI SDK” and “Public API” (Application Program Interface) are terms that are often used
interchangeably. The Public API mimics the usage of the graphical interface. For a better
understanding of the logic implemented in the SDK, ensure that you are familiar with the GUI.
Classes, methods, and parameters will be similar to the options available through the graphical
interfaces. The valid code will match valid objects in the ODI GUI.

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 4

SDK-Supported ODI Operations

• Master Repository:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

– Creation of Master / Work

Repositories
– Creation and management
of certain objects

• Work Repositories: ble

fe r a
– Creation and management
t r a ns
of metadata
n on-
– Creation and management s a
of ODI projects a
) h deฺ
– Retrieving values of c o m u i
l e ฺ G
flexfields rac ent o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce supported by SDK. The list is grouped by operations performed in the
e
Here are d e
the ODI l i
operations
n N
Master Repository and operations performed in the Work Repositories.
F lori• Master Repository:
- Creation of Master / Work Repositories (alternatively, use RCU in silent mode)
- Creation and management of the following objects:
• Data servers, Agents, Contexts
• Topology (Logical and Physical Architectures based on existing technologies)
• Work Repositories:
- Creation and management of metadata
• Models and submodels
• Datastores, Attributes (Columns), Constraints
- Creation and management of ODI projects
• Projects, Folders, Mappings, Variables, Packages
- Retrieving values of flexfields

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 5

ODI Operations Not Supported by SDK

• Master Repository:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

– Creation of new technologies

– Security management
• Work Repositories:
– Creation of new ODI procedures
(though procedures can be imported and used)
– Creation of new Knowledge Modules a b le
s f er
– Creation of new user functions
- t r an
– Locking/unlocking of ODI objects n on
– Versioning of ODI objects s a
a
) h deฺ
c o m
– Duplication of ODI objects in the repository
u i
e ฺ G
– Working with markers
r a cl ent
@o Stud
– Setting values of flexfields
cu his
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Master e del
Repository lice
i N
•n Modification of the data type conversion matrix
lo r
F • Creation of new technologies
• Security management (Only live users with the Supervisor role can use the SDK. This was
not for lack of know-how; this was a wise and conscious decision.)
Work Repository
• Creation of new ODI procedures (though procedures can be imported and used in packages)
• Creation of new Knowledge Modules
• Creation of new user functions
• Locking/unlocking of objects
• Versioning of objects
• No ability to “duplicate” an object in the repository (though this can be achieved through
code)
• No markers (smiley faces, thumbs up/down, and so on) or memos
• Setting the value of flexfields

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 6

Combining Different APIs
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

APIs Methods

ODI fe r a ble
ans
n - t r
a no
Tools h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d
• Public
eAPI (or l i
SDK)
i n N- Is a library of classes and methods for the manipulation of objects in the ODI
r
Flo Repository without accessing the repository tables directly. You can SELECT (at your
own peril) directly, but never modify the tables. Instead, use the APIs to modify objects.
- Developers relying on the SDK will be protected from repository evolutions over time.
• Sunopsis APIs
- Are represented in the packages as “Tools” (OdiSendMail, and so on)
- Are used to send emails, automate the reverse-engineering of models, invoke
scenarios, detect changes, and so on
- Each tool has a corresponding API version.
- Can be leveraged by packages built through the public APIs
• Substitution Methods
- They are used by developers to build generic code in Knowledge Modules and
procedures that will dynamically retrieve the names of all tables, attributes, and
mappings during the process of building mappings. Examples: odiRef.getColList,
odiRef.getTable, and so on.

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 7

Agenda

• Interacting Programmatically with ODI

 Using ODI Groovy Editor

• Performing SDK Tasks Using Java with Groovy Editor

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 8

Groovy: Introduction

• Groovy is a scripting language with

Java-like syntax for the Java platform.

• Groovy scripting language expressions
are dynamically compiled.
• ADF Business Components:
– Supports Groovy scripting language in
places where access to entity object and bl e
view object attributes is useful fe r a
ans
– Provides a limited set of built-in keywords
n - t r
that can be used in Groovy expressions
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Groovy eis d
el lice with Java-like syntax for the Java platform. The Groovy scripting
a scripting language
n N simplifies the authoring of code by employing dot-separated notation, yet still supporting
language
i
r
losyntax to manipulate collections, strings, and JavaBeans.
F
Groovy language expressions in Application Development Framework (ADF) Business
Components differ from the Java code that you might use in a Business Components custom Java
class because Groovy expressions are executed at run time, whereas the strongly typed language
of Java is executed at compile time. Because Groovy expressions are dynamically compiled, they
are stored in the XML definition files of the Business Components where you use them. “Dynamic”
is fast for development, but not as fast for production (not as fast as compiled).
ADF Business Components supports the use of the Groovy scripting language in places where
access to entity object and view object attributes is useful, including attribute validators (for entity
objects), attribute default values (for either entity objects or view objects), transient attribute value
calculations (for either entity objects or view objects), bind variable default values (in view object
query statements and view criteria filters), and placeholders for error messages (in entity object
validation rules). Additionally, ADF Business Components provides a limited set of built-in
keywords that can be used in Groovy expressions.
Note: Groovy can be used in ODI Procedures, so you could write a procedure with Groovy script
that does your work and calls the procedure through the regular channels, for example, as a
scenario through web services.

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 9

Groovy Editor: Introduction

The Groovy editor provides:

• A single environment for creating, editing, and executing
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Groovy scripts within the ODI Studio context

• Most of the standard features of editors, but no debugging

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce all standard features of a code editor such as comments, syntax
e
The Groovy d eeditor l i
provides
n N
highlighting,
i supportedandandcommon code editor commands except for debugging. The following commands
F lorare accessed through the context menu or through the Source main menu:
• Show Whitespace
• Indent Block, Unindent Block
• Text Edits
- Join Line, Delete Current Line
- Trim Trailing Whitespace
- Convert Leading Tabs to Spaces, Convert Leading Spaces to Tabs
- Macro Toggle Recording, Macro Playback
- Color coding of keywords

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 10

Executing Script with Groovy Editor

• You can execute one or several Groovy scripts at once

and also execute one script several times in parallel.

• You can execute only scripts opened in the Groovy editor.
– ODI Studio does not execute a selection of the script; it
executes the whole Groovy script.

bl e
fe r a
No
ans
Save
option
n - t r
here
a no
h a s
m ) deฺ
ฺ c o u i
c l e n t G Right-

o ra ude click

c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
del onelor
You caneexecute iceseveral Groovy scripts at once and also execute one script several times
inin N
parallel.
lo r
F You can only execute a script that is opened in the Groovy editor. ODI Studio does not execute a
selection of the script (unlike SQL Developer), it executes the whole Groovy script.
To execute a Groovy script in ODI Studio, select the script that you want to execute in the Groovy
editor. Click Run on the toolbar. The script is executed.
You can now follow the execution in the Log window.
Note that each script execution launches its own Log window. The Log window is named
according to the following format: Running <script_name>.
You do not save the script to the repository (there is no diskette icon on the main toolbar), instead
you save it to your local disk using Tools > Groovy > Save Script As. (See the previous slide for a
screenshot.)
Note: You can change the default comment header with your name/department and so on.

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 11

Agenda

• Interacting Programmatically with ODI

• Using ODI Groovy Editor

 Performing SDK Tasks Using Java with Groovy Editor
– Accessing external libraries
– Defining additional Groovy execution classpath
– Using odiInputStream
ble
– Using odiInstance
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 12

Performing an SDK Task Using Java
with Groovy Editor: Example
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

3 1

bl e
fe r a
an s
n - t r
a no
2 h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ceSDK is used to create a new ODI Project within the Work Repository that
d e l i
isin Ne within ODI Studio.
In this example,
connected
the ODI

F lor 1. Load and run the Groovy script.

2. Verify that the logs are clean and error-free. If there were errors, debug your script and rerun
the script (you may need to delete intermediate objects to avoid duplication errors). For
example, if you ran this script twice, you would see:
ODI-10143: Error while accessing the ODI repository.
ORA-00001: unique constraint (SOA_ODI_REPO.AK_PROJECT) violated
3. Verify that the new ODI project is created (or whatever was the goal of the script).

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 13

Using Custom Libraries

To use external custom libraries, do one of the following:

Custom Libraries

bl e
~/.odi/oracledi/userlib
fe r a
ans
n - t r
n o
s a
additional_path.txt
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ceto access external libraries; for example, if an external driver is needed.
e
The Groovyd eeditor is l i
able
n N
F lorToi use external libraries, do one of the following:
• Copy the custom libraries to the userlib folder. This folder is located:
- On Windows operating systems: %APPDATA%/odi/oracledi/userlib
- On UNIX operating systems: ~/.odi/oracledi/userlib
• Add the custom libraries to the additional_path.txt file. This file is located in the
userlib folder and has the following content:
; Additional paths file
; You can add here paths to additional libraries
; Examples:
; /opt/java/libs/myjar.jar
; /opt/java/libs/myzip.zip
; /opt/java/libs/*.jar will add all jars contained in the
/opt/java/libs/ directory
; /opt/java/libs/**/*.jar will add all jars contained in the
/opt/java/libs/ directory and subdirectories

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 14

Defining Additional Groovy Execution Classpath
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
2 n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
liceexecution classpath in addition to all classpath entries available to ODI
del a Groovy
You canedefine
n N
Studio.
i
lo r
F To define an additional Groovy execution classpath, perform the following steps:
1. Before executing the Groovy script, select Tools > Preferences.
2. In the Preferences dialog box, navigate to ODI > System > Groovy Preferences page.
3. Enter the Classpath and click OK.

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 15

Read Input with the odiInputStream Variable

ODI provides the odiInputStream variable to read input

streams. This variable is used as follows:

odiInputStream.withReader { println
(it.readLine())}

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce an Input text field is displayed at the bottom of the Log tab. Enter a
When thise d e
feature is l i
used,
i n
stringNtext and press Enter to pass this value to the script. The script is exited after the value is
F lorpassed to it.

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 16

Automating Development Tasks: Example 1

Creating an ODI project:

import oracle.odi.core.persistence.transaction.support.DefaultT
ransactionDefinition;
import oracle.odi.domain.project.OdiProject;
import oracle.odi.domain.project.OdiFolder;
def createProject(projectName, projectCode, folderName) {
txnDef = new DefaultTransactionDefinition();
tm = odiInstance.getTransactionManager()
ble
txnStatus = tm.getTransaction(txnDef)
fe r a
project = new OdiProject(projectName, projectCode)
ans
folder = new OdiFolder(project, folderName)
n - t r
odiInstance.getTransactionalEntityManager().persist Define
Definethe
for for
theparameters
Project
parameters
Project
Name,
Name,
a no
(project) Project
Project
Code,
Code,
a
andand
h s
tm.commit(txnStatus) FolderFolder
Name.
m ) deฺ
Name.
}
ฺ c o u i
e
cl ent
createProject("DATA WAREHOUSING", "EDW", "INITIALIZATION") G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce that shows how to create an ODI Project with a Groovy script. In
e
This slide d e
provides anl iexample
the
i n N
createProject method, you define parameters for Project Name, Project Code, and Folder
r
loName for your project.
F
ODI provides the variable odiInstance. It represents the ODI instance, more precisely the
connection to the repository, in which the script is executed.

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 17

Automating Development Tasks:
Example 2a (createModel)
Creating an ODI topology and a model:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

(The first 40 lines not shown for clarity, some in next slide)

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce of how to create an ODI model with the necessary topology using
e
This slide d e
provides anl iexample
then N
Groovy script.
F lori
To create the topology, enter:
lschema = createLogicalSchema("<Context>", "<Technology>", "<Logical
Schema Name>", "<Data Server Name>", "<DB UserName>",
ObfuscatedString.obfuscate("<DB User Password>"), "<URL>",
"<Driver>", "<DB Schema used for ODI Physical Schema>")
In this example, the following parameters are used:
lschema = createLogicalSchema("GLOBAL", "ORACLE", "ORACLE_EBS",
"ORACLE_HQ_DEV", "ODI", ObfuscatedString.obfuscate("ODI"),
"jdbc:oracle:thin:@localhost:1521:orcl", "oracle.jdbc.OracleDriver",
"ODI")
To create a model, enter:
createModel(lschema, "<Context>", "<ModelName>", "<ModelCode>")
In this example, you use the following parameters:
createModel(lschema, "GLOBAL", "ORACLE_WH", "ORACLE_WH")

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 18

Automating Development Tasks:
Example 2b (createLogicalSchema)
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e
createLogicalSchema: l i
n N
lori
• contextCode: The ODI code for the context used to map logical schema to physical
F • technologyCode: The ODI code for the technology
• nameForLogicalSchema: The name for the logical schema to create
• NameForDataserver: The name for the data server to create
• userNameForAuthentication: The username for the connection to the data server
• passwordForAuthentication: The password for the connection to the data server
• urlForAuthentication: The URL for the connection to the data server
• driverForAuthentication: The JDBC driver for the connection to the data server
• schemaForAuthentication: The schema to use for the ODI physical schema
createModel:
• logicalSchemaObject: The ODI logical schema object (instance of
ODILogicalSchema)
• contextCode: The ODI context code for reverse engineering
• nameForModel: The name for the model to create
• codeForModel: The code for the model to create
Note: With these two methods or variations of them, you can construct your topology objects and
models that will allow going straight to reverse engineering when the script has been run.

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 19

Automating Development Tasks:
Example 3a (Simple Mapping)
3
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

bl e
fe r a
s
Not shown are some initial
an
n - t r
housekeeping functions.

o
nDefine
s a source, target, and
1 h
) deฺa connections.

c o m u i
l e ฺ t G Define attribute

2 ra udec n expressions.

@ St o
l c u is
e d e t h
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ce that shows how to create an ODI mapping with a Groovy script.
e
This slide d e
provides anl iexample
n N
F lorToi create a mapping with Groovy:
1. In the Groovy script, define the variables with the values for the ODI project, the project
folder, and the path to the text file you created.
2. Define the attributes you want to map from source to target. Optionally, do transformation.
The whole script is defined in terms of methods, then you instantiate them by having the last
line of the script call the methods with hard-coded values. It might look similar to:
m1 = basic_mapping("DEMOS", "First Folder", "Basic_Mapping")
m2 = filter_mapping("DEMOS", "First Folder", "Filter_Mapping")
3. Run the Groovy script in the bottom half of the slide to create the ODI mapping shown in the
top half of the slide.
Note: You can create more that one mapping with Groovy script if you define more than one row
in the bottom of the script (one line of parameters for each mapping), as shown by the m1 and m2.

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 20

Automating Development Tasks:
Example 3b (Mapping and Filter)
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

bl e
fe r a
an s
n - t r
a no
h a s
m ) deฺ Add these lines to define a
o
ฺc t Gu i Filter component.
l e
o rac uden
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cefrom the previous slide, but with the addition of a filter. A more complete
e
This is the d e
same l i
mapping
n N
listing
i slidefound
is in the lab environment. (There are several pages of code required to do this task.
F lorThe shows only lines 158 – 170. A large amount of code is not shown.)

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 21

Automating Development Tasks: Example 4

Studio UI Automation with Java:

• Menu items
• Dialog boxes

import javax.swing.JMenuItem;
import javax.swing.Jmenu;
a b le
import oracle.ide.Ide; s f er
(JMenuItem)Ide.getMenubar().getGUI(false).getComp - t r an
onent(4)).doClick(); n on
s a
JMenu mnu = a
) h deฺ
c o m
((JMenu)Ide.getMenubar().getGUI(false).getCompo
u i
e ฺ G
nent(4));
r a cl ent
o ud
((JMenuItem)mnu.getMenuComponents()[0]).doClick()
u @ St
e l c h is
e d e t
i n ฺ n u s
l o r
Copyright ©
t o
2015, Oracle and/or its affiliates. All rights reserved.

u (f nse
c lice elements of ODI Studio user interface using Java with the Groovy
del creating
You caneautomate
n NThis slide provides an example that shows an implementation of ODI Studio GUI
editor.
i
r
loautomation using Swing.
F

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 22

Quiz

Which operation(s) are supported by SDK?

a. Creation of Master/Work Repositories

b. Creation of models and submodels
c. Creation of user functions
d. Creation of packages
e. Creation of Knowledge Modules ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
el
Answer:ea,db, d lice
i n N
lo r
Explanation: Choices c and e, Creation of user functions and KMs, are not supported by SDK.
F See page 8-5 and 8-6.

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 23

Summary

In this lesson, you should have learned to:

• Describe ODI SDK

• Use Groovy editor
• Automate ODI development tasks with Groovy

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 24

Practice 8-1 Overview:
Automating ODI Tasks with Groovy
This practice covers creating the ODI project, the topology and
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

model, and two ODI mappings with Groovy.

Oracle Data Integrator 12c: Advanced Integration and Development 8 - 25

Working with ASCII Files

After completing this lesson, you should be able to:

• Describe delimited files

• Describe complex files concepts
• Describe the technical requirements for using complex files
with ODI
• Set the topology for complex files and create a complex le
a b
files model by using the Native Format Builder fer s
- t r an
no n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 2

Agenda

 Complex Files Concepts: Overview

‒ XML
‒ JSON
• Complex Files: Configuring the Topology
• Complex Files in Integration Projects

ble
fe r a
an s
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce Master/Header/Detail/Trailer, COBOL or RPG, and so on.
e
Examples d e
of complex l i
files:
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 3

Types of Files in ODI

ODI supports the following types of text files:

• Flat files containing ASCII or EBCDIC data

(use File technology)
– KMs to read from such files:
— LKM File to SQL
— IKM SQL to File Append
RKM File (from EXCEL)
—

bl e
• XML files (use XML technology) fe r a
– XML driver loads the XML file into a RDBMS schema.tran
s
-
nonXML data.
– IKM XML Control Append KM is used for handling
a
• Complex (or native format) files, that areha s complex to be
too
m) uideฺ
parsed using the existing ODI Filecotechnology
c l eฺ nt G
o ra ude
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
del several
The ODIesupports licetext files types:
i N
•n Flat files (fixed or delimited): A File server corresponds to an ODI data server. In this File
r
lo server, a directory containing files corresponds to a physical schema. A group of flat files
F
within a directory corresponds to an ODI model, in which each file corresponds to a
datastore. The fields in the files correspond to the datastore columns. ODI provides the
Knowledge Modules (KM) listed in this slide for handling File data using the File driver.
- LKM File to SQL: Loads data from an ASCII or EBCDIC File to any ANSI SQL-92–
compliant database used as a Staging Area
- IKM SQL to File Append: Integrates data in a target file from any ANSI SQL-92–
compliant Staging Area in replace mode
- RKM File (from EXCEL): Retrieves metadata from Microsoft Office Excel
• XML files: An XML file corresponds to a data server in ODI. Within this data server, a single
schema maps the content of the XML file. The ODI Driver for XML (XML driver) loads the
hierarchical structure of the XML file into a relational schema. ODI provides the IKM XML
Control Append for handling XML data.
• Complex files (files with native format): For these files, you have to read or write file
structures that are too complex to be parsed using the existing ODI File technology.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 4

Complex Files Concepts

• For complex files, it is possible to build a native schema

description file that describes the file structure.

• Using this native schema (nXSD) description and the ODI
Driver for Complex Files, ODI is able to reverse-engineer,
read, and write information from complex files.

ble
XML Relational fe r a
Native File ODI ns
(Internal) Schema
-tra
n o n
s a
a
) h deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d
Conceptual
e
stages: l i
n N
lori
• Native File: Could be .xml, .csv, COBOL, .txt
F • XML (internal): .xsd (data dictionary)
• Relational Schema: ODI uses this to parse (reverse engineer)
• ODI: At the end, ODI represents the complex files as datastores.
For complex files, it is possible to build a native schema description file that describes the file
structure. Using this native schema (nXSD) description and the ODI Driver for Complex Files, ODI
is able to reverse-engineer, read, and write information from complex files. ODI Driver for Complex
Files (Complex File driver) converts native format to a relational structure and exposes this
relational structure as a data model in ODI.
The Complex File driver translates internally the native file into an XML (internal) structure, as
defined in the native schema (nXSD) description and from this XML file it generates a relational
schema that is used by ODI. The overall mechanism is shown in this slide.
Most concepts and processes that are used for complex files are equivalent to those used for XML
files. The main difference is the step that transparently translates the native file into an XML
structure that is used internally by the driver but never persisted.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 5

A complex file corresponds to an ODI data server. Within this data server, a single schema maps
the content of the complex file.
The ODI Driver for Complex File (Complex File driver) loads the complex structure of the native
file into a relational schema. This relational schema is a set of tables located in the schema that
can be queried or modified using SQL. The Complex File driver is also able to unload the
relational schema back into the complex file. The relational schema is reverse-engineered as a
data model in ODI, with tables, columns, and constraints. This model is used like a standard
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

relational data model in ODI. If the modified data within the relational schema needs to be written
back to the complex file, the driver provides the capability to synchronize the relational schema
into the file.
Note: For simple flat files formats (fixed and delimited), File technology is recommended, and for
XML files, the XML technology.

ble
fe r a
ans
n - t r
n o
s a
h a
) deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
l o r in to u
u (f nse
e l c l i ce
e d
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 6

Sample Complex XML File
6335722^Company One^First Street 999 San Jose 95129USCA650-
801-6250 ^Oracle^Bridge Parkway 1600 Redwood Shores
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

94065USCA650-506-7000
001|Oracle Data Integrator EE|20000,2,+40000+
002|Application Adapter for ODI|10000,5,+50000+
003|BPEL Process Manager Enterprise Edition|20000,2,+40000+
004|BPEL Process Manager Standard Edition|10000,5,+50000+
005|BPEL Process Manager Developer Edition|1000,20,+20000+#110000

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i SELLER
ADDRESS INVOICE LINE_ITEM e
cl ent
PURCHASER G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u ce
el
it is possiblelito infer what is going on by looking at the file, the real structure is defined in
Althoughed
n N nXSD file. The first record (header) contains several nested things:
aiseparate
lo r
F • Invoice number
• Purchaser
- Address (which in turn contains the fields Street, City, State, Zip, Country)
• Seller
- Address (another one)
The remaining records (detail) contain multiple line items associated with this invoice.
In some cases, the delimiter is a tab, in other cases it is a vertical bar, and in some other cases it
is a comma or a caret. The delimiters are specified in the nXSD.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 7

Sample Complex JSON file

{
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

"firstName": "Duke",
"lastName": "Java",
"age": 18,
"streetAddress": "100 Internet Dr",
"city": "JavaTown",
"state": "JA",
"postalCode": "12345", ble
fe r a
ns
"phoneNumbers": [
{ "Mobile": "111-111-1111" },
- t r a
{ "Home": "222-222-2222" }
PEOPLE nPHONES no
s a
]
h
) deฺ a
}
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce (JSON) has the following syntax:
e d
JavaScript
e
Object l i
Notation
n N
lori
• Objects are enclosed in curly braces “{}”, their name-value pairs are separated by a comma
F “,”, and the name and value in a pair are separated by a colon “:”. Names in an object are
strings, whereas values may be of any of the seven value types, including another object or
an array.
• Arrays are enclosed in square brackets “[]”, and their values are separated by a comma “,”.
Each value in an array may be of a different type, including another array or an object.
• When objects and arrays contain other objects or arrays, the data has a tree-like structure.
See the following sites for more information:
• https://docs.oracle.com/javaee/7/tutorial/jsonp.htm
• http://www.json.org/

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 8

Knowledge Modules for Complex Files

• You can use a Complex File data server as any SQL data
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

server.
• Complex File data servers support several kinds of KMs.
• You can also use the IKM XML Control Append when
writing to a Complex File data server.
– This Knowledge Module implements a specific option to
synchronize the data from the relational schema to the file, rabl
e
e
which is supported by the Complex File driver. nsf a
o n -tr
a n
a s
m ) h deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce support:
Complexe d e
File data l i
servers
n N
lori
• Technology-specific KMs sourcing or targeting SQL data servers
F • Generic KMs.
This is why you can use “IKM SQL to Oracle” instead of “IKM XML to Oracle.”
For more information about using KMs, see the chapter titled “Generic SQL” in Connectivity and
Knowledge Modules Guide for Oracle Data Integrator.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 9

Requirements for Working with Complex Files

• There are no technology-specific requirements for using

Complex files in ODI.

• Connectivity requirements:
– Complex files are accessed through the ODI Driver for
Complex File. This JDBC driver is installed with ODI.
– You must have the following connection information:
URL of the nXSD bl e
—

fe r a
— URL of the native complex file
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cofe the native schema (nXSD) file associated with your native file; the URL
The URLe d
is
e
the l
locationi
n
need N
not be local.
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 10

Agenda

• Complex Files Concepts: Overview

 Complex Files: Configuring the Topology

• Complex Files in Integration Projects

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 11

Setting Up the Topology

Setting up the topology:

• Create a Complex File data server, which corresponds to

one native file that is accessible to ODI:
– You create a data server for the Complex File technology
using the standard procedure.
– The fields specific for defining a Complex File data server:
JDBC Driver
—

JDBC URL ble

—

fe r a
• Create a Complex File Technology: s
– Same standard procedure for any technology - t r an
no n
– Physical schema a
– Logical schema h a s
– Context m ) deฺ
c o ui l eฺ nt G
ra udec
@ St o
l c u is
e d e t h
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
c u e
You seteup
el topology
dthe licfor complex files by creating a data server and physical schema. Create a
data N
in servera Data
for the Complex File technology by using the standard procedure, as described in
F lor“Creating Server” of the Oracle Fusion Middleware Developer’s Guide for Oracle Data
Integrator. This slide details only the fields required or specific for defining a Complex File data
server:
On the Definition tab:
• Name: Name of the data server that will appear in ODI
• User/Password: These fields are not used for Complex File data servers.
On the JDBC tab, enter the following values:
• JDBC Driver: oracle.odi.jdbc.driver.file.complex.ComplexFileDriver
• JDBC URL:
jdbc:snps:complexfile?f=<native_file_location>&d=<native_schema>&re=
<root_element_name>&s=<schema_name>[&<property>=<value>...]
(the “root element” is also called the “document root.”)
Note: Creating a Complex File physical schema is standard procedure for any technology.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 12

JDBC URL Properties

Mandatory key properties of the ODI Driver for Complex Files

(specified in the JDBC URL):

Property Value Notes

Native file location. Use slash “/” in the
<native file
f path name. It is possible to use HTTP,
name>
FTP, or file URL to locate the file.
bl e
d <native schema> Native schema (nXSD) file location
fe r a
Name of the element to take as the a s
nroot
re <root element> table of the schema. This value
t r
n-is case-
n o
sensitive. a
as
) h dschema
Name of the relational ฺ where
s <schema name>
o m i e
ฺc file willube loaded
the complex
le nt G
a c
@ or tude
e l cu his S
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
e d e
The following are the l i
mandatory key properties of the ODI Driver for Complex files:
N
•n f=<native file name>: This is native file location. Use slash (/) in the path name
F lori instead of back slash (\) even in Windows. It is possible to use an HTTP, FTP, or File URL
to locate the file. Files located by URL are read-only.
• d=<native schema>: This is the native schema (nXSD) file location (also could be http, url,
and so on).
• re=<root element>: The element to take as the root table of the schema. This value is
case-sensitive (most items in XML are case-sensitive). This property can be used for
reverse-engineering, for example, a specific section of the native schema.
• s=<schema name>: The name of the relational schema where the complex file will be
loaded. This schema will be selected when creating the physical schema under the Complex
File data server.
• tt=json: Used for JSON files
For the full set of ODI driver properties specified in the JDBC URL, refer to Appendix C “Oracle
Data Integrator Driver for Complex Files Reference” in Oracle Fusion Middleware Connectivity and
Knowledge Modules Guide for Oracle Data Integrator.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 13

JDBC URL: Example

The following example illustrates these URL properties:

• JDBC URL:
jdbc:snps:complexfile?f=/infiles/PROD20100125_0
01.csv&d=/infiles/products.nxsd&re=root&s=PRO
DUCTS
• Connects to the PROD20100125_001.csv file described
by products.nxsd and exposes this file as a relational a b le
structure in the PRODUCTS schema s f er
n a
o n -tr
n a
a s
m ) h deฺ
ฺ c o u i
e
cl ent G
r a
o tud
.csv + .xsd @
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u e
l used linicthis
e d e
The convention sample business is the PROD file gets a daily suffix. The one shown
here N
in would
is 2010-JAN-25, run number 001. So many .csv files can use the same .xsd description.
F lorThis be a good use of substitution variables!

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 14

Complex File Data Server Definition: Example
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
a no2
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cecreate a data server to implement Complex File technology. In ODI
e
This slide d e
shows howl i
to
n N
Studio:
F lori1. Open the Topology tab. In the Physical Architecture, expand the Complex File technology
node. Right-click Complex File and then select New Data Server.
2. Enter the name for this new data server (for example, PURCHASE_SAMPLE_CPLX_FILE),
and click the JDBC tab.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 15

Complex File Data Server Definition: Example
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

3
ble
fe r a
ans
n - t r
4 a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud 5
u @ S
d e lc this
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce uses a JDBC driver to read the original input file as well as the
e d
The Complex
e File l i
technology
n N
metadata
i definition of this complex file generated using the Native Format Builder, a SOA
F lorcomponent. To select the driver:
3. Click the magnifying glass icon and then select the JDBC Complex File driver:
oracle.odi.jdbc.driver.file.complex.ComplexFileDriver.
4. Edit the URL to point to the input file and to the XSD file:
jdbc:snps:complexfile?f=/home/oracle/labs/Files/Complex
Files/Purchase_sample.txt&d=/home/oracle/labs/Files/Complex
Files/Purchase_schema.xsd&re=invoice&s=PURCH
5. Test the connection (requires an agent).
Note: The native schema (nXSD) provided in the data server URL is used as the XSD file to
describe the XML structure. For more information, see Section 5.5.2 “Reverse-Engineering an
XML Model” in Oracle Fusion Middleware Connectivity and Knowledge Modules Guide for Oracle
Data Integrator.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 16

Definition of Physical
and Logical Schema: Example
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

1 2

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ 3
c o u i
c l eฺ nt G
o ra ude
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce you define physical and logical schemas the same way as you did
e
For Complexd e Files l i
technology,
n
for N
other technologies:
F lori1. Right-click the data server. Select New Physical Schema.
2. Create a new physical schema for this data server. Save physical schema.
3. Open the Logical Architecture bar. Expand Technologies > Complex File and create a new
logical schema as shown in the slide. Connect this logical schema to the physical schema (in
this example myComplexFile.PURCH) in all contexts. Save this logical schema.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 17

Agenda

• Complex Files Concepts: Overview

• Complex Files: Configuring the Topology

 Complex Files in Integration Projects

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 18

Setting Up an Integration Project
and Creating a Complex File Model: Example
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
2
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cusing
e d
For setting
e
up a l
projecti the Complex File technology, you follow the standard procedure.
See N
in “Creating an Integration project” of Oracle Fusion Middleware Developer’s Guide for Oracle
F lorData Integrator.
It is recommended to import the following knowledge modules into your project for getting started:
• LKM SQL to SQL
• LKM File to SQL
• IKM XML Control Append
A Complex File model groups a set of datastores. Each datastore typically represents an element
in the intermediate XML file generated from the native file using the native schema.
1. In Designer, create a New Model.
2. Name the new model, and select Technology: Complex File. The default logical schema will
be prepopulated. Save your work. The new model should be added to the list on the left.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 19

Setting Up an Integration project and
Creating a Complex File Model: Reverse Engineer
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
a no
s 4
h
) deฺ a
3 m
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u cesupports standard reverse engineering, which uses only the abilities of
Complexe d e
File l
technology i
then N
Complex File driver (the same process as for XML Files).
F lori
To perform a standard reverse engineering with a Complex File model, use the usual procedure,
as described in “Reverse-engineering a Model” in the Oracle Fusion Middleware Developer’s
Guide for Oracle Data Integrator.
3. Right-click the new model, select Reverse Engineer (you could have done this on the
previous model panel).
4. After a few moments, the new datastores will be shown below the new model. Notice
Address0 and Address1, corresponding to the seller address and the purchaser address.
You may want to change that in the schema to be a single address table.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 20

Designing a Mapping with Complex File Model

Sources Targets
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Mapping Complex
Complex File
File
Relational
Relational Table
Table ble
fe r a
ans
n - t r
KMs
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce file as a source or a target of an integration mapping.
• Youe d e
can use a l i
complex
i•n N
Complex file data models are handled in mappings similarly to XML structures.
r
Flo • The same Knowledge Modules can be used for both complex files and XML technologies.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 21

Creating an nXSD File

• Complex file models depend on the nXSD schema for the

given file.
• An nXSD file has to be created using a text editor (difficult)
or the Native Format Builder Wizard (easier) that is part of
Oracle BPEL Process Manager.
– This format extends the XML Schema standard by adding
additional parsing instructions to each element.
a b le
– Using nXSD parsing technology, the native file is converted s f er
into an internal XML format. - t r an
n n o
• The nXSD format extends the standard XML s aSchema
h a
Definition (XSD) format through nxsd attributes.
) eฺ m id
ฺco t Gthe
– nXSD is a valid XML schema,lebecause u XSD standard
c ennamespaces.
oratheir own
allows extra attributes with d @ Stu
l c u is
e d e t h
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u cineJDeveloper) is included in the ODI Suite, but not in the stand-alone ODI
e
Note: BPEL d e(a l
plug-in i
n N
Enterprise
i Edition. Namespaces are like qualifiers; they help avoid ambiguous references and
F lorcollisions.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 22

nXSD Schema: Example

<?xml version="1.0"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

xmlns:nxsd="http://xmlns.oracle.com/pcbpel/nxsd"
elementFormDefault="qualified"
xmlns:tns="http://xmlns.oracle.com/pcbpel/demoSchema/csv"
targetNamespace="http://xmlns.oracle.com/pcbpel/demoSchema/csv"
attributeFormDefault="unqualified"
nxsd:encoding="US-ASCII" nxsd:stream="chars" nxsd:version="NXSD">
<xsd:element name="Root">
<xsd:complexType><xsd:sequence>
<xsd:element name="Header"> ble
fe r a
<xsd:complexType><xsd:sequence>
ans
<xsd:element name="Branch" type="xsd:string"
n - t r
no
nxsd:style="terminated" nxsd:terminatedBy=","/>
<xsd:element name="ListDate" type="xsd:string"
s a
nxsd:style="terminated" nxsd:terminatedBy="${eol}"/>
h
) deฺ a
</xsd:sequence></xsd:complexType>
c o m u i
</xsd:element>
e ฺ G
r a
</xsd:sequence></xsd:complexType>cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce is shown in the slide and continued in the next slide. Some key
e d
An example
eof nXSD l i
schema
n N
terms:
Fl ori• <?xml version="1.0"?>: This is the “prolog,” which is optional in version 1.0, but
required in the latest version 1.1.
• <xsd:schema xmlns:xsd=>: Namespace URLs. Even though it looks like an HTTP web
address, you cannot go there with the browser.
• <xsd>: XML Schema Document.
• <xsd:complexType>: This is what makes it not plain old XML, but makes it “Complex.”
Note that it can be nested.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 23

nXSD Schema: Example

<xsd:element name="Customer" maxOccurs="unbounded">

<xsd:complexType><xsd:sequence>
<xsd:element name="Name" type="xsd:string"
nxsd:style="terminated" nxsd:terminatedBy=","/>
<xsd:element name="Street" type="xsd:string"
nxsd:style="terminated" nxsd:terminatedBy="," />
<xsd:element name="City" type="xsd:string"
nxsd:style="terminated" nxsd:terminatedBy="${eol}" />
</xsd:sequence></xsd:complexType>
bl e
</xsd:element>
fe r a
</xsd:sequence></xsd:complexType>
ans
</xsd:element>
n - t r
</xsd:schema>
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u e
cname="Customer"
e d e
• <xsd:element l i maxOccurs="unbounded">: You can limit the
N
in number of elements by type in an XML file. The Customer element can occur a maximum of
r
Flo unbounded (infinite) times. However, each customer may have only one name, one street,
one city, and so on. These are local business rules, not universal rules.
• <…nxsd:terminatedBy=…>: The nXSD schema annotates elements to describe their
position and delimiters within the flat text file. The schema in the slide uses almost
exclusively the nxsd:terminatedBy instruction to look for the next terminator characters.
There are various constructs in nXSD to parse fixed-length fields, look ahead in the
document for string occurrences, perform conditional logic, use variables to remember state,
and many more.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 24

Getting to the Native Format Builder Wizard
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

bl e
fe r a
ans
n - t r
a no
h a s
Type anything; you
m ) deฺ
only want to get to
ฺ c o u i
the Wizard.
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce Wizard as well as the nXSD language are described in the Oracle
e d
Both Native
eFormat l i
Builder
n N
Fusion Middleware User’s Guide for Technology Adapters.
F lori1. Start the Native Format Builder in BPEL by creating a new File Adapter.
2. On step 4 of 4, specify Operation Type: Read File. The steps renumber to now be 4 of 9.
3. On step 8 of 9, click the gear to start the Native Format Builder.
The nXSD schema is created by using the Native Format Builder Wizard in JDeveloper with SOA
Extensions installed. The Oracle File and FTP Adapters are automatically integrated with Oracle
BPEL PM. When you drag File Adapter for FTP Adapter from the Component Palette of the
JDeveloper BPEL Designer to the design area, the Adapter Configuration Wizard starts with a
Welcome page.
When you click the Define Schema for Native Format gear on the Messages page of the Adapter
Configuration Wizard, the Native Format Builder Wizard is displayed. The Messages page is the
last page that is displayed in the Adapter Configuration Wizard before the Finish page. For details,
refer to Oracle Fusion Middleware User’s Guide for Technology Adapters.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 25

Using the Native Format Builder Wizard
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
You can create a new fe
r a
schema using thea
t r ns
Native

on- file.
Format Builder Wizard for

a n
a complex-type

a s
m ) h deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
del new nXSD
You canecreate lice schema for your complex file, or edit an existing native schema
n N using the Native Format Builder Wizard by sampling a delimited, fixed-length, or
generated
i
r
locomplex-type file. To edit an existing native schema, select the “Edit existing” option on the
F Choose Type page of the Native Format Builder Wizard, and click Browse to navigate to the
location of the existing schema file and then select the native schema file that must be edited. The
Native Format Builder Wizard guides you through the editing of the native schema file.
Note: If the format is complex, it is often a good idea to “approximate” it with a similar simple
format and then add the complex components manually. The resulting *.xsd file can be copied
and used as the format for ODI. Using this technique, it is also possible to parse the same file
format in SOA Suite and ODI; for example, using SOA for small real-time messages, and ODI for
large batches.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 26

Reading the Source File
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Browse to the
location of the
existing schema
file and then select
the native schema
file to be edited.

ble
fe r a
an s
n - t r
a noFirst x records of
h a s file displayed here
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cschema
e d
Before you
e
edit a l
nativei file, you must ensure that the sample file specified in the
N
in is generated
annotation within the schema exists. This annotation is automatically added when the native
F lorschema the first time from the sample file.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 27

Using the Native Format Builder Wizard
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce schema for a complex file generated by the Native Format Builder
e
This slide d e
shows the l i
nXSD
n N
Wizard.
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 28

Quiz

Which KMs can Complex File data servers use?

a. Technology-specific KMs sourcing or targeting SQL data

servers
b. Generic KMs
c. XML KMs
d. All of the above e
r a bl
s fe
- t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Answer:edd
el lice
i n N
lo r
Explanation: Complex File data servers support both technology-specific KMs sourcing or
F targeting SQL data servers, as well as generic KMs.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 29

Summary

In this lesson, you should have learned to:

• Describe complex files concepts

• Describe the technical requirements for using complex files
with ODI
• Set the topology and create a complex files model

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 30

Practice 9-1 Overview: Configuring ODI Topology
and ODI Model with Complex Files
This practice covers exploring the XML Complex File
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

technology in ODI.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 31

Practice 9-2: Configuring Mappings to Support
JSON
This practice covers exploring the JSON Complex File
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

technology in ODI.

Oracle Data Integrator 12c: Advanced Integration and Development 9 - 32

Integration of ODI
in Enterprise Environment and SOA
ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori
Objectives

After completing this lesson, you should be able to:

• Describe the features of the three types of ODI Agents:

– Java EE Agents
– Collocated Agents
– Standalone Agents
• Use different types of web services with ODI
a b le
• Implement the integration of ODI with Service-Oriented fer
Architecture (SOA) t r a ns
- on
a n
a s
m ) h deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 2

Agenda

 ODI Integration with Java EE

• Using Web Services with ODI: Overview

• Invoking Web Services
• Integration of ODI with SOA

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 3

Three Types of ODI Agents

All Agents are multithreaded Java programs that can be

configured for high availability (HA).

• Local (no agent)
• Standalone Agent:
– Easiest to deploy
– Has built-in ODI Load Balancing feature
ble
• Collocated Agent: fe r a
– Needs WLS libraries, but indirectly ans
n - t r
• Java EE Agent: a no
– Slightly more complex to set up, needs) WLS has edomainฺ
o m i d
– Preferable for enterprise-scale deployment
l e ฺc t Gu
– HA provided by WLS Node c en
raManager
u @ o
S tud
d e lc this
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
e d el lice
There are Standalone Agents, Collocated Agents, and Java EE Agents. All three types of agents
n N
are multithreaded Java programs and can be configured for high availability. The main difference
i
r
Flo is where and how you install them (in WebLogic Server [WLS], with WLS, or on top of a Java
Virtual Machine [JVM]) and the benefits from this installation.
• Local (no agent) is not really an agent at all, but is adequate for much of development.
• Standalone Agents are easier to deploy anywhere, but do not have clustering or connection
pooling. WLS is not used at all. Standalone Agents run in their own JVM container (no
application server is required for Standalone Agents). Standalone Agents can be monitored
by EMCC. Do not confuse standalone agent (started external to ODI) with localagent
(started internal to ODI).
• Collocated Agents know that WLS is out there, and use some of the WLS libraries, but the
Agents are not deployed via WLS, and the Agents do not appear in the WLS console.
• Java EE Agents are slightly more complex to set up (you need to install WLS first, set up or
extend a domain, and so forth), but preferable in terms of enterprise-scale deployment:
clustering, load-balancing, high availability (HA), centralized monitoring (with Fusion
Middleware Console), and so forth.
Note that the choice between the types of agents is really a user’s choice, and it is easy to mix all
these types of agents seamlessly into an ODI environment.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 4

Java EE Agent Deployment Features

Deployment features:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

• WLS Integration
• Java EE Agent Template Generation
• WLS Configuration Wizard
• Automatic Data Source Creation for WLS
bl e
fe r a
WLS Domain ODI
t r a ns
• Admin Server n-
• Java EE Agent
n o
• Managed Svr • OtheraAgents
h a s
Consoles: • Managed Svr ) deฺ
• Managed Svr c o m u i
• FMWCC
e ฺ G
• EMCC r a cl ent
@ o tud
• ODI
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cbe
e d
The Runtime
e Agent l
cani deployed as a Java EE component within an application server. It
N
in in this
benefits configuration from the application server layer features such as clustering and
F lorconnection pooling for large configurations. This Java EE Agent exposes an MBeans interface,
enabling lifecycle operations (start/stop) from the application server console and metrics that can
be used by the application server console to monitor the agent activity and health.
• Oracle WebLogic Server Integration:
- ODI components integrate seamlessly with the Java EE application server.
• Java EE Agent Template Generation:
- ODI provides a wizard to automatically generate templates for deploying Java EE
Agents in WLS. Such a template includes the Java EE Agent and its configuration, and
can optionally include the JDBC data source definitions required for this agent, as well
as the drivers and library files for these data sources to work.
• By using the WLS Configuration Wizard, domain administrators can extend their domains or
create a new domain for the ODI Java EE Runtime Agents.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 5

Integration of ODI with Fusion Middleware Control
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
o r a d Java EE Agent

u@ is S t u
l c
n e de e th
i nฺ © t2015, s and/or its affiliates. All rights reserved.
uOracle
o r
Copyright
o
u (fl nse
d e lc lice
e an extension integrated into the Fusion Middleware Control Console (Enterprise
ODI provides
N
n
Manager, or EM). The default URL for EM is:
F lori http://<hostname>:<port>/em/
Example: http://localhost:7001/em/.
The ODI components can be monitored as a domain through this console, and administrators can
have a global view of these components along with other Fusion Middleware components from a
single administration console. To implement integration with Enterprise Manager:
• ODI Java EE Agent must be deployed and configured with the existing WLS domain
• Enterprise Manager and the ODI Enterprise Manager Plug-in must be deployed in the WLS
domain that has the ODI Java EE Agent deployed and configured

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 6

Using the ODI Console

Development
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Production Environment Production Servers

Agents

Topology/ Code Execution

Administrators
Security

Execution Execution Return

Operators Codes CRM Data
Log Log
Warehouse
ble
Production Repository

fe r a
ans
n - t r
Metadata n o ERP
Legacy
Business Lineage
s a
Users h
) deฺa
ODI c o m u i
Web Access ODI Console
l e ฺ t G
Consolea c n File / XML

@ or tude
e l cu his S
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
del provides
The ODIeConsole lice web access to ODI repositories. It enables an ODI developer to browse
ODI
i n N (such as projects, models, logs, and so on) and manage the ODI environment through
objects
r
lothe web service.
F
Business users, developers, operators, and administrators use their web browsers to access the
ODI Console. Note that with the ODI Console, you also can perform executions.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 7

Using the ODI Console: Example
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
To starte
the
l
deODI licein the browser, enter:
Console,
i n N http://<hostname>:<port>/odiconsole
o r
Fl Example:
http://localhost:8002/odiconsole
To log in, enter SUPERVISOR for user ID, and Welcome1 for password. Browse ODI objects in the
tree view, select the node, and then click the View icon (eyeglasses). The object or group of
objects will be displayed on the tab.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 8

Agenda

• ODI Integration with Java EE

 Using Web Services with ODI: Overview

• Invoking Web Services
• Integration of ODI with SOA

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 9

Types of Web Services

• Data Services: Are generated by ODI to give

you access to your data through web services

• Run-Time Web Services:
– Public Web Services: Access the ODI features through web
services deployed in a Java EE application server
– Agent Web Services:
Commands the ODI Agent to start and monitor a scenario, and a b le
er
—

to restart a session
n s f
— Built in the Java EE or Standalone Agent -tra on
a n
a s
m ) h deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce and Data Services are two different types of web services.
e d e
ODI Run-Time Web l i
Services
n N
i Run-Time Web Services:
F lorODI
• The Public Web Service connects to the repository to retrieve a list of contexts and
scenarios. This web service is deployed in a Java EE application server.
• The Agent Web Service is built in the Java EE or Standalone Agent.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 10

Data Services: Overview

• Data services are specialized web services.

• They provide access to:

– Data in datastores
– Captured changes in these datastores by using the Changed
Data Capture (CDC) feature
• They are generated by ODI and deployed to a web
services container. ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce web services that provide access to data in datastores via an SKM
e d e
Data services are l i
specialized
n N
(Service KM), and to the captured changes in these datastores by using the CDC feature using a
i (Journaling
F lorJKM KM). These web services are “automatically” generated by ODI and deployed to
a web services container—typically, a Java application server such as WLS. “Automatically” after
you do some initial configuration:
• Set up the topology.
• Set up the model.
• Import the SKM.
• And so on.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 11

Generation of Data Services

• Data services are generated by ODI.

• Generate the Java classes that expose the datastore’s

data as a data service from a datastore or a model, by
using a Service Knowledge Module (SKM).
• Compile and deploy these Java classes to a web services
container.
ble
fe r a
ODI J2EE ans
n - t r
Web services
a no
Datastore
Java classes container
h a s
Generation Data Service
o m i eฺ
) DatadService
l e ฺc t Gu
SKM
o rac uden
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cServices
e d e
ODI generates the l
Datai from a datastore or a model and deploys to a web services
n N
container in an application server.
F lori
This generation can be customized by using Service Knowledge Module (SKM). The resulting
data service is presented in the form of a Java package. You can compile and deploy these Java
classes to a web services container.
Data services can be generated and deployed into a web service stack implementing the Java API
for XML Web Services (JAX-WS), such as Oracle WebLogic Server.
Note: For more information about how to set up, generate, and deploy Data Services, refer to
Oracle Fusion Middleware Developer’s Guide for Oracle Data Integrator.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 12

Public Web Services: Overview

• ODI Public Web Services (WS) are web services that

leverage ODI features in a Service-Oriented Architecture

(SOA).
• WS uses WSDL to advertise.
• A Public Web Service enables applications to trigger
transformation developed with ODI through a secured web
service. ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ceprovide operations such as starting a scenario. Web Services (WS)
e d
ODI Public
e
Web l i
Services
n
uses N
Web Service Descriptor Language (WSDL) to advertise or catalog the existence of a service.
F lori
This secured web service can be:
• Deployed in any web services container, typically a Java 2 Platform, Enterprise Edition
(J2EE) application server
• Enabled with ODI Standalone Agent

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 13

Using Public Web Service OdiInvoke
and OdiInvokeWebService
To execute a scenario by using a Public Web Service:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u e
deal scenario
You canerun licfrom a web service calling in to ODI by using the ODI Public Web Service
in N
OdiInvoke. The opposite is an ODI tool OdiInvokeWebService used in a package to call out.
F lorOdiInvoke web service commands an agent to connect to a given Work Repository, and start a
specific scenario, whereas OdiInvokeWebService allows ODI to call someone else’s third-party
application. Summary: ODI can call out to you, you can call in to ODI.
Note that the agent (shown in the slide URL as oraclediagent) must be accessible from the
web services container host, and it must have access to the repositories.
The involved parameters are similar to the ones used when executing a scenario from an OS
command. The required parameters depend on the way you invoke the web service.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 14

Installing Public Web Services

Public Web Services can be:

• Deployed in any web services container (typically, a Java

application server)
• Provided by ODI Standalone Agent
• Provided as an archive file (.aar)

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u cebe deployed in any web services container installed in your machine,
e
Public Webd eServices l i
can
n
such N
as Java application server. Web services can also be provided by ODI Standalone Agent or
F lorasi an archive file (like a .jar or .zip file).

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 15

Agenda

• ODI Integration with Java EE

• Using Web Services with ODI: Overview

 Invoking Web Services
• Integration of ODI with SOA

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 16

OdiInvokeWebService Tool

• The OdiInvokeWebService tool can invoke any third-party

web service.
• It can be used in the tool step of a package or in a
Knowledge Module.
• It can save the response in an XML file that can be
processed with ODI.
• The OdiInvokeWebService tool provides an Advanced a b le
s f er
editor for generating its code. With this editor, you can:
an
– Connect to the WSDL on -tr
a n
– Specify parameters for the tool a
h eฺs
– Select a specific operation and specifym ) request
ฺ c o u id parameters
– Invoke a web service a c le nt G
– View the web service or tuindethe SOAP editor
response
@
u S
lc this
e d e
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ceservice request is written and saved in an XML file that can be used in
e d e
The response of the l i
web
n
ODI. N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 17

OdiInvokeWebService Tool in Action
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

1 Web service
2
Requests/invokes Port
ODI
OdiInvokeWebService Operation
package e
Response
r a bl
sfe
4
- t r an
n no
5 s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u e
delpackage,licscenario, or loadplan that contains an ODI tool is run.
1. AneODI
ri2.n NThe OdiInvokeWebService tool sends a client request to the web service through HTTP or
Flo HTTPS protocols.
3. OdiInvokeWebService invokes a specific operation on a port of the web service. The Web
Services Description Language (WSDL) file URL must be provided.
4. The response is written to a SOAP file. WSDL supports more than SOAP, but ODI supports
only SOAP.
5. The response is written to an XML file that can be processed with ODI.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 18

OdiInvokeWebService Tool in a Package
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
e
To create d
an
e l i
OdiInvokeWebService tool step in a package, perform the following steps:
N
Fl ori1.n Open the package where you want to create a tool step and click the Diagram tab.
2. From the toolbox, select Internet > OdiInvokeWebService. Click the diagram—a step
corresponding to your tool appears.
3. Click “Free choice” to be able to edit the step, then click the step icon in the diagram to open
the Properties panel.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 19

Agenda

• ODI Integration with Java EE

• Using Web Services with ODI: Overview

• Invoking Web Services
 Integration of ODI with SOA

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u e
del of this
The nextesection liclesson explores ways to integrate ODI within a Service-Oriented
n N
Architecture (SOA).
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 20

ODI with SOA Integration Scenarios

• Using Data Services to access data in the database tables:

– Generated by ODI and deployed as web services

• Exposing ODI processes as web services:
– Triggered as web services from any service-compliant
application (for example, Oracle BPEL, Oracle OSB, Oracle
BAM)
– Works with non-Oracle OEM ble
fe r a
• Invoking third-party web services (for example, a BPEL
t r a ns
process) as part of ODI workflow and using themon as- part of
n
data integration processes sa ha eฺ
)
ฺ c om uid
a c le nt G
@ or tude
e l cu his S
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
deintegrated
ODI canebe liseamlessly in a Service-Oriented Architecture (SOA) in several ways:
N
•n Data services that provide access to data stored in database tables. Coupled with the
F lori Changed Data Capture (CDC) capability, Data Services can also provide access to the
changed records for a given subscriber. Data services are automatically generated by Oracle
Data Integrator and deployed as web services to a web container, usually a Java application
server.
• ODI can also expose its transformation processes as web services to enable applications to
use them as integration services. For example, a LOAD_SALES batch process used to
update the CRM application can be triggered as a web service from any service-compliant
application, such as Oracle BPEL, Oracle Enterprise Service Bus (OSB), or Oracle Business
Activity Monitoring (BAM). Transformations developed by using ODI can therefore participate
in the broader Service-Oriented Architecture initiative.
• Third-party web services can be invoked as part of an ODI workflow and used as part of the
data integration processes. Requests are generated immediately and responses processed
through regular transformations. Suppose, for example, that your company subscribed to a
third-party service that exposes daily currency exchange rates as a web service.
If you want this data to update your multiple-currency data warehouse, ODI automates this task
with a minimum of effort. You would invoke the web service from your data warehouse workflow
and perform any appropriate transformation on the incoming data.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 21

Integration of ODI Within SOA in Action

Third-party Java EE Integration

process
application Source Target
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Web services
Business
process container Data
SOAP Access
Data Service

SOAP Third-party
OSB Public Web
Service Agent Web web service
Service
ble
SOAP Port
fe r a
SOAP Run-time
an
Operations
agent
n - t r
no
Start scenario
Restart sessions
Get sessions status s a
h a
List scenarios om
List contexts
i d eฺ
) OdiInvokeWebService Tool

l c
eฺ nt G u
ra udec Repository
@ o t
e l cu his S
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ceexample with the Data Services, Run-Time Web Services (Public Web
d e l i
eand Agent Web Service) and the OdiInvokeWebService tool.
This slide shows a simple
n N
Service
F lori
The Data Services and Run-Time Web Services components are invoked by a third-party
application, whereas the OdiInvokeWebService tool invokes a third-party web service.
Data Services provides access to data in datastores (both source and target datastores), as well
as changes trapped by the Changed Data Capture framework. This web service is generated by
ODI and deployed in a Java EE application server.
The Public Web Service connects to the repository to retrieve a list of context and scenarios. This
web service is deployed in a Java EE application server.
The Agent Web Service commands the ODI Agent to start and monitor a scenario and to restart a
session. Note that this web service is built in to the Java EE or Standalone Agent.
The OdiInvokeWebService tool is used in a package and invokes a specific operation on a port
of the third-party web service, for example, to trigger a BPEL process.
Oracle Data Integrator Run-Time Web Services and Data Services are two different types of web
services. Oracle Data Integrator Run-Time Web Services enable you to access the Oracle Data
Integrator features through web services, whereas Data Services are generated by Oracle Data
Integrator to give you access to your data through web services.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 22

Product Call web

mgmt process service
(BPEL)

2 ODI bulk data

service
1 Update data
ODI
ble
Operational Agent
fe r a
database SQL
ans
products
n - t r
3 a noWarehouse
Data

Join 4 h a s
Finance data WS, XML m ) deฺ
service discounts
ฺ c o u i
(BPEL) e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cean ODI data load that is controlled by SOA processes and uses web
e d e
In this example, you l i
see
n N
services as well as databases as sources.
F lori1. In this example, a business process in the product management department is changing
data in the product table of the operational database. This data needs to be propagated to
the data warehouse with minimal delay.
2. The process calls an ODI bulk data service to initiate a load based on the changes since the
last load.
3. The ODI Agent executes a scenario based on a package that first calls a data service from
the finance department to obtain price discount information.
4. Then this service joins this information with the changed product data from the operational
database. The joined and transformed data is stored in the data warehouse.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 23

Example 2: Exposing ODI Process
as a Web Service
1. Create the BPEL process to call your ODI scenario.
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

2. Create your ODI integration mapping.

3. Create a package with OdiInvokeWebService step
connected to your transformation mapping.
4. Generate a scenario from the package.
5. Edit the BPEL process to call your scenario. e
r a bl
s fe
- t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce can be integrated with a BPEL process for handling ODI errors by using
e
This slide d e
shows howl i
ODI
n
BPEL N Human Workflow.
F lori
In this example, you expose an ODI Transformation Process as a web service, which is called
from within a BPEL process:
1. Create a BPEL process, which will call your ODI scenario. (In the practices, you simply
enhance an existing BPEL process.)
2. Create your ODI transformation mapping.
3. Create a package with the OdiInvokeWebService step connected to your transformation
mapping. When adding the OdiInvokeWebService tool step, you should connect to the
appropriate BPEL process WSDL and specify the parameters for the tool.
4. Generate a scenario from the package.
5. Edit the BPEL process in JDeveloper by adding the Partner Links for integrating with ODI
and calling your scenario which in turn calls your package.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 24

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cneed
e d e
In JDeveloper, you l
may i to create the new application and the new project for your BPEL
N
orin
process. The project type you create is BPEL Process Project. Create your BPEL process
Fl according to your business project. This slide shows an example of a business process that
prepares the result that will be consumed and processed by ODI Interface. For details on creating
BPEL processes, refer to Oracle Fusion Middleware User’s Guide for Oracle Business Process
Management.
How to create a BPEL process from a blank starting point is outside the scope of this course. It is
covered in the SOA curriculum classes.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 25

Creating ODI Mapping and ODI Package

WSDL to call the

Advanced Editor for
SOAP Header and
ble
1 Response
fe r a
an s
n - t r
a no
h a s
m ) deฺ
ฺ 2
c o u i You will get all the
c l e n t G other information for
o a
r ude this in the lab.
u@ is S t
l c
n e de e th
i nฺ © t2015, s and/or its affiliates. All rights reserved.
uOracle
o r
Copyright
o
u (fl nse
d e lc lice
N e
1. Select OdiInvokeWebService and click the HTTP Analyzer (also known as the Advanced
rin Editor. See page 10-14 for an example). That populates the rest of the lower Properties
Flo panel.
2. Configure the OdiInvokeWebService step. Select the URL to connect to the BPEL
process WSDL. Enter other parameters for the OdiInvokeWebService tool as shown in
the slide. In the Properties panel, fill the following parameters:
- Storage mode for Response File: New File
- Response File: /tmp/processResponse.xml
- And others as directed by the Activity Guide
3. Connect to the OdiInvokeWebService step to the Mapping step in your diagram. Save
and close the package.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 26

bl e
fe r a
an s
1 n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce of creating the ODI scenario for the package and the process of editing
e
This slide d e
shows the l i
result
n
the N
BPEL process for invoking the scenario.
F lori1. Right-click the ODI package. Create the scenario for the ODI package.
2. Edit the BPEL process to ensure your invocation of your package.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 27

1
ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
3cl
e n t G
r a
o tud e
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
del process
After theeBPEL liciseready, it should be deployed to the application server and tested. To
n
deploy
i Nand test your BPEL process:
r
lo 1. In JDeveloper, select your BPEL process, right-click and then select Deploy.
F
2. Follow the screens to deploy your process to the WLS application server.
3. In Enterprise Manager, expand: SOA > soa-infra (AdminServer) > default > ODIInvoke.
Select ODIInvoke and click Test Service.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 28

ODI Package Business Process

“Error Hospital”

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cset
e
A businessd e
process l
is ia of coordinated tasks and activities, involving both human and system
N
orin
interactions, which leads to accomplishing a set of specific organizational goals. Business Process
Fl Execution Language (BPEL) is a programming abstraction that enables developers to compose
multiple discrete web services into an end-to-end process flow. BPEL enables the top-down
realization of a Service-Oriented Architecture (SOA) through composition, orchestration, and
coordination of web services.
In the BPEL process, the activities, called “Partner links” are used to integrate the business
process (BPEL) process with other applications within SOA. They link the BPEL process to
corresponding web services.
ODI can be integrated with the BPEL process by using the ODIInvokeWebservice tool. Thus, a
business process can be invoked from the ODI Package. After the execution, the BPEL process
can send a response back to ODI for subsequent data processing.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 29

Extract Transform

Load

bl e
fe r a
ans
n - t r
a no
• Rejected h a s
m ) deฺ
• Examine
ฺ c o u i
• Fixable? e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
e d e
For enterprise SOA l i
deployments, there is almost always a need for enterprise data extraction,
N
in taketransformation,
loading, and validation. By leveraging the native SOA architecture within ODI, you
F lorcan advantage of ODI to perform the ELT (Extract Load Transform). ODI provides the ability
to validate data during the load to the target by using ODI constraints or database constraints.
When this data is checked against a constraint by using Flow Control, any errors that are found
are not loaded to the target but are loaded to an errors table that is created and managed by ODI.
Each row of this table represents a record that did not pass a constraint. The row also has a
message column that explains why the record was rejected. This table can be edited within ODI
Designer or any other tool that can edit relational tables.
However, this is not always a convenient way for the end user or business user to edit the data.
Alternatively, the “ODI Error Hospital” can be created with BPEL Human Workflow. Any rows that
do not pass the constraint will be loaded to the error table on the target database. The ODI
scenario can be executed, and after the ODI ELT process is completed, the ODI scenario will then
call a BPEL web service to notify it of any errors during the load. The BPEL process will import the
errors and manage them by using BPEL human workflow tasks. A user can then use the BPEL
Worklist application to update bad records. On subsequent executions of the ODI scenario, the
updated records are recycled into the ELT process.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 30

1 2

bl e
fe r a
ans
n - t r
4a no
h a s
m ) deฺ 3
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ceconstraint are loaded into the target, and the error record is in the error
e
The recordsd ethat passl ithe
n N
table. In the slide, you see the steps to create such a process.
F lori
Because ODI manages the error table, you do not have to write any code to recycle the errors or
to delete the updated records when they are loaded successfully. Thus, you have to implement
the following steps:
1. First you need to build an ODI scenario that will validate data during the load to the target by
using ODI constraints or database constraints with a CKM. This ODI scenario will then call a
BPEL web service to notify it of any errors during the load.
2. You need to create a BPEL process that will import the errors and create BPEL human
workflow tasks (the green box in the middle). A user can then use BPEL Worklist to update
records that contain such errors.
3. You need to create an ODI package to execute the mapping and call a web service that
executes the BPEL process.
4. Finally, execute the ODI package, monitor execution of the BPEL process, and complete the
human workflow.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 31

ble
fe r a
a
an s
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e
1. Develop an ODIl i
transformation by using a mapping with an ODI constraint:
N
in a. Create a model and set up an ODI constraint that you need to enforce.
r
Flo b. Create a new mapping to load the source file (XML) to the target (RDBMS).

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 32

bl e
a c
fe r a
ans
n - t r
a no
b h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cprocess
e d e
2. Develop a BPELl i for invoking ODI from SOA and generating human workflow tasks.
N
in First create the BPEL process that will import data errors from the error table. The BPEL
r
Flo process will then create the human workflow tasks from the error records. You build a new
BPEL process to track the errors for each execution of the ODI package and present them to
a user for review. The user will be able to update each of the fields and correct any errors so
that on the next execution of the ODI package, the corrected rows are inserted into the
target. In this example, JDeveloper is the tool that you use to build the BPEL process and
deploy it to the application server:
a. Create a New Project for the BPEL process.
b. Create and configure the connection to an ODI data source.
c. Invoke database adapters to read errors and write error corrections back to the error
table.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 33

ble
fe r a
ans
n - t r
a no
e h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cprocess
e d e
2. Develop a BPELl i for invoking ODI from a SOA and generating human workflow
N
in tasks.
r
Flo d. Add Human Task (the green box) to your BPEL process.
e. Configure the Human Task parameters.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 34

a
ble
fe r a
ans
c
n - t r
b a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cprocess;
e d
3. Deploy
ethe l
BPEL i create and execute the ODI package. Deploy the BPEL process
N
in to the application server.
r
Flo a. Deploy the BPEL process.
b. Create the ODI package to execute the interface and invoke the web service.
c. Run your ODI package.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 35

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i c
a l e t G
o rac uden
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cofe the BPEL process from the BPEL Console, and complete the human
e d e
4. Monitor l
execution i
i n N
task:
r
Flo a. Using Oracle BPM Worklist application, perform an action on the human task to correct
errors.
b. Return to the BPEL Console page and refresh the browser. The process is now
updated and you see that it has progressed past the Error Hospital Human workflow
activity.
c. Click the Invoke Corrections activity to see the data that the error table has been
updated with.
You should now have a fully functional ODI to BPEL Human Workflow Error Hospital. If you rerun
your ODI scenario now, the corrected errors are picked up and resubmitted to the target table.

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 36

Quiz

Both types of agents, Java EE and Standalone, can be used for

integration of ODI within a SOA.

a. True
b. False

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 37

Summary

In this lesson, you should have learned how to:

• Describe the features of the three types of ODI Agents:

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 38

Practice 10-1 Overview: Integrating ODI in the
Enterprise Environment
This practice covers the following topics:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

• Performing the steps needed to configure the ODI Java EE

Agent with the existing WLS domain
• Performing monitoring of your ODI environment by using
Enterprise Manager

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 39

Practice 10-2 Overview: Exposing an ODI
Scenario as a Web Service
This practice covers the following topics:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

• Creating a new ODI scenario

• Configuring a web service
• Invoking the web service to execute the scenario

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 40

Practice 10-3 Overview: Integrating ODI
with a BPEL Process Within SOA
This practice covers the following topics:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

• Starting the SOA servers

• Using JDeveloper to edit existing BPEL processes
• Integrating ODI with a BPEL process within SOA

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce recommended RAM in your lab environment, this practice will be
e d e
Note: Because of thel i
less-than
n
slow.N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 10 - 41

Enhancing ODI Security

After completing this lesson, you should be able to:

• Describe the methods of enhancing security in ODI

• Use nongeneric profiles for implementing a strongly
secured approach
• Describe the methods of integrating ODI security
• Implement External Authentication with LDAP server e
r a bl
s fe
- t r an
no n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 2

Agenda

 ODI Security: Overview

‒ Profiles
‒ Users
‒ Objects
• Implementing a Strongly Secured Approach
• Security Integration e
• Configuring ODI External Authentication r a bl
s fe
- t r an
no n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 3

Security Concepts: Overview

• Object: The representation of an element that can be

handled through ODI

• Instance (object instance): A particular occurrence of an
object
• Method: Actions that can be performed on an object
• Profile: A generic rights model for working with ODI (set of
privileges) a b le
er f
– An authorization by a profile is placed on an object’s method
t r a ns
for a given profile.
n on-
– One or more profiles can be assigned to aas a to grant the
user
sum of these privileges to this user.m) h eฺ
ฺ c o u id
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce part of ODI object components (Java classes). It is necessary to
e d
• Objects
e are thel i
visible
i n N
combine the notions of object and object instance (or instances), which in ODI are similar to
r
Flo object-oriented programming notions. Examples are agents, models, and datastores.
• Instance: An example, MY_PROJ_1, is an instance of the project-type object. Similarly,
another instance of a project-type object is YOUR_PROJ_2. Thus, an instance is a particular
occurrence of an object. For example, the Datawarehouse project is an instance of the
Project object.
• Methods: Each object has a series of methods that are specific to it. The notion of a method
in ODI is similar to object-oriented programming. Methods include Create, Replace, Update,
Delete (CRUD).
• Profiles contain a set of privileges for working with ODI. One or more profiles can be
assigned to a user to grant the sum of these privileges to the user. An authorization by profile
is placed on an object’s method for a given profile. It allows a user with this profile to be
given, either optionally or automatically, the right to this object through the method. The
presence of an authorization by a profile for a method, under an object in the tree of a profile,
shows that a user with this profile is entitled (either optionally or automatically) to this object’s
instances through this method. The absence of authorization by a profile shows that a user
with this profile cannot invoke the method on an instance of the object.
Note: Objects and methods are predefined in ODI and must not be changed.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 4

Security Concepts: Overview

• User: Corresponds to the login name used for connecting

to a repository
• Authorization: Placed on a method of an object for a
given user
• User Method: A privilege granted to a user on a method of
an object type. It allows the user to perform an action.
• Profile Method: An authorization granted to a profile on a b le
r a
method of an object type sfe an
– Each granted method allows a user with this profile to - t r
perform an action (edit, delete, and so forth) on n
anoninstance
a
asand so forth).
of an object type (project, model, datastore,
) h deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
• Ae user lice Navigator module represents an ODI user and corresponds to the
deinl the Security
rin Nlogin name used for connecting to a repository. A user inherits the following rights:
Flo - All privileges granted to its various profiles
- Privileges on objects or instances given to this user
- Initially users have no privileges at all, not even Connect.
• Authorizations by the user are placed on a method of an object for a given user. It allows
the user to be given, either optionally or automatically, the right to this object through the
method.
• User Method is a privilege granted to a user on a method of an object type. Each granted
method allows the user to perform an action (edit, delete, and so forth) on instances of an
object type (project, model, datastore, and so forth). These methods are similar to the
Profiles Methods, applied to users.
Note: It is possible to grant users with privileges on instances on specific work repositories where
these instances exist. Think of it as a security hierarchy (top to bottom): repository, object, method,
instance. For example, you may grant a developer user the edit privilege on the
LOAD_DATAWAREHOUSE scenario on a DEVELOPMENT repository, and not on a PRODUCTION
repository. An authorization by object instance is granted to a user on an object instance. It allows
you to grant to this user certain methods of this object instance.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 5

Authorizing by Object Instance

• An authorization by object instance is granted to a user on

an object instance.
– It allows you to grant to this user methods of this object
instance.
– The presence of an authorization by object instance in a
user’s tree shows that the user is granted specific privileges
on the object methods for the given instance.
a b le
– If an instance is not visible in the tree of user instances, the er
User Method or Profile Method privileges for the objectatype n sf
tr
apply. on- n
• a
It is possible to grant users privileges on instances
s on
h a
specific work repositories where these eฺ exist.
m) instances
o id
– Authorizations by object instances
l e ฺc mayG u defined for one,
be
t
ac den attached to the Master
rRepositories
multiple, or all your Work
o tu
Repository. cu@ s S
el
e d e thi
r i n ฺn © 2015,uOracle
s and/or its affiliates. All rights reserved.
o Copyright
t o
u (fl nse
d e lc lice
N e
An authorization by object instance is granted to a user on an object instance. It allows you to

orin
grant to this user certain methods of this object instance.
Fl The presence of an authorization by object instance in a user’s tree for a given instance shows
that the user is granted specific privileges on the object methods for the given instance (these
privileges are specified in the Object Instance editor). If an instance is not visible in the tree of user
instances, the User Method or Profile Method privileges for the object type apply.
Because an instance may be replicated over the different Work Repositories that are attached to
the master repository, authorizations by object instances may be defined for one, multiple, or all
your work repositories attached to the master repository. For example, a LOAD_DATAWAREHOUSE
scenario instance may be replicated (using, for example, versioning) in the DEVELOPMENT, TEST,
and PRODUCTION repositories. Privileges on the instance will change depending on the
repository.
For example, it is common to replicate projects from a Development repository to a Test
repository. A developer may be granted edit privileges for his or her project on the Development
repository, but not on the Test repository. On the Test repository, the developer will be granted
only view privileges on the project.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 6

Create New
Profile

ble
Create New
fe r a
User
an s
n - t r
a no
h a s
ODI
m ) deฺ
Objects ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
l
deSecurity
By usingethe ice tab, you can manage security in ODI. The Security Navigator
lNavigator
n
module
i N allows ODI users and profiles to be created. It is used to assign user rights for methods
o r
Fl (edit, delete, and so on) on generic objects (data server, data types, and so on), and to fine-tune
these rights on the object instances (Server 1, Server 2, and so on).
The Security Navigator stores this information in a master repository. This information can be used
by all the other modules.
New with 12c is Roles. They are similar to Profiles, but externally available (for example, to OPSS
and LDAP).
Roles in OPSS have been there all along.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 7

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ceobjects available to the current user are organized by these tree views:
e d e
• Security l i
Navigator
i n N- The objects, describing each ODI elements type (datastore, model, and so on)
r
Flo - The users’ profiles, users, and their authorizations
• You can perform the following operations in each tree view:
- Insert or import root objects to the tree view by clicking the appropriate button in the
frame title.
- Expand and collapse nodes by clicking them.
- Activate the methods associated with the objects (Edit, Delete, and so on) through the
pop-up menus.
- Edit objects by double-clicking them or by dragging them on the Workbench.
• The windows for the object being edited or displayed appear in the Workbench.
Note: Each tree view appears in floatable frames that may be docked to the sides of the main
window. These frames can also be stacked up. When several frames are stacked up, tabs appear
at the bottom of the frame window to access each frame of the stack. Tree view frames can be
moved, docked, and stacked by selecting and dragging the frame title or tab. To lock the position
of views, select Lock window layout from the Windows menu. If a tree view frame does not appear
in the main window or has been closed, it can be opened by using the Windows > Show View
menu.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 8

Using Generic and Nongeneric Profiles

– They have the Generic privilege option selected for all

object methods.
– A user with such a profile is by default authorized for all
methods of all instances of an object.
• Nongeneric (NG_) profiles:
– By default they are not authorized for all methods of all a b le
instances of an object. s f er
- t r an
– The administrator must grant the user the rights on
o nthe
methods for each instance. a n
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u ceGeneric privilege option selected for all object methods. This implies that
del havelithe
Genericeprofiles
n Nwith such a profile is by default authorized for all methods of all instances of an object to
aiuser
r
lowhich the profile is authorized.
F
Nongeneric profiles are not by default authorized for all methods on the instances because the
Generic privilege option is not selected for all object methods. The administrator must grant the
user the rights on the methods for each instance.
If the security administrator wants a user to have the rights on all instances of an object type by
default, the user must be given a generic profile.
If the security administrator wants a user to have the rights on no instance by default, but wants to
grant the rights by instance, the user must be given a nongeneric profile.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 9

Built-in Profiles

These have an NG_ (Nongeneric) version as well:

• CONSOLE: Can use ODI Console (versus Studio)

• DESIGNER: To perform development operations
• METADATA_ADMIN: To manage metadata (models)
• VERSION_ADMIN: To create, restore, and edit versions
and solutions le
a b
These do not have an NG_ version:
s f er
• CONNECT: To connect to ODI - t r an
n on
• OPERATOR: To manage runtime objects (for a production
a s
users)
m ) h deฺ
• SECURITY_ADMIN: To edit security ฺ c o u i
e
cl ent G
r a
• TOPOLOGY_ADMIN: To edit tud and agents
o servers
u@ is S
l c
n e de e th
i nฺ © t2015, s and/or its affiliates. All rights reserved.
uOracle
o r
Copyright
o
u (fl nse
c ce that the security administrator can assign to the users he or she
del built-inliprofiles
ODI hasesome
n N This slide shows some built-in profiles delivered with ODI. For a complete list of built-in
creates.
i
r
loprofiles, see the Oracle Fusion Middleware Developer’s Guide for Oracle Data Integrator.
F
• CONNECT: Must be granted with another profile
• CONSOLE: Can use ODI web-based Console from WLS (versus Studio client)
• DESIGNER: Use this profile for users who work mainly on projects.
• METADATA_ADMIN: Use this profile for users who work mainly on models.
• OPERATOR: Use this profile for production users.
• SECURITY_ADMIN: Use this profile for security administrators.
• TOPOLOGY_ADMIN: To edit technologies, physical and logical servers, contexts, and agents
• VERSION_ADMIN: Use this profile for those entitled to perform version management
operations
SUPERVISOR is not a sum of all the profiles; it is its own distinct configuration, a kind of a user.
Do not modify the built-in profiles. You can copy them and modify the copies.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 10

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ceprivileges nor profiles (not even Connect) other than the one check box.
e d
Supervisor
e
has no l i
other
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 11

Agenda

• ODI Security: Overview

 Implementing a Strongly Secured Approach

‒ Generic
‒ Nongeneric
‒ Unused Cleanup
• Security Integration e
• Configuring ODI External Authentication r a bl
s fe
- t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ceto do this, you must configure it manually. This is intentional.
e
Note that d e
there is no l i
API
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 12

Security Policy Approach

There are two main approaches for defining ODI security:

• Generic • Strongly secured

– Simple – Complex
– Users inherit the – Users have no default
privileges of the profiles authorizations on
they have. objects. e
r a bl
s fe
- t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cwhere
e d
The generic
eapproach, l i users inherit the privileges of the profiles they have:
N
•n This policy is suitable for most cases, and is simple to configure.
F lori
The strongly secured approach, where users have no default authorizations on objects
• This approach uses only nongeneric profiles.
• The security administrator must grant the users authorizations on object instances.
• This policy is complex to configure because it requires you to manage privileges by instance.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 13

Defining Security Policies: Using Generic Profiles

1. Create the users.

2. Give the users generic profiles (built-in or customized).

ble
fe r a
2ans
n - t r
a no
3. Optionally, you can define a s
a password policy. h
) deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
e
Password d e
policies, l
to iencourage users to use a strong password, consist of combinations of
N
in length,
minimum UPPER/lowercase, expiration dates, and so on. If the passwords are
F lorauthenticated externally, you can use stronger methods such as OAAM.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 14

ble
4 fe r a
ans
n - t r
no
2 s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cesecured (and very granular) approach:
e d e
To implement a l i
strongly
n N
Fl ori1. Create the users as before.
2. Give the users nongeneric profiles (built-in or customized).
3. Grant the users privileges on object instances after these are created. This operation must
be repeated for every new instance. This requires being able to see both the Designer tab
and the Security tab at the same time. To do that, drag the Designer tab out of the navigation
panel into a separate area of the desktop. (You can put it back when you are done.) Drag an
instance (for example, a project) from Designer and drop it on the user.
4. A panel opens with the subobjects and methods listed. You can either Select All (shown by
the red square) or select individually.
Note: Only certain objects support the authorization by object instance. These object types are
listed under the Instances node for each user. The methods for which the user has generic
privileges are not listed in the Object Instance Editor.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 15

Using Nongeneric Profiles:
Granting an Authorization by Object Instance
Allow selected methods Allow selected methods
in all repositories. in selected repositories.
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

1 2

ble
fe r a
an s
n - t r
a no
h a s
3 m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ceslide, you have to grant users with authorizations to each object
e
As shown din
ethe l i
previous
n N
instance. To grant an authorization by object instance to a user:
Fl ori 1. In Security Navigator, expand the Users accordion. In the Designer, Operator, or Topology
Navigator, expand the accordion containing the object onto which you want to grant
privileges. Select this object and then drag it on the user in the Users accordion. Click Yes.
The Authorization by Object Instance editor appears.
2. This editor shows the list of methods available for this instance and the instances contained
in it. For example, if you grant privileges on a project instance, the folders, interfaces, and so
forth contained in this project will appear in the editor. Fine-tune the privileges granted per
object and method. You may want to implement the following simple privilege policies on
methods that you select from the list:
- To grant all these methods in all repositories, click Allow selected methods (the button
that looks like a check box) in all repositories.
- To deny all these methods in all repositories, click Deny selected methods in all
repositories.
3. To grant all these methods in certain work repositories, click Allow selected methods in
selected repositories and then select the repositories from the list. Click OK.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 16

Authorizations
Master Repo

Objects

bl e
Work Repos fe r a
ans
n - t r
a no
h a s
m ) deฺ
Security ฺ c o u i
Clean-up e
clTool ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce instance are stored in the Master Repository.
e d e
The authorizations byl i
object
n N
F lorIfiobjects are deleted from all Work Repositories, the authorizations are not necessarily deleted. In
Oracle Database, this is called "drop ... cascade," but ODI cannot do the “cascade” part.
• You may want to retain certain unused authorizations if they refer, for example, to objects
currently stored in an exported file or in a stored solution.
The Security Clean-up Tool should be used periodically to remove these unused authorizations
from the Master Repository.
• Unused authorizations are removed if they refer to objects that do not exist in the Master
Repository or in any Work Repository.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 17

bl e
fe r a
ans
n - t r
a no
2 h a s
m ) deฺ
ฺ c o u i
e
cl ent G 3
o r a d
c u @ Stu
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
l
deunused
To cleaneup lice
authorizations:
i n
1.
NFrom the Security Navigator toolbar menu, select Clean Up Security Settings... The Security
o r
Fl Clean-up Tool dialog box appears.
2. On the Cleanable tab, select Clean for the cleanable security settings you want to remove.
The security settings that cannot be removed are shown on the Non-cleanable tab.
3. Click OK to clean up the selected security settings.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 18

Agenda

• ODI Security: Overview

• Implementing a Strongly Secured Approach

 Security Integration
• Configuring ODI External Authentication

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 19

ODI Security Integration: Overview

• Wallet
• Authentication (OPSS)
• Password Storage (JPS Integration) RCU

bl e
fe r a
an s
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
jps-config.xml
cl ent G
r a
o tudExternal External Directory
u @
c his S Authentication Server
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u ce for login by default in ~/.odi/oracledi/cwallet.sso
lStores passwords
e d
• Wallet:e l i
i•n N
Internal Authentication: ODI stores by default all the user information as well as the users’
r
Flo privileges in the Master Repository. A user who logs in to ODI logs against the Master
Repository. This authentication method is called Internal Authentication.
• External Authentication: ODI can optionally use Oracle Platform Security Services (OPSS)
to authenticate its users against an external Identity Store, which contains enterprise users
and passwords. Such an identity store is used at the enterprise level by all applications, to
have centralized username and password definitions and Single Sign-On (SSO). In such a
configuration, the repository contains only references to these enterprise users. This
authentication method is called External Authentication. WLS or OUD can provide this.
• Internal Password Storage: ODI stores by default all security information in the Master
Repository. This password storage option is called Internal Password Storage. ODI can
optionally use Java Provisioning Service (JPS) for storing critical security information.
• External Password Storage: If you are using JPS with ODI, the data server passwords and
contexts are stored in the JPS Credential Store Framework (CSF). This password storage
option is called External Password Storage.
Note: When using External Password Storage, other security details such as usernames,
password, and privileges remain in the Master Repository.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 20

Implementing External Authentication (OPSS)

• Configuring ODI components for External Authentication

• Setting the authentication mode when:

– Creating the Master Repository with RCU
– Switching the authentication mode

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
jps-config.xml e n t G
cl $HOME/.odi/oracledi/
r a d e
system-jazn-data.xml @o tujps-config-jse.xml
u
lc this S
e d e
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ce for External Authentication:
e d
Configuring
e
ODI l i
components
n N
lori
• To use the External Authentication option, you need to configure an Enterprise Identity Store
F (for example, LDAP, Embedded LDAP in WLS, Oracle Unified Directory, and others), and
have this identity store configured for each ODI component to refer by default to it. The
configuration to connect and use the identity store is contained in an OPSS configuration file
called jps-config.xml. Refer to the Oracle Fusion Middleware Security Guide for more
information. Copy this file and also system-jazn-data.xml into the
$HOME/.odi/oracledi/ directory. ODI Studio reads the identity store configuration and
authenticates it against the configured identity store. Rename the JPS file as shown.
• ODI components deployed in a container (for example, Java EE agent, ODI Console) do not
require a specific configuration. They use the configuration of their container. Refer to the
Oracle Fusion Middleware Security Guide for more information about an OPSS configuration
in a Java EE context.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 21

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u cepassword storage in two ways:
deorl modifylithe
You caneset
i N
•n Creating the Master Repository enables you to define the authentication mode.
lo r
F • Switching the Authentication Mode modifies the authentication mode for an existing Master
Repository. The latter way requires you to be disconnected from any repositories.
Note: When you perform a password storage recovery, context and data server passwords are
lost and need to be re-entered manually in the Topology Navigator. If you are using External
Authentication, usernames and passwords are externalized. ODI privileges remain within the
repository. Data servers and context passwords also remain in the Master Repository. You can
externalize data server and context passwords by using the External Password Storage feature.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 22

Configuring External Authentication

To use the External Authentication option, you need to:

• Configure an enterprise identity store, such as:

— Embedded WLS LDAP
— Oracle Unified Directory
• Have this identity store configured for each ODI
component to refer to it by default
bl e
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
fe r a
<jpsConfig xmlns="http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd"
ans
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" schema-major-version="11"
schema-minor-version="1"
n - t r
xsi:schemaLocation="http://xmlns.oracle.com/oracleas/schema/11/jps-config-11_1.xsd
a no
jps-config-11_1.xsd">
: (many lines skipped)
h a s
m ) deฺ
<serviceProviders>
ฺ c o u i
e
<serviceProvider type="IDENTITY_STORE" name="idstore.ldap.provider"
cl ent G
r a
class="oracle.security.jps.internal.idstore.ldap.LdapIdentityStoreProvider">
o tud
<description>LDAP-based IdentityStore Provider</description>
</serviceProvider>
u @
c his S
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
e
To use the d e
External l i
Authentication option, you need to configure an enterprise Identity Store
N
in store
(LDAP, Oracle Unified Directory (OUD), Microsoft ActiveDirectory, and so on), and have this
F loridentity configured for each ODI component to refer by default to it. The configuration to
connect and use the identity store is contained in an OPSS Configuration file called the jps-
config.xml file.
Copy this file into the $HOME/.odi/oracledi directory. The Studio reads the identity store
configuration and authenticates against the configured identity store. If you want to locate this file
in a different location, edit the $ODI_HOME/studio/bin/odi.conf file and edit the option that
sets the location of the configuration file. This option is set in the following line: AddVMOption -
Doracle.security.jps.config=./jps-config.xml.
Copy the jps-config.xml file also to the agent directory. The agent and the command line
scripts will authenticate against the configured identity store.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 23

Oracle Unified Directory (OUD)

[oracle@edvmr1p0 bin]$ cd $FMW_HOME/asinst_1/OUD/bin

>>>> Specify Oracle Unified Directory LDAP connection parameters

Administrator user bind DN [cn=Directory Manager]: orcladmin
Password for user 'orcladmin': Welcome1
Unable to connect to the server at "edvmr1p0" on port 4444

--- Server Status --- Default name has a

Server Run Status: Started space in it, but that
Open Connections: 0 causes problems.
--- Server Details ---
ble
Host Name: edvmr1p0
fe r a
Administrative Users: cn=orcladmin

ans
Installation Path:
Instance Path:
/u01/app/oracle/Middleware/Oracle_OUD1
/u01/app/oracle/Middleware/asinst_1/OUD
n - t r
Version: Oracle Unified Directory 11.1.2.2.0
a no
Java Version: <not available> (*)

h a s
) deฺ
Administration Connector: Port 4444 (LDAPS)
--- Connection Handlers ---
c o m
Default port is :389
u i
Address:Port : Protocol : State
e ฺ unless you are not root.
G
-------------:----------:---------
r a cl ent
0.0.0.0:1389 : LDAP : Enabled
@ o tud
c his S
u
: (many lines skipped)

e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce can configure this product, but the way used in this course is the
e
There are d e
many waysl iyou
n N
simplest:
i VM (no completely stand-alone product using its own internal database and using a plain old
F lorJava WLS, no Oracle DB dependencies). Other more sophisticated environments and
other products could be used as well; the point is that ODI can contact some sort of LDAP. For
most purposes of ODI and this course, the LDAP is a black box.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 24

Agenda

• ODI Security: Overview

• Implementing a Strongly Secured Approach

• Security Integration
 Configuring ODI External Authentication
‒ XML files
‒ ldapsearch (or equivalent GUI) e
‒ r a bl
odi_credtool
s fe
‒ rcu
- t r an
no n
‒ Studio a
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 25

Configuring External Authentication with OUD

To configure the External Authentication option, you need to

perform the following steps:

1. Create a new LDAP user (or confirm existing user).
2. Edit jps-config-jse.xml to point to external LDAP
Server.
3. Run the script to set up the credentials to access LDAP in
the identity store. a b le
s f er
4. Create a new ODI Master Repository referencing a user
- t r an in
the external LDAP Server. n on
s a
5. Create a new ODI connection referencing
) a
h eฺ in the
a user
external LDAP Server.
ฺ c om uid
e nt G
6. Switch modes from InternalratoclExternal
e o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 26

1. Creating a New LDAP User with OUD

[oracle@edvmr1p0 bin]$ cd $FMW_HOME/asinst_1/OUD/bin

~/labs/ldap/pswd.txt -b dc=example,dc=com -s sub "uid=SUP*"

dn: uid=SUPERVISOR,ou=People,dc=example,dc=com
Good practice to not
objectClass: person
have the password in
objectClass: inetOrgPerson the command itself.
objectClass: organizationalPerson
objectClass: top
givenName: SUPERVISOR
uid: SUPERVISOR
That says “Welcome1”
ble
cn: SUPERVISOR
after encryption.
fe r a
sn: SUPERVISOR
ans
userPassword:
{SSHA512}JSPutuxKg2sfBgAW2JtjzBRx2ZrElifIBC93ogTSJlAagDMssXb0XRE5V
n - t r
NlD++aMp/JhMsqyzyzOh3yP2BHt7h+UzxA1U7++ no
ou: People
s a
mail: [email protected] h
) deฺ a
c o m u i
e ฺ G
cl ent
[oracle@edvmr1p0 bin]$ more ~/labs/ldap/seed_odi_oud.ldif
(only if you are curious what created this a
r
o tud
entry…)

@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u ce by the –j parameter) would be OS-protected.
l file (indicated
e d e
The password l i
n N
lori
[oracle@edvmr1p0 bin]$ more ~/labs/ldap/pswd.txt
F Welcome1
The LDIF seed file is shown on the next page.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 27

When the OUD database was created, it was “seeded” with the following file:
[oracle@edvmr1p0 bin]$ more ~/labs/ldap/seed_odi_oud.ldif

dc: example
description: Fictional example company

dn: ou=People,dc=example,dc=com
ou: People
objectclass: top
objectclass: organizationalUnit
ble
fe r a
dn: uid=SUPERVISOR,ou=People,dc=example,dc=com
ans
objectClass: person
n - t r
objectClass: organizationalPerson
n o
objectClass: inetOrgPerson s a
objectClass: top h a
) deฺ
c o m u i
cn: SUPERVISOR
e ฺ G
givenName: SUPERVISOR
r a cl ent
mail: [email protected]
@ o tud
ou: People
e l c his S
u
sn: SUPERVISOR
ฺ n ed se t
l o r in to u
uid: SUPERVISOR
u (f nseWelcome1
c
userPassword:
lice
e del
i n N dn: uid=OdiUser1,ou=People,dc=example,dc=com
o r
Fl objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: top
cn: OdiUser1
givenName: OdiUser1
mail: [email protected]
--More--(44%)

(and other entries for other people not shown)

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 28

2a. Editing jps-config-jse.xml
to Point to OUD LDAP Server

<serviceInstance name="idstore.oud" provider="idstore.ldap.provider">
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

<property name="subscriber.name" value="dc=example,dc=com" />

<property name="idstore.type" value="IPLANET" />
<property name="bootstrap.security.principal.map" value="BOOTSTRAP_JPS"/>
<property name="bootstrap.security.principal.key" value="bootstrap"/>
<property name="username.attr" value="uid"/>
<property name="groupname.attr" value="cn"/>
<property name="ldap.url" value="ldap://localhost:1389" />
<extendedProperty>
This matches the port
<name>user.search.bases</name>
on the output from
ble
<values> ./status
fe r a
<value>ou=People,dc=example,dc=com</value>
an s
</values>
n - t r
no
</extendedProperty>
<extendedProperty>
a
This matches the LDIF
s
<name>group.search.bases</name>
<values> h
) deฺ a
seed structure.

<value>cou=Groups,dc=example,dc=com</value>
c o m u i
e ฺ G
</values>
r a cl ent 1
</extendedProperty>
o t u d
</serviceInstance>
l c u@ is S
n e de e th
i nฺ © t2015, s and/or its affiliates. All rights reserved.
uOracle
o r
Copyright
o
u (fl nse
d e lc liceinto $HOME/.odi/oracledi/jps-config-jse.xml.
e
Copy jps-config.xml
N
n
i that you have the following three sections in the file. View properties in these three sections:
F lorVerify
1. JPS OUD LDAP Identity Store Service Instance
2. Basic Authenticator Login Module
3. <jpsContext name="default">
Edit the values in the JPS OUD LDAP Identity Store Service Instance section in the jps-
config-jse.xml file to point to your external OUD LDAP Server.
Note: The idstore.type property points to the type of the LDAP identity store with no default
value. The following are examples of some possible values for this parameter:
• OID: Oracle Internet Directory 10g and 11g
• OVD: Oracle Virtual Directory 10g and 11g
• ACTIVE_DIRECTORY: Microsoft Active Directory
• IPLANET: Sun Java System Directory Server and OUD
• WLS_OVD: WebLogic OVD (Embedded LDAP Server)

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 29

2b. Editing jps-config-jse.xml
to Point to OUD LDAP Protocols
<serviceInstance name="idstore.loginmodule" provider="jaas.login.provider">
<description>Identity Store Login Module</description>
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

<property name="loginModuleClassName"
value="oracle.security.jps.internal.jaas.module.idstore.IdStoreLoginModule"/>
<property name="jaas.login.controlFlag" value="REQUIRED"/>
</serviceInstance>
2

<jpsContext name="default">
<serviceInstanceRef ref="credstore"/>
ble

<serviceInstanceRef ref="policystore.xml"/>
fe r a
This matches the first
ans

<serviceInstanceRef ref="trust"/> highlighted box from the
n - t r
<serviceInstanceRef ref="pdp.service"/> previous slide.
a no
<serviceInstanceRef ref="attribute"/>
h a s
<serviceInstanceRef ref="idstore.oud"/>
m ) deฺ
<serviceInstanceRef ref="idstore.loginmodule"/>
ฺ c o u i
</jpsContext>
e
cl ent 3 G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ceinto $HOME/.odi/oracledi/jps-config-jse.xml.
e d e
Copy jps-config.xml l i
n N
i that you have the following three sections in the file. View properties in these three sections:
F lorVerify
1. JPS OUD LDAP Identity Store Service Instance
2. Basic Authenticator Login Module
3. <jpsContext name="default">
Notice that the LDAP userid and password are not in this file.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 30

3. Running odi_credtool to Set Up the
Credentials for the Identity Store
[OS]$ cd $ODI_HOME/sdk/bin
[OS]$ ./odi_credtool.sh
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Using JPS Configuration file /home/oracle/.odi/oracledi/jps-config-jse.xml

Sep 17, 2014 7:32:44 PM oracle.security.jps.JpsStartup start
INFO: Jps initializing.
Sep 17, 2014 7:32:48 PM oracle.security.jps.internal.common.util.JpsCommonUtil
initPolicyProvider
INFO: Initializing OPSS Java Policy Provider.
Sep 17, 2014 7:32:48 PM oracle.security.jps.internal.common.util.JpsCommonUtil
initPolicyProvider
INFO: OPSS Java Policy Provider initialized.
ble
Sep 17, 2014 7:32:48 PM oracle.security.jps.internal.common.util.JpsCommonUtil
initPolicyProvider
fe r a
INFO: OPSS Java Policy Provider is set as the system-wide Policy object.
ans
Sep 17, 2014 7:32:48 PM oracle.security.jps.JpsStartup start
n - t r
INFO: Jps started.
a no
User name: [cn=orcladmin]
h a s
Password: Welcome1
m ) deฺ
ฺ c o
The credentials have been successfully added to the bootstrap credential store.
u i
[OS]
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ctoe set up the credentials for the identity store, navigate to
e d e
To run odi_credtool l i
n N directory and then execute odi_credtool. Ignore possible warnings.
i input entries are provided
$ODI_HOME/sdk/bin/
F lorThe in the following:
• Map: oracle.odi.credmap
• Key: <OID Realm>
• Username: <OID Administrator User Name>
• Password: <OID Administrator Password>

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 31

OPSS cannot be
easily shared.
ble
fe r a
t r a ns
n o user-Repository
DB n
Master
for new

s a
a
) h deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u e
cRepository
e
To createda
e
new l
Masteri referencing a user in the external LDAP Server, start
N
orin
Repository Creation Utility, and then enter the necessary database connection information. You
Fl cannot extend an existing OPSS store to support both Internal and External Authentication
simultaneously, you need to create a new OPSS. There are several ways to do that; using RCU to
make a whole new set of repositories is the easiest.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 32

A user created in the

LDAP server

ble
fe r a
an s
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ceconnection to your new Master Repository. Configure Repository
You needdto
ecreate a l i
e with necessary parameters. In the ODI section, enter SUPERVISOR, the User and
new
n N
Connections
i
F lorPassword of the authenticated user in your external LDAP store. In the Repository Connection
section, enter the name created as part of RCU.
It may seem confusing to have the same name SUPERVISOR in both the internal and external user
stores, but there is an advantage to doing it this way as far as inheriting privileges.
Click Test to verify your entries.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 33

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o 2 ui
c l e n t G
a
or tude
@
cu his S
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u c>eODI > Switch Authentication Mode.
e d
1. Select
eMain l
Menu i
i n
2.
N
Enter the same information you used in the Connection login. Click Next.
r
Flo

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 34

3 4

ble
fe r a
5 ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce depending on the current Authentication Mode in use:
e
3. The d e
next action l i
varies
i n N- If currently using Internal Authentication, you are prompted to switch to External
r
Flo Authentication.
- If currently using External Authentication, you are prompted to switch to Internal
Authentication.
4. Click the green plus to add users from LDAP into the new RCU repository.
5. If the users already existed in the previous repository, then they map over as supervisors.
If you want to prove that you are really accessing the SUPERVISOR in the LDAP and not the
one in ODI itself, then you can log out, stop LDAP, and then try to log back in again and it
will say, “Bad credentials.”

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 35

Switching the Master Repository
Authentication Mode
This operation must be performed by a Supervisor user.
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

• When switching from an External to an Internal

Authentication:
– User passwords are not copied from the identity store to the
repository.
– All the user accounts are forcibly expired and must be re-
activated by a SUPERVISOR created during the switch.
– The context passwords are lost. Passwords for data servers able
and the JDBC password of the Work Repository are s f er
n
removed from their credential store. -tra n
• no
When switching from an Internal to an External
a
Authentication: has ) ฺ
ommatch
– Users that exist in the repositorycand
ฺ u idaeuser in the
identity store are automatically
c lemapped
n t G
a
r ustore e
– Unmatched users in the@oidentity t d are disabled
e l cu his S
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce mode of the ODI repository changes the way users authenticate. This
e
Switching d e
the l i
authentication
n N
operation must be performed by a Supervisor user.
F lori
When switching from an External to an Internal Authentication, user passwords are not copied
from the identity store to the repository. The passwords are nullified. All the user accounts are
marked as expired and must be re-activated by a SUPERVISOR that is created during the switch.
When switching from an Internal to an External Authentication, the users that exist in the
repository and match a user in the identity store are automatically mapped. Users that do not
match a user in the identity store are disabled. A Supervisor must edit the users so that their name
has a match in the identity store.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 36

1 e
3 r a bl
s fe
- t r an
n no
s a
2 h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ceAuthentication, you can re-activate ODI users (one at a time) that have
e d e
After switching to l i
Internal
n
been Nde-activated during the switch.
F lori
To reactivate a user:
1. In Security Navigator, expand the Users accordion. Select the user that you want to re-
activate from the list of users. Right-click and select Edit. The User editor appears.
2. Deselect Allow Expiration Date, or extend it past today.
3. If you want to set a password for this user, click Change Password and enter the new
password for this user.
4. From the File main menu, select Save.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 37

External Password Storage

• ODI stores by default all security information in the Master

Repository—Internal Password Storage.

• ODI can optionally use JPS for storing critical security
information—External Password Storage.
– When using JPS with ODI, the data server passwords and
contexts are stored in the JPS Credential Store Framework
(CSF).
a b le
– Other security details such as usernames, user passwords, s f er
and privileges remain in the Master Repository. - t r an
n
nohave
– It is possible to externalize the authenticationaand users
and passwords stored in an identity store s
haby using External
m ) e ฺ
Authentication. co uid l eฺ nt G
ra udec
@ St o
l c u is
e d e t h
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u cesecurity information in the master repository. This is internal password
e d
ODI stores
e
by default l i
all
n N
storage.
F lori
To use the External Password Storage option, you need to install a WebLogic Server instance
configured with JPS, and all ODI components (including the Runtime Agent) need to have access
to the remote credential store.
When using JPS with ODI, the data server passwords and contexts are stored in the JPS
Credential Store Framework (CSF).
To use the External Password Storage option, you need to install a WebLogic Server instance
configured with JPS and all ODI components (including the Runtime Agent) need to have access
to the remote credential store. For more information, refer to “Configuring Applications to Use
OPSS” in the Oracle Fusion Middleware Application Security Guide.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 38

Setting Up External Password Storage

There are four ways to set or modify password storage:

• Importing the Master Repository enables you to change

the password storage.
• Creating the Master Repository enables you to define the
password storage.
• Switching Password Storage modifies storage for an
existing Master Repository. a b le
s f er
• Recovering the Password Storage enables you to recover
- t r an
from a credential store crash. non a
a s
m ) h deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d
Supervisor
e
can force l i
authentication back to Internal temporarily for recovery.
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 39

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cstorage
e
To switch d e
the l
password i mode:
N
Fl ori1.n Disconnect ODI Studio from the repository. From the ODI main menu, select Password
Storage > Switch...
2. Specify the login details of your Oracle Data Integrator master repository, which are defined
when connecting to the master repository. Click Next.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 40

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce Storage (Internal/External).
e d
3. Select
eExternal l i
Password
i n
4.
N
With this mode, you have to provide the MBean Server parameters to access the credential
r
Flo store. These parameters are provided in the following. Click Finish. You can now reconnect
to the ODI repository.
- Host: MBeans Server Host (for example, abc115.oracle.com)
- Port: MBeans Server Port (for example, 7001)
- User: MBeans Server User Name (for example, weblogic)
- Password: MBeans Server Password (for example, weblogic)
Note: ODI offers a password recovery service that should be used only in case of an External
Password Storage crash. Using this procedure, password storage is forced to Internal Password
Storage because the external storage is no longer available. This operation should be performed
by a Supervisor user.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 41

Quiz

Which of the following is true about switching from an External

to an Internal Authentication?
a. User passwords are not copied from the identity store to
the repository.
b. All the user accounts are marked as expired.
c. All the user accounts must be re-activated by a
SUPERVISOR user that is created during the switch. a b le
s f er
d. All of the above tran
n on-
s a
a
) h deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Answer:edd
el lice
i n N
lo r
Explanation: When switching from an External to an Internal Authentication, user passwords are
F not copied from the identity store to the repository. The passwords are nullified. All the user
accounts are marked as expired and must be reactivated by a SUPERVISOR that is created
during the switch.

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 42

Summary

In this lesson, you should have learned to:

• Describe methods of enhancing security in ODI

• Use nongeneric profiles for implementing a strongly
secured approach
• Describe methods of integrating ODI security
• Implement External Authentication with LDAP Server e
r a bl
s fe
- t r an
no n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 43

Practice 11-1 Overview:
Implementing ODI External User Authentication
In this practice, you implement ODI External Authentication
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

with OUD LDAP Server.

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cbeerunning in WLS; in your practice environment it is running stand-alone.
e d
In real life,
e
OUD l
would i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 11 - 44

Choosing Integration Strategies:

Best Practices
ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori
Objectives

After completing this lesson, you should be able to:

• Describe typical integration processes

• Describe different integration strategies
• Implement integration strategies with Staging Area on the
target
• Implement integration strategies with Staging Area le
a b
different from the target
s f er
• Maintain the integrity of data - t r an
• Describe best practices for working on ODI projects n on
s a
• Implement real-time data integration by a
) husing
o m i d eฺODI with
Oracle GoldenGate eฺc Gu cl ent
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 2

Agenda

 Defining an ODI Integration Strategy

• Strategies with Staging Area on the Target

• Strategies with Staging Area Different from the Target
• Using ODI for Bulk Processing
• Real-Time Data Integration: Using ODI with GoldenGate
• Working on ODI Projects: Best Practices ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 3

Integration Strategies

Where to put the Staging Area?

Source Other Target

ble
fe r a
Factors in the decision:
t r a ns
• DB technology and capabilities
n on-
• Network speed and latency s a
a
h eฺ
• CPU horsepower and utilization om) id
e ฺ c G u
• Disk speed and utilization acl n t
o r d e
• Politics u@ Stu lc this
e d e
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ce some of the integration strategies that are used in ODI. They are
e d e
The following slides l i
explain
n N
grouped into two categories:
F lori• Strategies with the Staging Area on the Target
• Strategies with the Staging Area different from the Target
(could be on the source, could be on an optional middle platform)

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 4

Agenda

• Defining an ODI Integration Strategy

 Strategies with Staging Area on the Target

‒ KMs
‒ SCDs
• Strategies with Staging Area Different from the Target
• Using ODI for Bulk Processing le
a b
• Real-Time Data Integration: Using ODI with GoldenGatesfer
• Working on ODI Projects: Best Practices - t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 5

Strategies with Staging Area on the Target

The following strategies are used when the Staging Area

schema is located in the same data server as the target table

schema:
• Append
• Control Append
• Incremental Update
a b le
• Slowly Changing Dimension (SCD)
s f er
“Same data server” means same instance, but probably-tnotr an
same schema. non a
a s
m ) h deฺ
Source
ฺ c o u iTarget
e
cl ent G
r a
o tud
c u @ S Work
l i s
n e de e th
i nฺ © t2015, s and/or its affiliates. All rights reserved.
uOracle
o r
Copyright
o
u (fl nse
d e lc lice
e process uses an integration strategy that defines the steps required in the
An integration
N
orin
integration process. Examples of integration strategies are:
Fl • Append: Optionally delete/truncate all records in the target datastore and insert all (no Flow
Control, no Static Control) the flow into the target. No checking, no CKM.
• Control Append: Optionally delete all records in the target datastore and insert all the flow
into the target. This strategy includes an optional Flow Control (checking incoming records
for constraints/validation.).
• Incremental Update: Probably a SQL MERGE statement, which is more efficient than an
INSERT+UPDATE (one statement versus two separate SQL statements). Optionally
delete/truncate all records in the target datastore. Identify new and existing records by
comparing the flow with the target, and then insert new records and update existing records
in the target. This strategy includes an optional Flow Control.
• Slowly Changing Dimension: This strategy implements a Type 2 Slowly Changing
Dimension by identifying fields that require a simple update in the target record when
changed and fields that require to historize the previous record state.
This phase may involve one server when the Staging Area and the target are located in the same
data server. It may involve two servers when the Staging Area and target are on different servers.
Each of these four bullet options is covered in separate slides subsequently.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 6

Strategies with Staging Area
on the Target: Append
The Append strategy inserts the incoming data flow into the
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

target datastore, before possibly deleting the target.

1. Delete

bl e
fe r a
ans
2. Transform n - t r
and insert data a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
3. Commit r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ceincoming data flow into the target datastore, possibly deleting the content
e d
This strategy
e inserts l
thei
theN
ofin target beforehand.
F lor
This integration strategy includes the following steps:
1. Delete (or truncate) all records from the target table. This step usually depends on a KM
option.
2. Transform and insert data from sources located on the same server and from loading tables
in the Staging Area. When dealing with remote source data, LKMs will have already
prepared loading tables. Sources on the same server can be read directly. The integration
operation will be a direct INSERT/SELECT statement containing all the transformations
performed on the Staging Area in the SELECT clause and on all the transformations on the
target in the INSERT clause. There is no CKM in this mode, no “control.”
3. Commit the transaction. The operations performed on the target should be done within a
transaction and committed after they are all complete. Note that committing is typically
triggered by a KM option called COMMIT.
Note: The same integration strategy can be obtained by using the Control Append strategy (next
slide) and not choosing to activate Flow Control.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 7

1. Drop and create the integration table in the Staging Area.

2. Insert data into the loading table from the sources and
loading tables by using a single INSERT/SELECT.
3. Call the CKM for Flow Control.

bl e
fe r a
ans
n - t r
a no
h a s CKM
I$ I$ m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce by adding extra steps that will store the flow data in an integration
e d
This approach
e can bel iimproved
n
table N
(I$), and then call the CKM to isolate erroneous records in the error table (E$).
F lori
The Control Append integration strategy includes the following steps:
1. Drop (if it exists) and create the integration table in the Staging Area. (You can choose to
ignore the “error” of attempting to drop a nonexistent object.) The integration table is created
with the same columns as the target table so that it can be passed to the CKM for Flow
Control.
2. Insert data into the loading table from the sources and loading tables by using a single
INSERT/SELECT statement similar to the one loading the target in the Append strategy.
3. Call the CKM for Flow Control. The CKM will evaluate every constraint defined for the target
table on the integration table data. It will create an error table and insert the erroneous
records into this table. It will also remove erroneous records from the integration table. After
the CKM completes, the integration table will contain only valid records. Inserting them into
the target table can then be done safely. You can optionally fix the bad records (in an error
hospital?) and re-integrate them.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 8

Strategies with Staging Area
on the Target: Control Append
4. Remove all records from the target table.
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

5. Append the records from the integration table to the target

table in a single INSERT/SELECT statement.
6. Commit the transaction.
7. Drop the temporary integration table.

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i I$
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cefrom the target table. This step can be made dependent on an option
e d
4. Remove
e all l i
records
i n N
value that is set by the designer of the interface.
r 5.
Flo Append the records from the integration table to the target table in a single INSERT/SELECT
statement.
6. Commit the transaction.
7. Drop the temporary integration table.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 9

Strategies with Staging Area
on the Target: Incremental Update
The Incremental Update strategy is used to integrate data in
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

the target table by comparing the records of the flow with the
existing records in the target according to the “update key.”
1. Drop and create the integration table.
2. Transform and insert data using single INSERT/SELECT.
3. Recycle the rejected records from to the integration table?
a b le
4. Call the CKM for Flow Control. fer s
- t r an
n no
s a
I$ h
) deฺatrg
c o m u i CKM
l e ฺ t G
o rac uden E$
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u cestrategy is used to integrate data in the target table by comparing the
e d e
The Incremental l
Updatei
n N
records
i of the flow with existing records in the target according to a set of columns called the
F lor“update key.” Records that have the same update key are updated when their associated data is
not the same. Those that do not yet exist in the target are inserted. This strategy is often used for
dimension tables when there is no need to keep track of the records that have changed.
The challenge with such Integration Knowledge Modules (IKMs) is to use set-oriented SQL-based
programming to perform all operations rather than using a row-by-row approach that often leads to
performance issues. This method is described as follows:
1. Drop (if it exists) and create the integration table in the Staging Area. This is created with the
same columns as the target table (and additional columns as well) so that it can be passed
to the CKM for Flow Control. It also contains an IND_UPDATE column that is used to flag the
records that should be inserted (“I”) and those that should be updated (“U”).
2. Transform and insert data in the loading table from the sources and loading tables by using a
single INSERT/SELECT statement. The IND_UPDATE column is set by default to “I”.
3. Recycle the rejected records from the previous run to the integration table if the
RECYCLE_ERROR KM option is selected.
4. Call the CKM for Flow Control. The CKM will evaluate every constraint defined for the target
table on the integration table data. It will create an error table and insert the erroneous
records into this table. It will also remove erroneous records from the integration table.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 10

Strategies with Staging Area
on the Target: Incremental Update
5. Update integration table, set IND_UPDATE flag to:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

– “U” for all the records that have the same update key
– “N” for all records to ignore
6. Update the target with records flagged as “U”
7. Insert records into the integration table that are “I”
8. Commit the transaction. e
r a bl
9. Drop the temporary integration table. s fe
- t r an
n no
s a
h
) deฺ a
c o m u i I$
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u ce table to set IND_UPDATE to “U” for all the rows that have the same
lthe integration
e d
5. Updatee l i
i n N
update key values as the target ones. Therefore, rows that already exist in the target will
r
Flo have a “U” flag. This step is usually an UPDATE/SELECT statement, but could be a MERGE.
6. Update the integration table again to set the IND_UPDATE column to “N” for all records that
are already flagged as “U” and for which the column values are exactly the same as the
target ones. As these flow records match exactly the target records, they do not need to be
used to update the target data. After this step, the integration table is ready for applying the
changes to the target as it contains records that are flagged:
- “I”: These records should be inserted into the target.
- “U”: These records should be used to update the target.
- “N”: These records already exist in the target and should be ignored.
7. Update the target with records from the integration table that are flagged as “U”. Note that the
update statement is typically executed before the INSERT statement to minimize the volume
of data manipulated.
8. Insert records into the integration table that are flagged as “I” into the target.
9. Commit the transaction.
10. Drop the temporary integration table.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 11

• Teradata
• Oracle

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
e d
This approach
e can bel ioptimized depending on the underlying database. The following examples
N
n such optimizations:
illustrate
F lori• With Teradata, it may be more efficient to use a left-outer join between the flow data and the
target table to populate the integration table with the IND_UPDATE column already set
properly.
• With Oracle, it may be more efficient in some cases to use a MERGE INTO statement on the
target table instead of an UPDATE then INSERT.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 12

Working with Slowly Changing Dimensions

• Slowly Changing Dimension (SCD) is a technique that

stores and manages both current and historical data over

time in a Data Warehouse
• It is considered and implemented as one of the most
critical ETL (Extract, Transform, Load) tasks in tracking
the history of dimension records.
ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o Typetu2 d
Type 1 @ Type 3
c u s S
e d el thi
e
ฺn © 2015,uOracle
s and/or its affiliates. All rights reserved.
o r i n
Copyright
t o
u (fl nse
SCD is e
elc ice Kimball. The SCD choices are 1, 2, or 3.
sodnamed by lRalph
i N
•n Type 1 (default): Overwrites the row. The new data overwrites the existing data. Thus, the
r
lo existing data is lost because it is not stored anywhere else. This is the default type of
F
dimension that you create. You need not specify anything additional to create a Type 1 SCD.
• Type 2: Create new rows, identify certain fields that change, historize (track historical values
of either all or most recent) old/new values (audit in side records outside of the current row).
When the value of a chosen attribute changes, the current record is closed. A new record is
created with the changed data values and this new record becomes the current record. Each
record contains the effective time and expiration time to identify the time period between
which the record was active.
• Type 3: Add a field in the row for old/new values, and change the date field. Note it is all in
the same row. The column from the original table (in yellow) may be a column not expected
to change all that often, for example, marital status. The columns added to the end (in pink)
may be the date of a change, and the previous historical value. Two versions of values are
stored for selected attributes. Each record stores the previous value and the current value of
the selected attribute. When the value of any of the selected attributes changes, the current
value is stored as the old value and the new value becomes the current value.
See http://www.kimballgroup.com/ for information straight from the author of the term.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 13

Working with Type 2
Slowly Changing Dimensions
• It is often used for loading dimension tables, to keep track
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

of the changes that occurred on some of the columns.

• A typical SCD table would contain the following columns:
– Surrogate key
– Natural key
– Columns that may be overwritten
ble
– Columns that require a new record
fe r a
– Start date column ans
n - t r
– End date column
a no
– Current record flag
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u e
A Type e 2d
el icthe
SCD is onelof most well-known Data Warehouse loading strategies. It is often used
N
orin
for loading dimension tables, to keep track of changes that occurred on some of the columns. A
Fl typical SCD table would contain the following columns:
• A surrogate key that is calculated automatically. This is usually a numeric column that
contains an autonumber such as an identity column, a rank function, or a sequence.
• A natural key. List of columns that represent the actual primary key of the operational
system. This will become non-unique as history starts accumulating!
• Columns that may be overwritten on change (no need to historize)
• Columns that require the creation of a new record on change
• A start date column: When the record was created in the data warehouse
• An end date column: When the record became obsolete (closing date)
• A current record flag: Whether the record is the actual current one (1) or an old one (0). If
flag=0, then an end date must be present as well.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 14

bl e
Supplier is updated for product P1. Family is updated for product P2.
fe r a
ans
n - t r
a no
h a s 1
3
m ) deฺ 2
ฺ c o u i
4 e
cl ent G
Product P5 Name is updated r a
o tud
is added. for product P3. u @
c his S
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l
(f nseo
l c u ce of the behavior of the product SCD. In the operational system, a
e
This slide d e
shows an l i
example
n N
product is defined by its ID that acts as a primary key. Every product has a name, size, supplier,
i family.
F lorand In the Data Warehouse, you need to store a new version of this product whenever the
supplier or family is updated in the operational system.
In this example, the product dimension is first initialized in the Data Warehouse on March 12,
2006. All the records are inserted and are assigned a calculated surrogate key as well as the
ending date is set to January 1, 2400. Because these records represent the current state of the
operational system, their current record flag is set to 1.
After the first load, the following changes happen in the operational system: The supplier is
updated for product P1; the family is updated for product P2; the name is updated for product P3;
Product P5 is added.
These updates have the following impact on the Data Warehouse dimension:
1. The update of the supplier of P1 is translated into the creation of a new current record
(Surrogate Key 5) and the closing of the previous record (Surrogate Key 1).
2. The update of the family of P2 is translated into the creation of a new current record
(Surrogate Key 6) and the closing of the previous record (Surrogate Key 2).
3. The update of the name of P3 updates the target record with Surrogate Key 3.
4. The new product P5 is translated into the creation of a new current record (Surrogate Key 7).

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 15

Implementing Type 2 SCDs

Type 2 KMs need to know which columns act as SCD columns.

This is stored in the SCD Behavior field on the Description tab.

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cModule
e
To createda
e
Knowledge l i that implements this Type 2 SCD behavior, it is necessary to know
N
orin
which columns act as a surrogate key, a natural key, a start and end time stamp, and so on. ODI
Fl stores this information in the Slowly Changing Dimension Behavior field on the Description tab for
every attribute in the model.
When populating such a datastore in a mapping, the IKM has access to this metadata by using the
SCD_xx selectors on the getColList() substitution method.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 16

Implementing Type 2 SCDs: Steps

ODI implements Type 2 SCD as follows:

1. Drop and create the integration table.

2. Insert the flow data. Set the starting time stamp to the
current date and the ending time stamp to a constant.
3. Recycle previous rejected records.
4. Call the CKM to perform a data quality check on the flow. e
…continued… r a bl
s fe
- t r an
no n
s a
I$ trg
h
) deฺ a
c o m u i CKM
e ฺ G
r a cl ent E$
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cSCD
e d e
ODI implements Type l i2 as follows:
N
n Drop (if it exists) and create the integration table in the Staging Area.
lori
1.
F 2. Insert the flow data in the integration table by using only mappings that apply to the Natural
Key, overwrite on Change and Add Row on Change columns. Set the starting time stamp to
the current date and the ending time stamp to a constant, perhaps a fake date of 12/31/9999
to indicate “current.”
3. Recycle previous rejected records.
4. Call the CKM to perform a data quality check on the flow.
…continued…

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 17

Implementing Type 2 SCDs: Steps

5. Flag the records in the integration table.

6. Update the target with the columns flagged.

7. Close old records.
8. Insert the new changing records.
9. Drop the integration table.
ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ I$
ฺ c o u i
a c le nt G
@ or tude
e l cu his S
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
l recordsliincethe integration table as “U” when the natural key and the add row on
e
5. Flagd e
the
i n N
change columns have not changed compared to the current records of the target.
r 6.
Flo Update the target with the columns flagged overwrite on change by using the integration
table content filtered on the “U” flag.
7. Close old records (those for which the natural key exists in the integration table.)
− Set the current record flag to 0.
− Set the ending time stamp to the current date.
8. Insert the new changing records with their current record flag set to 1.
9. Drop the integration table.
Note: There may be some cases where the SQL produced requires further tuning and
optimization. For more information, refer to Knowledge Module Developer’s Guide for Oracle Data
Integrator.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 18

Agenda

• Defining an ODI Integration Strategy

• Strategies with Staging Area on the Target

 Strategies with Staging Area Different from the Target
• Using ODI for Bulk Processing
• Real-Time Data Integration: Using ODI with GoldenGate
• Working on ODI Projects: Best Practices ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 19

Strategies with Staging Area
Different from the Target
These strategies are used when the Staging Area cannot be
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

located on the same data server as the target datastore (files,

for example).
• Server to File Append
Target
Source
(file)

bl e
Work
fe r a
ans
n - t r
• RDBMS Table to JMS a no
h a s
m ) dTarget e ฺ
Source o
ฺc t Gu (JMS) i
l e
o rac uden
Work
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cwhen
There aree d e
some l
cases i the source is a single file that can be loaded directly into the target
N
n by using the most efficient method. By default, ODI suggests to locate the Staging Area on
table
F lorthei target server, use an LKM to stage the source file in a loading table, and then use an IKM to
integrate the loaded data to the target table.
If the source data is not transformed, the loading phase is not necessary.
In this situation, you would use an IKM that directly loads the file data to the target: this requires
setting the Staging Area on the source file logical schema. By doing this, ODI will automatically
suggest to use a “Multi-Connection” IKM that moves data between a remote Staging Area and the
target.
Such an IKM would use a loader, and include the following steps:
1. Generate the appropriate load utility script.
2. Run the loader utility.
An example of such a KM is the IKM File to Teradata (TTU).

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 20

Strategies with Staging Area Different from the
Target: Server to Server Append
• Server to Server Append:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

– When using a Staging Area different from the target and

when setting this Staging Area to an RDBMS, it is possible to
use an IKM that moves the transformed data from the
Staging Area to the remote target.
– This type of IKM is similar to an LKM and follows the same
rules. e
r a bl
s fe
- t r an
n no
s a
h
) deฺ a
Source c o m u iTarget
e ฺ G
r a cl ent
@ o tud
Work
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
Server e
el lice
todServer Append
i n Nusing a Staging Area different from the target and when setting this Staging Area to an
lo r
When
F RDBMS, it is possible to use an IKM that moves the transformed data from the Staging Area to the
remote target. This type of IKM is similar to an LKM and follows the same rules.
The steps when using the agent are usually:
1. Delete (or truncate) all records from the target table. This step usually depends on a KM
option.
2. Insert the data from the Staging Area to the target. This step has a SELECT statement on the
“Command on Source” tab that will be executed on the Staging Area. The INSERT statement
is written by using bind variables on the “Command on Target” tab and will be executed for
every batch on the target table.
The IKM SQL to SQL Append is a typical example of such a KM.
A variation of this strategy uses loaders or database-specific methods for loading data from the
Staging Area to the target, instead of the agent.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 21

Strategies with Staging Area Different from the
Target: Server to File or JMS Append
• Server to File or JMS Append:
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

– When the target datastore is a file or JMS queue, the Staging

Area is set on a different location than the target.
– You should use a “Multi-Connection” IKM that will integrate
the transformed data from your Staging Area to this target.
The method to perform this data movement depends on the
target technology e
r a bl
s fe
- t r an
n no
s a
h a
) dTarget ฺ
o m i e
Source
l e ฺc t Gu (JMS)
o rac uden
Work
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce is a file, or a JMS queue or topic, the Staging Area is set on a different
When thee d e
target l i
datastore
n N
location
i other than the target. This is because the Staging Area must be an RDBMS (not a JMS).
F lorTherefore, if you want to target a file or a queue datastore, you will have to use a “Multi-
Connection” IKM that will integrate the transformed data from your Staging Area to this target. The
method to perform this data movement depends on the target technology. For example, it is
possible to use the agent or specific features of the target (such as a Java API).
Typical steps of such an IKM will include:
1. Reset the target file or queue made dependent on an option.
2. Unload the data from the Staging Area to the file or queue.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 22

Agenda

• Defining an ODI Integration Strategy

• Strategies with Staging Area on the Target

• Strategies with Staging Area Different from the Target
 Using ODI for Bulk Processing
• Real-Time Data Integration: Using ODI with GoldenGate
• Working on ODI Projects: Best Practices ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 23

Using ODI for Bulk Processing

AIA-approved design patterns for using ODI with Oracle

Application Integration Architecture (AIA):

• Bulk data processing is the processing of a batch of
discrete records between participating applications.
• Oracle AIA uses ODI to perform bulk data integrations.

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
cset
e d
A business
e
process l
is ia of coordinated tasks and activities, involving both human and system.
N
orin
Bulk data processing is the processing of a batch of discrete records between participating
Fl applications. Oracle Application Integration Architecture (AIA) uses Oracle Data Integrator to
perform bulk data integrations.
See also new SOA 12c add-on Managed File Transfer.
http://www.oracle.com/us/products/middleware/soa/oracle-managed-file-transfer-ds-2215938.pdf

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 24

Using ODI for Bulk Processing: Design Patterns

AIA-approved design patterns for ODI with AIA architecture:

• Initial data loads

• High-volume transactions with XREF table
• Intermittent high-volume transactions
• Point-to-point integration with ODI
bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce of using AIA-approved design patterns for AIA ODI architecture.
e
This slide d e
provides anl ioverview
n N
Because
i ODI data transfer is always point-to-point, the source and target systems must be
F lorcapable of processing batch loads of data. An integration project should not adopt ODI as a
solution if there is a limitation in the number of rows that can be processed either on the source-
side or on the target-side application. There are four AIA-approved design patterns for using ODI
with AIA architecture:
• Initial data loads: The initial set of data of a particular object is loaded from the source
database to the target database. For example, loading Customer Account information or
loading Invoice information into a new application database from an existing source
application database. In the process, XREF data may or may not get established depending
on the integration requirement.
• High-volume transactions with XREF table: The interface tables on the source side have
a mechanism to indicate processed rows. Whenever a need arises for a high-volume data
transfer, AIA recommends using the Oracle Data Integrator solution for data transfer
between applications. Using this approach, the Oracle Data Integrator package transfers
data from source to target system on a regular basis. AIA recommends that the interface
tables on the source side have a mechanism to indicate processed rows.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 25

Using ODI for Bulk Processing: Design Patterns

AIA-approved design patterns for ODI with AIA architecture:

• Initial data loads

• High-volume transactions with XREF table
• Intermittent high-volume transactions
• Point-to-point integration with ODI
ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
e d e
• Intermittent l i
high-volume transactions: Batch loading coexists with regular online
N
in transactions (AIA recommends this approach). If you have a requirement that batch loading
r
Flo must coexist with regular online transactions, AIA recommends this approach.
• Point-to-point integration with ODI: For situations in which storing XREF data for high-
volume transactions does not make sense, AIA recommends using point-to-point integration
by using ODI. For example, the headquarters of a retail store chain receives data from
individual retail stores every day at the close of business. In this scenario, you need not store
XREF data between each individual local store with HQ because there are no DML
operations on those datasets. Local enterprise resource planning (ERP) applications load
their interface table and invoke an Oracle Data Integrator package to send data to the HQ
Interface table. After the data is processed, Oracle Data Integrator updates the local ERP
application’s Interface table with a Transferred or Processed status for each row.
For details, refer to “Using Oracle Data Integrator for Bulk Processing” in Developer’s Guide for
Oracle Application Integration Architecture Foundation Pack.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 26

Using Cross-Reference (XREF) Table: Overview

• Cross-references enable you to dynamically map values

for equivalent entities created in different applications.

• Cross-references are stored in the form of tables (XREF).
• The identifier mapping is required when information about
a customer is updated in one application and the changes
must be propagated to the other applications (Common).
– Add one more column: Common bl e
fe r a
• Example: ans
n - t r
a no
SAP EBS SBL Common
h a s
SAP_001 EBS_001 SBL001 m) CM001
o i d eฺ
l e ฺc t G u
rac uden
SAP_002 EBS_002 SBL002 CM002
@ St o
l c u is
e d e t h
ฺ n u s e
o r i n
Copyright ©
o
2015,
t Oracle and/or its affiliates. All rights reserved.
l
(f nse
l c u ce objects in one application, you may also want to propagate the
When youe d e
create or l i
update
n N
changes
i to the other applications. For example, when a new customer is created in an SAP
F lorapplication, you may want to create a new entry for the same customer in your Oracle E-Business
Suite application named EBS. However, the applications that you are integrating may be using
different entities to represent the same information. For example, for each new customer in an
SAP application, a new row is inserted in its Customer database with a unique identifier such as
SAP_001. When the same information is propagated to Oracle E-Business Suite and Siebel
applications, the new row should be inserted with different identifiers such as EBS_1001 and
SBL001. In such cases, you need some type of functionality to map these identifiers with each
other so that they can be interpreted by different applications to be referring to the same entity.
This can be done by using cross-references.
The identifier mapping is also required when information about a customer is updated in one
application and the changes must be propagated to other applications. You can integrate different
identifiers by using a common value integration pattern, which maps to all identifiers in a cross-
reference table. For example, you can add one more column named Common to the cross-
reference table shown in the slide.
Cross-referencing is an Oracle Fusion Middleware function, available through the Mediator
component, and leveraged typically by any loosely coupled integration that is built on Service
Oriented Architecture (SOA) principles. It is used to manage the runtime correlation between the
various participating applications of the integration.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 27

Using the XREF Knowledge Module

In ODI, the creation of XREF data is a two-step process. Each

step is a mapping:
• First: The user’s source table is the source in ODI and the
user’s target table is the target in ODI. The XREF data for
the source and common rows is created. Any column of
the target is populated with the COMMON identifier.
• Second: Target application identifies the datastore with the rable
mapping between the target identifier and the common n(or s fe
source) identifier. n - tra
o n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
Source u @ o Target
S tud
d e lc this
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
c u ce
In ODI, e
el
dcreation
the ofliXREF data is a two-step process. Each step is a mapping.
Inin
N
o r the first mapping, the user’s source table is the source in ODI and the user’s target table is the
Fl target in ODI.
While transporting data from the source table to the target table, create the XREF data for the
source and common rows. In this process, if you want to populate any column of the target with
the COMMON identifier, the ODI KM takes care of that, too.
In the second mapping, after data is posted from the mapping table to the base table, the target
application must identify a datastore where the mapping between target identifier and common (or
source) identifier that you have sent during previous mapping processing is available.
A second mapping must be run in which this datastore is the source in ODI and the XREF table is
the target. This creates the appropriate target columns in the XREF table.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 28

Using the XREF Knowledge Module

Source Table

1 XREF is
Source
updated

Target

PACKAGE 1
ble
fe r a
ans
Target application
creates data from
Target application updates a
datastore that contains the
n - t r
mapping to base table Common to Target mapping
a no
h a s
2 ) deฺ
Target and XREF
c o m u i
XREF
l e ฺ t G
PACKAGEa2c e n
r
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce names cannot be hard-coded, two variables must be defined to hold
e d e
Note: Because XREF l icolumn
the
i n N
source and target column names. Normally, these column names are derived from the AIA
F lorConfiguration file. This section does not describe how to get that from the XML but rather it
describes how to refresh this from a SQL SELECT statement.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 29

Agenda

• Defining an ODI Integration Strategy

• Strategies with Staging Area on the Target

• Strategies with Staging Area Different from the Target
• Using ODI for Bulk Processing
 Real-Time Data Integration: Using ODI with
GoldenGate e
r a bl
• Working on ODI Projects: Best Practices s fe
- t r an
no n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 30

Loading Data Patterns

– Data is loaded in full or incrementally.

– Capture: Filter Query for ODI
• Mini-batch:
– Data is loaded incrementally using intra-day loads.
– Capture: Filter Query for ODI
bl e
• Micro-batch: fe r a
– Source changes are captured and accumulated an s
- t r
to be loaded in intervals. n on
a
– Capture: CDC for ODI or OGG as ) h eฺ
• Real-time: c o m u id
e ฺ G
– Source changes capturedra and cl applied e n t immediately.
@ o tud
– Capture: CDC for OGG
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
Variouse del
architectures icecollecting transactional data (that is, only committed data from logs) from
lfor
n N sources have been used to populate Data Warehouses. These techniques vary mostly
operational
i
r
loon the latency of data integration, from daily batches to continuous real-time integration. The
F capture of data from sources is either performed through incremental queries that filter based on a
time stamp or flag, or through a CDC mechanism that detects any changes as they are happening.
Architectures are further distinguished between pull and push operations, where a pull operation
polls in fixed intervals for new data, whereas in a push operation, data is loaded into the target
after a change appears.
A daily batch mechanism is most suitable if intraday freshness is not required for the data, such as
longer-term trends or data that is calculated only once daily (for example, financial close
information). Batch loads might be performed in a down-time window, if the business model does
not require 24-hour availability of the Data Warehouse.
Approximate sizes:
• Batch: Weekends, nightly, or every 1000 rows
• Mini-batch: Hourly, or every 10 minutes. or every 100 rows
• Micro-batch: Less than every 10 seconds, or less than every 10 rows, whichever comes first
• Real-time: Sub-second, every row.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 31

Oracle GoldenGate: Overview

Oracle GoldenGate (OGG) provides low-impact capture,

routing, transformation, and delivery of database transactions

across heterogeneous environments in real time.

Key Capabilities: Additional Differentiators:

Real Time
Performance
bl e
Moves with sub-second latency
fe r a
ans
Heterogeneous n - t r
Moves changed data across
Extensibility &
a no
different databases and platforms
Flexibility
h a s
m ) deฺ
ฺ c o u i
Transactional e
cl entReliability G
r a
Maintains transaction integrityo
u @ S tud
d e lc this
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
e d e
Performance: l i
i n N
F lorLog-based capture moves thousands of transactions per second with low impact. It normally reads
logs, not actual tables, for performance reasons. This is in contrast to ODI, which normally reads
tables.
Extensibility:
Meets a variety of customer needs and data environments with open, modular architecture. OGG
supports lots of source technologies, but not nearly as many nor as diverse as ODI.
Reliability:
Resilient against interruptions and failures. OGG runs in memory, and also persists to trail files in
case of a power failure.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 32

Oracle GoldenGate Solutions: Overview

(Open & Active)

Zero–Down
Time Upgrades
New DB/OS/HW
and Migrations

Log-based,
Real-Time Change
Data Capture Live Reporting
Reporting

OGG
Database
ble
fe r a
ETL
ans
- t r
on BI
ODS EDW
Operational
ETL
a n
a
h eฺ s
Heterogeneous
m ) EDW

OLTP Systems
ฺ c o u id
e
cl ent G
BPM

o r a d BAM Transactional

u @ S tuMessage CEP
Data Integration

d e lc this Bus

ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce data replication solutions:
e d e
OGG provides the l i
following
n N
i Availability:
F lorHigh
• Live standby for an immediate failover solution that can later resynchronize with your primary
source. An option is to have heterogeneous standby: primary=Oracle, standby=DB/2.
• Active-active solutions for continuous availability and transaction load distribution between
two or more active systems. If Oracle-to-Oracle only, you also could use Data Guard and/or
RAC or other products in lieu of OGG.
Zero–down time upgrades and migrations:
• Eliminates down time for upgrades and migrations
Live reporting:
• Feeding a reporting database so that you do not burden your source production systems
Operational Business Intelligence (BI):
• Real-time data feeds to operational datastores or Data Warehouses, directly or via ETL
Transactional data integration:
• Real-time data feeds to messaging systems for business activity monitoring (BAM), business
process monitoring (BPM), and complex event processing (CEP)
• Uses event-driven architecture (EDA) and Service-Oriented Architecture (SOA)

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 33

Oracle GoldenGate Concepts
Capture: Committed transactions are captured (and can be filtered) as
they occur by reading the transaction logs.
Unauthorized reproduction or distribution prohibitedฺ Copyright© 2017, Oracle and/or its affiliatesฺ

Trail: Stages and queues data for routing

Pump: Distributes data for routing to target(s)
Route: Data is compressed,
encrypted for routing to target(s)
Delivery: Applies data with transaction
integrity, transforming the data as required

ble
fe r a
ans
Capture Trail
Pump
Trail
- t
Delivery
n r
LAN/WAN a no
Internet
h a s
m ) deฺ
TCP/IP
ฺ c o u i
e
cl ent G
Source r a
o tud Target
Database(s) @
Bi-directional
cu his S Database(s)
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce modules that are combined to create the best possible solution for
e d
OGG consists
e of l i
decoupled
n
your N
business requirements.
F lori
On the source system(s):
• OGG’s Capture (Extract) process reads data transactions as they occur, by reading the
native transaction log, typically the redo log. OGG moves only changed, committed
transactional data, which is only a percentage of all transactions—therefore operating with
extremely high performance and very low impact on the data infrastructure.
• Filtering can be performed at the source or target: at table, column, or row level.
• Transformations can be applied at the capture or delivery (also known as Replicat) stages.
Advanced queuing (trail files):
• To move the transactional data efficiently and accurately across systems, OGG converts the
captured data into an Oracle Canonical (that is, proprietary) Format in “trail” files. With both
source and target trail files moved by pumps (not to be confused with DB import/export
datapumps), OGG’s unique architecture eliminates any single point of failure and ensures
that data integrity is maintained—even in the event of a system error or outage.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 34

Transactional RDBMS
Source Tables Staging DB Target DB
Replicated Target Tables
Source Tables

ODI CDC
Framework

ble
fe r a
ODI an s
Extract
n - t r
Datapumps
Mappings
a no
Replicat
h a s ODI
WAN
m) uide ฺ
c o
Source Staging
c l eฺ nt G
trail trail
files@o
ra OGG u de
files t
e l cu his S
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce OGG, which provides a cross-platform data replication and changed
ODI canebedecombined liwith
data
i n N
capture (CDC).
r
lo • OGG works in a similar way to Oracle’s asynchronous CDC on ODI, but handles greater
F
volumes and works across multiple database platforms.
• OGG uses its own data logging (or “trails”) to record DML and DDL activities on a source
database, which it then replays on staging and target platforms to replicate changes in real-
time.
• Using OGG technology with ODI is beneficial, especially if you are reaching replication
performance limits.
Example: OGG captures and replicates changes from the source system, typically into a staging
schema on the target Data Warehouse platform. This staging schema contains a replicated copy
of the source table and the structures that are used by ODI’s own CDC framework. ODI then picks
up the changes in the structures of the ODI CDC framework, transforms the data, perhaps joins it
to look up other tables, and then loads it into the target schema. This technology does not require
ODI to be changed in any way, and uses OGG CDC with the regular existing ODI CDC
framework.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 35

2 3

ble
1 fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud 4
u @ S
d e lc this
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
1. IneTopology lice Architecture, create two new data servers, one for source and one for
del > Physical
in Ntarget.
r 2.
Flo On the JAgent tab, name them as appropriate, and configure the ports to match the ports
defined in the OGG mgr.prm file.
3. The source and target must have different host/port names. Test your connections.
4. When the tests are successful, both data servers will be displayed in the list.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 36

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent 2 G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce the directories for storing the trail files to be used by the data pumps.
e d e
This is where you l i
configure
n N
lori
1. Create the Physical Processes (source only, this is the Extract).
F 2. Create the Logical objects.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 37

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl 2ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce the directories for storing the trail files to be used by the data pumps.
e d e
This is where you l i
configure
n N
lori
1. Create the Physical Processes (target only, this is the Replicat).
F 2. Create the Logical objects.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 38

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
l
deOGG
This tiesethe lice to the ODI processes. There are many more screens not shown that
processes
n N CDC with JKMs. This was covered in the previous class, and is covered in the OLL
configure
i
r
loOBE.
F

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 39

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u cewith OGG, you have to create the package, which consists of a
e d
To initialize
e
CDC l i
process
n N
perpetual loop that is triggered by the log entry of a new journal and processes new changes with
Fl ori
the CDC Load Customer interface. Execute this package to initialize ODI CDC with OGG. The
name of this package is Sync_Data, and the last thing it does is to call itself, over and over again.
This recursive technique is cleaner than a return arrow loop.
Parameters of Note:
• Global Row Count=1 says to wait for only one row to be changed before proceeding to the
next step (that is, a “micro-batch”). OGG itself may have an optimum buffer size that thwarts
this micro-size.
• Polling Interval=1000 milliseconds, that is, 1 second (basically near-real-time).
• Logical Schema=ODI_DEMO_TRG and Subscriber=ODIDEMO match previous CDC
configuration values on screen in the OLL OBE.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 40

OGG Command Line: GGSCI

GGSCI (source) > info all

MANAGER RUNNING
JAGENT RUNNING
EXTRACT RUNNING CAP 00:00:00 00:00:00
EXTRACT RUNNING DELP 00:00:00 00:00:06

GGSCI (ODIGettingStarted) 12> Stats cap

Sending STATS request to EXTRACT CAP ...
ble
fe r a
GGSCI (target) > info all
ans
Program Status Group Lag at Chkpt Time Since Chkpt
n - t r
MANAGER RUNNING
a no
JAGENT RUNNING
h a s
REPLICAT RUNNING DEL1 00:00:00 00:00:07
m ) deฺ
ฺ c o u i
GGSCI (ODIGettingStarted) 8> Stats del1cl e G
Sending STATS request to REPLICAT o ra ...udent
DEL1
c u @ St
d e l t h is
ฺ n e s e
r i n
Copyright ©
o u
2015,
t Oracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce and monitored by ODI, but you can also see them from the OGG
e d e
The processes are l i
started
n N
command line, called GGSCI.
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 41

Agenda

• Defining an ODI Integration Strategy

• Strategies with Staging Area on the Target

• Strategies with Staging Area Different from the Target
• Using ODI for Bulk Processing
• Real-Time Data Integration: Using ODI with GoldenGate
 Working on ODI Projects: Best Practices ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 42

1. Use Context-Independent Design

 Do not use qualified object names in expressions:

DROP TABLE MY_SCHEMA.TBL_001

This is a context-dependent code.

 Use Substitution Methods (for example, DROP

TABLE <%odiRef.getObjectName("L",
"TBL_001","W")%>). ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
In manyeODI
l
deobjects lice
(procedure, variables, interfaces, and so on), you can add expressions and
N
orin
SQL codes. A very common mistake is to enter qualified object names, as in the following
Fl example, that drops a TBL_001 table in the staging schema MY_SCHEMA:
DROP TABLE MY_SCHEMA.TBL_001
If you run this code in the production environment where the staging schema is called
PRD_MY_SCHEMA, your code does not work. Note that the schema names are defined in the
topology, and the contexts would handle getting the right schema name depending on the
execution context. The Substitution Methods (also known as OdiRef API) exist to leverage in
your code the metadata stored in ODI and make the code context-independent.
In this case, it is assured that the qualified table name is generated according to the context you
are running your code into.
Using Substitution Methods, the generic code would be:
DROP TABLE <%odiRef.getObjectName("L", "TEMP_001","W")%>.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 43

2. Use Procedures Only When Needed

• Procedures contain manual code that needs to

be maintained manually.
• Procedures do not maintain cross-references to
other ODI artifacts such as datastores, models,
and so on.
– Their maintenance is complex compared to
interfaces. bl e
fe r a
 Procedures should never be used to move and
a ns
t r
transform data.
n on-
 Moving data operations should be performeds aby
h a
using mappings. m) deฺ c o u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ceperform very complex actions, including SQL statements. In addition,
e
Proceduresd e
allow youl i
to
n
they N
i data you
allow to use source and target connections, and support data binding. In short, you can
F lormove by using procedures.
Developers who feel at ease with SQL may be tempted to code their transformations and data
movements within procedures instead of using interfaces.
There are several issues with using procedures in such a way:
• Procedures contain manual code that needs to be maintained manually.
• Procedures do not maintain cross-references to other ODI artifacts such as datastores,
models, and so on, making their maintenance very complex compared to mappings.
Procedures should never be used to move and transform data. These operations should be
performed by using mappings.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 44

3. Always Enforce Data Quality

 It is a common mistake to not take into account the quality

of data.
 Enforce data quality of source data by using static checks.
 Enforce quality data before it is pushed into a target by
using flow checks.

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Project e del ofldata
developers ice integration sometimes do not take into account the quality of their data.
This
i n Na common mistake, because data integration itself may be to move and transform
is
r
loerroneous data, and propagate it over the applications.
F
ODI allows you to enforce data quality of source data by using static checks as well as the quality
of the data before it is pushed into a target via flow checks. By using both these checks, you can
make sure that the quality of the data is improved or enforced when the data is moved and
transformed.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 45

4. Handle Error Cases in Packages

• Every single step in a package may fail for some reason.

• You should always consider these different error cases

when designing packages.
• End the error path with a bogus command to error out the
entire package.

bl e
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
e d e
In a package, you l
can isequence any number of steps. Every single step in a package may fail for
some N
in always
reason, even if it is not your fault (for example, target or source database is down). You
F lormust consider these different error cases when designing packages. You achieve this by
having not only “OK” path (green arrows) in the packages but also the “KO” path (red arrow).
Suppose some middle step fails, then you go down the KO path and “successfully” send an alert.
If the alert itself is sent successfully, the package as a whole ends successfully. This makes it
more difficult to spot on the Operator tab. A way to “fix” this is after the Send Alert, to call an OS
command that does not exist, for example foobar, which then errors with no KO handler, so the
package as a whole ends with an error condition (now easy to spot) and you still have sent the
email alert.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 46

5. Choose Right Knowledge Module

• The KM choice is critical when designing mappings.

• Do not begin with complex KMs

• When switching to technology-specific KMs, read their
descriptions first and leverage all the features.
• KMs with extra features have an extra cost in terms of
performance.
le
– Default KM options are sufficient for running the KM providederab
out of the box. a n sf
o n -tr
a n
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
licewhen designing mappings. The KM choice conditions the features
del is critical
The KMechoice
n N and the performances of a mapping.
available
i
lo r
F Do not begin your mapping with complex KMs. Choosing, for example, technology-specific LKMs
by using loaders may lead to a nonworking mapping because the loader configuration or the
installation is not correct. A safe choice for starting to work is to use generic KMs (usually SQL
KMs) that work in most cases. After designing your first mappings with these KMs, you can start
switching to technology-specific KMs and leverage all the new features of these KMs.
Avoid over-engineering interfaces: KMs with extra features have an extra cost in terms of
performance. Examples:
• Performing a simple insert is faster than performing an incremental update.
• If you are deleting the data in the target before integration, using incremental update is over-
engineering, and causes some performance loss. Use the KM that fits your needs.
• Activating or deactivating some KM features may add extra steps that may degrade the
performance. Default KM options are sufficient for running the KM provided out of the box.
After running the KM with default options, it is always good to review the options and see
whether some of them can be changed for your needs. The KM and Option description are a
good documentation for understanding the optimization of KM usage.
• If the DB is already checking for NotNull, then do not have ODI also check for NotNull, a
duplicate check is extra overhead.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 47

6. Other Best Practices

• Do not import all the KMs, only import as required.

• Make sure that all repositories are created with strictly

different IDs (even on different sites). Document the
process for moving objects between repositories by using
import, export, or versioning.
• Self-document objects in the Repository. Add a
description, owner, date, and so on for each object a b le
r
(Variable, Mapping, and so on) for better understandingns offe
the relationship and usage of the object. n - tra
o n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
lice (GUIDs). Introduced with 12c, solve many of the potential duplicate
del Identification
GloballyeUnique
IDin N If you are running 11g or a mixed 11g/12c shop:
issues.
lo r
F • Two repositories with the same ID may possibly contain objects with the same internal ID,
which means the same object for ODI.
• Transferring objects between these repositories is similar to copying files with the same
names across different directories, and may lead to unintentional object replacement.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 48

Quiz

When using Oracle GoldenGate with ODI, you do not have to

change ODI CDC. You just use GoldenGate CDC with the
regular existing ODI CDC framework.
a. True
b. False

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Answer:ead
el lice
i n N
lo r
Explanation: This technology does not require ODI to be changed in any way, and uses
F GoldenGate CDC with the regular existing ODI CDC framework. However, while not required,
there is an advantage to changing ODI to “know about” (and start/stop/monitor) the OGG
processes.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 49

Quiz

Which product is aware of the status of the other product?

a. OGG knows about and can start/stop ODI.

b. ODI knows about and can start/stop OGG.
c. They both know about and can start/stop each other.
d. Neither one is aware of each other.
ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Answer:ebd
el lice
i n N
lo r
Explanation: ODI can be configured to know about the OGG servers, processes, agents,
F extracts, and replicats. OGG is blissfully unaware that ODI is out there.

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 50

Summary

In this lesson, you should have learned how to:

• Describe typical integration processes

Oracle Data Integrator 12c: Advanced Integration and Development 12 - 51

ODI and Big Data

After completing this lesson, you should be able to:

• Describe big data concepts

• Contrast terms such as Hadoop, NoSQL, Apache, Hive,
HBase, HDFS, and Connectors
• Implement ODI mapping for big data to Oracle SQL
• Locate BigDataLite VM for further research e
r a bl
s fe
- t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u e
clearning
e
Additionald e
resources l i
for about Oracle’s big data offerings can be found here:
N
•n Oracle Big Data Essentials [D79454GC10]
F lori http://education.oracle.com/pls/web_prod-plq-
dad/db_pages.getpage?page_id=609&get_params=dc:D79454,clang:EN
• Introduction to Big Data [D75058GC10]
http://education.oracle.com/pls/web_prod-plq-
dad/db_pages.getpage?page_id=609&p_org_id=1001&lang=US&get_params=dc:D75058G
C10,p_preview:N
• Oracle Big Data Learning Library (OLL)
https://apex.oracle.com/pls/apex/f?p=44785:141:0::NO:141:P141_PAGE_ID,P141_SECTIO
N_ID:27,615

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 2

Agenda

 Describe big data concepts

• Contrast terms
• Implement ODI mapping for big data to Oracle SQL
• Locate BigDataLite VM for further research

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 3

ACQUIRE ORGANIZE ANALYZE DECIDE

• HDFS • Hadoop • Data • Analytic
• Oracle (MapReduce) Warehouse Applications
NoSQL • Oracle
• Oracle DB Big Data e
Connectors r a bl
• ODI s fe
- t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u cedone on different hosts that are optimized for those functions. Different
e
Often the d e
four phasesl i
are
n N
software is responsible for different phases. The phases progress in order:
F lori1. ACQUIRE: Raw, unstructured data is captured or acquired using distributed file systems
(Hadoop Distributed File Systems [HDFSs]) and NoSQL databases (Oracle NoSQL
Database). Example formats of raw big data might be character, such as machine logs, or
binary, such as videos.
2. ORGANIZE: A programming paradigm called MapReduce is used to interpret and refine the
data. Only after it has been reduced can it be searched. This phase is where ODI
participates.
3. ANALYZE: The refined and organized data is fed into a relational database (SQL
databases) to enable proper analysis.
4. DECIDE: Business decisions can be based on the data.

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 4

Agenda

• Describe big data concepts

 Contrast terms
– Apache: Hadoop, HDFS
– Oracle: Databases, Connectors
• Implement ODI mapping for big data to Oracle SQL
• Locate BigDataLite VM for further research e
r a bl
s fe
- t r an
n no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
l c u ce
denames
Some ofethe oflithe components are very whimsical! For example, Hadoop was the name
that
i n N software creator’s child called his stuffed yellow toy elephant.
the
lo r
F

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 5

Open Source Projects

Apache Software Foundation is community-led development.

They manage the following big data projects:

• Hadoop
– HDFS
– Hadoop MapReduce
• Flume
ble
• Hive fe r a
ans
• HBase - t r
n no
• Pig a
h a s
• Sqoop ) deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Apachee del lice has many ongoing software development projects. The projects related
(http://apache.org)
bigN
toin data are:
lo r
F • Hadoop is a framework for distributed processing of large data sets across clusters.
HDFS (Hadoop Distributed File System) is a subproject of Hadoop. It is the file system that
stores large data sets, in the GB to TB range.
• MapReduce takes <key, value> pairs as input and produces other <key, value> pairs as
output. It also splits large input datasets into smaller chunks. The input and output data sets
are typically stored on a file system such as HDFS.
• Flume collects, aggregates, and moves large amounts of streaming log data.
• Hive is a data warehouse facility for querying and managing large distributed datasets.
• HBase is modeled after Google’s Bigtable. HBase queries very large structured tables
(billions of rows with millions of columns) stored in HDFS or Hadoop.
• Pig is a language for doing data analytics. Pig produces sequences of MapReduce
programs.
• Sqoop transfers bulk data between Hadoop and traditional structured RDBMS systems such
as Oracle Database.
The feather is a trademark of Apache Software Foundation.

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 6

Oracle Components

Oracle Databases Connectors

• Oracle Database • Oracle Direct Connector

• Oracle NoSQL for HDFS
• MySQL • Oracle Loader for Hadoop
• ODI Application Adapter
Analytics for Hadoop
a b le
• Oracle R Enterprise • Oracle R Connector for fer
s
Hadoop tran
• Oracle Exadata Intelligent
n on-
Warehouse Solutions s a
a
) h deฺ
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce components are Oracle components:
e
Combined d e
with the l i
Apache
n N
lori
• Oracle Database can do some big-data-like things in that it can have CLOB and BLOB
F columns mixed in with the traditional character and number columns. However, it is not
optimized for big data. It may be the target of a MapReduce run.
• Oracle NoSQL is based on Berkeley database Java Edition High-Availability (BDB JE/HA).
It uses a simple data model using key-value pairs with secondary indexes.
• Oracle R Enterprise is a component of the Oracle Advanced Analytics Option. “R” is a
statistical programming language that runs against Oracle Databases.
• Connectors are used to provide data communication between big data and Oracle
Databases. There are several available.
• Hardware: Not shown are Exadata appliances that run databases, both traditional RDBMS
and big data. Though, one of the design points of Hadoop is that it can run on scalable,
massively parallel, commodity PCs.

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 7

Agenda

• Describe big data concepts

• Contrast terms
 Implement ODI mapping for big data to Oracle SQL
– Topology
– Designer
– KMs e
• Locate BigDataLite VM for further research r a bl
s fe
- t r an
no n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 8

Big Data: Just Another Technology

• All of the complexity is in the KMs as a “black box."

• All of the MapReduce function is performed by the native,

optimized software/hardware, not by ODI.
• All of the models are used in a similar fashion regardless
of whether they are big data, relational data, or flat file.

bl e
fe r a
ACQUIRE ORGANIZE ANALYZE DECIDE
t r a ns
on-
• HDFS • Hadoop • Data • Analytic
• Oracle NoSQL (MapReduce) Warehouse
a n
Applications
• Oracle DB • Oracle
a s
Big Data
Connectors m ) h deฺ
ฺ c o u i
• ODI e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
l
dedata
After theebig lice components (physical/logical servers, datastores, and so on) are
topology
n N in ODI, the models are used just like any other models. All of the normal components and
defined
i
r
lotransformations are available.
F

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 9

ble
fe r a
Define Topology: ans
- t r
• Server
n on
• Physical a
h
• Logicala s
m ) deฺ
ฺ c o u i “normal”
just like other
e G
acl technologies
or tude nt
@
cu his S
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
c u
Accesse
el liceis by the usual methods: JDBC or JNDI.
todthe data server
i n N
lo r
The available schemas would be retrieved from the HBase metadata and displayed in the schema
F pull-down lists (where it currently says <Undefined>.)

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 10

Designer for Big Data

• Data Server (Topo)

• Logical Schema (Topo)
• Physical Schema(Topo)
• Model (Designer)
• Datastore (Designer)
ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce would be used to reverse-engineer the tables in your schema. After
e d
The metadata
e from l i
HBase
i n
that N
step, the datastores would be displayed in the list under the model, for example, under
F lormyBigData. After the datastores display, they can be dragged into a mapping diagram along with
other datastores from other technologies.

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 11

KMs for Big Data

• Lots of KMs available

• Lots more KMs coming

ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
Sample e dedata
big KMs:li
i N
•n RKM (Reverse Engineering)
lo r
F - RKM HBase
- RKM Hive
• LKM (Loading)
- LKM HBase to Hive
• CKM (Checking)
- CKM Hive
• IKM (Integration)
- IKM File to Hive
- IKM File-Hive to Oracle
- IKM File-Hive to SQL (Sqoop)
- IKM Hive Control Append
- IKM Hive Transform
- IKM Hive to HBase Incremental Update
- IKM SQL to Hive-HBase-File (Sqoop)

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 12

Agenda

• Describe big data concepts

• Contrast terms
• Implement ODI mapping for big data to Oracle SQL
 Locate BigDataLite VM for further research

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 13

ble
fe r a
ans
n - t r
a no
h a s
m ) deฺ
ฺ c o u i
e
cl ent G
r a
o tud
@
c his S
u
e l
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce sample code, and hands-on labs based on Oracle MoviePlex that will
e
Oracle hasd e
created l i
videos,
i n
help N
you learn how to develop big data applications using Oracle’s big data platform. Oracle
F lorMoviePlex is a fictitious online movie streaming company.
• Main site (do not let the “11g” in the URL fool you, it is current with 12c webcasts):
http://www.oracle.com/technetwork/middleware/data-integrator/odi-11g-webcast-archive-
367128.html
• Webcast – ODI 12.1.3 and Big Data:
https://oracleconferencing.webex.com/oracleconferencing/ldr.php?RCID=394c8e514957ba6
298c98d18f7ce5a09
• Webcast – ODI 12c and BigDataLite demo:
https://oracleconferencing.webex.com/oracleconferencing/ldr.php?RCID=c6de47feeccd7f2b
049d92775529c188
• Download – Oracle Big Data Lite Virtual Machine (approx 14 GB) and videos:
http://www.oracle.com/technetwork/database/bigdata-appliance/oracle-bigdatalite-
2104726.html

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 14

Quiz

Which of the big data components transfers bulk data between

Hadoop and traditional structured RDBMS ?

a. Flume
b. Pig
c. Sqoop
d. Hive e
r a bl
s fe
- t r an
no n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
Answer:ecd
el lice
i n N
lo r
Explanation: See page 13-6 for a list of what the other components do.
F

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 15

Quiz

a. ACQUIRE
b. ORGANIZE
c. ANALYZE
d. DECIDE
ble
fe r a
ans
n - t r
no
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r in © t2015,
Copyright
o uOracle and/or its affiliates. All rights reserved.
l o
(f nse
c u
Answer:ebd
el lice
i n N
lo r
Explanation: See page 13-4 for a list of what the other phases do.
F

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 16

Lesson Summary

In this lesson, you should have learned how to:

• Describe big data concepts

• Contrast terms such as Hadoop, NoSQL, Apache, Hive,
HBase, HDFS, Connectors
• Implement ODI mapping for big data to Oracle SQL
• Locate BigDataLite VM for further research e
r a bl
s fe
- t r an
no n
s a
h
) deฺ a
c o m u i
e ฺ G
r a cl ent
@ o tud
e l c his S
u
ฺ n ed se t
r
Copyright
o uOracle and/or its affiliates. All rights reserved.
in © t2015,
l o
(f nse
l c u ce
e d e l i
n N
F lori

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 17

Course Summary

In this course, you should have learned how to:

• Develop and enhance Knowledge Modules (KMs)

Oracle Data Integrator 12c: Advanced Integration and Development 13 - 18

Susanne K. Langer-Mind - An Essay On Human Feeling. 1-The Johns Hopkins University Press (1967)
100% (4)
Susanne K. Langer-Mind - An Essay On Human Feeling. 1-The Johns Hopkins University Press (1967)
514 pages
Oracle Database 19c: PL/SQL Workshop: Student Guide D108650GC10
100% (1)
Oracle Database 19c: PL/SQL Workshop: Student Guide D108650GC10
24 pages
Oracle 19c RAC Step by Step Part 1 Installation of RHEL 7.8
No ratings yet
Oracle 19c RAC Step by Step Part 1 Installation of RHEL 7.8
9 pages
Meaning and Definition of Education Educere, Educare Educatum
100% (1)
Meaning and Definition of Education Educere, Educare Educatum
1 page
Oracle Weblogic 12.2.1.X Configuration Guide: Oracle Utilities Application Framework
No ratings yet
Oracle Weblogic 12.2.1.X Configuration Guide: Oracle Utilities Application Framework
13 pages
ODI 12C Installation Step by Step PART 1
No ratings yet
ODI 12C Installation Step by Step PART 1
4 pages
Soa Suite 12c Exam Study Guide 2398385
No ratings yet
Soa Suite 12c Exam Study Guide 2398385
23 pages
Oracle SOA Suite Best Practices
No ratings yet
Oracle SOA Suite Best Practices
6 pages
Exam: 1Z0-931 1Z0-931-F: NO.1 A. B. C. D. E
No ratings yet
Exam: 1Z0-931 1Z0-931-F: NO.1 A. B. C. D. E
15 pages
Oracle Fusion Middleware Interview Questions: Edition
100% (1)
Oracle Fusion Middleware Interview Questions: Edition
16 pages
Active Data Guard Hands On Lab
100% (1)
Active Data Guard Hands On Lab
50 pages
Rman Restore and Recovery Scenarios
No ratings yet
Rman Restore and Recovery Scenarios
7 pages
Oracle Database 12c Install, Configure & Maintain Like a Professional: Install, Configure & Maintain Like a Professional
From Everand
Oracle Database 12c Install, Configure & Maintain Like a Professional: Install, Configure & Maintain Like a Professional
Ian Abramson
No ratings yet
D89113GC10 Ag
No ratings yet
D89113GC10 Ag
316 pages
D77758CN20 sg2
No ratings yet
D77758CN20 sg2
408 pages
D73668GC30 - Exadata Database Machine Administration Workshop Ed 3-02.03 A 06.03
No ratings yet
D73668GC30 - Exadata Database Machine Administration Workshop Ed 3-02.03 A 06.03
3 pages
Oracle E-Business R12.2 Step by Step Installation Guide
No ratings yet
Oracle E-Business R12.2 Step by Step Installation Guide
17 pages
Configuring OEM 13c For Oracle Database Firewall
No ratings yet
Configuring OEM 13c For Oracle Database Firewall
22 pages
Activity Guide OCI DB Service Professional WorkShop
No ratings yet
Activity Guide OCI DB Service Professional WorkShop
48 pages
09 - 08 - 21 - Oracle XML Gateway
No ratings yet
09 - 08 - 21 - Oracle XML Gateway
48 pages
EBS R12 Cloning Procedure
No ratings yet
EBS R12 Cloning Procedure
6 pages
TP Devops OCI Pro
No ratings yet
TP Devops OCI Pro
282 pages
Oracle Database Security
No ratings yet
Oracle Database Security
160 pages
OCI 1Z0-997 Verified 27july2020
No ratings yet
OCI 1Z0-997 Verified 27july2020
32 pages
Oracle RAC Commands
100% (1)
Oracle RAC Commands
2 pages
Oracle Autonomous Database Notes PDF
0% (1)
Oracle Autonomous Database Notes PDF
5 pages
Oracle Audit Vault Installation Manual
No ratings yet
Oracle Audit Vault Installation Manual
14 pages
Oracle Apps SysAdmin Configurations
100% (1)
Oracle Apps SysAdmin Configurations
23 pages
Oracle 19c - Important Feature For DBA
100% (2)
Oracle 19c - Important Feature For DBA
52 pages
ORacle Database Security 19c
No ratings yet
ORacle Database Security 19c
1 page
Odi 12.2.1: Rest
0% (1)
Odi 12.2.1: Rest
23 pages
Oracle Data Guard A Clear and Concise Reference
From Everand
Oracle Data Guard A Clear and Concise Reference
Gerardus Blokdyk
No ratings yet
r12 2 Oracle Applications System Administrator Fundamentals
No ratings yet
r12 2 Oracle Applications System Administrator Fundamentals
5 pages
Oracle Database 19c RAC Administration Workshop
No ratings yet
Oracle Database 19c RAC Administration Workshop
1 page
OEM 13c Cloud Control OUGN 2016
No ratings yet
OEM 13c Cloud Control OUGN 2016
45 pages
Installing RAC On AIX
No ratings yet
Installing RAC On AIX
80 pages
Qatar Courses Sep 2021
No ratings yet
Qatar Courses Sep 2021
1 page
Making The Move From Oracle Warehouse Builder To Oracle Data Integrator 12c
No ratings yet
Making The Move From Oracle Warehouse Builder To Oracle Data Integrator 12c
34 pages
OIC PAAS Developer Dumps
No ratings yet
OIC PAAS Developer Dumps
28 pages
Upgr Ora UX Guide 19c PDF
No ratings yet
Upgr Ora UX Guide 19c PDF
30 pages
Oracle Interview Questions:: Company: Interview Type: ZOOM Meeting Date: Interview Time: 45 Hours
No ratings yet
Oracle Interview Questions:: Company: Interview Type: ZOOM Meeting Date: Interview Time: 45 Hours
3 pages
Oracle 19c New Features
100% (1)
Oracle 19c New Features
67 pages
How To Create A New Wallet and Add A Signed Certficate Using Orapki
No ratings yet
How To Create A New Wallet and Add A Signed Certficate Using Orapki
2 pages
Oracle OVM Step by Step
No ratings yet
Oracle OVM Step by Step
29 pages
Oracle BPM 12c: Essentials: Activity Guide D84807GC10 Edition 1.0 - February 2015 - D89958
No ratings yet
Oracle BPM 12c: Essentials: Activity Guide D84807GC10 Edition 1.0 - February 2015 - D89958
270 pages
OEM 12c Day 5 2014
No ratings yet
OEM 12c Day 5 2014
123 pages
D97103GC10 Ag
No ratings yet
D97103GC10 Ag
394 pages
Oracle Database 19c RAC Installation On OL7 Part 1
No ratings yet
Oracle Database 19c RAC Installation On OL7 Part 1
120 pages
Oracle 19c DataGuard Step by Step
No ratings yet
Oracle 19c DataGuard Step by Step
6 pages
Practice 24 - PDB Cloning and Relocation Using DBCA
100% (1)
Practice 24 - PDB Cloning and Relocation Using DBCA
11 pages
OPSS Configuration
No ratings yet
OPSS Configuration
18 pages
Oracle Multitenant 19c - All About Pluggable D
0% (1)
Oracle Multitenant 19c - All About Pluggable D
67 pages
Aljaz+Mali+-+Secure+your+APEX+application Final HROUG
No ratings yet
Aljaz+Mali+-+Secure+your+APEX+application Final HROUG
148 pages
Oracle 19c RAC Step by Step Part 0
No ratings yet
Oracle 19c RAC Step by Step Part 0
8 pages
Oracle Cloud Infrastructure 2021
No ratings yet
Oracle Cloud Infrastructure 2021
482 pages
Oracle Apps R12 Architecture
No ratings yet
Oracle Apps R12 Architecture
26 pages
Oracle Exadata Complete Self-Assessment Guide
From Everand
Oracle Exadata Complete Self-Assessment Guide
Gerardus Blokdyk
No ratings yet
Oracle WebLogic Server Second Edition
From Everand
Oracle WebLogic Server Second Edition
Gerardus Blokdyk
No ratings yet
ODI Administration and Development Outline
No ratings yet
ODI Administration and Development Outline
7 pages
D48459GC40_sg3
No ratings yet
D48459GC40_sg3
346 pages
Oracle Data Integrator (ODI) Online Training and Support
No ratings yet
Oracle Data Integrator (ODI) Online Training and Support
9 pages
Oracle Data Integrator
No ratings yet
Oracle Data Integrator
4 pages
Timur Akhmadeev - Patching With Ansible - ConTech 2022
No ratings yet
Timur Akhmadeev - Patching With Ansible - ConTech 2022
150 pages
Roel Hartman - Enemy of The Session State - Compressed
No ratings yet
Roel Hartman - Enemy of The Session State - Compressed
10 pages
Dani - Schnider-Andrea - Kennel-Brewtracker Behind The Scenes of A Brewery App
No ratings yet
Dani - Schnider-Andrea - Kennel-Brewtracker Behind The Scenes of A Brewery App
40 pages
CHAPTER 8. Display Data From Multiple Tables
No ratings yet
CHAPTER 8. Display Data From Multiple Tables
6 pages
Leadership Challenge 2021 Workbook-Final
No ratings yet
Leadership Challenge 2021 Workbook-Final
25 pages
CHAPTER 4. Retrieving Data Using The SQL Select Statement
No ratings yet
CHAPTER 4. Retrieving Data Using The SQL Select Statement
4 pages
Cert4Prep 1Z0-448 Certification Sample Questions
No ratings yet
Cert4Prep 1Z0-448 Certification Sample Questions
7 pages
No Hard Feelings The Secret Power Of Embracing Emotions At Work Liz Fosslien download
100% (1)
No Hard Feelings The Secret Power Of Embracing Emotions At Work Liz Fosslien download
34 pages
Application For Admission Degree & Diploma: Program of Study
No ratings yet
Application For Admission Degree & Diploma: Program of Study
11 pages
RET MANUAL _front _Index Page..docx (2)
No ratings yet
RET MANUAL _front _Index Page..docx (2)
19 pages
Fundamentals of Reading Academic Texts
No ratings yet
Fundamentals of Reading Academic Texts
2 pages
Ils Notes 2021
No ratings yet
Ils Notes 2021
26 pages
State & Sovereignty Chapter
No ratings yet
State & Sovereignty Chapter
25 pages
Literature Review On Student Management System
100% (1)
Literature Review On Student Management System
8 pages
Discontinuous Learning
No ratings yet
Discontinuous Learning
232 pages
Austin Kleber Danielle Ohl ENGL297 Ethnography Final Draft
No ratings yet
Austin Kleber Danielle Ohl ENGL297 Ethnography Final Draft
8 pages
Chapter I
No ratings yet
Chapter I
12 pages
Ch4 - Learning and Transfer of Training
No ratings yet
Ch4 - Learning and Transfer of Training
58 pages
(Edwin Diller Starbuck) The Psychology of Religion
100% (1)
(Edwin Diller Starbuck) The Psychology of Religion
452 pages
GNED 6 Science Technology and Society
No ratings yet
GNED 6 Science Technology and Society
3 pages
Immediate Download The Philosophy of Elizabeth Anscombe 4th Edition Roger Teichmann Ebooks 2024
100% (17)
Immediate Download The Philosophy of Elizabeth Anscombe 4th Edition Roger Teichmann Ebooks 2024
84 pages
Intended Learning Outcome
No ratings yet
Intended Learning Outcome
57 pages
Philosophy of Technology
No ratings yet
Philosophy of Technology
17 pages
BS Nursing 2020
No ratings yet
BS Nursing 2020
16 pages
Contoh Personal Statement
100% (1)
Contoh Personal Statement
1 page
Module 1 - EDUC 110
100% (5)
Module 1 - EDUC 110
10 pages
Quality Analyst Job Description
No ratings yet
Quality Analyst Job Description
8 pages
NBA Handbook Tier 1
No ratings yet
NBA Handbook Tier 1
118 pages
Cooper - Recycling Modernity - Waste and Environmental History - 2010
No ratings yet
Cooper - Recycling Modernity - Waste and Environmental History - 2010
12 pages
Group Facilitatio N and Counselin G
No ratings yet
Group Facilitatio N and Counselin G
42 pages
Instant ebooks textbook Transparent Design in Higher Education Teaching and Leadership A Guide to Implementing the Transparency Framework Institution Wide to Improve Learning and Retention 1st Edition Mary-Ann Winkelmes download all chapters
100% (4)
Instant ebooks textbook Transparent Design in Higher Education Teaching and Leadership A Guide to Implementing the Transparency Framework Institution Wide to Improve Learning and Retention 1st Edition Mary-Ann Winkelmes download all chapters
55 pages
Many Dimensions (C.H. Hinton)
No ratings yet
Many Dimensions (C.H. Hinton)
11 pages
Assignment in ALS
No ratings yet
Assignment in ALS
3 pages
ICT SYLLABUS Compressed PDF
No ratings yet
ICT SYLLABUS Compressed PDF
40 pages
23-JI - Critical Thinking, Intelligence, & Unsubstantiated Beliefs - An Integrative Review
No ratings yet
23-JI - Critical Thinking, Intelligence, & Unsubstantiated Beliefs - An Integrative Review
15 pages