Oracle Data Integrator

Oracle Data Integrator (ODI) provides a fully unified solution for building, deploying, and
managing complex data warehouses or as part of data- centric architectures in a SOA or
business intelligence environment.
In addition, it combines all the elements of data integration - data movement, data
synchronization, data quality, data management, and data services - to ensure that
information is timely, accurate, and consistent across complex systems.
Traditional ETL (Extract, Transform, Load) tools operate by, first, extracting the data from
various sources, then, transforming the data in a proprietary, middle-tier ETL engine that is
used as the staging area, and finally loading the transformed data into the target data
warehouse, integration server, or Hadoop cluster. Hence the term ETL represents both the
names and the order of the operations performed.
ODI extracts the data from the various sources, and then loads it to the target where it's
transformed. This is known as E-LT: Extract, Load, Transform.
ODI features an active integration platform that includes all styles of data integration: data-
based, event-based and service-based.
ODI transforms large volumes of data efficiently, processing events in real time through its
advanced Changed Data Capture (CDC) framework, and providing data services to the
Oracle SOA Suite. It also provides robust data integrity control features, assuring the
consistency and correctness of data. With powerful core differentiators - heterogeneous E-
LT, Declarative Design and Knowledge Modules - ODI meets the performance, flexibility,
productivity, modularity and hot-pluggability requirements of an integration platform.

Architecture
The Oracle Data Integrator architecture is organized around a modular repository, which is
accessed in client-server mode by components such as the ODI Studio and execution Agents
that are written entirely in Java. The architecture also incorporates web-based components
such as the Management Pack for Oracle Data Integrator which integrates with the Oracle
Enterprise Manager Cloud Control infrastructure or the ODI Console which enables users to
access metadata information through a Web interface.
Studio
ODI Studio is a Java based development environment based on the JDeveloper framework.
Studio is installed on client machines and used to connect to the master and work repositories
to access the ODI metadata and perform object development.

Oracle Data Integrator Navigators

ODI Studio provides four Navigators for managing the different aspects and steps of an ODI
integration project,
 1. Topology Navigator
2. Designer Navigator
3. Operator Navigator
4. Security Navigator

The Topology Navigator is used to manage the data describing the information system's
physical and logical architecture. Through it you can manage the topology of your
information system, the technologies and their data types, the data servers linked to these
technologies and the schemas they contain, the contexts, the language and the agents, as well
as the repositories. The site, machine, and data server descriptions enable Oracle Data
Integrator to execute the same mappings in different environments.

The Designer Navigator is used to design data integrity checks and to build transformations,
such as automatic reverse-engineering of existing applications or databases, graphical
development and maintenance of transformations and mappings, visualization of data flows
in the mappings, automatic documentation generation, and customization of the generated
code. The main objects handled through it are Models and Projects.

The Operator Navigator is the production management and monitoring tool. It's designed
for IT production operators. Through it you can manage your executions in the sessions, as
well as the scenarios in production.

The Security Navigator is the tool for managing the security information in ODI. Through it
you can create users and profiles and assign user rights for methods on generic objects and
fine-tune these rights on the object instances.

Data Server
This object is the connection to your data source. Created under one of the many technologies
available in ODI, this is where the JDBC url, username, password, and other properties are all
created and stored.

Physical Architecture
The physical architecture defines the different elements of the information system, as well as
their characteristics considered by Oracle Data Integrator.
Each type of database (Oracle, DB2, etc.),
Big Data source (Hive, HBase),
file format (XML, Flat File), or application software is represented in Oracle Data Integrator
by a technology.

A technology handles formatted data. Therefore, each technology is associated with one or
more data types that allow Oracle Data Integrator to generate data handling scripts.

The physical components that store and expose structured data are defined as data servers.

A data server is always linked to a single technology.

A data server stores information according to a specific technical logic which is declared
into physical schemas attached to this data server.

Every database server, JMS message file, group of flat files, and so forth, that is used in
Oracle Data Integrator, must be declared as a data server.

Every schema, database, JMS Topic, etc., used in Oracle Data Integrator, must be declared as
a physical schema.

Contexts
Contexts bring together components of the physical architecture (the real Architecture) of the
information system with components of the Oracle Data Integrator logical architecture (the
Architecture on which the user works).

For example, contexts may correspond to different execution environments

(Development, Test and Production) or different execution locations (India Site, New-York
Site, and so forth.) where similar physical resource exist.

Note that during installation the default GLOBAL context is created.

Logical Architecture
The logical architecture allows to identify as a single Logical Schema a group of similar
physical schemas (that contain datastores that are structurally identical) that are located in
different physical locations.

Logical Schemas, like their physical counterparts, are attached to a technology.

Contexts allow logical schemas to resolve to physical schemas.

In a given context, a logical schema resolves to a single physical schema.

Agents
Oracle Data Integrator run-time Agents orchestrate the execution of jobs. These agents are
Java components.

The run-time agent functions as a listener and a scheduler agent. The agent executes jobs on
demand (model reverses, packages, scenarios, mappings, and so forth),

for example, when the job is manually launched from a user interface or from a command
line. The agent is also used to start the execution of scenarios according to a schedule defined
in Oracle Data Integrator.

ODI Studio can also directly execute jobs on demand. This internal "agent" can be used for
development and initial testing. However, it does not have the full production features of
external agents and is therefore unsuitable for production data integration. When running a
job, in the Run dialog, select Local (No Agent) as the Logical Agent to directly execute the
job using ODI Studio.

Note the following features are not available when running a job locally:

 Stale session clean-up

 Ability to stop a running session.
 Load balancing

If you need any of these features, you should use an external agent.

Agent Types

Oracle Data Integrator agents are available with three types:

standalone agents,

standalone collocated agents,

and Java EE agents.

Model
Once you have the Topology setup (Data Server, Physical Schema, Logical Schema), you can
then create your Model. This is going to be where the logical Datastores are grouped for a
given schema. There are many other functions of the Model object, such as journalizing
(CDC) setup, but we’ll save those features for another day.

Datastore
The Datastore is a logical representation of a table, file, XML element, or other physical
object. Stored in the form of a table, the Datastore has columns and constraints. This is the
object that will be used as a source or target in your ODI Mappings.

Repositories:

The Oracle Data Integrator (ODI) Repository is composed of a master repository and at least
one work repository. Objects developed or configured through the users are stored in one of
these repository types. It's installed on an OLTP relational database.

There's usually only one master repository that stores the Security information, Topology
information, and versioned and archived objects.

The work repository is the one that contains actual developed objects. Several work
repositories may coexist in the same ODI installation (for example, to have separate
environments or to match a particular versioning life cycle). A work repository stores
information for Models, Projects, and Scenario execution. When the work repository contains
only the execution information (typically for production purposes), it's called an execution
repository.

•Models (i.e. metadata)—including datastores, columns, data integrity constraints, cross

references, data lineage and impact analysis

• Projects—including mappings, packages, procedures, folders, knowledge modules, and

variables

• Runtime information—including scenarios, load plans, scheduling information, and

execution logs

Other Features

Oracle Data Integrator also includes the following components and features:

• Knowledge Modules make it possible to quickly and easily integrate technologies,

databases, and applications. They exist for a large range of platforms, including Oracle,
Exadata, Big Data, Oracle Applications, Oracle GoldenGate, Teradata, Netezza, SAP ERP,
SAP BW and XML.
 • The Changed Data Capture (CDC) framework tracks changes in source systems and
reduces the volume of processed data by extracting only the changed data. ODI can use
native database features to capture changes or Oracle GoldenGate.

• The integration with Oracle Business Intelligence Enterprise Edition provides business
users with report-to-source data lineage.

• Advanced Web Services capabilities offer best-in-class integration with Service-Oriented

Architecture (SOA) environments

• The Load Balancing feature enables large volumes of data to be processed by

automatically balancing the workload between several Agents.

• Advanced Version Management provides an interface to manage, safeguard, and replicate

revisions of units of work, even in the largest development environments.