Oracle Golden Gate
Oracle Golden Gate
Real-Time Replication
Fred Louis
Oracle Enterprise Architect
Ohio Valley Region
Agenda
Technology Overview
Architecture Deep Dive: Differentiators
Component Architecture
Performance & Scalability
Transaction Integrity & Reliability
Heterogeneity
Other Key Features
Instantiation
SQL Interface
Event Marker Infrastructure
ETL Integration
3
GoldenGate: A Strategic Acquisition
Best-in-class leader in real-time data movement
• Complements existing Oracle products
• Data replication for heterogeneous
databases
• Real-time information feeds to Oracle
BI/EPM
• Zero-downtime migrations and upgrades
to Oracle Database and Applications
• Over 400 customers with 4,000+
implementations across Fortune 500
companies:
• Top 3 of 5 largest commercial banks
• Top 7 of 10 financial data services
companies
• Top 4 of 5 telecommunications providers
• Top 3 of 5 largest food & drug stores
4
Oracle GoldenGate Technology Differentiators
Key Differentiators:
5
5
Oracle GoldenGate Solutions
Enterprise-wide Solution for Real Time Data Needs
• Standardize
Disaster
on Single
Recovery, Data Standby Technology for Multiple
Protection (Open & Active)
Requirements
Zero Downtime
Migration and
• Deploy for Continuous
Upgrades Availability and Real-time
Log Based, Real- Data Access for Reporting /
Time Change Data
Capture
Operational
Reporting Reporting BI
Database
OGG
ETL
ODS EDW
ETL
Heterogeneous
Source Systems
Real-time BI EDW • Highly Flexible
Query • Fast Deployments
Offloading
• Lower TCO & Improved ROI
Data
Distribution
6
Agenda
Technology Overview
Architecture Deep Dive: Differentiators
Component Architecture
Performance & Scalability
Transaction Integrity & Reliability
Heterogeneity
Other Key Features
Instantiation
SQL Interface
Event Marker Infrastructure
ETL Integration
7
Oracle GoldenGate
Simple, Flexible Instantiation
Transaction Logs
LAN / WAN /
Internet
(TCP/IP)
Capture Trail Collector Trail Deliver
Pump
File File
Online Backups
GG Initial Load
3rd party Tools
How GoldenGate Works: Modular “Building Blocks”
Capture
Source Trail
LAN / WAN / Target Trail
Deliver
Internet
Source Target
Database Database
9
How GoldenGate Works: Modular “Building Blocks”
Capture
Source Trail
LAN / WAN / Target Trail
Deliver
Internet
Source Target
Database Database
10
How GoldenGate Works: Modular “Building Blocks”
Capture
Source Trail
LAN / WAN / Target Trail
Deliver
Internet
Target Trail
Source Deliver
Source Trail Target
Capture
Database Database
Bi-directional
11
Oracle GoldenGate Heterogeneity Supports
Applications Running On…
Databases O/S and Platforms
Capture: Windows 2000, 2003, XP
Oracle
Linux
DB2
MicrosoftSQL Server Sun Solaris
Sybase ASE HP NonStop
Teradata
Enscribe
HP-UX
SQL/MP HP TRU64
SQL/MX
HP OpenVMS
Delivery: IBM AIX
All listed above, plus: IBM z/OS
MySQL, TimesTen, Netezza, Greenplum, HP
Neoview and any ODBC compatible databases
ETL products
JMS message queues
12
Oracle GoldenGate Topologies
Broadcast
Integration/Consolidation Cascading
Data Distribution Data Warehouse Data Marts
13
Agenda
Technology Overview
Architecture Deep Dive: Differentiators
Component Architecture
Performance & Scalability
Transaction Integrity & Reliability
Heterogeneity
Other Key Features
Instantiation
SQL Interface
Event Marker Infrastructure
ETL Integration
14
Architecture Components (Processes)
CAPTURE DELIVERY
Log-based Native database API
VAM-based ODBC
Local/Remote Queuing Batched Operations
Filtering Filtering
Parallel Coordination Transaction/Operations Grouping
Local/Remote* Parallel Coordination
Local/Remote*
PUMP MANAGER
Filtering Queue (Trail) Management
Local/Remote Process Management (Dynamic)
TCP/IP Monitoring and Administration (Lag reports)
Compression Reporting (Events, Errors, Thresholds)
Encryption Receive/Route requests from UI
On-disk Components
Trail Files
Checkpoint Files (Capture, Pump, Delivery)
Data Definition Files (SOURCEDEFS/TARGETDEFS)
Configuration Files
Discard Files
Report/Log Files
Binaries
15
Oracle GoldenGate Components
Communications and Startup
GGSCI GGSCI
Interactive Manager Manager Interactive
Interface Interface
Network
Source Target
TCP/IP
Process Start
Shared Memory (for Monitoring)
16
Oracle GoldenGate Components
Communications and Startup
GGSCI GGSCI
Interactive Manager Manager Interactive
Interface Interface
Network
Source Target
TCP/IP
Process Start
Shared Memory (for Monitoring)
17
Oracle GoldenGate Components
Communications and Startup
GGSCI GGSCI
Interactive Manager Manager Interactive
Interface Interface
Network
Source Target
TCP/IP
Process Start
Shared Memory (for Monitoring)
18
Oracle GoldenGate Components
Communications and Startup
GGSCI GGSCI
Interactive Manager Manager Interactive
Interface Interface
Network
Source Target
TCP/IP
Process Start
Shared Memory (for Monitoring)
19
Oracle GoldenGate Components
Communications and Startup
GGSCI GGSCI
Interactive Manager Manager Interactive
Interface Interface
Network
Source Target
TCP/IP
Process Start
Shared Memory (for Monitoring)
20
Oracle GoldenGate Capture
GoldenGate GoldenGate
Event Log
Parameter Messaging Subsystem
File
Checkpoint Manager
Parser
Memory Manager
GoldenGate
nsaction Checkpoint
Logs File
VAM
API/Redo
Transaction API Transformation Serialization
Formatting Filtering
Log Reader and Mapping and Routing
(log reader
interface)
GoldenGate
SQL
Trail Files
Metadata Metadata Transformation
Manager Manager Interface Engine
Capture GoldenGate
Source
Command
Database Interpreter
21
Oracle GoldenGate Delivery
GoldenGate
GoldenGate
Event Log
Parameter Messaging Subsystem
File
Checkpoint Manager
Parser
Memory Manager
GoldenGate
Checkpoint
File
GoldenGate SQL
Transformation
Trail Read API Formatting Filtering
and Mapping Generation
Reader
GoldenGate
Trail Files
Metadata SQL Transformation
Manager Interface Engine
Command
and
Control Database Access
22
Agenda
Technology Overview
Architecture Deep Dive: Differentiators
Component Architecture
Performance & Scalability
Transaction Integrity & Reliability
Heterogeneity
Other Key Features
Instantiation
SQL Interface
Event Marker Infrastructure
ETL Integration
23
Differentiator: Performance and Scalability
GoldenGate Capture
24
Differentiator: Performance and Scalability
GoldenGate Delivery
Decoupled architecture
Multiple delivery processes may be used to scale
Possible to split “hot” tables into a separate delivery process
Possible for multiple delivery processes to split the work for a
single table
Transaction grouping
Small transactions are grouped by default to reduce commit
overhead
Record batching
Records are batched by table and operation within a single
SQL execution
Automatic reordering of batches
25
Agenda
Technology Overview
Architecture Deep Dive: Differentiators
Component Architecture
Performance & Scalability
Transaction Integrity & Reliability
Heterogeneity
Other Key Features
Instantiation
SQL Interface
Event Marker Infrastructure
ETL Integration
26
Differentiator: Transaction Integrity
Transaction boundaries
Recorded in GoldenGate trail file
Transactions ordered in commit sequence
Boundaries adjusted automatically due to
record filtering or trail splits
Transaction integrity
Checkpointing and recovery are based on
transaction boundaries
Original commit sequence is maintained
by the delivery process
27
Differentiator: Reliability
Decoupled architecture
Individual processes can be restarted automatically
Tolerance to network outages (configurable)
Recovery
Recovery ensures that no operations are skipped or
duplicated after failure of any kind
Recovery of the Capture process is more involved than
recovery the Delivery process
Although GoldenGate processes are completely decoupled,
the Delivery process must be aware of a recovery performed
by the Capture process or pump
28
GoldenGate Checkpointing
Capture, Pump, and Delivery save positions to a checkpoint file so they can
recover in case of failure
Start of Oldest Open (Uncommitted)
Begin, TX 1 Transaction
Insert, TX 1
Begin, TX 2 Begin, TX 2
Update, TX 1 Insert, TX 2
Insert, TX 2 Commit, TX 2
Commit, TX 2
Capture
Begin, TX 3
Checkpoint
Begin, TX 3 Insert, TX 3
Insert, TX 3 Commit, TX 3
Current
Begin, TX 4 Write
Commit, TX 3
Position
Delete, TX 4
Current Read
Position
Commit Ordered
Capture Source Trail
Source
Database
29
GoldenGate Checkpointing
Capture, Pump, and Delivery save positions to a checkpoint file so they can
recover in case of failure
Start of Oldest Open (Uncommitted)
Begin, TX 1 Transaction
Insert, TX 1
Begin, TX 2 Begin, TX 2
Pump
Begin, TX 2
Checkpoint
Update, TX 1 Insert, TX 2 Insert, TX 2
Insert, TX 2 Commit, TX 2 Commit, TX 2
Capture Current Current
Commit, TX 2 Begin, TX 3
Checkpoint Read Write
Begin, TX 3 Position
Insert, TX 3 Position
Insert, TX 3 Commit, TX 3
Current
Begin, TX 4 Write
Commit, TX 3
Position
Delete, TX 4
Current Read
Position
30
GoldenGate Checkpointing
Capture, Pump, and Delivery save positions to a checkpoint file so they can
recover in case of failure
Start of Oldest Open (Uncommitted)
Begin, TX 1 Transaction
Insert, TX 1
Begin, TX 2 Pump Delivery
Begin, TX 2 Begin, TX 2
Checkpoint Checkpoint
Update, TX 1 Insert, TX 2 Insert, TX 2
Insert, TX 2 Commit, TX 2 Commit, TX 2
Capture Current Current Current
Commit, TX 2 Begin, TX 3
Checkpoint Read Write Read
Begin, TX 3 Position
Insert, TX 3 Position Position
Insert, TX 3 Commit, TX 3
Current
Begin, TX 4 Write
Commit, TX 3 Position
Delete, TX 4
Current Read
Position
31
Agenda
Technology Overview
Architecture Deep Dive: Differentiators
Component Architecture
Performance & Scalability
Transaction Integrity & Reliability
Heterogeneity
Other Key Features
Instantiation
SQL Interface
Event Marker Infrastructure
ETL Integration
32
Differentiator: Heterogeneity
Core Infrastructure
Core product components are utilized for all databases
Canonical trail file format
Canonical metadata definitions
Automatic mapping of common datatypes
Capture Methods
Log scraping (direct file access)
Database built-in log interface (transaction log API)
Intercepts/exits
Vendor-provided access modules
Delivery utilizes native database interfaces
33
Differentiator: Heterogeneity
Oracle Capture
34
Differentiator: Heterogeneity
Microsoft SQL Server Capture
35
Differentiator: Heterogeneity
DB2 LUW Capture
Commit Ordered
Capture
Trail
DB2 CLI
36
Differentiator: Heterogeneity
DB2 z/OS Capture
37
Differentiator: Heterogeneity
HP NonStop Capture (Enscribe and SQL/MP)
38
Differentiator: Heterogeneity
HP NonStop Capture (SQL/MX)
39
Differentiator: Heterogeneity
Teradata Capture
40
Differentiator: Heterogeneity
Delivery
Commit Ordered
Flat file integration for data
Delivery
Trail
Target Database warehousing appliances (Netezza
and Greenplum)
41
Agenda
Technology Overview
Architecture Deep Dive: Differentiators
Performance & Scalability
Transaction Integrity
Reliability
Heterogeneity
Other Key Features
Instantiation
SQL Interface
Event Marker Infrastructure
ETL Integration
42
GoldenGate Instantiation
Start change data capture on Source Database
Current source Commit Sequence Number (CSN) is 222
Current CSN is
222 CSNs 222
through …
43
GoldenGate Instantiation
Wait for any open transactions to close
Use a backup (or export/import) to copy source to target as of a consistency point
Current source Commit Sequence Number (CSN) is 245
Consistent as
Current CSN is of CSN 245
245 CSNs 222
through …
44
GoldenGate Instantiation
Start delivery at or after given backup CSN (245 in this case)
Current source Commit Sequence Number (CSN) is 356
Once Delivery is current the source and target are in sync
Consistent as
Current CSN is Deliver
of CSN 356
356 CSNs 222 transactions In Sync!
through 356 after CSN 245
45
Agenda
Technology Overview
Architecture Deep Dive: Differentiators
Performance & Scalability
Transaction Integrity
Reliability
Heterogeneity
Other Key Features
Instantiation
SQL Interface
Event Marker Infrastructure
ETL Integration
46
Other Key Features
SQL Interface
Standalone execution
Add parameter to the root level of a parameter file
Executes independently of a table or map statement
Execution order is based on the order it appears in the parameter file
Execute once
SQLEXEC “call prc_job_count ()”
SQLEXEC “truncate table scott.target_table”
SQLEXEC “SET TRIGGERS OFF”
47
Other Key Features
SQL Interface
Inline execution
Add a clause to the table/map statement in the parameter file
Executes for records that meet the map condition
The procedure or query can accept input parameters and return
output parameters
Executed within the target database transaction context
Execute once (always execute by default)
MAP sales.cust, TARGET sales.cust_extended, &
SQLEXEC (SPNAME lookup, PARAMS &
(long_name = birth_state),EXEC ONCE), &
COLMAP (custid = custid, birth_state_long = lookup.long_name);
48
Agenda
Technology Overview
Architecture Deep Dive: Differentiators
Performance & Scalability
Transaction Integrity
Reliability
Heterogeneity
Other Key Features
Instantiation
SQL Interface
Event Marker Infrastructure
ETL Integration
49
Other Key Features
Event Marker Infrastructure
EVENT EVENT
PROCESSING PROCESSING
50
Other Key Features
Event Marker Infrastructure
51
Agenda
Technology Overview
Architecture Deep Dive: Differentiators
Performance & Scalability
Transaction Integrity
Reliability
Heterogeneity
Other Key Features
Instantiation
SQL Interface
Event Marker Infrastructure
ETL Integration
52
Other Key Features
ETL Integration
Integration via:
Staging tables ETL
Flat files Integration via Staging Tables
Messaging Delivery
OLTP
Integration via Messaging
Delivery
ETL
JMS / MoM
Queue or
Topic
53
Other Key Features
Delivery to JMS (Java Message System)
XML
XML
XML Third Party
source JMS
tx Data
JMS Systems
txlogs Partners
defs
txlogs Pump UE Queue or Topic
logs (meta Customers
data) Applications
etc
54
Other Key Features
Delivery to Flat File
Time/size-based rollover
One file, or one table
per file
55
Sun/Oracle Information
http://tinyurl.com/sunorcl
or
oracle.com
Oracle GoldenGate
Other Oracle Products..
http://oracle.com/goldengate
Customer Case
Studies
Customer Case Study
Real-time Data for Better BI and Reduce Customer Churn
66
Continuous Availability Use Case
Siebel Application Upgrade & Database Migration
67