Oracle9i Change Data Capture

Dan White Manager Technology Solutions Group Innovative Consulting Inc.

intelligent solutions for your business
Innovative Consulting, Inc., 2001

ICI Fast Facts

Serving Clients Headquartered in the Greater Philadelphia, New Jersey, and Delaware Area
Headquartered in Malvern, PA

Representative Clients Industries served

Technology 11% Finance & Insurance 17% American Institute for CPCU Aventis Campbell Soup CDNOW Online Centocor Cigna Colorcon Comcast Delaware Valley Financial Services Fleet Genesis Health Ventures GlaxoSmithKline IKEA IMS Health Intracorp MeDecision Primavera Systems Reliance Standard Life Siemens (Health Services) Towers Perrin US Interactive Syngenta

Pharmaceutical, Life Sciences 53%

Manufacturing, Retail 19%

Partnerships and Alliances

arcplan

Innovative Consulting, Inc., 2001

ICI At OracleWorld
Dan White - Oracle 9i Change data Capture- A technical Overview
Monday Nov 11, 2002 1:00pm 2:00pm

Dwight DeVera - Delivering Complete Business Intelligence With Oracle9iAS Portal

Wed Nov 13, 2002 1:00pm 2:00pm

Dwight DeVera - Utilizing Oracle9iAS Portal to deploy Internet Facing Applications

Thurs Nov 14, 2002 4:00pm 5:00pm

Innovative Consulting, Inc., 2001

Agenda
What is Change Data Capture? Traditional Data Capture Methods. CDC Architecture. Functional Overview of the DBMS_CDC packages. CDC Publish Methods. CDC Subscribe Methods. Conclusions.

Innovative Consulting, Inc., 2001

What is Change Data Capture

intelligent solutions for your business

Innovative Consulting, Inc., 2001

What is Change Data Capture?

Way of extracting delta changes from a source system Can be used to optimize the extraction process Not only in data warehousing

Innovative Consulting, Inc., 2001

Traditional Data Capture

intelligent solutions for your business

Innovative Consulting, Inc., 2001

Traditional Data Capture Methods

Extract all data to flat files
Cumbersome Prone to formatting errors Large OS files

Triggers
Introduce a point of failure Have to maintain multiple objects on other systems
Triggers Capture Tables Database Links. Custom code

Innovative Consulting, Inc., 2001

Traditional Data Capture Methods

Merge Purge Method
Extract entire table See if record exists
If yes, did it change? If no insert it!!

You have to extract very large amounts of data every time. Very complex compare code

Innovative Consulting, Inc., 2001

CDC Architecture

intelligent solutions for your business

Innovative Consulting, Inc., 2001

CDC Architecture
Source System
Source Databases Redo Logs PDML triggers Change Tables Subscription Views

Data Warehouse Staging DW

Innovative Consulting, Inc., 2001

Change Data Capture Views

Change_sources Change_sets Change_tables Dba_source_tables Dba_published_columns Dba_subscriptions
Innovative Consulting, Inc., 2001

Functional Overview

intelligent solutions for your business

Innovative Consulting, Inc., 2001

Functional Flow: Publish

CREATE CHANGE TABLE drop subscriber view

Yes

Does the change table need to be altered?

No

drop subscription

Alter change table Drop change table

Innovative Consulting, Inc., 2001

Functional Flow: Subscription

Innovative Consulting, Inc., 2001

DBA
Typically uses the DBMS_CDC_PUBLISH package Creates change tables Maintains change tables
Using alter change table **Note** The change table is a physical object so it must be maintained by the DBA or development staff

Innovative Consulting, Inc., 2001

ETL developers
Uses DBMS_CDC_SUBSCRIBE Sets up and maintains subscriptions Sets up and maintains subscription views Maintains the subscription windows
Innovative Consulting, Inc., 2001

DBMS_CDC_PUBLISH
Methods

intelligent solutions for your business

Innovative Consulting, Inc., 2001

DBMS_CDC_PUBLISH
Create_change_table Alter_change_table Drop_change_table Drop_subscriber_view Drop_Subscription Purge

Innovative Consulting, Inc., 2001

Creating a change table

A change table is a database object that contains change data that resulted from DML statements made on the source table. A change table is a database table that maintains data in two types of columns.
Source columns identify data from the source table to capture. Control columns contain special metadata.

The Publisher can control the change tables physical properties. Do not attempt to control a change tables partitioning properties. CDC Manages partitions.

Innovative Consulting, Inc., 2001

Create_change_Table
DBMS_CDC_PUBLISH.CREATE_CHANGE_TABLE (OWNER => CDC, CHANGE_TABLE_NAME => SALES_CDC', CHANGE_SET_NAME => 'SYNC_SET', SOURCE_SCHEMA => SRC1', SOURCE_TABLE => SALES, COLUMN_TYPE_LIST =. prod_id NUMBER(6), cust_id NUMBER, time_id DATE, channel_id CHAR(1), promo_id NUMBER(6), quantity_sold NUNBER(3), amount_sold NUMBER(10,2) ', CAPTURE_VALUES => BOTH', RS_ID => N' , ROW_ID => N', USER_ID => Y', TIMESTAMP => Y', OBJECT_ID => N', SOURCE_COLMAP => N', TARGET_COLMAP => N', OPTIONS_STRING => TABLESPACE DW STORAGE INITIAL 50M NEXT 12M);

Innovative Consulting, Inc., 2001

Change tables
SQL> desc SALES_CDC Name Null? Type ----------------------------------------- -------- -----------OPERATION$ CHAR(2) CSCN$ NUMBER COMMIT_TIMESTAMP$ DATE USERNAME$ VARCHAR2(30) TIMESTAMP$ DATE SOURCE_COLMAP$ RAW(128) TARGET_COLMAP$ RAW(128) PROD_ID NUMBER(6) CUST_ID NUMBER TIME_ID DATE CHANNEL_ID CHAR(1) PROMO_ID NUMBER(6) QUANTITY_SOLD NUMBER(3) AMOUNT_SOLD NUMBER(10,2) SQL>
Innovative Consulting, Inc., 2001

Alter_change_table
Allows you to add or drop change columns Allows you to add or drop metadata columns You can not add and drop user columns in the same call, these schema changes require separate calls Do not specify the name of the control columns in the user column lists

Innovative Consulting, Inc., 2001

Alter Change Table

DBMS_CDC_PUBLISH.ALTER_CHANGE_TABLE ( Owner=>CDC, change_table_name =>SALES_CDC , operation => DROP, column_list => channel_id CHAR(1), rs_id =>N row_id =>N user_id =>N Timestamp =>N object_id =>N source_colmap =>N target_colmap =>N);

Innovative Consulting, Inc., 2001

Drop Change Table

Used To drop a change table
DBMS_CDC_PUBLISH.DROP_CHANGE_TABLE(Owner=>CDC', Change_table_name =>'SALES_CDC', Force=>N');

*** NOTE *** Force flag = Y will drop a change table even if there are subscriptions referencing it

Innovative Consulting, Inc., 2001

Drop Subscriber view

This procedure allows a publisher to drop a subscriber view in the subscriber's schema. Same as

DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIBER_VIEW

DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIBER_VIEW( SUBSCRIPTION_HANDLE =>:subhandle, SOURCE_SCHEMA =>SRC1, SOURCE_TABLE => SALES);

Innovative Consulting, Inc., 2001

Drop subscription
This procedure will drop a subscription Same as DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIPTION Why Can we do this from the publish?
DBMS_CDC_PUBLISH.DROP_SUBSCRIPTION ( SUBSCRIPTION_HANDLE => :subhandle);

Innovative Consulting, Inc., 2001

Purge
This procedure will purge data from change tables Automatically monitors subscriptions You can run this procedure in 2 ways Manual at command line Run in a script to proactively maintain change tables Again do not attempt to control a change tables partition properties DBMS_CDC_PUBLISH.PURGE

Innovative Consulting, Inc., 2001

WAKE UP !!!!!!!!!!!

Question for putty ! Who is this?

(and dont say Dan at the bar after the presentation)

Innovative Consulting, Inc., 2001

Question for Putty

Q: Can YOU manage a change tables partitions? A: NO, The CDC system manages partitions by it self!

Innovative Consulting, Inc., 2001

DBMS_CDC_SUBSCRIBE
Methods

intelligent solutions for your business

Innovative Consulting, Inc., 2001

DBMS_CDC_SUBSCRIBE
GET_SUBSCRIPTION_HANDLE SUBSCRIBE ACTIVATE_SUBSCRIPTION EXTEND_WINDOW PREPARE_SUBSCRIBER_VIEW DROP_SUBSCRIBER_VIEW PURGE_WINDOW DROP_SUBSCRIPTION
Innovative Consulting, Inc., 2001

GET_SUBSCRIPTION_HANDLE
Creates a subscription handle This is the first step in obtaining a subscription Subscription handles:
Never get reused Are tracked from the time of creation Not shared by subscribers Validated against the users login id Can be used to access multiple change tables
Innovative Consulting, Inc., 2001

GET_SUBSCRIPTION_HANDLE

DBMS_CDC_SUBSCRIBE.GET_SUBSCRIPTION_HANDLE( CHANGE_SET=>'SYNC_SET', DESCRIPTION=>'Change data for SALES', SUBSCRIPTION_HANDLE=>:subhandle);

Innovative Consulting, Inc., 2001

SUBSCRIBE
Specifies source table for change data Specifies source columns for change data Subscribe procedure is overloaded
Version1: If you know the publication_id Version2: If you do not know the publication_id

Allows you to subscribe to one or more source tables and select columns in each table To see all published columns and tables query the ALL_PUBLISHED_COLUMNS view

Innovative Consulting, Inc., 2001

Subscribe
SQL> desc ALL_PUBLISHED_COLUMNS Name ----------------------------------------CHANGE_SET_NAME SOURCE_SCHEMA_NAME SOURCE_TABLE_NAME PUB_ID COLUMN_NAME DATA_TYPE DATA_LENGTH DATA_PRECISION DATA_SCALE NULLABLE SQL> Null? -------NOT NULL NOT NULL NOT NULL NOT NULL NOT NULL Type -------------VARCHAR2(30) VARCHAR2(30) VARCHAR2(30) NUMBER VARCHAR2(30) VARCHAR2(106) NOT NULL NUMBER NUMBER NUMBER VARCHAR2(1)

Innovative Consulting, Inc., 2001

Subscribe
Must be created before before you need the data You can subscribe to published columns only

Innovative Consulting, Inc., 2001

SUBSCRIBE
DBMS_CDC_SUBSCRIBE.SUBSCRIBE( SUBSCRIPTION_HANDLE=>:subhandle, SOURCE_SCHEMA=>SRC1, SOURCE_TABLE=>SALES, COLUMN_LIST=>prod_id ,cust_id, time_id, channel_id ,promo_id ,quantity_sold);

DBMS_CDC_SUBSCRIBE.SUBSCRIBE ( subscription_handle => :subhandle, publication_id =>31310, COLUMN_LIST=>prod_id ,cust_id, time_id, channel_id ,promo_id ,quantity_sold);

Innovative Consulting, Inc., 2001

ACTIVATE_SUBSCRIPTION
Will signify that we are ready to access data

DBMS_CDC_SUBSCRIBE.ACTIVATE_SUBSCRIPTION( SUBSCRIPTION_HANDLE=>:subhandle);

Innovative Consulting, Inc., 2001

EXTEND_WINDOW
Sets subscription window boundaries First call will establish boundaries Subsequent call will extend High Water Mark
DBMS_CDC_SUBSCRIBE.EXTEND_WINDOW( subscription_handle=>:subhandle);

Innovative Consulting, Inc., 2001

PREPARE_SUBSCRIBER_VIEW
Creates a subscriber view You can now access data defined within your window
DBMS_CDC_SUBSCRIBE.PREPARE_SUBSCRIBER_VIEW( SUBSCRIPTION_HANDLE =>:subhandle, SOURCE_SCHEMA =>SRC1', SOURCE_TABLE => SALES, VIEW_NAME => :viewname);

Innovative Consulting, Inc., 2001

PURGE_WINDOW
Used to purge the subscription window Sets low watermark to zero

DBMS_CDC_SUBSCRIBE.PURGE_WINDOW( SUBSCRIPTION_HANDLE=>:subhandle);

Innovative Consulting, Inc., 2001

DROP_SUBSCRIBER_VIEW
Same as
DBMS_CDC_PUBLISH.DROP_SUBSCRIBER_VIEW

Subscription must have been created by prepare_subscriber_view Must drop the subscriber view before dropping subscription
DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIBER_VIEW( SUBSCRIPTION_HANDLE =>:subhandle, SOURCE_SCHEMA =>SRC1', SOURCE_TABLE => SALES');
Innovative Consulting, Inc., 2001

DROP_SUBSCRIPTION
Drops a subscription Must use the drop subscriber view first Same as DBMS_CDC_PUBLISH.DROP_SUBSCRIPTION

DBMS_CDC_SUBSCRIBE.DROP_SUBSCRIPTION( SUBSCRIPTION_HANDLE => :subhandle);

Innovative Consulting, Inc., 2001

Question for Putty

Q:Can you drop a change table with active subscriptions? A: YES! if the force flag is set to Y Q:Is it a good idea? A:Not if you want to stay in the development teams good graces

Innovative Consulting, Inc., 2001

Conclusions

intelligent solutions for your business

Innovative Consulting, Inc., 2001

Thank you
[email protected]

intelligent solutions for your business

Innovative Consulting, Inc., 2001