100% found this document useful (1 vote)

171 views24 pages

Incremental Load

Incremental load is used to load only new or changed records from a database. It can handle: 1) Append only data sources by loading new records added to log files 2) Insert only databases by selecting records added since the last load time 3) Insert and update changes by selecting new and updated records while excluding existing keys 4) Insert, update and delete but it is slow or requires a deletion flag, making it more complex to implement. The document provides SQL scripts for implementing various incremental load scenarios and discusses handling the last load time and error handling.

Uploaded by

samuari49

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PPT, PDF, TXT or read online on Scribd

100% found this document useful (1 vote)

171 views24 pages

Incremental Load

Uploaded by

samuari49

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PPT, PDF, TXT or read online on Scribd

You are on page 1/ 24

Incremental Load

using qvd files

Incremental Load
Is sometimes called
Incremental Load Differential Load Delta Load

Incremental Load
Goal: Load only the new or the changed records from the database. The rest should already be available, one way or another.

Comments on Buffer Load

Buffer (Incremental) Load is a solution only for Log files (text files), but not for DBs. Buffer (Stale after 7 days) Select is not a good solution. It makes a full Load after 7 days. And nothing in between

Incremental Load
Load new data from Database table (slow, but few records) Load old data from QVD file (many records, but fast) Create new QVD file Procedure must be repeated for each table

Different DB-changes
If source allows
1) 2) 3) 4) Append only. (Logfiles) Insert only. (No Update or Delete) Insert and Update. (No Delete) Insert, Update and Delete.

1) Append only
Must be Log file Loads records added in the end of the file

1) Append only

Buffer (Incremental) Load * From LogFile.txt (ansi, txt, delimiter is '\t', embedded labels);

Done!
But it should be renamed to Buffer (Append) Load

2) Insert only
Can be any DB Loads INSERTed records Needs the field ModificationDate

2) Insert only

QV_Table: SQL SELECT PrimaryKey, X, Y FROM DB_TABLE WHERE ModificationTime >= #$(LastExecTime)#;

2) Insert only

QV_Table: SQL SELECT PrimaryKey, X, Y FROM DB_TABLE WHERE ModificationTime >= #$(LastExecTime)#; Concatenate LOAD PrimaryKey, X, Y FROM File.QVD;

2) Insert only

QV_Table: SQL SELECT PrimaryKey, X, Y FROM DB_TABLE WHERE ModificationTime >= #$(LastExecTime)#; Concatenate LOAD PrimaryKey, X, Y FROM File.QVD;

STORE QV_Table INTO File.QVD;

Almost done
But there is a small chance that a record gets loaded twice

2) Insert only

QV_Table: SQL SELECT PrimaryKey, X, Y FROM DB_TABLE WHERE ModificationTime >= #$(LastExecTime)# AND ModificationTime < #$(BeginningThisExecTime)#; Concatenate LOAD PrimaryKey, X, Y FROM File.QVD; STORE QV_Table INTO File.QVD;

Done!

3) Insert and Update

Can be any DB Loads INSERTed and UPDATEd records Needs the fields ModificationDate and PrimaryKey

3) Insert and Update

QV_Table: SQL SELECT PrimaryKey, X, Y FROM DB_TABLE WHERE ModificationTime >= #$(LastExecTime)#; Concatenate LOAD PrimaryKey, X, Y FROM File.QVD WHERE NOT Exists(PrimaryKey); STORE QV_Table INTO File.QVD;

Done!

4) Insert, Update and Delete

Can be any DB Loads INSERTed and UPDATEd records Removes DELETEd records Needs the fields ModificationDate and PrimaryKey Tricky to implement

4) Insert, Update and Delete

QV_Table: SQL SELECT PrimaryKey, X, Y FROM DB_TABLE WHERE ModificationTime >= #$(LastExecTime)#; Concatenate LOAD PrimaryKey, X, Y FROM File.QVD WHERE NOT EXISTS(PrimaryKey); Inner Join SQL SELECT PrimaryKey FROM DB_TABLE; STORE QV_Table INTO File.QVD;

OK, but slow

4) Insert, Update and Delete

ListOfDeletedEntries: SQL SELECT PrimaryKey AS Deleted FROM DB_TABLE WHERE DeletionFlag = 1 and ModificationTime >= #$(LastExecTime)#;
QV_Table: SQL SELECT PrimaryKey, X, Y FROM DB_TABLE WHERE ModificationTime >= #$(LastExecTime)#; Concatenate LOAD PrimaryKey, X, Y FROM File.QVD WHERE NOT Exists(PrimaryKey) AND NOT Exists(Deleted,PrimaryKey);

Drop Table ListOfDeletedEntries;

STORE QV_Table INTO File.QVD;

OK, but needs a DeletionFlag

LastExecutionTime & Error handling

Let ThisExecTime = Now(); { Load sequence } If ScriptErrorCount = 0 then Let LastExecTime = ThisExecTime; End If

Final Script
Let ThisExecTime = Now(); QV_Table: SQL SELECT PrimaryKey, X, Y FROM DB_TABLE WHERE ModificationTime >= #$(LastExecTime)# AND ModificationTime < #$(ThisExecTime)#; Concatenate LOAD PrimaryKey, X, Y FROM File.QVD WHERE NOT EXISTS(PrimaryKey); Inner Join SQL SELECT PrimaryKey FROM DB_TABLE; If ScriptErrorCount = 0 then STORE QV_Table INTO File.QVD; Let LastExecTime = ThisExecTime; End If

Summary 1