Lab: Qlik Replicate for

Oracle Database to Synapse

TABLE OF CONTENTS
Overview ..................................................................................................................................................... 3
Introduction ............................................................................................................................................... 3
Accessing Replicate Environment ............................................................................................................. 4
Oracle Source Configuration ..................................................................................................................... 5
Azure Synapse Target Configuration ......................................................................................................... 8
Configure Replication Task ........................................................................................................................ 9
Run Task ................................................................................................................................................... 15
Summary .................................................................................................................................................. 18
Tips & Tricks ............................................................................................................................................. 19
Oracle Database as Target 19
 Overview
This Document covers the process involved in configuring Qlik Replicate to extract data out of an
Oracle Database and store it into Microsoft Synapse Environment. The areas covered include:

• Configuring Endpoints (Source and Target)

• Creating Replicate Task

• Execute Task

Introduction
Replicate Tasks are used to manage the extraction and loading from various Source
Systems/Databases to targeted Databases. Although a Replicate task can only manage one source
and one target system, it does not limit the development of multiple tasks within a project. Example:
A client may have three different Database/File Systems in their environment, (Oracle, SAP HANA,
IBM VSAM files) and would like to extract all these data into a centralized database environment
(Microsoft Synapse) for Analysis. In this scenario, three Replicate Tasks would be required.
What will surprise those who worked with other data extraction tools, is the seamlessness with
which Qlik Replicate performs these processes.

In this Lab, you will define the steps needed to configure Qlik Replicate to extract data from Oracle
Database to Microsoft Synapse. Below are the steps required.

Oracle Database to Synapse | Overview 3

 Accessing Replicate Environment
Steps

1. Ensure that the URL for the Replicate Server is available, and access granted.
- This will be provided by your Systems Administrator.

2. Open browser** and enter the URL of the Replication Server.

The format is: https://ReplicateComputerName:3552/attunityreplicate/

- This will prompt you to log into Replicate.

3. Enter your Username and Password.

- This will take you to the Replicate Console.

Oracle DatabaseOracle Database

to Synapse to Microsoft
| Accessing Replicate Synapse in Qlik Replicate Environment
Environment 4
4
 Oracle Source Configuration

4. Select Manage Endpoint Connections.

The following window will appear.

We will now create an Oracle Endpoint.

Oracle DatabaseOracle Database

to Synapse | OracletoSource
Microsoft Synapse in Qlik Replicate Environment
Configuration 5
5
 5. Select + New Endpoint Connection.

6. Enter a meaningful Endpoint Name and Description for the Endpoint Connector.

7. With Source button selected, Select dropdown arrow to select Oracle .

You will notice as we proceed that the content of the configuration window is context sensitive.

Oracle DatabaseOracle Database

to Synapse | OracletoSource
Microsoft Synapse in Qlik Replicate Environment
Configuration 6
6
 8. Fill in both Source, Target Servers and Azure Database credentials as provided
by your Systems Administrator.

Server:
Port:
User:
Password:
Security/SSL Mode:

9. Select Test Connection.

This tells whether your configuration is correct.

Look for the “Test Connection succeeded” message. Any other message means something may be
incorrect with your Server/Database definitions, or the Server/Database is unavailable.

10. Select Save.

Oracle DatabaseOracle Database

to Synapse | OracletoSource
Microsoft Synapse in Qlik Replicate Environment
Configuration 7
7
 Azure Synapse Target Configuration
Next, we need to configure our Azure Synapse target endpoint. The process is much the same as you saw with
the endpoints, and once again you will note that the configuration process is context sensitive as we move
along. As before, the first step in the configuration process is to tell Replicate that we want to create a new
endpoint.

1. In the Replicate Console, Select Manage Endpoint Connections.

2. Select + New Endpoint Connection.

3. Enter a meaningful Endpoint Name and Description for the Endpoint Connector.

4. Select Target button selected in Role.

5. Select Test Connection.

Again, look for the “Test Connection succeeded” message. Any other message means something
may be incorrect with your Server/Database definitions, or the Server/Database is unavailable.

6. Select Save.

Oracle DatabaseOracle Database

to Synapse | Azureto Microsoft
Synapse Synapse
Target in Qlik Replicate Environment
Configuration 8
8
 Configure Replication Task
Now that we have configured our Oracle source and Azure Synapse target endpoints, we need to tie them
together in what we call a Replicate task. In short, a task defines the following:

1. A source endpoint.

2. A target endpoint.

3. The list of tables that we want to capture.

4. Any transformations we want to make on the data.

We start by doing the following:

1. In the Replication Console, Select +New Task to create a New Task.

The following window will appear:

2. Enter a meaningful Task Name.

3. Select Unidirectional radio button.

- This indicates data flows from Source to Target – with no writebacks to source system.

4. Ensure that Full Load and Apply Changes are shaded.

Oracle DatabaseOracle Database

to Synapse to Microsoft
| Configure Synapse
Replication Task in Qlik Replicate Environment 9
9
 You should have the following filled in:

• Unidirectional
• Full Load: enabled (Blue highlight is enabled; Select to enable / disable.)
• Apply Changes: enabled (Blue highlight is enabled; Select to enable / disable.)
• Store Changes: disabled (Blue highlight is enabled; Select to enable / disable.)

5. Select OK.
- This closes the New Task dialog box.
Once completed, the following window will appear:

We will now assign the newly created endpoints - source endpoint (Oracle) and target endpoint (Azure
Synapse).

At this point the Source and Target Endpoints created above will be assigned, in order for Replicate to
extract and load the data as defined.

Oracle DatabaseOracle Database

to Synapse to Microsoft
| Configure Synapse
Replication Task in Qlik Replicate Environment 10
10
 1. On the left of the Replicate Console panel, Select Source.

2. Locate the Source Endpoint created above or one which meets your Source definitions.

3. Drag and drop to Source Endpoint on the right, as indicated in diagram.

4. On the left of the Replicate Console panel, Select Target.

5. Locate the Target Endpoint created above or one which meets your Target definitions.

Oracle DatabaseOracle Database

to Synapse to Microsoft
| Configure Synapse
Replication Task in Qlik Replicate Environment 11
11
 6. Drag and drop to Target Endpoint on the right, as indicated in diagram.

7. Select Save.
Replicate is ready to extract the data from Oracle into Azure Synapse.

Steps

1. In the Qlik Replicate Console, Select Table Selection.

Oracle DatabaseOracle Database

to Synapse to Microsoft
| Configure Synapse
Replication Task in Qlik Replicate Environment 12
12
 2. Enter % in Table.

3. Enter HR in Schema.

4. Select Search.
A list of available files/tables will appear.

5. Select Files/tables to be transfer into the Qlik Replicate Console.

Oracle DatabaseOracle Database

to Synapse to Microsoft
| Configure Synapse
Replication Task in Qlik Replicate Environment 13
13
 6. Select HR. Departments.

7. Select HR. Employee.

8. Select > icon to transfer data to the Selected Tables column.

9. Select OK.
That completes configuration of the task. We are now ready to save our task and run it.

Oracle DatabaseOracle Database

to Synapse to Microsoft
| Configure Synapse
Replication Task in Qlik Replicate Environment 14
14
 Run Task

1. Select down arrow beside Run.

2. Select Start Processing.

• If this is not the first time this Task is being used to extract data, Reload Target must be used –
Reload Target will not be an option.

3. Select Yes to reload Data Target.

Replicate switches to Monitor mode to allow for monitoring of Data Loads.

Oracle DatabaseOracle Database

to Synapse to Microsoft Synapse in Qlik Replicate Environment
| Run Task 15
15
 After Full Load is complete, select on the Completed bar to display the tables. There

is DML activity running in the background. Select on the Change Processing tab to

see it in action.

Note: Changes to the tables occur somewhat randomly in the background. You may

need to wait a few minutes before you will see changes appear in the tables that we

selected.

4. Open Azure data studio and Select on the Create new Connection.

Enter Synapse logon credentials as provided by your System Administrator.

Oracle DatabaseOracle Database

to Synapse to Microsoft Synapse in Qlik Replicate Environment
| Run Task 16
16
 5. Select on the Connect to test the connectivity. Connection should be successful.
6. Open a new query workbench and validate the records.

When you have seen enough, you can declare Victory! for this part of the Test Drive.

7. Press Stop in the top left corner of the Replicate console to end the task.

8. Select Yes in the confirmation dialog.

9. Close the Oracle to Azure Synapse tab or Select on the TASKS tab to return to the main window.

Oracle DatabaseOracle Database

to Synapse to Microsoft Synapse in Qlik Replicate Environment
| Run Task 17
17
 Summary
By the end of this lab we have covered:
• Defined access and authentication into an Oracle source and an Azure Synapse target.
• Defined the source tables from which you want to transfer data into Azure Synapse system.
• Configured the Oracle to Azure Synapse task.
• Captured initial data from the source without while maintaining business continuity (DML activity was
going on in the background to simulate users working on the source database).
• Captured all new transactions which were happening while the initial load was running.
• Observed change data being recorded as it is sent to and applied at the target.

Oracle DatabaseOracle Database

to Synapse to Microsoft Synapse in Qlik Replicate Environment
| Summary 18
18
 Tips & Tricks
Oracle Database as Target
1. Performance fine tuning From patch p01 on 3.1.1

Oracle target performance enhancement:

There are two new Oracle target advanced properties, similar to the options
exposed by the SQL Loader:

• directPathColumnArrayRows – This is the max number of rows

loaded in batch. Default 5000 rows.

• directPathStreamSize – The is the direct path stream buffer size.

Default 256000 bytes.

Increasing those parameters should give better performance.

Newer performance params

Oracle target Parameters:

bulkUseParallel - default true – Use Parallel hint for bulk DML statements

bulkArraySize – for direct insert

2 direct path params.

Task settings:

“batch_apply_use_direct_insert”=true

“batch_apply_direct_insert_min_events”=100 (default)

notes:

Insert into net changes:

· Uses direct path

· Influenced by the direct path params

CDC direct insert:

· Uses array insert

· Influenced by “bulkArraySize” param

Oracle DatabaseOracle Database

to Synapse | Tips &to Microsoft Synapse in Qlik Replicate Environment
Tricks 19
19
 2. More Performance fine tuning From patch p02 on 3.1.1 and up:

An option was added in the provider syntax to support using an append INSERT
statements when working in bulk apply. This will cause Oracle to use Direct
Path when inserting the rows from the net changes table to the target table and
therefore, should give a boost when there are many inserts. Especially with
Exadata.
To make the target Oracle endpoint use the append inserts bulk statements,
add the following to the connection string:

$info.query_syntax.bulk_insert_syntax=USE_APPEND

Tests done in our lab proved improvement of 43%!!!!!!!!

3. Stream Parameters Stream Parameters

Please try to export the repository, play with the following parameters and
import back:

"common_settings": {

"lob_max_size": 1,

"change_table_settings": {

"table_suffix": "__ct",

"column_prefix": "header__"

},

"audit_table_settings": {

"table_name": ""

},

"stream_buffers_number" : 5,

"stream_buffer_size" : 20

Oracle DatabaseOracle Database

to Synapse | Tips &to Microsoft Synapse in Qlik Replicate Environment
Tricks 20
20
 Please note that the defaults are:

· "stream_buffers_number" : 3

· "stream_buffer_size" : 8 (MB)

4. To set the Upsert/Merge mode upsertMode=true;

5. Bi Directional privileges In Oracle target, when the target bi-directional schema is different that the
Replicate user account you have two options:

1. Manually run the following on the target assuming HR is the bi-

directional schema and BIDI is the Replicate user account:

CREATE TABLE "HR"."attrep_loopback_prevention" ( "task_name"

VARCHAR2(128) not null, "timestamp" TIMESTAMP(0), CONSTRAINT
"HRattrep_loopback_prevention" PRIMARY KEY ( "task_name" ) );

grant select on "HR"."attrep_loopback_prevention" to bidi;

grant insert on "HR"."attrep_loopback_prevention" to bidi;

2. Else grant the following to the Replicate user account:

grant create any table to bidi;

grant create any index to bidi;

grant select any table to bidi;

grant insert any table to bidi;

6. Set varchar(char n) in target To enable the fix, add in target connection string: "charLengthSemantics".
database
Values: DEFAULT, CHAR, BYTE

Default - create target tables with default value of Oracle

BYTE- Default - create target tables with VARCHAR (XX BYTE) CHAR - create
target tables with VARCHAR (XX CHAR)

Oracle DatabaseOracle Database

to Synapse | Tips &to Microsoft Synapse in Qlik Replicate Environment
Tricks 21
21
 Default will be as before (use Oracle default)

7. directPathNoLog=true Will cause direct path inserts to not be logged.

8. createNetChangesTempTable

9. EmptyStringValue By default an Oracle target changes an 'empty' string on the source to 'space'.

This is implemented using an internal parameter "emptyStringValue" which

defaults to a SPACE.

If an empty string is preferred (which will become a NULL in Oracle), one cannot
directly enter emptiness in the GUI, as the GUI interprets that as an attempt to
revert to default.

To revert back to an empty string one must:

- Set internal parameter emptyStringValue for Oracle Target to 'Hello World'

- Export (any) task with that target

- Edit JSON, look for 'Hello World' replace with nothing leaving

"emptyStringValue": ""

- Save.

- Import task.

- Stop and Resume any tasks which needed this fix.

10. max_transaction_size This is the JSON variable behind full-load-tuning - "Commit rate during full
load:" Contrary to that description this is also in effect for CDC.

For some targets is it fine to set this to 1 Million or so.

Not for Oracle targets.

Values like that will cause the task to use excessive memory (28 GB !?) as visible
by (windows) 'taskmanager and the (5.2) new "memory_usage_kb" porperty in
gettaskstatus. With REPCTL GETSNAPSHOT I observed
the ORACLE_CDC pool ORACLE_STREAM_COMPONENT grow linear with this:
10K --> 41033728 bytes, 100K --> 410394624 bytes, 1M --> 4104003584 bytes.

Or how about: Pool ORACLE_CDC , address 000000FE4A1305E8 , prev size -

190963712 , cur size

Oracle DatabaseOracle Database

to Synapse | Tips &to Microsoft Synapse in Qlik Replicate Environment
Tricks 22
22
 About Qlik
Qlik’s vision is a data-literate world, where everyone can use data and analytics to improve decision-making and solve their most challenging
problems. Qlik provides an end-to-end, real-time data integration and analytics cloud platform to close the gaps between data, insights and action.
By transforming data into active intelligence, businesses can drive better decisions, improve revenue and profitability, and optimize customer
relationships. Qlik does business in more than 100 countries and serves over 50,000 customers around the world.
qlik.com
© 2021 QlikTech International AB. All rights reserved. All company and/or product names may be trade names, trademarks and/or registered trademarks of the respective owners with which they are
associated. CODE NEEDED FOR PUBLISHING

Oracle DatabaseOracle Database

to Synapse | Tips &to Microsoft Synapse in Qlik Replicate Environment
Tricks 23
23