Advanced ETL Processor

User manual

© DB Software Laboratory 2008-2009

www.dbsoftlab.com
Advanced ETL Processor User Manual

Contents

Contents....................................................................................................................................... 2
1. Introduction ............................................................................................................................. 9
2. Key features ........................................................................................................................... 10
2.1 Extraction Process ........................................................................................................... 10
2.1.1 Multiple Data Formats.............................................................................................. 10
2.1.2 Multiple Databases and Table Processing................................................................ 10
2.1.3 Other Database Features........................................................................................... 10
2.2 Summary of the Extraction Process: ............................................................................... 10
2.3 Validation Process ........................................................................................................... 11
2.4 Summary of Validation Processes:.................................................................................. 11
2.5 Transformation Process ................................................................................................... 11
2.6 Powerful Data Transformation ........................................................................................ 11
2.7 Summary of Transformation Processes:.......................................................................... 12
2.8 Loading Process............................................................................................................... 13
2.9 Summary of the Loading Process:................................................................................... 13
3. Requirements......................................................................................................................... 14
4. Advanced ETL Processor Architecture .............................................................................. 16
5. Processing Data ..................................................................................................................... 16
5. Processing Data ..................................................................................................................... 17
5.1 Screen Overview ............................................................................................................. 17
Main tool bar ..................................................................................................................... 18
5.2 Transformation Properties ............................................................................................... 19
5.3 Template tab .................................................................................................................... 20
5.4 Execution Log Tab .......................................................................................................... 21
5.5 Rejected Records Tab...................................................................................................... 22
5.6 Creating new transformation and working with Objects................................................. 23
5.7 Working with Reader ...................................................................................................... 25
5.7.1 Universal Data Reader.............................................................................................. 25
5.7.2 Data source is a Text File. ........................................................................................ 26
5.7.3 Data source is MS Access or Excel File................................................................... 31
5.7.4 Data source is a DBF File......................................................................................... 35
5.7.5 Data source is a ODBC Connection ......................................................................... 37
5.7.6 Data source is a MS SQL Server .............................................................................. 40
5.7.7 Data source is an Oracle Database ........................................................................... 41
5.7.8 Data source is an MySql........................................................................................... 42
5.7.9 Data source is an PostgreSQL Database................................................................... 43
5.7.10 Data source is an Interbase or Firebird Database ................................................... 44
5.8 Working with Validator................................................................................................... 45
5.8.1 Debugging Validation............................................................................................... 47
5.9 Working with Transformer.............................................................................................. 50
5.9.1 Auto mapping ........................................................................................................... 52
5.10 Working with Grouper .................................................................................................. 53
5.11 Working with Sorter ...................................................................................................... 54
5.12 Working with Deduplicator........................................................................................... 56
5.13 Working with UnPivot .................................................................................................. 57
5.14 Working with Pivot ....................................................................................................... 62
5.15 Working with Writer ..................................................................................................... 67
5.15.1 Target type is a Text File........................................................................................ 67
Copyright © 2009 DB Software Laboratory Page 2 of 194
Advanced ETL Processor User Manual

5.15.2 Target type is MS Access or Excel File.................................................................. 70

5.15.3 Target type is a DBF File........................................................................................ 72
5.15.4 Target type is ODBC connection............................................................................ 74
5.15.5 Target type is MS SQL Server................................................................................ 75
5.15.6 Target type is Oracle............................................................................................... 77
5.15.7 Target type is MySql .............................................................................................. 78
5.15.8 Target type is PostgreSQL...................................................................................... 79
5.15.9 Target type is Interbase/Firebird............................................................................. 80
5.15.10 How to Update/Delete Records ............................................................................ 82
5.15.11 Running SQL Scripts............................................................................................ 84
6. Validation Rules .................................................................................................................... 85
6.1 Strings.............................................................................................................................. 86
6.1.1 Is Null ....................................................................................................................... 86
6.1.2 Is Empty String......................................................................................................... 86
6.1.3 Is Alpha .................................................................................................................... 86
6.1.4 Is Alpha Numeric...................................................................................................... 86
6.1.4 Is Hex........................................................................................................................ 87
6.1.5 Is Equal To ............................................................................................................... 87
6.1.6 Is Starts With ............................................................................................................ 88
6.1.7 Is Ends With ............................................................................................................. 88
6.1.8 Contains.................................................................................................................... 88
6.1.9 In List........................................................................................................................ 88
6.2 Numbers .......................................................................................................................... 92
6.2.1 Is Number ................................................................................................................. 92
6.2.2 Is Integer ................................................................................................................... 92
6.2.3 Is Positive ................................................................................................................. 92
6.2.4 Is Negative................................................................................................................ 92
6.2.5 Is Number Between .................................................................................................. 93
6.2.6 Is Number Less Than................................................................................................ 94
6.2.7 Is Number More Than .............................................................................................. 95
6.3 Date.................................................................................................................................. 96
6.3.1 Is Date....................................................................................................................... 96
6.3.2 Is Date Between........................................................................................................ 97
6.3.3 Is Date Less Than ..................................................................................................... 98
6.3.4 Is Date More Than.................................................................................................... 98
6.3.5 Is Monday, Is Tuesday, Is Wednesday, Is Thursday, Is Friday, Is Saturday, Is
Sunday, Is Today, Is Yesterday, Is Tomorrow, Is Weekend, Is Weekday..................... 99
6.3.6 Is January, Is February, Is March, Is April, Is May, Is June, Is July, Is August, Is
September, Is October, Is November, Is December, Is Current Month, Is Last Month, Is
Next Month........................................................................................................................ 99
6.3.7 Is 1st Quarter, Is 2nd Quarter, Is 3rd Quarter, Is 4th Quarter, Is Current Quarter, Is
Last Quarter, Is Next Quarter .......................................................................................... 100
6.3.8 Is Leap Year, Is Current Year, Is Last Year, Is Next Year ..................................... 100
6.3.9 Is Current Week, Is Last Week, Is Next Week....................................................... 100
6.3.10 Is Within Past Minutes, Is Within Past Hours, Is Within Past Days, Is Within Past
Weeks, Is Within Past Months ........................................................................................ 101
6.4 Time............................................................................................................................... 102
6.4.1 Is Time.................................................................................................................... 102
6.4.2 Is Time Between..................................................................................................... 103
6.4.3 Is Time Less Than .................................................................................................. 104

Copyright © 2009 DB Software Laboratory Page 3 of 194

Advanced ETL Processor User Manual

6.4.4 Is Time More Than ................................................................................................. 104

6.4.5 Is Second ................................................................................................................ 105
6.4.6 Is Minute................................................................................................................. 105
6.4.7 Is Hour 24 ............................................................................................................... 105
6.4.8 Is Hour 12 ............................................................................................................... 105
6.4.9 Is PM ...................................................................................................................... 106
6.4.10 Is AM.................................................................................................................... 106
6.5 Regular Expressions ...................................................................................................... 107
6.5.1 Regular Expression................................................................................................. 108
6.5.2 Is IP Address V4..................................................................................................... 109
6.5.3 Is IP Address V6..................................................................................................... 109
6.5.4 Is Email................................................................................................................... 109
6.5.5 Is ISBN 10 .............................................................................................................. 109
6.5.6 Is ISBN 13 .............................................................................................................. 110
6.5.7 Is Credit Card ......................................................................................................... 110
6.5.8 Is URL .................................................................................................................... 110
6.5.9 Is UNC.................................................................................................................... 110
6.5.10 Is File Name.......................................................................................................... 111
6.5.11 Is UK Post Code ................................................................................................... 111
6.5.12 Is UK VAT Number ............................................................................................. 111
6.5.13 Is UK Bank Sort Code .......................................................................................... 111
6.5.14 Is UK Vehicle Registration Number .................................................................... 112
6.5.15 Is UK Driver Licence............................................................................................ 112
6.5.16 Is UK National Insurance Number ....................................................................... 112
6.5.17 Is UK Phone Number ........................................................................................... 112
6.5.18 Is UK Mobile Number.......................................................................................... 113
6.5.19 Is US Zip Code ..................................................................................................... 113
6.5.20 Is US State ............................................................................................................ 113
6.5.21 Is US Social Security Number.............................................................................. 113
6.5.22 Is US Phone Number ............................................................................................ 114
6.5.23 Is Canadian Zip Code ........................................................................................... 114
6.5.24 Is Canadian Provincial Code ................................................................................ 114
6.5.25 Is Canadian Branch-Transit/ Bank code ............................................................... 114
6.5.26 Is Canadian Social Insurance Number.................................................................. 115
6.5.27 Is Australian Post Code ........................................................................................ 115
6.5.28 Is Australian Phone Number................................................................................. 115
6.5.29 Is Australian Mobile Number ............................................................................... 116
6.5.30 Is Argentinean Post Code ..................................................................................... 116
6.5.31 Is Dutch Post Code ............................................................................................... 116
6.5.32 Is Dutch Phone Number ....................................................................................... 116
6.5.33 Is Dutch Car Number Plate................................................................................... 117
6.5.34 Is Belgian Post Code............................................................................................. 117
6.5.35 Is Belgian VAT Number....................................................................................... 117
6.5.36 Is French Post Code.............................................................................................. 117
6.5.37 Is French VAT Number........................................................................................ 118
6.5.38 Is French Phone Number ...................................................................................... 118
6.5.39 Is German Post Code ............................................................................................ 118
6.5.40 Is German VAT Number ...................................................................................... 118
6.5.41 Is German Car License Plate ................................................................................ 119
6.5.43 Is German IBAN................................................................................................... 119

Copyright © 2009 DB Software Laboratory Page 4 of 194

Advanced ETL Processor User Manual

6.5.44 Is Italian Post Code............................................................................................... 119

6.5.45 Is Italian VAT Number......................................................................................... 119
6.5.46 Is Italian IBAN ..................................................................................................... 120
6.5.47 Is Italian Phone Number ....................................................................................... 120
6.5.48 Is Italian Mobile Number ..................................................................................... 120
6.5.49 Is Spanish Post Code ............................................................................................ 120
6.5.50 Is Spanish VAT Number ...................................................................................... 121
6.5.51 Is Spanish IBAN................................................................................................... 121
6.5.52 Is Spanish Phone Number .................................................................................... 121
6.5.53 Is Swedish Post Code ........................................................................................... 121
6.5.54 Is Swedish VAT Number ..................................................................................... 121
6.5.55 Is Swedish IBAN .................................................................................................. 122
6.5.56 Is Swedish Phone Number.................................................................................... 122
6.5.57 Is Swedish Person Number................................................................................... 122
6.5.58 Is Austrian Mobile Phone Number....................................................................... 123
6.5.59 Is Danish Post Code.............................................................................................. 123
6.5.60 Is Danish VAT Number........................................................................................ 123
6.5.61 Is Danish Phone Number...................................................................................... 123
6.5.62 Is Danish Civil Registration Number ................................................................... 124
6.5.63 Is Finnish VAT Number ....................................................................................... 124
6.5.64 Is Greek VAT Number ......................................................................................... 124
6.5.65 Is Irish VAT Number............................................................................................ 124
6.5.66 Is Luxembourg VAT Number .............................................................................. 125
6.5.67 Is Portuguese VAT Number ................................................................................. 125
6.5.68 Is Estonian Post Code........................................................................................... 125
6.5.69 Is Estonian VAT Number ..................................................................................... 125
6.5.70 Is Latvian VAT Number....................................................................................... 126
6.5.71 Is Lithuanian VAT Number.................................................................................. 126
6.5.72 Is Slovenian VAT Number ................................................................................... 126
6.5.73 Is Slovakian VAT Number ................................................................................... 126
6.5.74 Is Slovakian IBAN................................................................................................ 127
6.5.75 Is Polish VAT Number......................................................................................... 127
6.5.76 Is Polish IBAN...................................................................................................... 127
6.5.77 Is Czech Republic VAT Number.......................................................................... 127
6.5.78 Is Czech Republic IBAN ...................................................................................... 128
6.5.79 Is Netherlands VAT Number................................................................................ 128
6.5.81 Is Maltese VAT Number ...................................................................................... 128
6.5.82 Is Algerian Post Code ........................................................................................... 129
6.5.83 Is Andorran Post Code.......................................................................................... 129
6.5.84 Is Andorran IBAN ................................................................................................ 129
6.5.85 Is Azerbaijani Post Code ...................................................................................... 129
6.5.86 Is Bahraini Post Code ........................................................................................... 130
6.5.87 Is Bangladeshi Post Code ..................................................................................... 130
6.5.88 Is Belarusian Post Code........................................................................................ 130
6.5.89 Is Brunei Post Code .............................................................................................. 130
6.5.90 Is Chilean Post Code ............................................................................................ 131
6.5.91 Is Croatian Post Code ........................................................................................... 131
6.5.92 Is Cyprus Post Code ............................................................................................. 131
6.5.93 Is Cyprus VAT Number........................................................................................ 131
6.5.94 Is Egyptian Post Code........................................................................................... 132

Copyright © 2009 DB Software Laboratory Page 5 of 194

Advanced ETL Processor User Manual

6.5.95 Is Ethiopian Post Code ......................................................................................... 132

6.5.96 Is Faroe Islands Post Code.................................................................................... 132
6.5.97 Is French Polynesia Post Code ............................................................................. 132
6.5.98 Is Guadeloupe Post Code...................................................................................... 133
6.5.99 Is Guinean Post Code ........................................................................................... 133
6.5.100 Is Guinea-Bissau Post Code ............................................................................... 133
6.5.101 Is Russian Post Code .......................................................................................... 133
6.5.102 Is Russian Phone Number .................................................................................. 134
6.5.103 Is Israeli Phone Number ..................................................................................... 134
6.5.104 Is Israeli Mobile Phone Number......................................................................... 134
6.5.105 Is Peru Phone Number........................................................................................ 134
6.5.106 Is Peru Cellular Number ..................................................................................... 135
6.5.107 Is Indian Post Code............................................................................................. 135
6.5.108 Is Indian Phone Number ..................................................................................... 135
6.5.109 Is Indian Mobile Number ................................................................................... 135
6.5.110 Is Indian Vehicle Registration Number .............................................................. 136
6.5.111 Is South African Phone Number......................................................................... 136
6.5.112 Is South African Mobile Number ....................................................................... 136
6.5.113 Is South African VAT Number .......................................................................... 136
6.5.114 Is US or CA Zip code ......................................................................................... 137
7. Transformation Functions ................................................................................................... 138
7.1 Strings............................................................................................................................ 138
7.1.1 Upper Case ............................................................................................................. 138
7.1.2 Lower Case ............................................................................................................. 138
7.1.3 Proper Case............................................................................................................. 138
7.1.4 First Up................................................................................................................... 138
7.1.5 Delete Spaces.......................................................................................................... 139
7.1.6 Trim ........................................................................................................................ 139
7.1.7 Left Trim ................................................................................................................ 139
7.1.8 Right Trim .............................................................................................................. 139
7.1.9 Sub String ............................................................................................................... 140
7.1.10 Replace ................................................................................................................. 141
7.1.11 Ensure Prefix ........................................................................................................ 142
7.1.12 Ensure Suffix ........................................................................................................ 142
7.1.13 Ensure No Prefix .................................................................................................. 142
7.1.14 Ensure No Suffix .................................................................................................. 143
7.1.15 Left Pad ................................................................................................................ 143
7.1.16 Right Pad .............................................................................................................. 143
7.1.17 Escape String ........................................................................................................ 144
7.1.18 UnEscape String ................................................................................................... 144
7.1.19 Delete.................................................................................................................... 144
7.1.20 Left ....................................................................................................................... 145
7.1.21 Right ..................................................................................................................... 145
7.1.22 Left Delete ............................................................................................................ 145
7.1.23 Right Delete.......................................................................................................... 146
7.2 Numbers ........................................................................................................................ 147
7.2.1 Round ..................................................................................................................... 147
7.2.3 Abs.......................................................................................................................... 148
7.2.4 Sign......................................................................................................................... 148
7.3 Date................................................................................................................................ 149

Copyright © 2009 DB Software Laboratory Page 6 of 194

Advanced ETL Processor User Manual

7.3.1 Date Format ............................................................................................................ 149

7.3.2 Current Date ........................................................................................................... 149
7.4 Miscellaneous ................................................................................................................ 150
7.4.1 Length..................................................................................................................... 150
7.4.2 Literal ..................................................................................................................... 150
7.4.3 User......................................................................................................................... 151
7.4.4 Splitter .................................................................................................................... 151
7.4.5 Joiner ...................................................................................................................... 152
7.4.6 Calculation.............................................................................................................. 153
7.4.7 Lookup.................................................................................................................... 182
7.4.8 Previous Value........................................................................................................ 185
7.4.9 Reader Parameter.................................................................................................... 186
7.4.10 Sequence............................................................................................................... 187
7.4.11 Running Total....................................................................................................... 188
8. Date formats ........................................................................................................................ 189
9. Command Line .................................................................................................................... 190
10. Support Procedure ............................................................................................................. 191
11. License Agreement ............................................................................................................ 192

Copyright © 2009 DB Software Laboratory Page 7 of 194

Advanced ETL Processor User Manual

Copyright

Copyright © 2009 DB Software Laboratory Limited. All rights reserved.

No portion of this document may be reproduced, transmitted, transcribed, stored in

a retrieval system, or translated into any language, in any form or by any means,
without prior written consent of DB Software Laboratory Limited.

Note to U.S. Government users:

Documentation and programs related to restricted rights - use, duplication or
disclosure is subject to restrictions set forth in:
GSA FMSS Schedule Contract No. GS00K92AFS2505-PS05.

License Information

You have access to Advanced ETL Processor software and documentation

pursuant to the terms of a Software License Agreement granted by DB Software
Laboratory Limited. As a user of this software and documentation, you are bound by
the terms of the Software License Agreement. All rights, title, and interest to this
software remain with DB Software Laboratory Limited.
Requests for copies of this publication and for technical information about DB
Software Laboratory products should be made directly to DB Software Laboratory
Limited.

Disclaimer

All information in this manual is subject to periodic change and revision without
notice. While every effort has been made to ensure that this manual is accurate, DB
Software Laboratory Limited excludes its liability for errors or inaccuracies (if any)
contained herein.

Registered Marks

Any products or services mentioned or depicted in this document are identified by

the trademarks or service marks of their respective companies or organisations.

Edition Information

This document refers to Advanced ETL Processor version 2.4.2.

Copyright © 2009 DB Software Laboratory Page 8 of 194

Advanced ETL Processor User Manual

1. Introduction

The Advanced ETL Processor is a Codeless ETL Tool.

The Advanced ETL Processor is an end to end database extraction and importing
tool. The beauty of the system is that it saves the drudgery and manual tasks
normally required for tasks of this type, such as the writing of code, and all the
transformations, validations and general checks normally performed. Using traditional
methods, operations of this type can only be performed in stages, and not as one
smooth operation!

For example, the traditional method of importing data from one system to another is
to write specific code to extract data from the source database, e.g. an Oracle
database, by creating a CSV comma de-limited file, and then writing code in the new
language or system, for instance Microsoft Access, and then performing the import.
However, the operation does not end there. Any data imported has to be sorted,
duplicated and loaded into the database using appropriate primary and foreign key
constraints. This is only possible by creating code designed to achieve this process.
You then need to manually send an e-mail to the administrator when the process is
complete. In other words, each stage cannot be left to run in an automated fashion
and has to be completed before proceeding to the next.

As you can see, the process is not straightforward. The Advanced ETL Processor
automates all these processes in a simple and transparent fashion, and all without
writing any code whatsoever.

As stated, the tool to handle any kind of database, including Oracle, Microsoft
Access, SQL Server, DB2, MySQL, Excel spreadsheets, and a wide range of others.
It is an excellent tool for those organisations who work with data warehouses, and
where this involves working with a number of disparate databases.

Existing users find that the tool provides several benefits over existing tools such as
Oracle SQL Loader, BCP, DTS or SSIS, such as the ability to update records
automatically via utilisation of the primary key.

Copyright © 2009 DB Software Laboratory Page 9 of 194

Advanced ETL Processor User Manual

2. Key features

The ability of the Advanced ETL Processor to work with a number of disparate
systems means that it is provided with a rich set of tools and functionality, which can
be used in isolation or combined in a powerful way with other toolsets, either within
the processor or with other third party tools. It is in effect, an "engineering"
environment for the movement of data to and from different sources.

We will now explain and define the variety of features which are provided as part of
the toolset. Let us first have a look at the data extraction process.

2.1 Extraction Process

2.1.1 Multiple Data Formats

The extraction process can handle a variety of data formats, including multiple
delimited or fixed width text files. The power of this system however, is in its ability to
find files to load using mask.

2.1.2 Multiple Databases and Table Processing

It can easily interpret and manipulate Microsoft Access data, from a number of
different databases. Again the end user can use a mask to find the tables to load the
data from. It does also apply to Excel and DBF/FoxPro files.

2.1.3 Other Database Features

The Advanced ETL Processor also has other useful database features, such as
the ability to connect to any Object Database Connectivity (ODBC) database. ODBC
was intended to enable developers' access to any data through any application,
regardless of the DBMS used for managing that data. ODBC boasts platform
independence since it has been purposefully designed in a way that makes it distinct
from database systems, programming languages and operating systems. The
Advanced ETL Processor utilises this technology to great effect.

So what if you don’t have an Oracle or MySQL database? What about SQL Server?
No problem, the Advanced ETL Processor can handle SQL Server data as
efficiently as data from any other type of database.

2.2 Summary of the Extraction Process:

• Multiple Delimited or Fixed width Text files

• Multiple Excel files + Multiple Excel Spreadsheets
• Multiple MS Access Databases + Multiple Tables
• Multiple DBF Files
• Any ODBC compliant database
• Direct connection to Oracle, SQL Server, MySql, PostgreSQL and Interbase/Firebird
• Multiple Tables

Copyright © 2009 DB Software Laboratory Page 10 of 194

Advanced ETL Processor User Manual

2.3 Validation Process

The Advanced ETL Process has a robust validation process built in. The types and
nature of the validations taking place can be tweaked and configured by the user. A
full range of validation functions are included. Validations can be performed on the
basis of data type, lists of values, and regular expressions, which can be individually
changed according to requirements.

2.4 Summary of Validation Processes:

Validation:

• 190 Validation Functions

• String Validation
• Number Validation
• Date Validation
• Time Validation
• Validation Against list of values
• Regular Expressions: Post codes, Phone Numbers, VAT Numbers ETC

2.5 Transformation Process

The process of “transformation” involves the manipulation of data so that it can be

converted from one format or data type to another. The Advanced ETL Processor
provides a useful set of “built in” transformation functions. Transformations can be
performed on strings, numbers, dates, coded values and whole chunks of data. The
transformation functionality allows for the sorting and grouping of data according to
requirements. Data can also be duplicated as required.

2.6 Powerful Data Transformation

In addition to the standard data transformation abilities, the processor can also
perform complex translation functions. An example would be if an integer variable =
“1”, then set a text variable to “yes”. Once data is translated, it is possible to join data
into a completely new format or present it in a new way. The Advanced ETL
Processor provides many flexible alternatives for data manipulation, and these are
not difficult to take advantage of.

The Advanced ETL Processor also provides the ability to derive calculated values,
join data together from multiple fields, summarise multiple rows at once, or can split
or merge columns at will.

The flexibility and power of the processor means that you can customize data
transformation and conversion functions according to your requirements with a click
of the mouse. This saves hours and hours of coding.

Copyright © 2009 DB Software Laboratory Page 11 of 194

Advanced ETL Processor User Manual

2.7 Summary of Transformation Processes:

Transformation:

• 39 Transformation Functions
• String Transformation
• Number Transformation
• Date Transformation
• Sorting
• Grouping
• Deduplication
• Translating coded values (e.g., if the source system stores 1 for male and 2 for female,
but the warehouse stores M for male and F for female)
• Deriving a new calculated value (e.g., sale amount = qty * unit price)
• Joining together data from multiple fields
• Summarizing multiple rows of data (e.g., total sales for each store, and for each region)
• Generating surrogate key values
• Transposing or pivoting (turning multiple columns into multiple rows or vice versa)
• Splitting a column into multiple columns (e.g., putting a comma-separated list
specified as a string in one column as individual values in different columns)
• Customised Transformation
• Primary Key Generation
• Running Totals

Copyright © 2009 DB Software Laboratory Page 12 of 194

Advanced ETL Processor User Manual

2.8 Loading Process

The loading capability of the Advanced ETL Processor is superior to other basic
tools such as SQL Loader, because it provides the administrator with several options
and ways of providing database load capability without creating any code.

Other tools allow you to load data into a single database table at a time, under a
single instance. The ETL processor on the other hand allows you to specify multiple
upload targets which mean you can save time loading each individual table. Another
useful feature is that you can execute SQL scripts either prior to the load or after the
load has completed. This is useful for tidying up the data or providing a report on the
result of the load process once it completes.

It does not matter if the file to be loaded is a fixed or variable length text file, the
Advanced ETL Processor can handle it. It works with Access, DBF files, Oracle,
SQL Server and any ODBC compliant database.

It is one of the most powerful database loading tools in the marketplace.

2.9 Summary of the Loading Process:

Loading:

• Multiple Data Targets

• SQL scripts execution before and after loading
• Delimited or Fixed width Text files
• MS Access Database
• Microsoft Excel File
• DBF File
• Any ODBC compliant database
• Oracle database
• Microsoft SQL Server database
• MySQL
• PostgreSQL
• Interbase/Frebird

Oracle

There are two ways of loading available:

• Oracle direct path loading
• Conventional path

SQL server

This software uses the same API as Microsoft DTS and SSIS service.

Copyright © 2009 DB Software Laboratory Page 13 of 194

Advanced ETL Processor User Manual

3. Requirements

Below is the list of Software that must be installed before installation of Advanced
ETL Processor:

Software Version )otes

Microsoft Windows 98 or higher
MDAC 2.6 or higher Part of OS on W2K,
XP, Vista.
MS Excel ODBC 4.00.6305.00 or Part of OS on W2K, Only to work with
driver higher XP, Vista. Excel 3.0-2003 files
MS Excel 2007 12.00.4518.1014 or Separate download Only to work with
ODBC driver higher Excel 2007 files
MS Access ODBC 4.00.6364.00 or Part of OS on W2K, Only to work with MS
driver higher XP, Vista. Access 95-2003
Databases
MS Access 2007 12.00.4518.1014 or Separate download Only to work with MS
ODBC driver higher Access 2007
Databases
FoxPro ODBC 6.1.8629.1 or Separate download Only to work with
driver higher DBF/FoxPro Files
SQL ODBC driver 2000.81.9041.40 Part of OS on W2K, Only to work with
XP, Vista. MS SQL Server
7/2008
Oracle Client 7.3.4 or higher Provided by Oracle Only to work with
Oracle Databases
Interbase client GDS32.DLL Only to work with
Interbase or Firebird
Databases

Separate Downloads:

FoxPro ODBC driver

http://msdn.microsoft.com/en-us/vfoxpro/bb190233.aspx

Office 2007 Data Access Components

http://www.microsoft.com/downloads/details.aspx?FamilyID=7554F536-8C28-4598-9B72-
EF94E038C891&displaylang=en

Copyright © 2009 DB Software Laboratory Page 14 of 194

Advanced ETL Processor User Manual

Working with Oracle:

Oracle client 8.1.7 to load data into/from Oracle

Or
Oracle client 9 to load data into/from Oracle
Or
Oracle client 10 to load data into/from Oracle
Or
Oracle client 11 to load data into/from Oracle

)ote:
Depending on the Requirements you may or may not need to have all components installed
There is no need to install clients for MySql and PostgreSQL they are integrated into the
software itself.

Copyright © 2009 DB Software Laboratory Page 15 of 194

Advanced ETL Processor User Manual

4. Advanced ETL Processor Architecture

The following graphical depiction of a typical ETL installation shows how the
software interacts with other components and interfaces belonging to various
databases. The processor sits in the middle of the various databases and carries out
its tasks, such as converting, transforming and validating data from various sources.

As you can see that the Advanced ETL Processor uses native low level API's for
specific databases, such as the Oracle Call Interface (OCI), or, in order to handle the
Microsoft SQL Server database the BCP API. Other API's can also be "plugged in"
such as the ODBC API which allows MS Access, DBF to be processed.

Note:
One of major benefit of using native low level API' is a great performance boost it
gives to Advanced ETL Processor.

Oracle OCI API Oracle

SQL SQL
Advanced ETL
BCP API
ODBC ODBC

Excel Excel
ODBC
Reader
API
Validator
MS Access Transformer MS Access
Groupper
Sorter
DBF Files Deduplicator DBF Files
Pivot
Unpivot
MySQL Writer MySQL
Integrated
Clients
PostgreSQL PostgreSQL

Interbase Interbase

Flat Files Flat Files

Extraction Loading
Copyright © 2009 DB Software Laboratory Page 16 of 194
Advanced ETL Processor User Manual

5. Processing Data

In order to load data from the data source into the data target you must define data
mapping between target table and data source.

5.1 Screen Overview

The data processing screen provides a number of settings, for the three stages of
reading, transformation and writing. This is facilitated by the main tool bar, and a list
of available objects. The main reader toolbar provides the user with all the necessary
functionality to setup the reading process, according to the settings provided.

Main Tool bar

Available Objects

Source file/Table
Field Width
Reader Toolbar Field No Reader Fields Reader data

Copyright © 2009 DB Software Laboratory Page 17 of 194

Advanced ETL Processor User Manual

Main tool bar

The main tool bar provides a number of icons which allow the user to create the
steps required to carry out the automatic functions required.

1 2 3 4 5 6 7 8

1. Map Properties
2. New Map
3. Load Template From the file
4. Saves Template to the file
5. Saves Template under the new name
6. Prints Map
7. Print Preview Map
8. Process Data

Copyright © 2009 DB Software Laboratory Page 18 of 194

Advanced ETL Processor User Manual

5.2 Transformation Properties

Click to change Transformation properties, please note that Default Date format
applies to all templates. Default Date format only used for new Date related validation
and transformation functions for example Is Date, Is Date between ETC

Copyright © 2009 DB Software Laboratory Page 19 of 194

Advanced ETL Processor User Manual

5.3 Template tab

The template shows the actual transformation script source.

Copyright © 2009 DB Software Laboratory Page 20 of 194

Advanced ETL Processor User Manual

5.4 Execution Log Tab

The execution log provides information about the Advanced ETL Processor and
the actions it took during its operations. This is useful when you wish to analyse the
activities of individual processes during their execution.

Copyright © 2009 DB Software Laboratory Page 21 of 194

Advanced ETL Processor User Manual

5.5 Rejected Records Tab

Occasionally, records will be rejected by the Advanced ETL Processor. This may
be due to things like corrupt records which have been read in a format not expected
by the processor. The rejected records tab allows the user to see a list of all the
rejected records, and other information about the nature of the rejection and where
this occurred in the process.

Copyright © 2009 DB Software Laboratory Page 22 of 194

Advanced ETL Processor User Manual

5.6 Creating new transformation and working with Objects

To create new transformation click . New transformation consists of Reader,

Validator, Transformer and Writer. You may add new object at any time by dragging
it from the Objects panel.

Objects Panel

To join two objects together, first click on the source Output button then drag it on to
the Input button

Output Buttons

Drag and Drop to Join

Input Button

To remove the Join, repeat the same procedure again.

To delete an object right click on the object and click delete.

Copyright © 2009 DB Software Laboratory Page 23 of 194

Advanced ETL Processor User Manual

To change object properties double click on it

Deletes All Links Deletes Links for Selected objects

Copyright © 2009 DB Software Laboratory Page 24 of 194

 Advanced ETL Processor User Manual

5.7 Working with Reader

The Reader extracts data from various data sources.

Description

Records/ Lines Read Records Rejected

Every Transformation created must have one reader object. The Reader connects to
the Data source and extracts data from it. Depending on the Data source type some
options may not be available. To change the Reader properties click or double
click on the Reader object.

5.7.1 Universal Data Reader

One of the benefits of using Advanced ETL Processor is a concept of Universal

Data Reader

Consider following scenario:

You have several extracts from your accounting system

• Text files
• Excel Files
• Access Databases

All files have same format such as:

• Same field’s order
• Same field’s names
• Same date and number formats

Provided that you are using Advanced ETL Processor all you need to do is to
change connection type no mapping will be lost.

Other tools use different connector for different databases and some of then even
sell separate licences for them. That mean that user have to recreate mapping for
new files/ databases.

Copyright © 2009 DB Software Laboratory Page 25 of 194

Advanced ETL Processor User Manual

5.7.2 Data source is a Text File.

The Reader is capable of extracting data from delimited or fixed width files. All
parameters are user definable. It can also skip a number of Header and Footer lines

Copyright © 2009 DB Software Laboratory Page 26 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 27 of 194

Advanced ETL Processor User Manual

The Rejected records file can have a pre-defined format. The following dialogue
allows the user to set this up.

Copyright © 2009 DB Software Laboratory Page 28 of 194

Advanced ETL Processor User Manual

Data View

One of the useful features of the Advanced ETL Processor is the ability to view the
resultant data prior, or subsequently processed. The data view looks like a
spreadsheet view as follows:

Source file

Number of records to show Field Width Field Number Field Name

Data View Toolbar

The data view toolbar allows you to change various aspects of the data view, such
as refreshing data as it changes and setting properties for how the data will look. You
can also switch between viewing of the data and checking to see how the data is
defined i.e. the data dictionary, via the “Switch to Data Definition View”.

1 2 3 4 5 6 7

1. Reader Properties
2. Refresh Data
3. Edit file in external editor
4. Add new column
5. Delete last column
6. Switch to Data View
7. Switch to Data Definition View

Copyright © 2009 DB Software Laboratory Page 29 of 194

Advanced ETL Processor User Manual

Data Definition View

)ote:
You may rename fields and change field’s width here. (Works only for text files)

Data Definition View Toolbar

Within the Data Definition view you can perform a number of actions. These allow
you to change how you want data to be represented in the data view screens. The
navigation also allows switching between views.

1 2 3 4 5 6 7 8 9 10

1. Reader Properties
2. Refresh Data
3. Print Data Definition
4. Print Preview Data Definition
5. Find
6. Edit file in external editor
7. Add new column
8. Delete last column
9. Switch to Data View
10. Switch to Data Definition View

Copyright © 2009 DB Software Laboratory Page 30 of 194

Advanced ETL Processor User Manual

5.7.3 Data source is MS Access or Excel File

Copyright © 2009 DB Software Laboratory Page 31 of 194

Advanced ETL Processor User Manual

Query Builder

)ote:
It is also possible to use Query builder to design queries

Copyright © 2009 DB Software Laboratory Page 32 of 194

Advanced ETL Processor User Manual

Data View

Copyright © 2009 DB Software Laboratory Page 33 of 194

Advanced ETL Processor User Manual

Data View Toolbar

1 2 3 4 5

1. Reader Properties
2. Refresh Data
3. Edit file in external editor
4. Switch to Data View
5. Switch to Data Definition View

Data Definition View

Data Definition View Toolbar

1 2 3 4 5 6 7 8

1. Reader Properties
2. Refresh Data
3. Print Data Definition
4. Print Preview Data Definition
5. Find
6. Edit file in external editor
7. Switch to Data View
8. Switch to Data Definition View

Copyright © 2009 DB Software Laboratory Page 34 of 194

Advanced ETL Processor User Manual

5.7.4 Data source is a DBF File

Source Directory

Copyright © 2009 DB Software Laboratory Page 35 of 194

Advanced ETL Processor User Manual

Data View
Source Directory

Data View Toolbar

1 2 3 4

1. Reader Properties
2. Refresh Data
3. Switch to Data View
4. Switch to Data Definition View

Data Definition View

Data Definition View Toolbar

1 2 3 4 5 6 7

1. Reader Properties
2. Refresh Data
3. Print Data Definition
4. Print Preview Data Definition
5. Find
6. Switch to Data View
7. Switch to Data Definition View

Copyright © 2009 DB Software Laboratory Page 36 of 194

Advanced ETL Processor User Manual

5.7.5 Data source is a ODBC Connection

Copyright © 2009 DB Software Laboratory Page 37 of 194

Advanced ETL Processor User Manual

ODBC Connection Strings

It also possible to use ODBC connection strings for both Reader and Writer
connections.
For example for MS SQL Server connection string is:

Driver={SQL Native Client};

Server=myServerAddress;Database=myDataBase;Uid=myUsername;Pwd=myPassword;

One of the major benefits of using connection strings that it makes it no longer necessary to
create ODBC Dsn’s manually on every single computer where Advanced ETL Processor is
installed. It also gives a greater control over the connection parameters.

)ote:
Leave user name and password blank and provide it within connection string

More information about connection strings can be found at:

http://www.connectionstrings.com

The simplest way to create ODBC connection string is to use ODBC Connection builder
dialog. Double click on ODBC driver name to create a connection string

Copyright © 2009 DB Software Laboratory Page 38 of 194

Advanced ETL Processor User Manual

Data View

Source Table

Data View Toolbar

1 2 3 4

1. Reader Properties
2. Refresh Data
3. Switch to Data View
4. Switch to Data Definition View

Data Definition View

Data Definition View Toolbar

1 2 3 4 5 6 7

1. Reader Properties
2. Refresh Data
3. Print Data Definition
4. Print Preview Data Definition
5. Find
6. Switch to Data View
7. Switch to Data Definition View

Copyright © 2009 DB Software Laboratory Page 39 of 194

Advanced ETL Processor User Manual

5.7.6 Data source is a MS SQL Server

)ote:
All properties are very similar to ODBC connection

Copyright © 2009 DB Software Laboratory Page 40 of 194

Advanced ETL Processor User Manual

5.7.7 Data source is an Oracle Database

)ote:
All properties are very similar to ODBC connection

Copyright © 2009 DB Software Laboratory Page 41 of 194

Advanced ETL Processor User Manual

5.7.8 Data source is an MySql

)ote:
All properties are very similar to MS SQL Server connection

Copyright © 2009 DB Software Laboratory Page 42 of 194

Advanced ETL Processor User Manual

5.7.9 Data source is an PostgreSQL Database

)ote:
All properties are very similar to MS SQL Server connection

Copyright © 2009 DB Software Laboratory Page 43 of 194

Advanced ETL Processor User Manual

5.7.10 Data source is an Interbase or Firebird Database

)ote:
All properties are very similar to ODBC/MS Access connection

Copyright © 2009 DB Software Laboratory Page 44 of 194

 Advanced ETL Processor User Manual

5.8 Working with Validator

Advanced ETL Processor Data Validator guarantees to your application database

that every data value is correct and accurate.

There are several types of data validation.

• Data type validation

• Range checking
• Code checking
• Complex validation
• Pattern checking

One of the simplest forms of data validation is verifying the data type. Data type
validation answers such simple questions as "Is the string alphabetic?" and "Is the
number valid?"

As an extension of simple type validation, range checking ensures that the provided
value is within allowable minimums and maximums. For example, a character data
type service code may only allow the alphabetic letters A through Z. All other
characters would not be valid.

Code checking is a bit more complicated, typically requiring a lookup table. For
example, maybe your application calculates sales tax for only certain state codes.
You would need to create a lookup object to hold the authorized, taxable state codes.

Pattern checking when you checking structure of the data field for example social
security number format or car registration number. Regular expressions used quite
often for pattern checks.

It does not matter which business you are in sooner or later you will discover that
there is something wrong with the data ant it has to be validated. Here when
Advanced ETL Processor Validation can help.
Records Rejected by validation Rules

Processed Records

Discarded Records

)ote:
• Records can also be rejected by the Server.
• If you have several validation rules and one of them rejects record and another
discards it, record will be discarded

Copyright © 2009 DB Software Laboratory Page 45 of 194

Advanced ETL Processor User Manual

To change Validator properties double click on it

Inputs
Validator Toolbar Objects Panel
Validation Rules

Data Sample

Copyright © 2009 DB Software Laboratory Page 46 of 194

Advanced ETL Processor User Manual

Validator Toolbar

1 2 3 4 5 6 7 8 9 10 11

1. Print
2. Print Preview
3. Delete All objects
4. Delete All Links
5. Process Data
6. First Record
7. Previous Record
8. Next Record
9. Last Record
10. Show Data
11. Show Objects Panel

5.8.1 Debugging Validation

To start debugging validation press Process Data button . To test data edit it in
the Data grid.

Pass Warning Failed

To test different value edit data in the Data grid

Copyright © 2009 DB Software Laboratory Page 47 of 194

Advanced ETL Processor User Manual

To Change Validation Rule properties double click on it

)ote:
Use <value> to include actual value into default value

To add new Validation rule drag and drop it from the Validation rules panel

Copyright © 2009 DB Software Laboratory Page 48 of 194

Advanced ETL Processor User Manual

There more than 190 Validation Functions at the moment. They are grouped in five
different categories

1. String
2. Number
3. Date
4. Time
5. Regular Expressions

To choose appropriate category click on Category toolbar

It is also possible to apply several validation rules to Input field by joining them

Data is considered as validated when all validation rules are succeeded.

Validated Data

Failed Data

)ote:
If you have several validation rules and one of them rejects record and another
discards it, record will be discarded

Copyright © 2009 DB Software Laboratory Page 49 of 194

Advanced ETL Processor User Manual

5.9 Working with Transformer

Records Transformed

)ote:
To change Transformer properties double click on it

Transformer Toolbar Inputs

Outputs Objects Panel

Data Sample Transformations

Copyright © 2009 DB Software Laboratory Page 50 of 194

Advanced ETL Processor User Manual

Transformer Toolbar

1 2 3 4 5 6 7 8 9 10 11 12 13

1. Print
2. Print Preview
3. Auto Map
4. Delete All objects
5. Delete All Links
6. Process Data
7. First Record
8. Previous Record
9. Next Record
10. Last Record
11. Copies Inputs to Outputs (Only visible if transformer is connected to any object other
than writer)
12. Show Data
13. Show Objects Panel

)ote:
If transformer is connected to any object other than writer it is possible to modify
list of Outputs. When transformer is connected to writer list of Outputs is taken
from Writer.

Examples:

Example below splits date field into Day, Month and Year using ‘/’ as a delimiter

Another Example Converts Customer Name into More readable format:

Copyright © 2009 DB Software Laboratory Page 51 of 194

Advanced ETL Processor User Manual

5.9.1 Auto mapping

If the Inputs and Outputs have got the same names you may use Auto map feature.

Click , Fill in all necessary data and click map.

Copyright © 2009 DB Software Laboratory Page 52 of 194

Advanced ETL Processor User Manual

5.10 Working with Grouper

Records Out
Records In

To change grouper properties double click on the object. Choose appropriate

grouping order and grouping type. Fields with grouping type Ignore are not passed to
the next object. Grouper does not perform sorting.

Copyright © 2009 DB Software Laboratory Page 53 of 194

Advanced ETL Processor User Manual

5.11 Working with Sorter

Records Out
Records In

About Sorting

In computer science and mathematics, a sorting algorithm is an algorithm that puts

elements of a list in a certain order. All sorting algorithms are very well documented
and it is hard to invent something new or much faster that existing algorithm. To
achieve the best performance is important to load all the data into the memory which
is not possible for large amount of data. Advanced ETL Processor uses
combination of Quicksort and Merge sort. Portion of Data is loaded in the memory
and sorted using Quicksort algorithm than data is saved into temporary file. Once all
portions are sorted temporary files merged together using Merge sort algorithm.

Quicksort

Quicksort is a divide and conquer algorithm which relies on a partition operation: to

partition an array, we choose an element, called a pivot, move all smaller elements
before the pivot, and move all greater elements after it. This can be done efficiently in
linear time and in-place. We then recursively sort the lesser and greater sublists.
Efficient implementations of quicksort (with in-place partitioning) are typically unstable
sorts and somewhat complex, but are among the fastest sorting algorithms in
practice. Together with its modest O(log n) space usage, this makes quicksort one of
the most popular sorting algorithms, available in many standard libraries. The most
complex issue in quicksort is choosing a good pivot element; consistently poor
choices of pivots can result in drastically slower (O(n²)) performance, but if at each
step we choose the median as the pivot then it works in O(n log n).

Merge sort

Merge sort takes advantage of the ease of merging already sorted lists into a new
sorted list. It starts by comparing every two elements (i.e., 1 with 2, then 3 with 4...)
and swapping them if the first should come after the second. It then merges each of
the resulting lists of two into lists of four, then merges those lists of four, and so on;
until at last two lists are merged into the final sorted list. Of the algorithms described
here, this is the first that scales well to very large lists, because its worst-case running
time is O(n log n).

Copyright © 2009 DB Software Laboratory Page 54 of 194

Advanced ETL Processor User Manual

To change Sorter properties double click on the object. Tick sort, select field order
and sort order. Field order must be unique for sort field. Data is loaded in the memory
first than sorted and passed to the next object.

Copyright © 2009 DB Software Laboratory Page 55 of 194

Advanced ETL Processor User Manual

5.12 Working with Deduplicator

Records Out
Records In

To change Deduplicator properties double click on the object. Tick Deduplicate for
the fields you wish to depuplicate, only ticked fields are passed to the next object.

)ote:
There is no need to sort data before deduplication

Copyright © 2009 DB Software Laboratory Page 56 of 194

Advanced ETL Processor User Manual

5.13 Working with UnPivot

Records In Records In

Example of Pivoted Data

Group 1 Group 2 Group 3 Group 4

To change UnPivot properties double click on the object. Fill in Description, Group
field name, create all necessary Groups and Outputs. Once it is done Map input fields
to outputs and Groups

UnPivoted Data
Group field

Copyright © 2009 DB Software Laboratory Page 57 of 194

Advanced ETL Processor User Manual

UnPivot Properties

Copyright © 2009 DB Software Laboratory Page 58 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 59 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 60 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 61 of 194

Advanced ETL Processor User Manual

5.14 Working with Pivot

Records In Records Out

UnPivoted Data Fields to Pivot

Pivot Key

Set Key

Pivoted Data

Copyright © 2009 DB Software Laboratory Page 62 of 194

Advanced ETL Processor User Manual

Since there are more than one field to pivot let’s join Profit and Sales fields together first
using tab a delimiter than pivot the data and split it again

Joining Data

Copyright © 2009 DB Software Laboratory Page 63 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 64 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 65 of 194

Advanced ETL Processor User Manual

Data after Pivoting

Splitting Pivoted Data

Copyright © 2009 DB Software Laboratory Page 66 of 194

Advanced ETL Processor User Manual

5.15 Working with Writer

Inserted Records Updated Records

Deleted Records Errors

Every Transformation created must have at least one writer object. Writer connects
to Target database and loads data into it. Depending of the target type some options
may not be available. To change the Writer properties click or double click on the
Writer object.

5.15.1 Target type is a Text File

Writer is capable of saving data into delimited or fixed width files. All parameters are
user definable.

Copyright © 2009 DB Software Laboratory Page 67 of 194

Advanced ETL Processor User Manual

Data View

Target File
Number of records to show

Copyright © 2009 DB Software Laboratory Page 68 of 194

Advanced ETL Processor User Manual

Data View Toolbar

1 2 3 4 5 6 7

1. Reader Properties
2. Refresh Data
3. Edit file in external editor
4. Add new column
5. Delete last column
6. Switch to Data View
7. Switch to Data Definition View

Data Definition View

)ote:
You may rename fields and change field’s width here. (Works only for text files)

Data Definition View Toolbar

1 2 3 4 5 6 7 8 9 10

1. Reader Properties
2. Refresh Data
3. Print Data Definition
4. Print Preview Data Definition
5. Find
6. Edit file in external editor
7. Add new column
8. Delete last column
9. Switch to Data View
10. Switch to Data Definition View

Copyright © 2009 DB Software Laboratory Page 69 of 194

Advanced ETL Processor User Manual

5.15.2 Target type is MS Access or Excel File

Data View

Copyright © 2009 DB Software Laboratory Page 70 of 194

Advanced ETL Processor User Manual

Data View Toolbar

1 2 3 4 5 6 7 8

1. Reader Properties
2. Refresh Data
3. Print
4. Print Preview
5. Find
6. Edit file in external editor
7. Switch to Data View
8. Switch to Data Definition View

Data Definition View

Data Definition View Toolbar

1 2 3 4 5 6 7 8

1. Reader Properties
2. Refresh Data
3. Print Data Definition
4. Print Preview Data Definition
5. Find
6. Edit file in external editor
7. Switch to Data View
8. Switch to Data Definition View

Copyright © 2009 DB Software Laboratory Page 71 of 194

Advanced ETL Processor User Manual

5.15.3 Target type is a DBF File

Data View

Copyright © 2009 DB Software Laboratory Page 72 of 194

Advanced ETL Processor User Manual

Data View Toolbar

1 2 3 4 5 6 7

1. Reader Properties
2. Refresh Data
3. Print
4. Print Preview
5. Find
6. Switch to Data View
7. Switch to Data Definition View

Data Definition View

Copyright © 2009 DB Software Laboratory Page 73 of 194

Advanced ETL Processor User Manual

5.15.4 Target type is ODBC connection

Copyright © 2009 DB Software Laboratory Page 74 of 194

Advanced ETL Processor User Manual

5.15.5 Target type is MS SQL Server

Check constraints

Ensure that any constraints on the destination table are checked during the bulk copy
operation. By default, constraints are ignored.

Keep identity
Specify that there are values in the data file for an identity column.

Keep )ULLS
Specify that any columns containing a null value should be retained as null values, even if a
default value was specified for that column in the destination table.

Batch size

Specify the number of rows in a batch. The default is the entire data file.

The following values for the Batch size property have these effects:

If you set Batch size to zero, the data is loaded in a single batch. The first row that fails will
cause the entire load to be cancelled, and the step fails.

Copyright © 2009 DB Software Laboratory Page 75 of 194

Advanced ETL Processor User Manual

If you set Batch size to one, the data is loaded a row at a time. Each row that fails is counted
as one row failure. Previously loaded rows are committed.

If you set Batch size to a value greater than one, the data is loaded one batch at a time. Any
row that fails in a batch fails that entire batch; loading stops and the step fails. Rows in
previously loaded batches are either committed or, if the step has joined the package
transaction, provisionally retained in the transaction, subject to later commitment or rollback.

Copyright © 2009 DB Software Laboratory Page 76 of 194

Advanced ETL Processor User Manual

5.15.6 Target type is Oracle

)ote:
Option ‘Commit every Array’ works only for Oracle conventional path loading

Copyright © 2009 DB Software Laboratory Page 77 of 194

Advanced ETL Processor User Manual

5.15.7 Target type is MySql

Copyright © 2009 DB Software Laboratory Page 78 of 194

Advanced ETL Processor User Manual

5.15.8 Target type is PostgreSQL

Copyright © 2009 DB Software Laboratory Page 79 of 194

Advanced ETL Processor User Manual

5.15.9 Target type is Interbase/Firebird

Copyright © 2009 DB Software Laboratory Page 80 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 81 of 194

Advanced ETL Processor User Manual

5.15.10 How to Update/Delete Records

In order to Update/Delete records you must specify update key.

Update Key

For the example provided below, Advanced ETL Processor will execute the
following SQL
(Update key is CustomerId, OrderNo).

Add )ew And Update Old Records

Select count(*)
from [DEMO].[dbo].[orders]
where CustomerId=? And OrderNo=?

If any records found Advanced ETL will update them by executing

Copyright © 2009 DB Software Laboratory Page 82 of 194

Advanced ETL Processor User Manual

Update [DEMO].[dbo].[orders]
set orderdate=?,
amount=?
where customerid=? And OrderNo=?

If no records found Advanced ETL will add new records

Update Records

Update [DEMO].[dbo].[orders]
set OrderDate=?,
Amount=?
where CustomerId=? And OrderNo=?

Delete Records

Delete from [DEMO].[dbo].[orders]

Where CustomerId=? And OrderNo=?

)ote:
“Add New And Update Old Records” is not supported for SQL Server
Connection use ODBC connection instead.

Copyright © 2009 DB Software Laboratory Page 83 of 194

Advanced ETL Processor User Manual

5.15.11 Running SQL Scripts

Advanced ETL Processor is capable of running SQL Scripts before and after the
transformation.

SQL Editor Toolbar

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1. Loads Script from a file

2. Saves Script to a file
3. Print
4. Print Preview
5. Clear Script
6. Cut
7. Copy
8. Paste
9. Undo
10. Find
11. Find Next
12. Find Previous
13. Replace
14. Run
15. Stop on error
16. Show/Hide Log

Copyright © 2009 DB Software Laboratory Page 84 of 194

Advanced ETL Processor User Manual

6. Validation Rules

This chapter represents list of validation rules together with short description
grouped by category. Every Validation Rule/function could have some specific
parameters and common parameters to control data flow of Advanced ETL. Most if it
is self explanatory see the example below:

)ote:
To change validation rule properties double click on it

Copyright © 2009 DB Software Laboratory Page 85 of 194

Advanced ETL Processor User Manual

6.1 Strings

6.1.1 Is Null

Category: Validation Function

Description: Checks if the data is equal to null.
Properties: Default

6.1.2 Is Empty String

Category: Validation Function

Description: Checks if the data is Empty String. (Length of the string is zero).
Properties: Default

6.1.3 Is Alpha

Category: Validation Function

Description: Checks if the data contains only Alpha characters.
Properties: Default
Pass: FCD|||mmm|||ABC
Fail: adc1|||ad b||12 D

6.1.4 Is Alpha Numeric

Category: Validation Function

Description: Checks if the data contains only Alpha Numeric characters.
Properties: Default
Pass: FCD12|||mm11m|||1ABC
Fail: adc-1|||ad_b||12 D

Copyright © 2009 DB Software Laboratory Page 86 of 194

Advanced ETL Processor User Manual

6.1.4 Is Hex

Category: Validation Function

Description: Checks if the data contains valid HEX string
Properties: Default
Pass: 3D|||FF|||CE
Fail: DK|||KK||12D

6.1.5 Is Equal To

Category: Validation Function

Description: Checks if the data is equal to supplied value
Properties:

Copyright © 2009 DB Software Laboratory Page 87 of 194

Advanced ETL Processor User Manual

6.1.6 Is Starts With

Category: Validation Function

Description: Checks if the data starts with supplied value
Properties: Same as ‘Is Equal To’

6.1.7 Is Ends With

Category: Validation Function

Description: Checks if the data ends with supplied value
Properties: Same as ‘Is Equal To’

6.1.8 Contains

Category: Validation Function

Description: Checks if the data contains supplied value
Properties: Same as ‘Is Equal To’

6.1.9 In List

Category: Validation Function

Description: Checks if the data exists within supplied list values
Properties: List of values can be entered by the user or extracted for the file(s) or database.
When source data for the “List of values” consists of more than 1 field only first one is used.

Copyright © 2009 DB Software Laboratory Page 88 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 89 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 90 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 91 of 194

Advanced ETL Processor User Manual

6.2 Numbers

6.2.1 Is Number

Category: Validation Function

Description: Checks if the data is a Number
Properties: Default
Pass: 12|||123.6|||1
Fail: adc1|||ad b||11,112.D

6.2.2 Is Integer

Category: Validation Function

Description: Checks if the data is an Integer
Properties: Default
Pass: 12|||123|||1
Fail: 12.5|||ad b||11,112.8

6.2.3 Is Positive

Category: Validation Function

Description: Checks if the data is a Positive Number
Properties: Default

6.2.4 Is Negative

Category: Validation Function

Description: Checks if the data is a Negative Number
Properties: Default

Copyright © 2009 DB Software Laboratory Page 92 of 194

Advanced ETL Processor User Manual

6.2.5 Is Number Between

Category: Validation Function

Description: Checks if the data is a number between two values
Properties:

Copyright © 2009 DB Software Laboratory Page 93 of 194

Advanced ETL Processor User Manual

6.2.6 Is Number Less Than

Category: Validation Function

Description: Checks if the data is a number less than value
Properties:

Copyright © 2009 DB Software Laboratory Page 94 of 194

Advanced ETL Processor User Manual

6.2.7 Is Number More Than

Category: Validation Function

Description: Checks if the data is a number more than value
Properties:

Copyright © 2009 DB Software Laboratory Page 95 of 194

Advanced ETL Processor User Manual

6.3 Date

6.3.1 Is Date

Category: Validation Function

Description: Checks if the data is a Date
Properties:

Copyright © 2009 DB Software Laboratory Page 96 of 194

Advanced ETL Processor User Manual

6.3.2 Is Date Between

Category: Validation Function

Description: Checks if the data is a Date between two values
Properties:

Copyright © 2009 DB Software Laboratory Page 97 of 194

Advanced ETL Processor User Manual

6.3.3 Is Date Less Than

Category: Validation Function

Description: Checks if the data is a Date less than value
Properties:

6.3.4 Is Date More Than

Category: Validation Function

Description: Checks if the data is a Date more than value
Properties: Same as ‘Is Date Less Than’

Copyright © 2009 DB Software Laboratory Page 98 of 194

Advanced ETL Processor User Manual

6.3.5 Is Monday, Is Tuesday, Is Wednesday, Is Thursday, Is Friday, Is Saturday,

Is Sunday, Is Today, Is Yesterday, Is Tomorrow, Is Weekend, Is Weekday

Category: Validation Function

Description: Checks if the data is a Date value which belongs to specific day of week
Properties: Same as ‘Is Date’

6.3.6 Is January, Is February, Is March, Is April, Is May, Is June, Is July, Is

August, Is September, Is October, Is November, Is December, Is Current
Month, Is Last Month, Is Next Month

Category: Validation Function

Description: Checks if the data is a Date value which belongs to specific month of a year
Properties: Same as ‘Is Date’

Copyright © 2009 DB Software Laboratory Page 99 of 194

Advanced ETL Processor User Manual

6.3.7 Is 1st Quarter, Is 2nd Quarter, Is 3rd Quarter, Is 4th Quarter, Is Current
Quarter,
Is Last Quarter, Is Next Quarter

Category: Validation Function

Description: Checks if the data is a Date value which belongs to specific quarter of a year
Properties: Same as ‘Is Date’

6.3.8 Is Leap Year, Is Current Year, Is Last Year, Is Next Year

Category: Validation Function

Description: Checks if the data is a Date value which belongs to specific year
Properties: Same as ‘Is Date’

6.3.9 Is Current Week, Is Last Week, Is Next Week

Category: Validation Function

Description: Checks if the data is a Date value which belongs to specific week
Properties: Same as ‘Is Date’

Copyright © 2009 DB Software Laboratory Page 100 of 194

Advanced ETL Processor User Manual

6.3.10 Is Within Past Minutes, Is Within Past Hours, Is Within Past Days, Is
Within Past Weeks, Is Within Past Months

Category: Validation Function

Description: Checks if the data is a Date value within in past minutes etc.
Properties:

Copyright © 2009 DB Software Laboratory Page 101 of 194

Advanced ETL Processor User Manual

6.4 Time

6.4.1 Is Time

Category: Validation Function

Description: Checks if the data is a time value
Properties:

Copyright © 2009 DB Software Laboratory Page 102 of 194

Advanced ETL Processor User Manual

6.4.2 Is Time Between

Category: Validation Function

Description: Checks if the data is a time between two values
Properties:

Copyright © 2009 DB Software Laboratory Page 103 of 194

Advanced ETL Processor User Manual

6.4.3 Is Time Less Than

Category: Validation Function

Description: Checks if the data is a time less than value
Properties:

6.4.4 Is Time More Than

Category: Validation Function

Description: Checks if the data is a time more than value
Properties: Same as ‘Is Time Less Than’

Copyright © 2009 DB Software Laboratory Page 104 of 194

Advanced ETL Processor User Manual

6.4.5 Is Second

Category: Validation Function

Description: Checks if data is a number between 0 and 59
Properties: Default

6.4.6 Is Minute

Category: Validation Function

Description: Checks if data is a number between 0 and 59
Properties: Default

6.4.7 Is Hour 24

Category: Validation Function

Description: Checks if data is a number between 0 and 23
Properties: Default

6.4.8 Is Hour 12

Category: Validation Function

Description: Checks if data is a number between 0 and 12
Properties: Default

Copyright © 2009 DB Software Laboratory Page 105 of 194

Advanced ETL Processor User Manual

6.4.9 Is PM

Category: Validation Function

Description: Checks if the data is a PM Date
Properties: Same as ‘Is Date’

6.4.10 Is AM

Category: Validation Function

Description: Checks if the data is a AM Date
Properties: Same as ‘Is Date’

Copyright © 2009 DB Software Laboratory Page 106 of 194

Advanced ETL Processor User Manual

6.5 Regular Expressions

Regular expression is a string that is used to describe or match a set of strings,

according to certain syntax rules.

Regular expressions are used by many text editors, utilities, and programming
languages to search and manipulate text based on patterns. For example, Perl and
Tcl have a powerful regular expression engine built directly into their syntax. Several
utilities provided by Unix distributions—including the editor ed and the filter grep—
were the first to popularize the concept of regular expressions. "Regular expression"
is often shortened to regex or regexp (singular), or regexes, regexps, or regexen
(plural). Some authors distinguish between regular expression and abbreviated forms
such as regex, restricting the former to true regular expressions, which describe
regular languages, while using the latter for any regular expression-like pattern,
including those that describe languages that are not regular. As only some authors
observe this distinction, it is not safe to rely upon it.

As an example of the syntax, the regular expression \bex can be used to search for
all instances of the string "ex" that occur at word boundaries (signified by the \b).
Thus in the string, "Texts for experts," \bex matches the "ex" in "experts," but not in
"Texts" (because the "ex" occurs inside the word there and not immediately after a
word boundary).

Source: Wikipedia.

For more information on regular expressions please visit:

www.regularexpression.info

Most of regular expressions were taken from www.regexlib.com. We are not able to
include full list of contributors because it is too big. We did our best to test and modify
if necessary pattern strings however as time goes by standards changes and some of
them could became no longer valid. Therefore Regular Expression Validation Rules
should be used with caution. If you have any useful Regular expressions please let
us know we would be more than happy to include it into the next release of
Advanced ETL Processor.

Copyright © 2009 DB Software Laboratory Page 107 of 194

Advanced ETL Processor User Manual

6.5.1 Regular Expression

Category: Validation Function

Description: Validates data against regular expression
Properties:

Copyright © 2009 DB Software Laboratory Page 108 of 194

Advanced ETL Processor User Manual

6.5.2 Is IP Address V4

Category: Validation Function

Description: Validates IP addresses in the dotted quad form
Properties: Default
Pass: 192.168.1.1|||10.2.234.1|||66.129.71.122
Fail: 192.168.1|||10.2.1234.1|||66 129 71 122

6.5.3 Is IP Address V6

Category: Validation Function

Description: Validates all IPv6 text representations as defined within RFC 2373
Properties: Default
Pass: ::0:0:0:FFFF:129.144.52.38|||FEDC:BA98::3210:FEDC:BA98:7654:3210|||::13.1.68.3
Fail:
FEDC:BA98:7654:3210:FEDC:BA98:7654:3210:1234|||3210:FEDC:BA98:7654:3210:1234|||:
FEDC:BA98:7654:3210:

6.5.4 Is Email

Category: Validation Function

Description: Email Address validation that allows both IP addresses and regular domains. In
the case of an IP address it makes sure that it is no more than 255 for each part.
Properties: Default
Pass: [email protected]|||[email protected]|||[email protected]
Fail: john@doe|||[email protected]|||@john.doe.com

6.5.5 Is ISBN 10

Category: Validation Function

Description: Validation of 10 digits ISBN. The ISBN number must be preceded by the text
"ISBN:" or "ISBN-10:", the colon is optional.
Properties: Default
Pass: ISBN-10: 0-93028-923-4
Fail: ISBN-13: 978-0-5960-0289-3

Copyright © 2009 DB Software Laboratory Page 109 of 194

Advanced ETL Processor User Manual

6.5.6 Is ISBN 13

Category: Validation Function

Description: Validation of new 13 digits ISBN. The ISBN number must be preceded by the
text "ISBN:" or "ISBN-13:", the colon is optional.
Properties: Default
Pass: ISBN-13: 978-0-5960-0289-3
Fail: ISBN-10: 0596002890

6.5.7 Is Credit Card

Category: Validation Function

Description: Validation of all major credit cards formats
Properties: Default
Pass: 341-1111-1111-1111|||5431-1111-1111-1111|||30569309025904
Fail: 30-5693-0902-5904|||5631-1111-1111-1111|||31169309025904

6.5.8 Is URL

Category: Validation Function

Description: Validation of URL
Properties: Default
Pass: http://www.download.com
Fail: http://999.999.999.999/dfd

6.5.9 Is UNC

Category: Validation Function

Description: Validation of UNC
Properties: Default
Pass: \\server\c$
Fail: //server

Copyright © 2009 DB Software Laboratory Page 110 of 194

Advanced ETL Processor User Manual

6.5.10 Is File Name

Category: Validation Function

Description: Validation of file name format, file name must have a drive letter and an
extension
Properties: Default
Pass: c:\1.txt
Fail: ftc

6.5.11 Is UK Post Code

Category: Validation Function

Description: Validates United Kingdom Post Code
Properties: Default
Pass: SW1W 8EU
Fail: SW1WK8EU

6.5.12 Is UK VAT Number

Category: Validation Function

Description: United Kingdom VAT Numbers format verification with support for optional
member state definition.
Properties: Default
Pass: GB123123412|||GBGD123|||123123412123
Fail: GB1231234124|||GBGD1234|||023 1234 12 123

6.5.13 Is UK Bank Sort Code

Category: Validation Function

Description: Validation of United Kingdom Bank Sort code
Properties: Default
Pass: 09-01-29|||05-06-25
Fail: 90125

Copyright © 2009 DB Software Laboratory Page 111 of 194

Advanced ETL Processor User Manual

6.5.14 Is UK Vehicle Registration Number

Category: Validation Function

Description: Validation of United Kingdom Vehicle registration Number
Properties: Default
Pass: ABC 123 A|||A 123 ABC|||AB 12 ABC
Fail: AB 123 C|||A 123 AB|||AB 12 AB

6.5.15 Is UK Driver Licence

Category: Validation Function

Description: Validates the United Kingdom Drivers License format as described by the
DVLA.
See http://www.govtalk.gov.uk/gdsc/html/frames/default.htm.
Matches: 1. Must be 16 characters 2. First 5 characters are alphanumeric. 3. Next 6 characters
must be numeric 4. Next 3 characters are alphanumeric 5. Last 2 characters are alpha 6.
Second character of numeric section can only be 0, 1, 5 or 6. 7. Fourth and fifth characters of
numeric section must be in the range 01 to 31
Properties: Default
Pass: JOHNS711215GG9SY
Fail: JOHNS731215GG9SY

6.5.16 Is UK National Insurance Number

Category: Validation Function

Description: Validation of United Kingdom National Insurance Number
Properties: Default
Pass: SP939393H|||PX123456D|||SW355667G
Fail: 12SP9393H|||S3P93930D|||11223344SP00ddSS

6.5.17 Is UK Phone Number

Category: Validation Function

Description: Validation of United Kingdom phone number, also allows +44 national code in
addition to extension numbers of 3 or 4 digits
Properties: Default
Pass: +447222555555|||+44 7222 555 555|||(0722) 5555555 #2222
Copyright © 2009 DB Software Laboratory Page 112 of 194
Advanced ETL Processor User Manual

Fail: (+447222)555555|||+44(7222)555555|||(0722) 5555555 #22

6.5.18 Is UK Mobile Number

Category: Validation Function

Description: United Kingdom mobile phone number validation with optional +44 national
code, also allows optional brackets and spaces at appropriate positions
Properties: Default
Pass: 07222 555555|||(07222) 555555|||+44 7222 555 555
Fail: 7222 555555|||+44 07222 555555|||(+447222) 555555

6.5.19 Is US Zip Code

Category: Validation Function

Description: Validates US Zip Code
Properties: Default
Pass: 48222|||48222-1746
Fail: 4632|||Blake|||37333-32

6.5.20 Is US State

Category: Validation Function

Description: Validates US State 2 letter abbreviations
Properties: Default
Pass: CO|||GA|||TX
Fail: A|||ZZ|||Florida

6.5.21 Is US Social Security Number

Category: Validation Function

Description: Validates US Social security number
Properties: Default
Pass: 123-45-6789|||123 45 6789|||123456789
Fail: 12345-67-890123|||1234-56-7890|||123-45-78901

Copyright © 2009 DB Software Laboratory Page 113 of 194

Advanced ETL Processor User Manual

6.5.22 Is US Phone Number

Category: Validation Function

Description: US Phone Number Validation
Properties: Default
Pass: 5305551212|||(530) 555-1212|||530-555-1212
Fail: 0010011212|||1991991212|||123) not-good

6.5.23 Is Canadian Zip Code

Category: Validation Function

Description: Canadian postal code format verification. The format of a Canadian postal code
is LDL DLD where L are alpha characters and D are numeric digits. But there are some
exceptions. The letters D, F, I, O, Q and U never appear in a postal code because of their
Advanced similarity to 0, E, 1, 0, 0, and V respectively. In addition to avoiding the six
"forbidden" letters W and Z also do not appear as the first letter of a postal code (at least not at
present).
Properties: Default
Pass: M1R 4B0|||L0R 1B1|||L0R1B9
Fail: W1R 4B0|||L0R 1D1|||LOR1B9

6.5.24 Is Canadian Provincial Code

Category: Validation Function

Description: Validates Canadian provincial codes
Properties: Default
Pass: ON|||PE|||NB
Fail: OB|||NM

6.5.25 Is Canadian Branch-Transit/ Bank code

Category: Validation Function

Description: Validates Canadian Branch-Transit number. The branch number must be 3 or 4
digits then '-' then five digits
Properties: Default
Pass: 654-45654|||5544-65486
Fail: 455645564|||44-55545|||55455-54565

Copyright © 2009 DB Software Laboratory Page 114 of 194

Advanced ETL Processor User Manual

6.5.26 Is Canadian Social Insurance Number

Category: Validation Function

Description: Validates Canadian Social Insurance Number
Properties: Default
Pass: 046 454 286|||123-456-789
Fail: 046FF 454 286|||123GG-456-789

6.5.27 Is Australian Post Code

Category: Validation Function

Description: Australian postal code verification. Australia has 4-digit numeric postal codes
with the following state based specific ranges. ACT: 0200-0299 and 2600-2639. NSW: 1000-
1999, 2000-2599 and 2640-2914. NT: 0900-0999 and 0800-0899. QLD: 9000-9999 and 4000-
4999. SA: 5000-5999. TAS: 7800-7999 and 7000-7499. VIC: 8000-8999 and 3000-3999. WA:
6800-6999 and 6000-6799
Properties: Default
Pass: 0200|||7312|||2415
Fail: 0300|||7612|||2915

6.5.28 Is Australian Phone Number

Category: Validation Function

Description: Validation of Australian phone number. Accepts all forms of Australian phone
numbers in different formats (area code in brackets, no area code, spaces between 2-3 and 6-
7th digits, +61 international dialling code). Checks that area codes are valid (when entered).
Properties:
Pass: 0403111111|||(03) 1111 1111|||+61403111111|||02 9111 1111|||0403 111 111|||91111111
Fail: 9111 11111|||99 1111 1111

Copyright © 2009 DB Software Laboratory Page 115 of 194

Advanced ETL Processor User Manual

6.5.29 Is Australian Mobile Number

Category: Validation Function

Description: Validates Australian Mobile Number
Properties: Default
Pass: 0412345678
Fail: 0299999999

6.5.30 Is Argentinean Post Code

Category: Validation Function

Description: Validates Argentinean Post Code
Properties: Default
Pass: C1406HHA|||A4126AAB|||c1406hha
Fail: c1406HHA|||4126|||C1406hha

6.5.31 Is Dutch Post Code

Category: Validation Function

Description: Validates Dutch Postal Code
Properties: Default
Pass: 1234AB|||1234 AB|||1001 AB
Fail: 0123AB|||1234A B|||0123 AB

6.5.32 Is Dutch Phone Number

Category: Validation Function

Description: Validates Dutch Phone Number
Properties: Default
Pass: +31235256677|||+31(0)235256677|||023-5256677
Fail: +3123525667788999|||3123525667788|||232-2566778

Copyright © 2009 DB Software Laboratory Page 116 of 194

Advanced ETL Processor User Manual

6.5.33 Is Dutch Car Number Plate

Category: Validation Function

Description: Validates Dutch Car Number Plate
Properties: Default
Pass: ABC123|||12-AB-AB|||12-ABC-1
Fail: 12ABAB|||12ABC1|||1ABC12

6.5.34 Is Belgian Post Code

Category: Validation Function

Description: Validates Belgian Post Code
Properties: Default
Pass: 1234
Fail: 123|||123A

6.5.35 Is Belgian VAT Number

Category: Validation Function

Description: Belgian VAT Numbers format verification with support for optional member
state definition.
Properties: Default
Pass: BE123456789|||0123456789
Fail: BE12345678|||O123456789

6.5.36 Is French Post Code

Category: Validation Function

Description: French Post Code Validation
Properties: Default
Pass: 12345|||F-12345|||F-2B100
Fail: F12345|||F-123456|||123456

Copyright © 2009 DB Software Laboratory Page 117 of 194

Advanced ETL Processor User Manual

6.5.37 Is French VAT Number

Category: Validation Function

Description: French VAT Numbers format verification with support for optional member
state definition.
Properties: Default
Pass: FRAB 123456789|||L7 123456789
Fail: FRAB123456789|||L7 L23456789

6.5.38 Is French Phone Number

Category: Validation Function

Description: French Phone Number Validation
Properties: Default
Pass: 01 46 70 89 12|||01-46-70-89-12|||0146708912
Fail: 01-46708912|||01 46708912|||+33235256677

6.5.39 Is German Post Code

Category: Validation Function

Description: Validates German Post Code format
Properties: Default
Pass: 12556|||01550|||80796
Fail: 05234|||8973|||62980

6.5.40 Is German VAT Number

Category: Validation Function

Description: German VAT Numbers format verification with support for optional member
state definition.
Properties: Default
Pass: DE123456789|||224466880
Fail: DE12345678|||22446688B

Copyright © 2009 DB Software Laboratory Page 118 of 194

Advanced ETL Processor User Manual

6.5.41 Is German Car License Plate

Category: Validation Function

Description: German Car License Plate Validation
Properties: Default
Pass: F-RM373|||WÜ-Z1|||K- 07333|||HN-WK11H
Fail: MOS-AS

6.5.43 Is German IBAN

Category: Validation Function

Description: German International Bank Account Number Validation
Properties: Default
Pass: DE89 3704 0044 0532 0130 00|||DE89370400440532013000
Fail: DE89-3704-0044-0532-0130-00

6.5.44 Is Italian Post Code

Category: Validation Function

Description: Postcode check for Italy (including possible Vatican/Italy indications)
Properties: Default
Pass: 1234|||V-1234
Fail: 12345

6.5.45 Is Italian VAT Number

Category: Validation Function

Description: Italian VAT Numbers format verification with support for optional member state
definition.
Properties: Default
Pass: IT12345678901|||12345678901
Fail: IT1234567890|||1234567890I

Copyright © 2009 DB Software Laboratory Page 119 of 194

Advanced ETL Processor User Manual

6.5.46 Is Italian IBAN

Category: Validation Function

Description: Italian International Bank Account Number Validation
Properties: Default
Pass: IT28 W800 0000 2921 0064 5211 151|||IT28W8000000292100645211151
Fail: IT28-W800-0000-2921-0064-5211-151

6.5.47 Is Italian Phone Number

Category: Validation Function

Description: Validates Italian phone number
Properties: Default
Pass: 02-343536|||02/343536|||02 343536
Fail: 02a343536|||02+343536

6.5.48 Is Italian Mobile Number

Category: Validation Function

Description: Validates Italian mobile phone number with (or without) the International code
Properties: Default
Pass: +393471234567|||3381234567
Fail: +39 3401234567|||347 1234567|||338-1234567

6.5.49 Is Spanish Post Code

Category: Validation Function

Description: Spanish Post Code Validation
Properties: Default
Pass: 12345|||10234|||01234
Fail: 00123

Copyright © 2009 DB Software Laboratory Page 120 of 194

Advanced ETL Processor User Manual

6.5.50 Is Spanish VAT Number

Category: Validation Function

Description: Spanish VAT Numbers format verification with support for optional member
state definition.
Properties: Default
Pass: ES01234567A|||A12345678
Fail: ES012345678|||AB2345678

6.5.51 Is Spanish IBAN

Category: Validation Function

Description: Spanish IBAN Validation
Properties: Default
Pass: ES91 2100 0418 4502 0005 1332|||ES9121000418450200051332
Fail: ES91-2100-0418-4502-0005-1332

6.5.52 Is Spanish Phone Number

Category: Validation Function

Description: Spanish Phone Number Validation
Properties: Default
Pass: 972367087|||97-3770866|||972-377086
Fail: 9988-989898|||989898988989|||9 9898989898

6.5.53 Is Swedish Post Code

Category: Validation Function

Description: Validates Swedish Post Code Format
Properties: Default
Pass: 12345|||932 68|||S-621 46
Fail: 5367|||425611|||31 545

6.5.54 Is Swedish VAT Number

Copyright © 2009 DB Software Laboratory Page 121 of 194

Advanced ETL Processor User Manual

Category: Validation Function

Description: Swedish VAT Numbers format verification with support for optional member
state definition.
Properties: Default
Pass: SE123456789012|||123456789012
Fail: SE12345678901|||123456789O12

6.5.55 Is Swedish IBAN

Category: Validation Function

Description: Swedish International Bank Account Number IBAN
Properties: Default
Pass: SE35 5000 0000 0549 1000 0003|||SE3550000000054910000003
Fail: SE35-5000-0000-0549-1000-0003

6.5.56 Is Swedish Phone Number

Category: Validation Function

Description: Validates Swedish Phone Number
Properties: Default
Pass: +46 8 123 456 78|||08-123 456 78|||0123-456 78
Fail: +46 08-123 456 78|||08 123 456 78|||0123 456 78

6.5.57 Is Swedish Person Number

Category: Validation Function

Description: Validates Swedish Person Number
Properties: Default
Pass: 19740609-7845|||19811116-7845|||20010913-7598
Fail: 21003612-9999|||18790505-4545|||19740641-5559

Copyright © 2009 DB Software Laboratory Page 122 of 194

Advanced ETL Processor User Manual

6.5.58 Is Austrian Mobile Phone Number

Category: Validation Function

Description: Validates Austrian Mobile Phone Number
Properties: Default
Pass: 0664-1234567|||06641234567|||0699-11234567
Fail: 06991-1234567|||+43650-1234567|||07661234567

6.5.59 Is Danish Post Code

Category: Validation Function

Description: Validates Danish Post Code
Properties: Default
Pass: DK-1234|||dk 1234|||Dk-1234
Fail: DK 12345

6.5.60 Is Danish VAT Number

Category: Validation Function

Description: Validates Danish VAT Numbers format with support for optional member state
definition.
Properties: Default
Pass: DK11 22 33 44
Fail: DK11223344

6.5.61 Is Danish Phone Number

Category: Validation Function

Description: Validates Danish Phone Number
Properties: Default
Pass: (+45) 35 35 35 35 ||| +45 35 35 35 35 ||| 35 35 35 35 ||| 35353535
Fail: (45)35353535 ||| 4535353535

Copyright © 2009 DB Software Laboratory Page 123 of 194

Advanced ETL Processor User Manual

6.5.62 Is Danish Civil Registration Number

Category: Validation Function

Description: Validates Danish civil registration number, which consists of 10 numeric
characters and one special character '-'. The format is DDMMYY-XXXX
Properties: Default
Pass: 010380-2343 || 120454-5467 || 390178-3434
Fail: 1223876776 || 4023100923

6.5.63 Is Finnish VAT Number

Category: Validation Function

Description: Finnish VAT Numbers format verification with support for optional member
state definition.
Properties: Default
Pass: FI12345678|||22446688
Fail: FI1234567|||224466BB

6.5.64 Is Greek VAT Number

Category: Validation Function

Description: Greek VAT Numbers format verification with support for optional member state
definition.
Properties: Default
Pass: EL123456789|||224466880
Fail: EL12345678|||22446688B

6.5.65 Is Irish VAT Number

Category: Validation Function

Description: Irish VAT Numbers format verification with support for optional member state
definition.
Properties: Default
Pass: IE4*12345Z|||0012345K
Fail: IE4-12345Z|||00123456

Copyright © 2009 DB Software Laboratory Page 124 of 194

Advanced ETL Processor User Manual

6.5.66 Is Luxembourg VAT Number

Category: Validation Function

Description: Luxemburg VAT Numbers format verification with support for optional member
state definition
Properties: Default
Pass: LU12345678|||22446688
Fail: LU1234567|||224466BB

6.5.67 Is Portuguese VAT Number

Category: Validation Function

Description: Portuguese VAT Numbers format verification with support for optional member
state definition.
Properties: Default
Pass: PT123456789|||224466880
Fail: PT12345678|||22446688B

6.5.68 Is Estonian Post Code

Category: Validation Function

Description: Validates Estonian Post Codes
Properties: Default
Pass: 00000|||99999
Fail: 00 000

6.5.69 Is Estonian VAT Number

Category: Validation Function

Description: Estonian VAT Numbers format verification with support for optional member
state definition.
Properties: Default
Pass: EE123456789|||224466880
Fail: EE12345678|||22446688B

Copyright © 2009 DB Software Laboratory Page 125 of 194

Advanced ETL Processor User Manual

6.5.70 Is Latvian VAT Number

Category: Validation Function

Description: Validates Latvian VAT Number
Properties: Default
Pass: LV12345678901|||12345678901
Fail: IT1234567890|||1234567890I

6.5.71 Is Lithuanian VAT Number

Category: Validation Function

Description: Lithuanian VAT Numbers format verification with support for optional member
state definition.
Properties: Default
Pass: LT123456789|||123456789012
Fail: LX123456789|||123466789O12

6.5.72 Is Slovenian VAT Number

Category: Validation Function

Description: Slovenian VAT Numbers format verification with support for optional member
state definition
Properties: Default
Pass: SI12345678|||22446688
Fail: SI1234567|||224466BB

6.5.73 Is Slovakian VAT Number

Category: Validation Function

Description: Slovakian VAT Numbers format verification with support for optional member
state definition
Properties: Default
Pass: SK1234567890|||1234567890
Fail: SK1234567890|||1234567890

Copyright © 2009 DB Software Laboratory Page 126 of 194

Advanced ETL Processor User Manual

6.5.74 Is Slovakian IBAN

Category: Validation Function

Description: Validates Slovakian International Bank Account Number
Properties: Default
Pass: SK31 1200 0000 1987 4263 7541|||SK3112000000198742637541
Fail: SK31-1200-0000-1987-4263-7541

6.5.75 Is Polish VAT Number

Category: Validation Function

Description: Polish VAT Numbers format verification with support for optional member state
definition
Properties: Default
Pass: PL1234567890|||1234567890
Fail: PL123456789|||123456789O

6.5.76 Is Polish IBAN

Category: Validation Function

Description: Validates Polish International Bank Account Number
Properties: Default
Pass: PL27 1140 2004 0000 3002 0135 5387|||PL27114020040000300201355387
Fail: PL27-1140-2004-0000-3002-0135-5387

6.5.77 Is Czech Republic VAT Number

Category: Validation Function

Description: Czech Republic VAT Numbers format verification with support for optional
member state definition.
Properties: Default
Pass: CZ12345678|||1234567890
Fail: CZ1234567|||12345678901

Copyright © 2009 DB Software Laboratory Page 127 of 194

Advanced ETL Processor User Manual

6.5.78 Is Czech Republic IBAN

Category: Validation Function

Description: Validates Czech International Bank Account Number
Properties: Default
Pass: CZ65 0800 0000 1920 0014 5399|||CZ6508000000192000145399
Fail: CZ65-0800-0000-1920-0014-5399

6.5.79 Is Netherlands VAT Number

Category: Validation Function

Description: The Netherlands VAT Numbers format verification with support for optional
member state definition.
Properties: Default
Pass: NL123456789B12
Fail: NL123456789012

6.5.80 Is Hungarian VAT Number

Category: Validation Function

Description: Hungarian VAT Numbers format verification with support for optional member
state definition
Properties: Default
Pass: HU12345678|||22446688
Fail: HU1234567|||224466BB

6.5.81 Is Maltese VAT Number

Category: Validation Function

Description: Maltese VAT Numbers format verification with support for optional member
state definition
Properties: Default
Pass: MT12345678|||22446688
Fail: MT1234567|||224466BB

Copyright © 2009 DB Software Laboratory Page 128 of 194

Advanced ETL Processor User Manual

6.5.82 Is Algerian Post Code

Category: Validation Function

Description: Validates Algerian Post Code Format
Properties: Default
Pass: 12345|||99999|||00000
Fail: 1|||1234|||999999

6.5.83 Is Andorran Post Code

Category: Validation Function

Description: Validates Andorran Post Code format
Properties: Default
Pass: AB 123|||AB123
Fail: 12 abc|||12345

6.5.84 Is Andorran IBAN

Category: Validation Function

Description: Validates Andorran International Bank Account Number
Properties: Default
Pass: AD12 0001 2030 2003 5910 0100|||AD1200012030200359100100
Fail: AD12-0001-2030-2003-5910-0100

6.5.85 Is Azerbaijani Post Code

Category: Validation Function

Description: Validates Azerbaijani Post Code format
Properties: Default
Pass: AZ 1234|||AZ1234
Fail: 12 abcd|||AB 1234

Copyright © 2009 DB Software Laboratory Page 129 of 194

Advanced ETL Processor User Manual

6.5.86 Is Bahraini Post Code

Category: Validation Function

Description: Validates Bahraini Post Code format
Properties: Default
Pass: 123|||1299|||199
Fail: 1300|||000|||99

6.5.87 Is Bangladeshi Post Code

Category: Validation Function

Description: Validates Bangladeshi Post Code format
Properties: Default
Pass: 1000|||9999|||1234
Fail: 123456|||0123|||99

6.5.88 Is Belarusian Post Code

Category: Validation Function

Description: Validates Belarusian Post Code format
Properties: Default
Pass: 123456|||000000|||999999
Fail: 0|||9|||1234

6.5.89 Is Brunei Post Code

Category: Validation Function

Description: Validates Brunei Post Code format
Properties: Default
Pass: BT2328|||KA1234
Fail: AB1234|||AB 1234

Copyright © 2009 DB Software Laboratory Page 130 of 194

Advanced ETL Processor User Manual

6.5.90 Is Chilean Post Code

Category: Validation Function

Description: Validates Chilean Post Code format
Properties: Default
Pass: 872-0019|||000-0000|||999-9999
Fail: 000 0000|||000000

6.5.91 Is Croatian Post Code

Category: Validation Function

Description: Validates Croatian Post Code format
Properties: Default
Pass: HR-00000|||HR-99999
Fail: HR 00000|||00000

6.5.92 Is Cyprus Post Code

Category: Validation Function

Description: Validates Cyprus Post Code format
Properties: Default
Pass: 1000|||9999|||0000
Fail: 1|||99999|||99 000

6.5.93 Is Cyprus VAT Number

Category: Validation Function

Description: Cyprus VAT Numbers format verification with support for optional member
state definition
Properties: Default
Pass: CY12345678A|||12345678A
Fail: CY1234567A|||123456789

Copyright © 2009 DB Software Laboratory Page 131 of 194

Advanced ETL Processor User Manual

6.5.94 Is Egyptian Post Code

Category: Validation Function

Description: Validates Egyptian Post Code format
Properties: Default
Pass: 00000|||99999
Fail: 0 0 0 00|||00

6.5.95 Is Ethiopian Post Code

Category: Validation Function

Description: Validates Ethiopian Post Code format
Properties: Default
Pass: 1000|||9999
Fail: 1 999

6.5.96 Is Faroe Islands Post Code

Category: Validation Function

Description: Validates Faroe Islands Post Code format
Properties: Default
Pass: FO100|||FO000|||FO 000
Fail: AB 123|||AB123

6.5.97 Is French Polynesia Post Code

Category: Validation Function

Description: Validates French Polynesia Post Code format
Properties: Default
Pass: 00000|||99999
Fail: 00 000|||99 9 9 9

Copyright © 2009 DB Software Laboratory Page 132 of 194

Advanced ETL Processor User Manual

6.5.98 Is Guadeloupe Post Code

Category: Validation Function

Description: Validates Guadeloupe Post Code format
Properties: Default
Pass: 97100|||98099
Fail: 12345|||97 100

6.5.99 Is Guinean Post Code

Category: Validation Function

Description: Validates Guinean Post Code format
Properties: Default
Pass: 001 BP 123|||499 BP 999
Fail: 001BP999|||999 BP 000

6.5.100 Is Guinea-Bissau Post Code

Category: Validation Function

Description: Validates Guinea-Bissau Post Code format
Properties: Default
Pass: 1000|||1999|||9999
Fail: 0000|||0123

6.5.101 Is Russian Post Code

Category: Validation Function

Description: Validates Russian Post Code
Properties: Default
Pass: 150016
Fail: wert12

Copyright © 2009 DB Software Laboratory Page 133 of 194

Advanced ETL Processor User Manual

6.5.102 Is Russian Phone Number

Category: Validation Function

Description: Validates Russian Phone Number
Properties: Default
Pass: +7(916)9985670|||8-912-268-5440|||8905148-3339|||8(913)448-51-90|||903-345-34-34|||
903-34-334-34|||903-34-33434
Fail: +33903-34-33434

6.5.103 Is Israeli Phone Number

Category: Validation Function

Description: Validates Israeli Phone Number Format
Properties: Default
Pass: +972-4-8253055|||972-4-8615466|||972-04-8564521
Fail: 04-5487665|||05-455665545|||456884624

6.5.104 Is Israeli Mobile Phone Number

Category: Validation Function

Description: Validates Israeli Mobile Phone Number
Properties: Default
Pass: 050-1234567, 0501234567, 501234567
Fail: 0601234567, 5412345678

6.5.105 Is Peru Phone Number

Category: Validation Function

Description: Validates Peru Phone Number format
Properties: Default
Pass: 263-8854|||5660578 1235|||622-4588 21
Fail: 1227585|||4224586 50124|||0554499

Copyright © 2009 DB Software Laboratory Page 134 of 194

Advanced ETL Processor User Manual

6.5.106 Is Peru Cellular Number

Category: Validation Function

Description: Validates Peru Cellular Number format
Properties: Default
Pass: 8874-2544|||99106800|||84509955
Fail: 6540-9985|||77329390|||725-2763

6.5.107 Is Indian Post Code

Category: Validation Function

Description: Validates Indian Post Code
Properties: Default
Pass: 400 099|||400099|||400050
Fail: 2345678|||12345|||asdf

6.5.108 Is Indian Phone Number

Category: Validation Function

Description: Validates Indian Phone Number
Properties: Default
Pass: 0493 - 3227341|||0493 3227341|||493 3227341
Fail: 93 0227341|||493 322734111|||493 -- 3227341

6.5.109 Is Indian Mobile Number

Category: Validation Function

Description: Validates Indian Mobile Phone Number
Properties: Default
Pass: +919847444225|||+91-98-44111112|||98 44111116
Fail: +919800444225|||+947444225|||44111116

Copyright © 2009 DB Software Laboratory Page 135 of 194

Advanced ETL Processor User Manual

6.5.110 Is Indian Vehicle Registration Number

Category: Validation Function

Description: Validates Indian Vehicle Registration Number
Properties: Default
Pass: mh 12 bj 1780|||mmx 1234
Fail: mmm 123 1234|||mm 12 bj 345|||mm 12345

6.5.111 Is South African Phone Number

Category: Validation Function

Description: Validates South African Phone Number
Properties: Default
Pass: 0333872119|||033 386162|||074 101 2850
Fail: 033386 2612

6.5.112 Is South African Mobile Number

Category: Validation Function

Description: only allows valid cell phones from South African service providers
Properties: Default
Pass: 0848298477|||084 829 8477|||084-829 8477
Fail: 084829 8477

6.5.113 Is South African VAT Number

Category: Validation Function

Description: Validates South African VAT Number
Properties: Default
Pass: 2001/067995/23|||1989/064445/73
Fail: LX123456789|||123466789O12

Copyright © 2009 DB Software Laboratory Page 136 of 194

Advanced ETL Processor User Manual

6.5.114 Is US or CA Zip code

Category: Validation Function

Description: Validates US or CA Zip code format
Properties: Default
Pass: 00501|||84118-3423|||n3a 3B7
Fail: 501-342|||123324|||Q4B 5C5

Copyright © 2009 DB Software Laboratory Page 137 of 194

Advanced ETL Processor User Manual

7. Transformation Functions

)ote:
To change Transformation function properties double click on it

7.1 Strings

7.1.1 Upper Case

Category: Transformation Function

Description: Converts all characters of a string to Upper Case
My String => MYSTRING
Properties: None

7.1.2 Lower Case

Category: Transformation Function

Description: Converts all characters of a string to Lower Case
My String => my string
Properties: None

7.1.3 Proper Case

Category: Transformation Function

Description: Converts first character of all words within a string to Upper Case and the rest to
the lower case
my strinG => My String
Properties: None

7.1.4 First Up

Category: Transformation Function

Description: Converts first character a string to Upper Case and the rest to the lower case
my strinG => My string
Properties: None

Copyright © 2009 DB Software Laboratory Page 138 of 194

Advanced ETL Processor User Manual

7.1.5 Delete Spaces

Category: Transformation Function

Description: Makes sure that there is only one space between words
My string => My string
Properties: None

7.1.6 Trim

Category: Transformation Function

Description: Removes Leading and Trailing Spaces from a string
Properties: None

7.1.7 Left Trim

Category: Transformation Function

Description: Removes Leading Spaces from a string
Properties: None

7.1.8 Right Trim

Category: Transformation Function

Description: Removes Trailing Spaces from a string
Properties: None

Copyright © 2009 DB Software Laboratory Page 139 of 194

Advanced ETL Processor User Manual

7.1.9 Sub String

Category: Transformation Function

Description: Returns part of a string.
Properties:

Copyright © 2009 DB Software Laboratory Page 140 of 194

Advanced ETL Processor User Manual

7.1.10 Replace

Category: Transformation Function

Description: Replaces a one or more characters from the data with zero or more new
characters.
Properties:

Copyright © 2009 DB Software Laboratory Page 141 of 194

Advanced ETL Processor User Manual

7.1.11 Ensure Prefix

Category: Transformation Function

Description: Check string for prefix and if it is missing adds it
Properties:

7.1.12 Ensure Suffix

Category: Transformation Function

Description: Check string for suffix and if it is missing adds it
Properties: Same as ‘Ensure Prefix’

7.1.13 Ensure No Prefix

Category: Transformation Function

Description: Check string for prefix and if it is present removes it
Properties: Same as ‘Ensure Prefix’

Copyright © 2009 DB Software Laboratory Page 142 of 194

Advanced ETL Processor User Manual

7.1.14 Ensure No Suffix

Category: Transformation Function

Description: Check string for suffix and if it is present removes it
Properties: Same as ‘Ensure Prefix’

7.1.15 Left Pad

Category: Transformation Function

Description: Returns the string left-padded to the length with spaces or specified character.
Properties:

7.1.16 Right Pad

Category: Transformation Function

Description: Returns the string right-padded to the length with spaces or specified character.
Properties: Same as ‘Right Pad’

Copyright © 2009 DB Software Laboratory Page 143 of 194

Advanced ETL Processor User Manual

7.1.17 Escape String

Category: Transformation Function

Description: Replaces special characters such as TAB, CR,
LF,NULL,BEL,BS,ESC,HT,FF,VT with there C-Style escape equivalents.
Properties: None

7.1.18 UnEscape String

Category: Transformation Function

Description: Opposite to Escape String, Replaces C-Style escape characters with special
characters such as TAB, CR, LF,NULL,BEL,BS,ESC,HT,FF,VT.
Properties: None

7.1.19 Delete

Category: Transformation Function

Description: Deletes part of a string.
Properties:

Copyright © 2009 DB Software Laboratory Page 144 of 194

Advanced ETL Processor User Manual

7.1.20 Left

Category: Transformation Function

Description: Returns left part of a string.
Properties:

7.1.21 Right

Category: Transformation Function

Description: Returns right part of a string
Properties: Same as ‘Left’

7.1.22 Left Delete

Category: Transformation Function

Description: Deletes left part of a string
Properties: Same as ‘Left’

Copyright © 2009 DB Software Laboratory Page 145 of 194

Advanced ETL Processor User Manual

7.1.23 Right Delete

Category: Transformation Function

Description: Deletes right part of a string
Properties: Same as ‘Left’

Copyright © 2009 DB Software Laboratory Page 146 of 194

Advanced ETL Processor User Manual

7.2 Numbers

7.2.1 Round

Category: Transformation Function

Description: Rounding is the process of reducing the number of significant digits in a number.
The result of rounding is a "shorter" number having fewer non-zero digits yet similar in
magnitude. The result is less precise but easier to use.
For example: 73 rounded to the nearest ten is 70, because 73 is closer to 70 than to 80.
Properties:

Copyright © 2009 DB Software Laboratory Page 147 of 194

Advanced ETL Processor User Manual

7.2.3 Abs

Category: Transformation Function

Description: Returns absolute value of a number. Absolute value is always positive. For
example -2 will give 2
Properties:

7.2.4 Sign

Category: Transformation Function

Description: Returns Sign of numeric value -1 for negative values, 1 for positive values and 0
for 0
Properties: Same as ‘Abs’

Copyright © 2009 DB Software Laboratory Page 148 of 194

Advanced ETL Processor User Manual

7.3 Date

7.3.1 Date Format

Category: Transformation Function

Description: Returns Date in ‘yyyy-mm-dd hh:nn:ss.zzz' format.
In order to load Date fields into target database they should be converted into ‘yyyy-mm-dd
hh:nn:ss.zzz' format
Properties:

7.3.2 Current Date

Category: Transformation Function

Description: Returns Current Date in ‘yyyy-mm-dd hh:nn:ss.zzz' format
Properties: None

Copyright © 2009 DB Software Laboratory Page 149 of 194

Advanced ETL Processor User Manual

7.4 Miscellaneous

7.4.1 Length

Category: Transformation Function

Description: Returns Length of String
Properties: None

7.4.2 Literal

Category: Transformation Function

Description: Populates field with a constant value
Properties:

Copyright © 2009 DB Software Laboratory Page 150 of 194

Advanced ETL Processor User Manual

7.4.3 User

Category: Transformation Function

Description: Returns Current Windows User Name
Properties: None

7.4.4 Splitter

Category: Transformation Function

Description: Splits data into number of values
Properties:

Copyright © 2009 DB Software Laboratory Page 151 of 194

Advanced ETL Processor User Manual

7.4.5 Joiner

Category: Transformation Function

Description: Joins several values into one value delimited Delimiter and surrounded by
Qualifier if specified
Properties:

Copyright © 2009 DB Software Laboratory Page 152 of 194

Advanced ETL Processor User Manual

7.4.6 Calculation

Category: Transformation Function

Description: Allows end user to write scripts to transform data. Input values represented as
[F00N] Where N is values number. For example calculation you can see above has got two
input values [F001] and [F002]. Programming language used is very close to Pascal.

The Structure of a Calculation

Below is the basic structure that every Calculation transformation must follow:

Var
VariableName : VariableType;
VariableName : VariableType;
...
// Some single line Comment if necessary

{
Multi line
Comment
}

Procedure ProcedureName;
variables here if necessary
Begin
Some Code;
End;

Function FunctionName(variableList): VariableType;

variables here if necessary
Begin
Some Code if necessary;
Result := some expression
More Code if necessary;
End;

... some more functions and procedures if necessary ...

Begin
the main program block.
Result:= some calculation
End.

)ote:

Copyright © 2009 DB Software Laboratory Page 153 of 194

Advanced ETL Processor User Manual

The functions and procedures can appear in any order. The only requirement is that if one
procedure or function uses another one, that latter one must have been defined already.

Declaring Variables

var // This starts a section of variables

LineTotal : Integer; // This defines an Integer variable called LineTotal
First,Second : String; // This defines two variables to hold strings of text

Assigning Values to Variables

Variables are simply a name for a block of memory cells in main memory. If a value is
assigned to a variable, that value must be of the same type as the variable, and will be stored in
the memory address designated by the variable name. The assignment statement is the
semicolon-equal :=.

• Variables must be declared at the beginning of the program, a procedure, or a function

• Variables must be initialized before they can be used.
• Variables can be reused as often as necessary. Their old value is simply overwritten by
a new assignment.

Example:

Var
i : Integer: { variable name is i, type is integer)
Begin
i := 10; { valid integer number assigned to variable i }
End.

Copyright © 2009 DB Software Laboratory Page 154 of 194

Advanced ETL Processor User Manual

Variable Types

)umeric Data Types

Type Storage size Range

Byte 1 0 to 255
ShortInt 1 -127 to 127
Word 2 0 to 65,535
SmallInt 2 -32,768 to 32,767
LongWord 4 0 to 4,294,967,295
Cardinal 4* 0 to 4,294,967,295
LongInt 4 -2,147,483,648 to 2,147,483,647
Integer 4* -2,147,483,648 to 2,147,483,647
Int64 8 -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
Single 4 7 significant digits, exponent -38 to +38
Currency 8 50+ significant digits, fixed 4 decimal places
Double 8 15 significant digits, exponent -308 to +308
Extended 10 19 significant digits, exponent -4932 to +4932

Assigning to and from number variables

Number variables can be assigned from other numeric variables, and expressions:

var
Age : Byte; // Smallest positive integer type
Books : SmallInt; // Bigger signed integer
Salary : Currency; // Decimal used to hold financial amounts
Expenses : Currency;
TakeHome : Currency;

begin
Expenses := 12345.67; // Assign from a literal constant
TakeHome := Salary; // Assign from another variable
TakeHome := TakeHome - Expenses; // Assign from an expression
end;

)umerical operators

Number calculations, or expressions, have a number of primitive operators available:

+ Add one number to another

- Subtract one number from another
* Multiply two numbers
/ Divide one decimal number by another
div Divide one integer number by another
mod Remainder from dividing one integer by another

Copyright © 2009 DB Software Laboratory Page 155 of 194

Advanced ETL Processor User Manual

When using these multiple operators in one expression, you should use round brackets to
wrap around sub-expressions to ensure that the result is obtained. This is illustrated in the
examples below:

var
myInt : Integer; // Define integer and decimal variables
myDec : Single;
begin
myInt := 20; // myInt is now 20
myInt := myInt + 10; // myInt is now 30
myInt := myInt - 5; // myInt is now 25
myInt := myInt * 4; // myInt is now 100
myInt := 14 div 3; // myInt is now 4 (14 / 3 = 4 remainder 2)
myInt := 14 mod 3; // myInt is now 2 (14 / 3 = 4 remainder 2)
myInt := 12 * 3 - 4; // myInt is now 32 (* comes before -)
myInt := 12 * (3 - 4); // myInt is now -12 (brackets come before *)
myDec := 2.222 / 2.0; // myDec is now 1.111
end;

Character Types

var
Str1 : Char; // Holds a single character, small alphabet
Str2 : WideChar; // Holds a single character, International alphabet
Str3 : AnsiChar; // Holds a single character, small alphabet
Str4 : ShortString; // Holds a string of up to 255 Char's
Str5 : String; // Holds strings of Char's of any size desired
Str6 : AnsiString; // Holds strings of AnsiChar's any size desired
Str7 : WideString; // Holds strings of WideChar's of any size desired

Some simple text variable usage examples are given below:

+ Concatenates two strings together

= Compares for string equality
< Is one string lower in sequence than another
<= Is one string lower or equal in sequence with another
> Is one string greater in sequence than another
>= Is one string greater or equal in sequence with another
<> Compares for string inequality

Copyright © 2009 DB Software Laboratory Page 156 of 194

Advanced ETL Processor User Manual

Variants

The Variant data type provides a flexible general purpose data type.

It can hold anything but structured data and pointers.

Variants are useful in very specific circumstances, where data types and their content are
determined at run time rather than at compile time.

Example:

var
myVar : Variant;

Date Variables

TDateTime

Description

The TDateTime type holds a date and time value.

It is stored as a Double variable, with the date as the integral part, and time as fractional
part. The date is stored as the number of days since 30 Dec 1899. Quite why it is not 31 Dec is
not clear. 01 Jan 1900 has a days value of 2.

Because TDateTime is actually a double, you can perform calculations on it as if it were a

number. This is useful for calculations such as the difference between two dates.

)ote:
No local time information is held with TDateTime - just the day and time values.

Example:
Finding the difference between two dates

var
day1, day2 : TDateTime;
diff : Double;
begin
day1 := StrToDate('12/06/2002');
day2 := StrToDate('12/07/2002');
diff := day2 - day1;
Result:='day2 - day1 = '+FloatToStr(diff)+' days';
end;

day2 - day1 = 30 days

Logical data types

Copyright © 2009 DB Software Laboratory Page 157 of 194

Advanced ETL Processor User Manual

These are used in conjunction with programming logic. They are very simple:

Var
Log1 : Boolean; // Can be 'True' or 'False'

Boolean variables are a form of enumerated type. This means that they can hold one of a
fixed number of values, designated by name. Here, the values can be True or False.

Logical Operation

Simple if then else

Here is an example of how the if statement works:

var
number : Integer;
text : String;
begin
number := Sqr(17); // Calculate the square of 17
if number > 400
then text := '17 squared > 400' // Action when if condition is true
else text := '17 squared <= 400'; // Action when if condition is false
result:=text;
end;

text is set to : '17 squared <= 400'

There are a number of things to note about the if statement. First that it spans a few lines -
remember that statements can span lines - this is why it insists on a terminating ;
Second, that the then statement does not have a terminating ; -this is because it is part of the if
statement, which is finished at the end of the else clause.

Third, that we have set the value of a text string when the If condition is successful - the
Then clause - and when unsuccessful - the Else clause. We could have just done a then
assignment:

if number > 400

then text := '17 squared > 400';

Note that here, the then condition is not executed (because 17 squared is not > 400), but
there is no else clause. This means that the if statement simply finishes without doing anything.

Note also that the then clause now has a terminating ; to signify the end of the if statement.

Compound if conditions, and multiple statements

We can have multiple conditions for the if condition. And we can have more than one
statement for the then and else clauses. Here are some examples:

If (condition1) And (condition2) // Both conditions must be satisfied

Copyright © 2009 DB Software Laboratory Page 158 of 194
Advanced ETL Processor User Manual

then
begin
statement1;
statement2;
...
end // Notice no terminating ';' - still part of 'if'
else
begin
statement3;
statement4;
...
end;

We used And to join the if conditions together - both must be satisfied for the then clause to
execute. Otherwise, the else clause will execute. We could have used a number of different
logical primitives, of which And is one, covered under logical primitives below.

)ested if statements:

There is nothing to stop you using if statements as the statement of an if statement. Nesting
can be useful, and is often used like this:

if condition1
then statement1
else if condition2
then statement2
else statement3;

However, too many nested if statements can make the code confusing. The Case statement,
discussed below, can be used to overcome a lot of these problems.

Logical primitives

Before we introduce these, it is appropriate to introduce the Boolean data type. It is an

enumerated type that can have one of only two values: True or False. We will use it in place of
a condition in the if clauses below to clarify how they work:

begin
if false And false
then Result:='false and false = true';

if true And false

then Result:= 'true and false = true';

if false And true

then Result:= 'false and true = true';

if true And true

Copyright © 2009 DB Software Laboratory Page 159 of 194

Advanced ETL Processor User Manual

then Result:= 'true and true = true';

if false Or false
then Result:= 'false or false = true';

if true Or false
then Result:= 'true or false = true';

if false Or true
then Result:= 'false or true = true';

if true Or true
then Result:= 'true or true = true';

if false Xor false

then Result:= 'false xor false = true';

if true Xor false

then Result:= 'true xor false = true';

if false Xor true

then Result:= 'false xor true = true';

if true Xor true

then Result:= 'true xor true = true';

if Not false
then Result:= 'not false = true';

if Not true
then Result:= 'not true = true';
end;

true and true = true

false or true = true
true or false = true
true or true = true
false xor true = true
true xor false = true
not false = true

Note that the Xor primitive returns true when one, but not both of the conditions are true.

Case statements

The If statement is useful when you have a simple two way decision. Ether you go one way
or another way. Case statements are used when you have a set of 3 or more alternatives.

A simple numerical case statement:

Copyright © 2009 DB Software Laboratory Page 160 of 194

Advanced ETL Processor User Manual

var
i : Integer;
begin
i := [F1];
Case i of
15 : Resut := ('Number was fifteen');
16 : Resut := ('Number was sixteen');
17 : Resut := ('Number was seventeen');
18 : Resut := ('Number was eighteen');
19 : Resut := ('Number was nineteen');
20 : Resut := ('Number was twenty');
end;
end;

Number was fifteen

The case statement above routes the processing to just one of the statements. OK, the code
is a bit silly, but it is used to illustrate the point.

Using the otherwise clause

Supposing we were not entirely sure what value our case statement was processing? Or we
wanted to cover a known set of values in one fell swoop? The Else clause allows us to do that:

var
i : Integer;
begin
i := [F1];
Case i of
15 : Resut := ‘Number was fifteen';
16 : Resut := 'Number was sixteen';
17 : Resut := 'Number was seventeen';
18 : Resut := 'Number was eighteen';
19 : Resut := 'Number was nineteen';
20 : Resut := 'Nuumber was twenty';
else
Resut := 'Unexpected number‘;
end;
end;

Unexpected number : 10

Repeating sets of commands

Why loops are used in programming

Copyright © 2009 DB Software Laboratory Page 161 of 194

Advanced ETL Processor User Manual

One of the main reasons for using computers is to save the tedium of many repetitive tasks.
One of the main uses of loops in programs is to carrry out such repetitive tasks. A loop will
execute one or more lines of code (statements) as many times as you want.

Your choice of loop type depends on how you want to control and terminate the looping.

The For loop

This is the most common loop type. For loops are executed a fixed number of times,
determined by a count. They terminate when the count is exhausted. The count (loop) is held
in a variable that can be used in the loop. The count can proceed upwards or downwards, but
always does so by a value of 1 unit. This count variable can be a number or even an
enumeration.

Counting up

Here is a simple example counting up using numeric values:

var
count : Integer;
begin
For count := 1 to 5 do
Result:= 'Count is now '+IntToStr(count);
end;

Counting down

Here is a simple example counting up using numeric values:

var
count : Integer;
begin
For count := 5 downto 1 do
Result:= 'Count is now '+IntToStr(count);
end;

The For statements in the examples above have all executed one statement. If you want to
execute more than one, you must enclose these in a Begin and End pair.

Copyright © 2009 DB Software Laboratory Page 162 of 194

Advanced ETL Processor User Manual

The Repeat loop

The Repeat loop type is used for loops where we do not know in advance how many times
we will execute. For example, when we keep asking a user for a value until one is provided, or
the user aborts. Here, we are more concerned with the loop termination condition.

Repeat loops always execute at least once. At the end, the Until condition is checked, and
the loop aborts of condition works out as true.

A simple example

var
stop : Boolean; // Our exit condition flag
i : Integer;
begin
i := 1;
exit := False; // do not exit until we are ready
repeat
i := i+1; // Increment a count
if Sqr(i) > 99
then stop:= true; // Exit if the square of our number exceeds 99
until stop; // Shorthand for 'until exit := true'
result:=I;
end;

Upon exit, i will be 10 (since Sqr(10) > 99)

Here we exit the repeat loop when a Boolean variable is true. Notice that we use a
shorthand - just specifying the variable as the condition is sufficient since the variable value is
either true or false.

Using a compound condition

var
i : Integer;
begin
i := 1;
repeat
i := i+1; // Increment a count
until (Sqr(i) > 99) or (Sqrt(i) > 2.5);
result:=i;
end;

Upon exit, i will be 7 (since Sqrt(7) > 2.5)

Notice that compound statements require separating brackets. Notice also that Repeat
statements can accomodate multiple statements without the need for a begin/end pair. The
repeat and until clauses form a natural pairing.
While loops

Copyright © 2009 DB Software Laboratory Page 163 of 194

Advanced ETL Processor User Manual

While loops are very similar to Repeat loops except that they have the exit condition at the
start. This means that we use them when we wish to avoid loop execution altogether if the
condition for exit is satisfied at the start.

Var
i : Integer;
begin
i := 1;
while (Sqr(i) <= 99) and (Sqrt(i) <= 2.5) do
i := i+1; // Increment a count
result:=i;
end;

Upon exit, i will be 7 (since Sqrt(7) > 2.5)

Notice that our original Repeat Until condition used Or as the compound condition joiner -
we continued until either condition was met. With our While condition, we use And as the
joiner - we continue whilst neither condition is met. Have a closer look to see why we do this.
The difference is that we repeat an action until something or something else happens. Whereas
we keep doing an action while neither something nor something else have happened.

Functions

Functions provide a flexible method to apply one formula many times to possibly different
values. They are comparable to procedures but

• functions are of always of a certain type

• functions usually have one or more input variable(s)
• the function name must appear at least once inside the definition

The general form of the function statement looks like this:

Function Function)ame(Variable)ame: VariableType): VariableType;

Begin
some code, if necessary;
Result := some computation;
more code if necessary;
End;

Copyright © 2009 DB Software Laboratory Page 164 of 194

Advanced ETL Processor User Manual

Basic Calculation Examples

1. Adding one number to another

Begin
Result:= [F001] + [F002];
End;

2. Strings Concatenation

Begin
Result:= ‘[F001]’ + ‘[F002]’;
End;

Note single quotes around field names for string values.

2. If Statement

Begin
If '[F002]'='' then
Result:= 0
else If [F002]=0 then
Result:= 0
Else
Result:= [F001] mod [F002];
End;

3. Variables

var
MyVariable : integer;
Begin
MyVariable:=10;
Result :=[F001] mod MyVariable;
end;

Copyright © 2009 DB Software Laboratory Page 165 of 194

Advanced ETL Processor User Manual

Properties:

Copyright © 2009 DB Software Laboratory Page 166 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 167 of 194

Advanced ETL Processor User Manual

Calculation Function List

LowerCase(S)

LowerCase returns a string with the same text as the string passed in S, but with all letters
converted to lowercase. The conversion affects only 7-bit ASCII characters between 'A' and 'Z'.
To convert 8-bit international characters, use AnsiLowerCase.

AnsiUpperCase(S)

AnsiUpperCase returns a string that is a copy of S, converted to upper case.

AnsiLowerCase(S)

AnsiLowerCase returns a string that is a copy of the given string converted to lower case.

AnsiCompareStr(S1,S2)

AnsiCompareStr compares S1 to S2, with case sensitivity.

The return value is:

Conditio
n Return Value
S1 > S2 >0
S1 < S2 <0
S1 = S2 =0

AnsiCompareText(S1,S2)

AnsiCompareText compares S1 to S2, without case sensitivity. AnsiCompareText returns a

value less than 0 if S1 < S2, a value greater than 0 if S1 > S2, and returns 0 if S1 = S2.

AnsiStrLIComp (S1,S2,MaxLen)

AnsiStrLIComp compares S1 to S2, without case sensitivity. If S1 or S2 is longer than

MaxLen characters, AnsiStrLIComp only compares up to the first MaxLen characters.
The return value is:

Condition Return Value

S1 > S2 >0
S1 < S2 <0
S1 = S2 (up to MaxLen characters) = 0

AnsiLastChar(S)

Call AnsiLastChar to obtain the last character in a string.

Copyright © 2009 DB Software Laboratory Page 168 of 194
Advanced ETL Processor User Manual

Trim(S)

Trim removes leading and trailing spaces and control characters from the given string S.

TrimLeft(S)

TrimLeft returns a copy of the string S with leading spaces and control characters removed.

TrimRight(S)

TrimRight returns a copy of the string S with trailing spaces and control characters
removed.

QuotedStr(S)

Use QuotedStr to convert the string S to a quoted string. A single quote character (') is
inserted at the beginning and end of S, and each single quote character in the string is repeated.

AnsiQuotedStr(S,Quote)

Use AnsiQuotedStr to convert a string (S) to a quoted string, using the provided Quote
character. A Quote character is inserted at the beginning and end of S, and each Quote
character in the string is doubled.

AnsiExtractQuotedStr(S,Quote)

AnsiExtractQuotedStr removes the quote characters from the beginning and end of a quoted
string, and reduces pairs of quote characters within the string to a single quote character. The
Quote parameter defines what character to use as a quote character. If the first character in S is
not the value of the Quote parameter, AnsiExtractQuotedStr returns an empty string.

The function copies characters from S to the result string until the second solitary quote
character or the first null character in S. The S parameter is updated to point to the first
character following the quoted string. If S does not contain a matching end quote character, the
S parameter is updated to point to the terminating null character.

IntToStr(S)

IntToStr converts an integer into a string containing the decimal representation of that
number.

IntToHex(I,Digits)

IntToHex converts a number into a string containing the number's hexadecimal (base 16)
representation. Value is the number to convert. Digits indicates the minimum number of
hexadecimal digits to return.

StrToInt(S)

Copyright © 2009 DB Software Laboratory Page 169 of 194

Advanced ETL Processor User Manual

StrToInt converts the string S, which represents an integer-type number in either decimal or
hexadecimal notation, into a number. If S does not represent a valid number, StrToInt raises an
exception.

StrToIntDef(S,Default)

StrToIntDef converts the string S, which represents an integer-type number in either

decimal or hexadecimal notation, into a number. If S does not represent a valid number,
StrToIntDef returns Default.

FileAge(File)ame)

Call FileAge to obtain the OS timestamp of the file specified by FileName. The return value
can be converted to a TDateTime object using the FileDateToDateTime function. The return
value is -1 if the file does not exist.

FileExists(File)ame)

FileExists returns true if the file specified by FileName exists. If the file does not exist,
FileExists returns false.

DeleteFile(File)ame)

DeleteFile deletes the file named by FileName from the disk. If the file cannot be deleted or
does not exist, the function returns false.

RenameFile(OldFile,)ewFile)

RenameFile attempts to change the name of the file specified by OldFile to NewFile. If the
operation succeeds, RenameFile returns true. If RenameFile cannot rename the file (for
example, if the application does not have permission to modify the file), it returns false.

ChangeFileExt(File)ame,EXT)

ChangeFileExt takes the file name passed in FileName and changes the extension of the file
name to the extension passed in Extension. Extension specifies the new extension, including
the initial dot character.

ChangeFileExt does not rename the actual file, it just creates a new file name string.

ExtractFilePath(File)ame)

The resulting string is the leftmost characters of FileName, up to and including the colon or
backslash that separates the path information from the name and extension. The resulting
string is empty if FileName contains no drive and directory parts.

ExtractFileDir(File)ame)

Extracts Directory part from the File Name provided

Copyright © 2009 DB Software Laboratory Page 170 of 194

Advanced ETL Processor User Manual

ExtractFileDrive(File)ame)

ExtractFileDrive returns a string containing the drive portion of a fully qualified path name
for the file passed in the FileName. For file names with drive letters, the result is in the form
"drive". For file names with a UNC path the result is in the form "\servername\sharename". If
the given path contains neither style of path prefix, the result is an empty string.

ExtractFile)ame(File)ame)

The resulting string is the rightmost characters of FileName, starting with the first character
after the colon or backslash that separates the path information from the name and extension.
The resulting string is equal to FileName if FileName contains no drive and directory parts.

ExtractFileExt(File)ame)

Use ExtractFileExt to obtain the extension from a file name.

ExpandFile)ame(File)ame)

ExpandFileName converts the relative file name into a fully qualified path name.
ExpandFileName does not verify that the resulting fully qualified path name refers to an
existing file, or even that the resulting path exists.

ExpandU)CFile)ame(File)ame)

ExpandUNCFileName returns the fully-qualified file name for a specified file name.

ExtractRelativePath(File)ame)

Call ExtractRelativePath to convert a fully qualified path name into a relative path name.
The DestName parameter specifies file name (including path) to be converted. BaseName is
the fully qualified name of the base directory to which the returned path name should be
relative. BaseName may or may not include a file name, but it must include the final path
delimiter.

DiskFree(Drive)

DiskFree returns the number of free bytes on the specified drive, where 0 = Current, 1 = A,
2 = B, and so on.

DiskSize(Drive)

DiskSize returns the size in bytes of the specified drive, where 0 = Current, 1 = A, 2 = B,
etc. DiskSize returns -1 if the drive number is invalid.

GetCurrentDir(Directory)

GetCurrentDir returns the fully qualified name of the current directory.

SetCurrentDir(Directory)

Copyright © 2009 DB Software Laboratory Page 171 of 194

Advanced ETL Processor User Manual

The SetCurrentDir function sets the current directory. The return value is true if the current
directory was successfully changed, or false if an error occurred.

CreateDir(Directory)

CreateDir creates a new directory. The return value is true if a new directory was
successfully created, or false if an error occurred.

RemoveDir(Directory)

Call RemoveDir to remove the directory specified by the Dir parameter. The return value is
true if a new directory was successfully deleted, false if an error occurred. The directory must
be empty before it can be successfully deleted.

FloatToStr(F)

FloatToStr converts the floating-point value given by Value to its string representation. The
conversion uses general number format with 15 significant digits.

StrToFloat(S)

Use StrToFloat to convert a string, S, to a floating-point value. S must consist of an

optional sign (+ or -), a string of digits with an optional decimal point, and an optional
mantissa. The mantissa consists of 'E' or 'e' followed by an optional sign (+ or -) and a whole
number. Leading and trailing blanks are ignored.

EncodeDate(Year,Month,Day)

EncodeDate returns a TDateTime value from the values specified as the Year, Month, and
Day parameters. The year must be between 1 and 9999. Valid Month values are 1 through 12.
Valid Day values are 1 through 28, 29, 30, or 31, depending on the Month value. For example,
the possible Day values for month 2 (February) are 1 through 28 or 1 through 29, depending on
whether or not the Year value specifies a leap year.

EncodeTime(Hour,Min,Sec,MSec)

EncodeTime encodes the given hour, minute, second, and millisecond into a TDateTime
value. Valid Hour values are 0 through 23. Valid Min and Sec values are 0 through 59. Valid
MSec values are 0 through 999. If the specified values are not within range, EncodeTime
raises an EConvertError exception. The resulting value is a number between 0 and 1
(inclusive) that indicates the fractional part of a day given by the specified time or (if 1.0)
midnight on the following day. The value 0 corresponds to midnight, 0.5 corresponds to noon,
0.75 corresponds to 6:00 pm, and so on.

DayOfWeek(D)

Copyright © 2009 DB Software Laboratory Page 172 of 194

Advanced ETL Processor User Manual

DayOfWeek returns the day of the week of the specified date as an integer between 1 and 7,
where Sunday is the first day of the week and Saturday is the seventh.

Date

Use Date to obtain the current local date as a TDateTime value. The time portion of the
value is 0 (midnight).

Time

Use Time to return the current time as a TDateTime value. The two functions are
completely equivalent.

)ow

Returns the current date and time.

IncMonth(D)

IncMonth returns the value of the Date parameter, incremented by NumberOfMonths

months. NumberOfMonths can be negative, to return a date N months previous. If the input
day of month is greater than the last day of the resulting month, the day is set to the last day of
the resulting month. The time of day specified by the Date parameter is copied to the result.

IsLeapYear(D)

Call IsLeapYear to determine whether the year specified by the Year parameter is a leap
year. Year specifies the calendar year. Use YearOf to obtain the value of Year for IsLeapYear
from a TDateTime value.

DateToStr(D)

Use DateToStr to obtain a string representation of a date value that can be used for display
purposes.

TimeToStr(D)

TimeToStr converts the Time parameter, a TDateTime value, to a string.

DateTimeToStr(D)

Converts a TDateTime value to a string.

StrToDate(S)

Call StrToDate to parse a string that specifies a date. If S does not contain a valid date,
StrToDate raises an exception.

StrToTime(S)

Copyright © 2009 DB Software Laboratory Page 173 of 194

Advanced ETL Processor User Manual

Call StrToTime to parse a string that specifies a time value. If S does not contain a valid
time, StrToTime raises an exception.

StrToDateTime(S)

Call StrToDateTime to parse a string that specifies a date and time value. If S does not
contain a valid date, StrToDateTime raises an exception.

FormatDateTime(Format,DateTime)

FormatDateTime formats the TDateTime value given by DateTime using the format given
by Format. See the table below for information about the supported format strings.

Abort

Use Abort to escape from an execution path without reporting an error.

Beep

Beep generates a conventional message beep.

AnsiPos(Substr,S)

Call AnsiPos to obtain the byte offset of the Substr parameter, as it appears in the string S.
For example, if Substr is the string "AB", and S is the string "ABCDE", AnsiPos returns 1. If
Substr does not appear in S, AnsiPos returns 0.

Chr(X)

Returns the character for a specified ASCII value.

Abs(X)

Returns an absolute value.

Length(X)

Returns the number of characters in a string or elements in an array.

Copy(S,Index,Count)

S is an expression of a string or dynamic-array type. Index and Count are integer-type

expressions. Copy returns a substring or sub array containing Count characters or elements
starting at S[Index]. The substring or sub array is a unique copy (that is, it does not share
memory with S, although if the elements of the array are pointers or objects, these are not
copied as well.) If Index is larger than the length of S, Copy returns an empty string or array. If
Count specifies more characters or array elements than are available, only the characters or
elements from S[Index] to the end of S are returned.

Round(X)

Copyright © 2009 DB Software Laboratory Page 174 of 194

Advanced ETL Processor User Manual

Returns the value of X rounded to the nearest whole number.

Trunc(X)

Truncates a real number to an integer.

Pos(Substr,Str)

Pos searches for Substr within S and returns an integer value that is the index of the first
character of Substr within S. Pos is case-sensitive. If Substr is not found, Pos returns zero

Delete(S,Index,Count)

Delete removes a substring of Count characters from string S starting with S[Index]. S is a
string-type variable. Index and Count are integer-type expressions. If index is larger than the
length of the string or less than 1, no characters are deleted. If count specifies more characters
than remain starting at the index, Delete removes the rest of the string. If count is less than or
equal to 0, no characters are deleted

Insert(Substr,Dest,Index)

Insert merges Source into S at the position S[index]. Source is a string-type expression. S is
a string-type variable of any length. Index is an integer-type expression. It is a character index
and not a byte index. If Index is less than 1, it is mapped to a 1. If it is past the end of the
string, it is set to the length of the string, turning the operation into an append. If the Source
parameter is an empty string, Insert does nothing

Sqr(X)

the Sqr function returns the square of the argument. X is a floating-point expression. The
result, of the same type as X, is the square of X, or X*X.

Sqrt(X)

The result is the square root of X.

Exp(X)

Exp returns the value of e raised to the power of X, where e is the base of the natural
logarithms

Ln(X)

Ln returns the natural logarithm (Ln(e) = 1) of the real-type expression X.

Sin(X)

Sin returns the sine of the angle X in radians.

Copyright © 2009 DB Software Laboratory Page 175 of 194

Advanced ETL Processor User Manual

Cos(X)

Cos returns the cosine of the angle X. X expression that represents an angle in radians

Tan(X)

Tan returns the tangent of X. Tan(X) = Sin(X) / Cos(X).

ArcTan(X)

ArcTan returns the arctangent of X. X is a real-type expression that gives an angle in

radians

SetLength(S,Length)

Set Length of dynamic array or string

High(X)

Call High to obtain the upper limit of an Array

Low(X)

Call Low to obtain the lowest value or first element of an Array.

PI

Represents the mathematical value pi, the ratio of a circle's circumference to its diameter. Pi
is approximated as 3.1415926535897932385.

ArcCos(X)

ArcCos returns the inverse cosine of X. X must be between -1 and 1. The return value is in
the range [0..Pi], in radians.

ArcCosh(X)

ArcCosh returns the inverse hyperbolic cosine of X. The value of X must be greater than or
equal to 1.

ArcCot(X)

ArcCot returns the inverse cotangent of X.

ArcCotH(X)

ArcCot returns the inverse hyperbolic cotangent of X.

ArcCsc(X)

Copyright © 2009 DB Software Laboratory Page 176 of 194

Advanced ETL Processor User Manual

ArcCsc returns the inverse cosecant of X.

ArcCscH(X)

ArcCsc returns the inverse hyperbolic cosecant of X.

ArcSec(X)

ArcSec returns the inverse secant of X.

ArcSecH(X)

ArcSec returns the inverse hyperbolic secant of X.

ArcSin(X)

ArcSin returns the inverse sine of X. X must be between -1 and 1. The return value will be
in the range [-Pi/2..Pi/2], in radians.

ArcSinh(X)

ArcSinh returns the inverse hyperbolic sine of X.

ArcTan(X)

ArcTan returns the arctangent of X. X is a real-type expression that gives an angle in

radians.

ArcTanh(X)

ArcTanh returns the inverse hyperbolic tangent of X. The value of X must be between -1
and 1 (inclusive).

Ceil(X)

Call Ceil to obtain the lowest integer greater than or equal to X. The absolute value of X
must be less than MaxInt. For example: Ceil(-2.8) = -2 Ceil(2.8) = 3 Ceil(-1.0) = -1

Cosecant(X)

Use the Cosecant to calculate the cosecant of X, where X is an angle in radians. The
cosecant is calculated as 1/ Sin(X).

Cosh(X)

Use the Cosh to calculate the hyperbolic cosine of X.

Cot(X)

Copyright © 2009 DB Software Laboratory Page 177 of 194

Advanced ETL Processor User Manual

Call Cot to obtain the cotangent of X. The cotangent is calculated using the formula 1 / Tan
(X).

Cotan(X)

Call Cotan to obtain the cotangent of X. The cotangent is calculated using the formula 1 /
Tan (X)
Do not call Cotan with X = 0

CotH(X)

Call CotH to obtain the hyperbolic cotangent of X, where X is an angle in Radians.

Csc(X)

Use the Csc to calculate the cosecant of X, where X is an angle in radians.

CscH(X)

Use the CscH to calculate the hyperbolic cosecant of X, where X is an angle in radians.

CycleToDeg(X)

CycleToDeg converts angles measured in cycles into degrees, where degrees = cycles * 360.

CycleToGrad(X)

CycleToGrad converts angles measured in cycles into grads.

CycleToRad(X)

CycleToRad converts angles measured in cycles into radians, where radians = 2pi * cycles.

DegToCycle(X)

Use DegToCycle to convert angles expressed in degrees to the corresponding value in

cycles.

DegToGrad(X)

Use DegToGrad to convert angles expressed in degrees to the corresponding value in grads.

DegToRad(X)

Use DegToRad to convert angles expressed in degrees to the corresponding value in

radians, where radians = degrees(pi/180).

Floor(X)

Copyright © 2009 DB Software Laboratory Page 178 of 194

Advanced ETL Processor User Manual

Call Floor to obtain the highest integer less than or equal to X. For example: Floor(-2.8) = -
3 Floor(2.8) = 2 Floor(-1.0) = -1

GradToCycle(X)

GradToCycle converts angles measured in grads into cycles.

GradToDeg(X)

GradToDeg converts angles measured in grads into degrees.

GradToRad(X)

GradToRad converts angles measured in grads into radians, where radians = grads(pi/200).

Hypot(X,Y)

Hypot returns the length of the hypotenuse of a right triangle. Specify the lengths of the
sides adjacent to the right angle in X and Y. Hypot uses the formula Sqrt(X**2 + Y**2)

IntPower(Base,Exponent)

IntPower raises Base to the power specified by Exponent.

Ldexp(X)

Ldexp returns X times (2 to the power of P).

LnXP1(X)

LnXP1 returns the natural logarithm of (X+1). Use LnXP1 when X is a value near 0.

Log10(X)

Log10 returns the log base 10 of X.

Log2(X)

Log2 returns the log base 2 of X.

Log)(Base,X)

LogN returns the log base Base of X.

Max(A,B)

Call Max to compare two numeric values. Max returns the greater value of the two.

Min(A,B)

Copyright © 2009 DB Software Laboratory Page 179 of 194

Advanced ETL Processor User Manual

Call Min to compare two numeric values in Delphi. Min returns the smaller value of the
two.

Power(Base,Exponent)

Power raises Base to any power. For fractional exponents or exponents greater than MaxInt,
Base must be greater than 0.

RadToCycle(X)

Use RadToCycle to convert angles measured in radians into cycles, where cycles =
radians/(2pi).

RadToDeg(X)

Use RadToDeg to convert angles measured in radians to degrees, where degrees =

radians(180/pi).

RadToGrad(X)

Use RadToGrad to convert angles measured in radians to grads, where grads =

radians(200/pi).

RandG(Mean,StdDev)

RandG produces random numbers with Gaussian distribution about the Mean. This is useful
for simulating data with sampling errors and expected deviations from the Mean.

RandomRange(AFrom,ATo)

RandomRange returns a random integer from the range that extends between AFrom and
ATo (non-inclusive). RandomRange can handle negative ranges (where AFrom is greater than
ATo). To initialize the random number generator, add a single call Randomize or assign a
value to the RandSeed variable before making any calls to RandomRange.

Sec(X)

Call Sec to obtain the secant of X, where X is an angle in radians. The secant is calculated
using the formula 1 / Cos(X).

SecH(X)

Call SecH to obtain the hyperbolic secant of X, where X is an angle in Radians.

Sinh(X)

Sinh calculates the hyperbolic sine of X.

Tan(X)

Copyright © 2009 DB Software Laboratory Page 180 of 194

Advanced ETL Processor User Manual

Tan returns the tangent of X. Tan(X) = Sin(X) / Cos(X).

Tanh(X)

Tanh calculates the hyperbolic tangent of X.

Copyright © 2009 DB Software Laboratory Page 181 of 194

Advanced ETL Processor User Manual

7.4.7 Lookup

Category: Transformation Function

Description: This function substitutes values using mapping table. Mapping table can be
populated manually or loaded from a file/Database.
Properties:

Copyright © 2009 DB Software Laboratory Page 182 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 183 of 194

Advanced ETL Processor User Manual

Copyright © 2009 DB Software Laboratory Page 184 of 194

Advanced ETL Processor User Manual

7.4.8 Previous Value

Category: Transformation Function

Description: This function returns previous value of field. For the first record it returns null.
This function can be used to calculate running totals
Properties:

Copyright © 2009 DB Software Laboratory Page 185 of 194

Advanced ETL Processor User Manual

7.4.9 Reader Parameter

Category: Transformation Function

Description: This function returns selected metadata information from the Reader object
Properties:

Copyright © 2009 DB Software Laboratory Page 186 of 194

Advanced ETL Processor User Manual

7.4.10 Sequence

Category: Transformation Function

Description: This function returns a number incremented by one starting from “Start With”
value
Properties:

Copyright © 2009 DB Software Laboratory Page 187 of 194

Advanced ETL Processor User Manual

7.4.11 Running Total

Category: Transformation Function

Description: The running total represents an instantaneous total of all the "quantities" of stuff
that can be reviewed at any time during the recording period to determine what the total was at
that time of review.

Here's a sample of a running total of that store's receipts:

Sale 1: $6.49 -- Running total: $6.49

Sale 2: $2.81 -- Running total: $9.30
Sale 3: $1.37 -- Running total: $10.67
Sale 4: Refund: $0.89 -- Running total: $9.78
Sale 5: $5.26 -- Running total: $15.04

Properties:

Copyright © 2009 DB Software Laboratory Page 188 of 194

Advanced ETL Processor User Manual

8. Date formats

Date/Time format strings control the conversion of strings into date time type.

Date/Time format strings are composed from specifiers which describe values to be
converted into the date time value.

In the following table, specifiers are given in lower cases. Case is ignored in formats,
except for the "am/pm" and "a/p" specifiers.

Specifier Description

d Day as a number without a leading zero (1-31).

dd Day as a number with a leading zero (01-31).

m Month as a number without a leading zero (1-12).

mm Month as a number with a leading zero (01-12).
mmm Month as an abbreviation (Jan-Dec).
mmmm Month as a full name (January-December).

yy Year as a two-digit number (00-99).

yyyy Year as a four-digit number (0000-9999).

h Hour without a leading zero (0-23).

hh Hour with a leading zero (00-23).

n Minute without a leading zero (0-59).

nn Minute with a leading zero (00-59).

s Second without a leading zero (0-59).

ss Second with a leading zero (00-59).

fff Fraction of Second with a leading zero (000-999).

tt Uses the 12-hour clock for the preceding h or hh specifier, 'am' for any hour
before noon, and 'pm' for any hour after noon.

Important thing is to understand that this format has nothing to do with your target database.
This is the format of the source data. It is there to help to covert string into date time type
inside of the software, so it can be loaded later into date or timestamp field

So if source data is:

16/08/2009 than the format is DD/MM/YYYY
1/31/2009 than the format is M/D/YYYY
2006-05-23 22:34:42.096 than the format is YYYY-MM-DD HH:NN:SS.FFF
1992/mar/12 00:00 than the format is YYYY/MMM/DD HH:NN

Copyright © 2009 DB Software Laboratory Page 189 of 194

Advanced ETL Processor User Manual

9. Command Line

To run an transformation from the command line type AdvancedETL.exe trasformation.ats.

Copyright © 2009 DB Software Laboratory Page 190 of 194

Advanced ETL Processor User Manual

10. Support Procedure

We welcome your suggestions and comments to make our software better.

Please do contact us through email or visit our website

For General Sales and License Queries

Email to: [email protected]

For Technical Help

Support Forum: http://www.dbsoftlab.com/index.php/Support-Forum/
Email To: [email protected]

Online Contact Form: http://www.dbsoftlab.com/index.php/Contact-Us/

While submitting a bug or problem please include the following to make it easier to solve the
problem as soon as possible:

• All the symptoms of a Problem

• Exact steps required to reproduce it
• Version number
• Operation system version
• Database version
• Log files
• Script
• Sql script to create tables
• Screenshots

We intend to resolve 90 percent of the reported problems with in 48 hours.

Most difficult issues or functionality extension are resolved with in one week

Copyright © 2009 DB Software Laboratory Page 191 of 194

Advanced ETL Processor User Manual

11. License Agreement

Advanced ETL Processor by
DB Software Laboratory
www.dbsoftlab.com
[email protected]

END-USER LICENSE AGREEMENT FOR THIS SOFTWARE

IMPORTANT - READ CAREFULLY:

This End-User License Agreement ("EULA") is a legal agreement between you (either
an individual or a single entity) and DB Software Laboratory for the SOFTWARE
PRODUCT identified above, which includes computer software and may include
associated media, printed materials, and "online" or electronic documentation. By
installing, copying, or otherwise using the SOFTWARE PRODUCT, you agree to be
bound by the terms of this EULA. If you do not agree to the terms of this EULA, you
may be subject to civil liability if you install and use this SOFTWARE PRODUCT.

SOFTWARE PRODUCT LICENSE

The SOFTWARE PRODUCT is protected by copyright laws and international

copyright treaties, as well as other intellectual property laws and treaties.

The SOFTWARE PRODUCT is licensed, not sold.

INSTALLATION AND USAGE

Once SOFTWARE PRODUCT is installed you may use it for 30 days. After
evaluation period ends, you have to purchase a license or stop using the
SOFTWARE PRODUCT.

If this is an EVALUATION VERSION of the SOFTWARE PRODUCT, you may copy

and distribute an unlimited number of copies of the SOFTWARE PRODUCT;
provided that each copy shall be a true and complete copy, including all copyright and
trademark notices, and shall be accompanied by a copy of this EULA.

If this is a REGISTERED VERSION of the SOFTWARE PRODUCT, you may install

and use it for your personal use only. You may not reproduce or distribute the
SOFTWARE PRODUCT for use by others.

Copyright © 2009 DB Software Laboratory Page 192 of 194

Advanced ETL Processor User Manual

LICENSING

There are two types of licenses available

1. A single computer usage license. The user purchases one license to use the
SOFTWARE PRODUCT on one computer.

2. A SITE usage license. The user purchases a single usage license, authorising the
use of SOFTWARE PRODUCT, by the purchaser, the purchaser's
employees or accredited agents, on an unlimited number of computers at the same
physical site location. This site location would normally be defined as a single
building, but could be considered to be a number of buildings within the same,
general, geographical location, such as an industrial estate or small town.

OTHER RIGHTS AND LIMITATIONS

You may not reverse engineer, decompile, or disassemble the SOFTWARE

PRODUCT, except and only to the extent that such activity is expressly permitted by
applicable law notwithstanding this limitation. Any such unauthorized use shall result
in immediate and automatic termination of this license and may result in criminal
and/or civil prosecution. All rights not expressly granted here are reserved by DB
Software Laboratory.

The SOFTWARE PRODUCT is licensed as a single product. Its component parts

may not be separated for use on more than one computer.

You may permanently transfer all of your rights under this EULA, provided the
recipient agrees to the terms of this EULA.

SEVERABILITY

In the event of invalidity of any provision of this license, the parties agree that such
invalidity shall not affect the validity of the remaining portions of this license.

COPYRIGHT

The SOFTWARE PRODUCT is protected by copyright laws and international treaty

provisions. All title and copyrights related to the SOFTWARE PRODUCT (including
but not limited to any images, photographs, animations, video, audio, music, text, and
"applets" incorporated into the SOFTWARE PRODUCT), the accompanying printed
materials, and any copies of the SOFTWARE PRODUCT are owned by DB Software
Laboratory.

MISCELLANEOUS

Should you have any questions concerning this EULA, or if you desire to contact the
author of this Software for any reason, please contact DB Software Laboratory (see
contact information at the top of this EULA).
LIMITED WARRANTY

Copyright © 2009 DB Software Laboratory Page 193 of 194

Advanced ETL Processor User Manual

DB Software Laboratory expressly disclaims any warranty for the SOFTWARE

PRODUCT. The SOFTWARE PRODUCT and any related documentation is provided
"as is" without warranty of any kind, either express or implied, including, without
limitation, the implied warranties or merchantability, fitness for a particular purpose, or
no infringement. The entire risk arising out of use or performance of the SOFTWARE
PRODUCT remains with you.

In no event shall DB Software Laboratory be liable for any damages whatsoever or

refund any money (including, without limitation, damages for loss of business profits,
business interruption, loss of business information, or any other pecuniary loss)
arising out of the use of or inability to use this product, even if DB Software
Laboratory has been advised of the possibility of such damages. Because some
states/jurisdictions do not allow the exclusion or limitation of liability for consequential
or incidental damages, the above limitation may not apply to you.

SUPPORT AND MAINTENACE

The annual maintenance fee is 20 percent of initial software license cost.

Users with a fully paid annual maintenance fee get the following benefits:

Priority Support
Free software enhancements, updates and upgrades during the maintenance period
Advanced and exclusive notification of software promotions
"Maintenance Owner ONLY" product promotions

ENTIRE AGREEMENT

This is the entire agreement between you and DB Software Laboratory which
supersedes any prior agreement or understanding, whether written or oral, relating to
the subject matter of this license.

Thank you for using the Advanced ETL Processor.

DB Software Laboratory

Copyright © 2009 DB Software Laboratory Page 194 of 194