Data Conversion

Sybase Database to Oracle

Database

Authors: Sandeep Arsid, Balwinder Saini, Kartik

Josyula
Date: 20 February 2012

Contents
Introduction

Problem Definition

High Level Solution

Implementation

BCP Export

bcp_tables_nm.ksh

bcp_tables_m.ksh

SQL Loader Load into Oracle

Benefits

10

Conclusion

11

Appendix A: bcp_tables.ksh script

12

Appendix B: setup.env Environment setup file

16

Appendix C: bcp_tables_nm.ksh script

17

Appendix D: bcp_tables_m.ksh script

19

Appendix E: keywords.lst

21

Appendix F: sqlldr_tables_trim.ksh script

24

Data conversion success Siebel batch scripting and EIM automation

Problem Definition

Introduction
Data Migration is an important part of application migration. Some of the common drivers for data
migrations are:

Movement of legacy systems to a new and improved platform

Consolidation of applications and data sources after Mergers and Acquisitions

Implement new application functionality

Data load across various applications with different data source requirements

Data migration goes through the following lifecycle

After business impact analysis and setting up data mappings and DDL conversion, the next step is
to convert the data definitions from one platform to another. Data migrations can be very
challenging, particularly if data is migrated from heterogeneous data sources and large amount of
tables are involved.
This whitepaper will talk about the process and tasks involved to simplify the data migration from
Sybase to Oracle.

Data Loading: Sybase Database to Oracle Database

Problem Definition

Problem Definition
There are many ways to perform a data load, including external vendor tools.
But, there could be issues with vendor tools such as
Data structure incompatibility
Licensing Cost
Insufficient support between the external tool vendor and the database vendor etc.
If the internal database tools are used for data load, then the aforementioned problems do not arise.
But, another major issue could be the compatibility and coherence between the data load tools of
the source and target database vendors. One of the options in such cases is to follow the below
steps during data load:

1.Extractthedatafromthesourcedatabaseusingthecorrespondingdataloadutilityandexportitintoa
datafile.
2.Loadthedatafromthedatafileintothetargetdatabaseusingthecorrespondingdataloadutility

For data loads from Sybase to Oracle, the same steps translate as below:
1. ExtractdatafromaSybasetableusingBCPutilityinapredefinedformat.
2. MakecontrolfileforthecorrespondingtableinOracleasperthedesiredloadconditions,formatofthe
datafilecreatedfromBCPutility,andtablestructuredetails.
3.RunSQLLoadertoloadthedatafromthedatafileusingthecontrolfileintothetargettableinOracle.

Some of the major challenges observed in the above process are as stated
below:
1. DefiningtheformatoftheflatfilewhichcanbewrittenbytheBCPutilityandreadbySQLLoader
utility.
Itshouldbekeptinmindthattheformat,especiallytherowandcolumndelimiters,arenotmixedup
withthedataitselfandremainunique.
2.DefiningtheparametersfortheBCPextractsothattheloadontheSybaseserverisoptimized.
3.Sybasedatabasestorestrailingwhitespacesattheendofcharacterdataifthecharacterfieldshave
additionalbytesthantheactualdata.Sometimes,applicationsalsoinsertrecordswithleadingand
trailingspacesalongwiththecharacterdata.But,whencharacterdataisqueried,Sybaseignoresthe
trailingwhitespacesandincludestherecordswiththetrailingspacesinitsresultset;whileOracle
expectsthedatatobeexacttobeincludedinthequeryresults.Itisimportanttonotethatleading
whitespacesaretreatedthesamewaybothinSybaseandOracle.

Data Loading: Sybase Database to Oracle Database

Problem Definition

4.WhenBCPutilityexportsdatawithMONEY,SMALLMONEYandMONEYNdatatypesfrom
Sybasetables,itextractsdatatill2decimalplacesonlyandroundsoffanyextradecimalspresent.
Sincedataisextractedonlytill2decimalplaces,SQLLoaderloadsonly2decimalplaces.But,while
queryingonSybase,thedataisaccurateto4decimalspaces.Hence,thereisadiscrepancyinthedata
betweenSybaseandOracle.
5.TheSybaseuserwhichisusedtoextractdatamighthaveatimeoutlimitonthedurationforwhicha
querycanrun.ThiscouldleadtosuddendisconnectbetweenBCPutilityanddatabasewhenextracting
data.
6.Definingthecolumnformatsandloadingconditionsinthecontrolfile.
7.Automationofthecompletedataloadprocesswithminimalmanualintervention.
8.Identifyingandresolvingadhocerrorsduetodifferenceindatastorageanddataformats.Thesecould
beraresincedatastructureissuesareassumedtoberesolvedbeforehand.

Data Loading: Sybase Database to Oracle Database

Problem Definition

High Level Solution

All the challenges presented in the previous section require a deep analysis on
1. TheserverloadcapabilitiesofSybaseandOracle
2.Formatofthedatafile
3.LoadingparametersandColumnformatsforOracle
To come to a conclusion on the above parameters, it is imperative to get the approval of the
stakeholders on some issues and to do some proof of concept for others.
It is recommended to strive for automation of data load because the data load process is, in general,
a very routine process. The more the manual intervention, the more the chances of errors in the data
load process. But, to achieve automation, a clear logic should be in place for resolving most of the
highly frequent issues. Further, the automation can be done at various levels depending on the
number of tables and schemas to be loaded and the frequency of issues that need manual
intervention.

1. Ifthenumberoftablestoloadisless,thentheautomationcanbeatthetablelevelwhereintheload
parametersarepreestablishedandtheloadisrunforeachtable.
2.Ifthenumberoftablestoloadisveryhigh,thentheautomationcanbeatdatabaseorschemalevel
whereintheBCPrunsinparallelwiththeSQLLoaderandSQLLoaderkicksoffassoonasBCPis
completeforatable.
Inthiscase,ifanexportthroughBCPtakeslongforaparticulartable,thenitdoesnotholdtheSQL
Loaderfromstartingtheloadfortableswithlessdata,thereby,releasingburdenontheusertosort
theBCPexportsasperthetablesdata.

Data Loading: Sybase Database to Oracle Database

Problem Definition

Implementation
In this paper, the discussion is mainly based on automating the data loads from Sybase to
Oracle when the number of tables is extremely high to the extent of migrating databases of
Sybase itself. The basic analysis around the data loads culminated in the below decisions on
Sybase side:
1. Regardingtheformatofthedatafile,itwasdeemedpreferabletousecharacterformatforBCPexport
(coptionintheBCPcommand)sincethenativeformatisnotinreadableformandcannotbe
understoodbySQLLoaderorBCPitselfincaseofimportingdatabacktoSybase.
2.ThedefaultdelimitersofBCPare\t(TAB)forcolumndelimiterand\n(NewLine)forrow
delimiter.BothofthecharactersinterferedwiththeactualdatapresentinSybasedatabase.Sincethis
couldresultinerrorproneandsometimesinvaliddataloads,thedelimiters,<EOFD>and
<EORD>wereusedascolumn(toption)androw(roption)delimitersforBCPexport.
ThesamewereusedintheControlfiledefinitionofOracleLoads.
3.Thebatchsizeoption(b)ofBCPcommandoffersanoptiontospecifycommitintervalswhile
insertingdataintoatablesinSybase.ThoughthishasnoeffectontheexportingofdatausingBCP,a
valueof50000wasdecidedontheoptionvaluesoastomaintainacommonBCPcommandforexport
andimport.
4.InordertohelptheSybaseserverhandleparallelexportsoftablesfromSybase,itwasdecidedtorun
2550BCPexportsatapointoftimeforagivendatabase.Dependingonthesizeoftablesina
particulardatabase,thenumbercanbechangedtosuitetheneed.
5.Inthecaseoftableshavingmorethan1billionrecordsorhavingmorethan10millionrecordswith
over3040columns,theuseofoneBCPthreadtoexportthewholedataintoasingledatafileseemed
veryloadintensive.Hence,insuchcasestheuseofParallelBCPthreadstoexportdataintomultiple
datafileswasimplemented.
Inthisscenario,itisnecessarytospecifytheF(thenumberofthefirstrowinthetabletostartthe
export)andLoptions(thenumberofthelastrowinthetablewhichistobeexported).Thishelps
inquickerexportandreducesthechancesoflargesizesofdatafiles.
Further,themultipledatafilesthuscreatedcanbeagainloadedinparallelintoOracle.
6.InordertocountertheissueofdatawithMONEY,SMALLMONEY,andMONEYNdatatypes,such
tableswereinitiallyfoundoutbyqueryingthesystemtablesofSybase.Sinceitisknownthatthedata
incolumnswiththesedatatypesisaccurateupto4decimals,viewswerecreatedoverthesetables
whileconvertingthedatatypeofthecorrespondingcolumnstoNUMERIC(19,4).Aftercreatingthe
views,thedatawasexportedfromtheviewsandthismadeitpossibletogetthedatatill4decimal
valuesfromtheSybasetables,therebyeliminatingthedatavalidationissues.
7.ThereisalsoanoptiontospecifytheinterfacesfileandtheSybaseservernamefromwhichtheBCP
exportistotakeplace.
TheinterfacesfilecontainstheconnectioninformationforvariousSybaseserversandtheSybase
servernameshouldbespecifiedexactlyaspresentintheinterfacesfilefortheBCPutilitytomake
theconnection.

On the Oracle side, the following decisions were made:

Data Loading: Sybase Database to Oracle Database

Problem Definition

1. TheDirectPathLoadoptionisusedtoloaddataintoOracle.SincethisrequiressettinguptheNLS
parameter,theNLS_LANGenvironmentvariableisalsoset.Theloadisruninunrecoverablemodeto
speedupthedataload.
2. Thecommitpointfordataloadissetat1000000.Thiswouldreducetheloadontheserverasitneed
notputthewholeincomingdatainbufferbeforeacommit.
3. Sincethedataloadisperformedatdatabaselevelwithlotoftablesbeingloaded,theindex
maintenanceisskippedduringthedataloadintoOracle.Itisperformedafterthecompletionofalldata
loadsatthedatabaseorindexlevelasperconvenience.
4. Duringthedataload,severalconstraintsaredisabledsoastoaidinfasterloads.Theseconstraintsare
reenabledattheendofthedataload.Thesuccess/failureoftheenablingprocessdependsonthedata
consistencyandthestatusisindicatedintheSQLLoaderlogfile.
5. Thefieldandrowdelimitersarespecifiedas<EOFD>and<EORD>toremainconsistentwiththe
Sybasespecifications.
6. Columnformattingforvariousdatatypesislistedbelow:

Column
Data type
TIMESTAMP(3
)

CLOB
DATE

Column Format
TIMESTAMP "Mon dd yyyy
HH12:MI:SS:FF3AM"
TRANSLATE (:<COLUMN_NAME in lower
case>,'1,','1')
CHAR(2147483647) TERMINATED BY
{'<EORD>'|'<EOFD>'}
DATE "Mon dd yyyy HH12:MIAM"

VARCHAR2
CHAR
Others

"case when rtrim(:<column_name>) is null

and :<column_name> is not null then ' ' else
rtrim(:<column_name>) end"
PRESERVE BLANKS
No Formatting parameters specified

NUMBER

TheformattingforVARCHAR2datatypetakescareofthetrailingspacesissuethatwas
encounteredbetweenSybaseandOracle.
7. TheloadisdoneinINSERTINTOformat.Hence,itisimperativetocheckthatthetargettableis
emptybeforetheSQLLoaderloadsdataintothetable.Ifthetargettableisnotempty,theSQLLoader
willfail.
8. InordertohelptheOracleserverhandleparallelloadingintotables,itwasdecidedtorun200300
SQLLoaderloadsatapointoftimeforagivendatabase.Dependingonthesizeoftablesina
particulardatabase,thenumbercanbechangedtosuitetheneed.
9. ThereisalsoanoptiontospecifythepathoftheTNSNAMES.ORAfileandtheOracledatabasename
towhichtheSQLLoaderloads.
TheTNSNAMES.ORAcontainstheconnectioninformationforvariousOracledatabasesandthe
OracledatabasenameshouldbespecifiedexactlyaspresentinTNSNAMES.ORAfortheSQL
Loaderutilitytomaketheconnection.
Thedecisionsmentionedfrom17areappliedduringtheControlfilecreationoftheOracleTable.The
BCPandSQLLoaderscriptsarestartedindependentofeachother.TheSQLLoaderscriptiswrittenin
suchawaythatitwaitsforthesuccessfulcompletionoftheBCPexportforaparticulartablebefore
startingtheloadintoOracle.

Data Loading: Sybase Database to Oracle Database

Problem Definition

BCP Export
The export of data from Sybase tables to data file using BCP utility is primarily kicked off by using
the bcp_tables.ksh script. The script takes the name of the database for which the export is to be
done as a parameter. The detailed script is given in Appendix A. It further uses the tables_syb.lst as
the list of tables for which data should be exported. It further uses setup.env file to look up the
environment settings, the typical example of which is present in Appendix B. As far as BCP utility is
concerned, the variables of importance in the environment setup file are:

a)

uid: The user using which the BCP utility logs into Sybase database

b)

passwd: The password for the user.

c)

server: The Sybase server the BCP utility connects to.

d)

interfaces_file: The interfaces file location for the BCP utility.

e)

data_dir: The directory where the data file and BCP log is stored. For each database, a directory
by the database name is made in this location and the data files and BCP logs for the particular
database are present in the corresponding folder.

f)

max_proc_syb: The number of parallel BCP exports per database.

The syntax for running the bcp_tables.ksh command is as follows:

nohup ./bcp_tables.ksh <Sybase DB name> &
The bcp_tables.ksh script divides the tables into two parts depending on the presence of MONEY,
SMALLMONEY and MONEYN data types. The tables not having these data types are over written
on the tables_syb.lst file while the ones with these data types are written into tables_m.lst and
processed further. To start the BCP export of the tables without the above data types, this script calls
bcp_tables_nm.ksh script. The details of this script are present in Appendix C.
For the tables with MONEY, SMALLMONEY and MONEYN data types, the bcp_tables.ksh creates a
view with the above data type columns converted as NUMERIC (19, 4). The original table name and
view name combination is maintained in change_tnames.lst file. The script copies the new table list,
change_tnames.lst, setup.env and a new script called bcp_tables_m.ksh to a folder, temp, and runs
the bcp_tables_m.ksh to start the exports for these tables. The details of bcp_tables_m.ksh are
present in Appendix D.

bcp_tables_nm.ksh
The bcp_tables_nm.ksh creates one BCP export file for each table in tables_syb.lst in temp_syb
folder and runs them in parallel as per the settings in setup.env file. For tables having the names
specified in a file, keywords.lst, present in Appendix E, this script renames them to <table_name>_1.
This is performed since it was decided not to have any object names in Oracle with the listed
keywords and all such words shall be renamed as <NAME>_1. The data file name is maintained as
<TABLE NAME>.dat and the BCP log file is maintained as bcp_<TABLE NAME>.log at
<data_dir>/<Sybase DB name> folder. The data_dir parameter is obtained from setup.env file.
Depending on status of the BCP export the BCP log file name is renamed as:

1. TABLENAME>.readyincasetheBCPexportissuccessful.
2. TABLENAME>.errorincasetheBCPexportisnotsuccessful.
Depending on the information in the log file for the errored exports, the BCP export is rectified and
run again by the user.

Data Loading: Sybase Database to Oracle Database

Problem Definition

bcp_tables_m.ksh
The bcp_tables_m.ksh creates one BCP export file for each of the views created by the
bcp_tables.ksh script. If the view is created in a database called viewdb, the data file and the BCP
log file for the export are created in <data_dir>/<viewdb>_<Sybase db> as <VIEW NAME>.dat and
bcp_<VIEW NAME>.log. Depending on the success or failure of the BCP exports, the BCP export
log file is renamed as <VIEW NAME>.ready and <VIEW NAME>.error as specified earlier.
As soon as the BCP export is complete for the view, and the .ready or .error file is
created, the bcp_tables.ksh script moves the data file and the log file to <data_dir>/<Sybase DB>
directory as <TABLE NAME>.dat and <TABLE NAME>.{error|ready} by using the mapping created in
change_tnames.lst file.

SQL Loader Load into Oracle

At the time when the BCP exports are started, the SQL Loader script is also started. The SQL
Loader script is kicked off using the sqlldr_tables_trim.ksh script. The script uses the list of tables
present in tables_ora.lst to create the control file and the SQL Loader script. The script takes the
schema name as a mandatory parameter. Further, the user has the option to specify:

1. OnlycreatecontrolfilesandSQLLoaderscripts.ThesyntaxforrunningtheSQLLoaderwiththis
optionisasfollows:
nohup./sqlldr_tables_trim.kshs<Oracleschemaname>C&
2. OnlyruntheSQLLoaderscripts.ThesyntaxforrunningtheSQLLoaderwiththisoptionisas
follows:
nohup./sqlldr_tables_trim.kshs<Oracleschemaname>L&
3. CreatethecontrolfileandSQLLoaderscriptsandthenruntheSQLLoaderscripts.Thesyntaxfor
runningtheSQLLoaderwiththisoptionisasfollows:
nohup./sqlldr_tables_trim.kshs<Oracleschemaname>&
The script uses setup.env, the environment settings file for the following parameters:

a)

ORA_SERVER: The Oracle server the SQL Loader connects to.

b)

ORA_USR: The user using which SQL Loader logs into the Oracle database.

c)

ORA_PWD: The password for the user.

d)

ORA_DATA_DIR: The path wherein the SQL Loader looks for the data file and the successful
completion of the BCP log file. For each database, a directory by the database name is made in
this location and the SQL Loader looks for the files in the corresponding database folder.

e)

CTL_FILE_DIR: The path wherein the SQL Loader creates the control files. For each database, a
directory by the database name is made in this location and the control files for the particular
database are present in the corresponding folder.

f)

LOG_FILE_DIR: The path wherein the SQL Loader creates the SQL Loader log files. For each
database, a directory by the database name is made in this location and the log files for the
particular database are present in the corresponding folder.

g)

BAD_FILE_DIR: The path wherein the SQL Loader creates the bad files in which the rejected
records of the data load are sent. For each database, a directory by the database name is made
in this location and the bad files for the particular database are present in the corresponding
folder.

Data Loading: Sybase Database to Oracle Database

Problem Definition

h)

NLS_LANG: The NLS_LANG parameter for the SQL Loader is specified here.

i)

TNS_ADMIN: The location of the TNSNAMES.ORA is specified here.

j)

max_proc_ora: The number of parallel SQL Loader threads per database.

The details of the script are given in Appendix F.

Once the script is started, as long as -L option is not specified, it starts of by creating
Control Files in <CTL_FILE_DIR>/<DB name> folder and SQL Loader scripts in temp_ora folder. If C option is specified, the script stops after this point.
In case -C option is not specified, the script starts to look for *.ready files in the
<ORA_DATA_DIR>/<DB Name> directory. Typically, this directory points to the same location as the
<data_dir>/<Sybase DB Name> from the BCP exports. Once it finds a file with the .ready
extension and the file name matches a table name present in tables_ora.lst, it does the following:
1)
2)
3)
4)

Renames the extension of the file to .next.

Starts the corresponding SQL Loader script for the table present in temp_ora.
This script again renames the file to .running.
Starts the SQL Loader for the table. As the load runs, the SQL Loader log is created as <TABLE
NAME>.log in <LOG_FILE_DIR>/<DB Name> and the bad file, if rejected records are present, is
created in <BAD_FILE_DIR>/<DB Name>.
5) If the Oracle load completes successfully,
a. The BCP log file at <ORA_DATA_DIR>/<DB Name> which is with .running extension is
renamed with .done extension.
b. The SQL Loader log file at <LOG_FILE_DIR>/<DB Name> which is with .log extension is
renamed with .done extension.
6) If the Oracle load does not complete successfully,
a. The BCP log file at <ORA_DATA_DIR>/<DB Name> which is with .running extension is
renamed with .skip extension.
b. The SQL Loader log file at <LOG_FILE_DIR>/<DB Name> which is with .log extension is
renamed with .error extension.
Once the number of files with .done, .skip and .error extensions in <ORA_DATA_DIR>/<DB
name> location is equal to the number of tables listed in tables_ora.lst, the sqlldr_tables_trim.ksh
stops. In case the sqlldr_tables_trim.ksh script needs to stopped on urgency, then it is enough to
create an empty stop file as shown below in the <ORA_DATA_DIR>/<DB name> location:
touch stop
This would kill the sqlldr_tables_trim.ksh and in order to resume the load again, delete the stop file
before starting the script.
It is important to note that the sqlldr_tables_trim.ksh script keeps looking for .ready files as
long as the equality is not established. In case there are older logs in the <ORA_DATA_DIR>/<DB
Name> location, then, the script would run in a loop forever since the equality is never established.
In such scenarios, it is important to kill the script manually or create the stop file to kill the script.
Another advantage of the script is the advantage during SQL Loader errors. If a SQL Loader
load fails due to some reason, it is not needed to wait for the sqlldr_tables_trim.ksh to complete or
kill it and then restart again. All one needs to do is the following:
1)

Resolve the Load error issue.

2)

Truncate the table.

3)

Change the extension of the BCP log file at <ORA_DATA_DIR>/<DB Name> from .skip to
.ready.
Data Loading: Sybase Database to Oracle Database

Problem Definition

This will make the sqlldr_tables_trim.ksh to automatically pick up the table again for the load and the
table is loaded.
But, this is only possible as long as the main script, sqlldr_tables_trim.ksh, is still running. In case,
by the time the issue is resolved and the target table is truncated, the main script is completed, then,
it is enough run the script, sqlldr_tables_trim.ksh, in the Load-Only mode (with -L option) since the
control files and SQL Loader scripts are already created in the first round.

Data Loading: Sybase Database to Oracle Database

10

Problem Definition

Benefits
The following benefits can be realized after following the process mentioned above:

ImprovesAccuracy:Asitisevident,theprocessrequiresalotofroutinechecksfilemanipulationsto
beperformedforeachTablesload.Iftakenupmanually,therewouldbelotofissueslikeskipping
fewcheckstobeperformed,mistakesinperformingthefilemanipulationsetc.Byperformingallthe
tasksinanautomatedscript,theaccuracyisimproved.
ReducesComplexity:Inthecaseswhenperformingthechecksorfilemanipulationsinvolvesa
complexprocess,theresultcouldbeerrorprone,ifhandledmanually.Byusingtheautomatedscript,
suchcomplexitiesarehandledautomaticallyandaccurately.
Reduces time and effort: In case of a manual effort, the various checks, their resolution, tracking
the completion of the numerous BCP or SQLLDR processes, and a final check on the complete
data load would take a lot of time and effort of the resources. The automation of the process
makes it possible for the resources to utilize their time and effort on more innovative and
challenging problems.

Scalable:AsthewholeprocessisbeingperformedonUNIXbasedscripts,thesamecanbeusedwith
someminimalchangesforDataLoadprocessesacrossvariousSybaseandOracleversions.

Data Loading: Sybase Database to Oracle Database

11

Problem Definition

Conclusion
Though there are some client specific parts in the BCP export and SQL Loader load scripts, they are
very easy to understand by a person with introductory knowledge in UNIX, Sybase and Oracle.
These can be changed very easily as per the clients requirements or can further be suggested to
the clients as a possible solution.
In conclusion, the BCP export and SQL Loader load of data from Sybase to Oracle are handled in
UNIX scripts which start the BCP exports as per the load bearing capability of Sybase and start the
SQL Loader as and when the BCP export is completed. The extensions of the log files of BCP and
SQL Loader provide a simple way for the user to understand the status of the load of a table. The
script also provides the user the chance to rectify the SQL Loader issues and reload the table
without the need to restart the load. In the event that a restart is necessary, the user has the option
to directly start the load with existing control files and SQL Loader scripts rather than recreating them
every time.

Data Loading: Sybase Database to Oracle Database

12

Problem Definition

Appendix A: bcp_tables.ksh script

#!/bin/ksh
path=`pwd`
. ${path}/setup.env
db=$1
if [ -e ${path}/change_tnames.lst ];then rm -f ${path}/change_tnames.lst;fi
## Find out if tables contain MONEY or SMALL MONEY Columns
tab=`cat ${path}/tables_syb.lst|sed "s/^/ '/g;s/$/' /g"|sed "s/,$//g"|tr '\012' ':'|gawk '{gsub(" : ",",",
$0);print}'|sed "s/://g"`
isql -U ${uid} -P "${passwd}" -S${server} -I ${interfaces_file}<<EOF>${path}/tables_m.lst
use ${db}
go
select distinct b.name as tablename
from syscolumns a join sysobjects b on (a.id = b.id)
join systypes c on (a.type=c.type)
and b.name in (${tab})
and b.type in ('U','V')
and c.name in ('moneyn','money','smallmoney')
go
EOF
cat ${path}/tables_m.lst|sed -n "3,$ p"|grep -iwv "rows affected"|grep -ivw "row affected"|gawk
'{gsub(" ","",$0);print}'|grep .>${path}/tables_m.lst_tmp
mv ${path}/tables_m.lst_tmp ${path}/tables_m.lst
## Get the tables without the MONEY columns
cat ${path}/tables_syb.lst|sort -u>${path}/t_tmp
cat ${path}/tables_m.lst|sort -u>${path}/m_tmp
diff ${path}/t_tmp ${path}/m_tmp|grep -i ^\<|gawk '{gsub("< ","",$0);print}'>tables_nm.lst
rm -f ${path}/t_tmp
rm -f ${path}/m_tmp
mv ${path}/tables_syb.lst ${path}/main_tables.lst
mv ${path}/tables_nm.lst ${path}/tables_syb.lst
## Take out tables greater than 28 characters from Non-Money Tables
if [ -e ${path}/tables_nm.lst ];then rm -f ${path}/tables_nm.lst;fi
for tab in `cat ${path}/tables_syb.lst`
do
if [ `echo ${tab}|wc -c` -gt 29 ]
then

Data Loading: Sybase Database to Oracle Database

13

Problem Definition

echo ${tab}>>${path}/tables_m.lst
else
echo ${tab}>>${path}/tables_nm.lst
fi
done
if [ ! -e ${path}/tables_nm.lst ];then touch ${path}/tables_nm.lst;fi
if [ ! -e ${path}/tables_m.lst ];then touch ${path}/tables_m.lst;fi
chmod 777 ${path}/tables_m.lst
chmod 777 ${path}/tables_nm.lst
mv ${path}/tables_nm.lst ${path}/tables_syb.lst
## Kick off BCP for Non_money Tables
nohup ./bcp_tables_nm.ksh ${db} &
### Create Views for the Money Tables
cnt=1
for tab in `cat ${path}/tables_m.lst`
do
if [ `echo ${db}_${tab}_vw|wc -c` -gt 29 ]
then
tname=${db}_`echo ${tab}|cut -c1-10`_vw
if [ ! -e ${path}/change_tnames.lst ];then touch ${path}/change_tnames.lst;fi
chmod 777 ${path}/change_tnames.lst
if [ `cat ${path}/change_tnames.lst|grep -iw "${tname}"|wc -l` -eq 0 ]
then
echo ${tab},${tname}>>${path}/change_tnames.lst
else
tname=${db}_`echo ${tab}|cut -c1-10`${cnt}_vw
echo ${tab},${tname}>>${path}/change_tnames.lst
cnt=`expr ${cnt} + 1`
fi
else
tname=${db}_${tab}_vw
echo ${tab},${tname}>>${path}/change_tnames.lst
fi
if [ -d ${path}/logs ];then rm -rf ${path}/logs;fi
mkdir ${path}/logs
chmod -R 777 logs
## Get the Columns to build the view
isql -U ${uid} -P "${passwd}" -w 800 -S${server} -I ${interfaces_file}<<EOF>${path}/logs/$
{tname}.log
use ${db}
go
select columnname from (
select distinct case when a.colid<>d.cid then case when c.name in ('moneyn','money','smallmoney')
then 'convert(numeric(19,4),'||a.name||') as '||a.name||',' else a.name||',' end
else

Data Loading: Sybase Database to Oracle Database

14

Problem Definition

case when c.name in ('moneyn','money','smallmoney') then 'convert(numeric(19,4),'||a.name||') as '||

a.name else a.name end
end
as columnname,a.colid
from syscolumns a join sysobjects b on (a.id = b.id)
join systypes c on (a.type=c.type)
,(select max(colid) cid from syscolumns a join sysobjects b on (a.id = b.id)
where b.name='${tab}'
and b.type in ('U','V'))d
where b.name='${tab}'
and b.type in ('U','V')
)a
order by colid
go
EOF
cat ${path}/logs/${tname}.log|sed -n "3,$ p"|grep -ivw "rows affected"|grep -ivw "row affected"|tr '\012'
' '|grep .>${path}/logs/${tname}_tmp
echo " ">>${path}/logs/${tname}_tmp
cols=`cat ${path}/logs/${tname}_tmp`
## Create the view with the above columns
isql -U ${uid} -P "${passwd}" -S${server} -I ${interfaces_file}<<EOF > ${path}/logs/${tab}.log
use scratchdb
go
IF OBJECT_ID('${tname}') IS NOT NULL
BEGIN
DROP VIEW ${tname}
END
go
create view ${tname} as select ${cols} from ${db}..${tab}
go
EOF
done

## Move the Money-Tables to TEMP Directory.

if [ -d ${path}/temp ];then rm -rf ${path}/temp;fi
mkdir ${path}/temp
cp ${path}/bcp_tables_m.ksh ${path}/setup.env ${path}/change_tnames.lst ${path}/keywords.lst $
{path}/temp/
cat ${path}/temp/change_tnames.lst|cut -d"," -f2>${path}/temp/tables_syb.lst
chmod -R 777 ${path}/temp
## Run the BCP for the Money-Tables
nohup ${path}/temp/bcp_tables_m.ksh scratchdb ${db} &
cn=1
total=`cat ${path}/temp/tables_syb.lst|wc -l`
### Move the completed BCP files from scratchdb folder to the main db folder
Data Loading: Sybase Database to Oracle Database

15

Problem Definition

while [[ ${cn} -le ${total} ]]

do
if [ `ls ${data_dir}/scratchdb_${db}/*.ready 2>/dev/null|wc -l` -gt 0 ]
then
file=`ls ${data_dir}/scratchdb_${db}/*.ready|head -1|cut -d"." -f1|gawk -F"/" '{print $NF}'`
nm=`cat ${path}/temp/change_tnames.lst|grep -iw "${file}"|cut -d"," -f1|tr '[A-Z]' '[a-z]'`
mv ${data_dir}/scratchdb_${db}/${file}.dat ${data_dir}/${db}/${nm}.dat
mv ${data_dir}/scratchdb_${db}/${file}.ready ${data_dir}/${db}/${nm}.ready
cn=`expr ${cn} + 1`
elif [ `ls ${data_dir}/scratchdb_${db}/*.error 2>/dev/null|wc -l` -gt 0 ]
then
file=`ls ${data_dir}/scratchdb_${db}/*.error|head -1|cut -d"." -f1|gawk -F"/" '{print $NF}'`
nm=`cat ${path}/temp/change_tnames.lst|grep -iw "${file}"|cut -d"," -f1|tr '[A-Z]' '[a-z]'`
mv ${data_dir}/scratchdb_${db}/${file}.error ${data_dir}/${db}/${nm}.error
cn=`expr ${cn} + 1`
else
sleep 30
fi
done

Data Loading: Sybase Database to Oracle Database

16

Problem Definition

Appendix B: setup.env
Environment setup file
uid=svc_deloitte
passwd='svc_deloitte'
server=SYBRPTU
interfaces_file=/appl/ora_stagedr/eps/loading/interfaces
data_dir=/appl/epsbackupfs1/dirdat/
ORA_SERVER=ep01pims
ORA_USR=pm_own
ORA_PWD=solongsybase
ORA_DATA_DIR=/appl/epsbackupfs1/dirdat/
CTL_FILE_DIR=/appl/epsbackupfs1/dirctl
LOG_FILE_DIR=/appl/epsbackupfs1/dirlog
BAD_FILE_DIR=/appl/epsbackupfs1/dirbad
export NLS_LANG='AMERICAN_AMERICA.WE8MSWIN1252'
export TNS_ADMIN=/appl/ora_stagedr/eps/loading
max_proc_ora=300
max_proc_syb=25

Data Loading: Sybase Database to Oracle Database

17

Problem Definition

Appendix C: bcp_tables_nm.ksh
script
#!/bin/ksh
path=`pwd`
schema=$1
. ${path}/setup.env
table_list=tables_syb.lst
data_dir_new=${data_dir}/${schema}
if [ ! -d ${data_dir_new} ];then mkdir ${data_dir_new};chmod -R 777 ${data_dir_new};fi
#if [[ -Z ${schema} ]];then echo "Usage : bcp_tables.ksh schema";exit 1;fi
###Creating the BCP standard file
if [ -e ${path}/bcp_standard.txt ];then rm -f ${path}/bcp_standard.txt;fi
if [ ${server} != DELTA2PMCYC1 ]
then
echo "bcp \"[${schema}].[dbo].[TABLE]\" out \"${data_dir_new}/LTAB.dat\" -b 50000 -c -t \"<EOFD>\"
-r \"<EORD>\" -U ${uid} -P \"${passwd}\" -S${server} -I ${interfaces_file}>$
{data_dir_new}/bcp_TABLE.log" > ${path}/bcp_standard.txt
else
echo "bcp \"[${schema}].[dbo].[TABLE]\" out \"${data_dir_new}/LTAB.dat\" -b 50000 -c -t \"<EOFD>\"
-r \"<EORD>\" -U ${uid} -P \"${passwd}\" >${data_dir_new}/bcp_TABLE.log">$
{path}/bcp_standard.txt
fi
chmod 777 ${path}/bcp_standard.txt

####Create individual files for tables for running them paralllely

if [ -d ${path}/temp_syb ];then rm -rf ${path}/temp_syb;fi
if [ ! -d ${path}/temp_syb ];then mkdir ${path}/temp_syb;chmod -R 777 ${path}/temp_syb;fi
for tab in `cat ${path}/${table_list}`
do
##Remove existing error or ready files for these tables
if [ -e ${data_dir_new}/${tab}.ready ];then rm -f ${data_dir_new}/${tab}.ready;fi
if [ -e ${data_dir_new}/${tab}.error ];then rm -f ${data_dir_new}/${tab}.error;fi
if [ -e ${data_dir_new}/${tab}.done ];then rm -f ${data_dir_new}/${tab}.done;fi

Data Loading: Sybase Database to Oracle Database

18

Problem Definition

if [ `echo ${tab}|wc -c` -gt 29 ]

then
echo "Table name greater than 28 characters not supported by BCP">${data_dir_new}/${tab}.error
touch ${path}/temp_syb/bcp_${tab}.ksh
chmod 777 ${path}/temp_syb/bcp_${tab}.ksh
chmod 777 ${data_dir_new}/${tab}.error
else
ltab_tmp=`echo ${tab}|tr '[A-Z]' '[a-z]'`
if [ `cat ${path}/keywords.lst|grep -iw ${ltab_tmp}|wc -l` -eq 1 ]
then
ltab=${ltab_tmp}_1
else
ltab=${ltab_tmp}
fi
cat ${path}/bcp_standard.txt|sed "s/TABLE/${tab}/g;s/LTAB/${ltab}/g" > ${path}/temp_syb/bcp_$
{tab}.ksh
chmod 777 ${path}/temp_syb/bcp_${tab}.ksh
echo "if [ \$? -eq 0 ];then mv ${data_dir_new}/bcp_${tab}.log ${data_dir_new}/${ltab}.ready;else mv
${data_dir_new}/bcp_${tab}.log ${data_dir_new}/${ltab}.error;fi">>${path}/temp_syb/bcp_${tab}.ksh
echo "chmod -R 777 ${data_dir_new}/*${tab}*">>${path}/temp_syb/bcp_${tab}.ksh
echo "chmod -R 777 ${data_dir_new}/*${ltab}*">>${path}/temp_syb/bcp_${tab}.ksh
fi
done

####Run the BCP commands for the tables paralelly

for tab in `cat ${path}/${table_list}`
do
${path}/temp_syb/bcp_${tab}.ksh &
x=1
while [[ $x -eq 1 ]]
do
proc=`ps -elf|grep bcp_*.ksh|wc -l`
if [ ${proc} -gt ${max_proc_syb} ]
then
sleep 20
echo "sleeping"
else
x=0
fi
done
done

Data Loading: Sybase Database to Oracle Database

19

Problem Definition

Appendix D: bcp_tables_m.ksh
script
#!/bin/ksh
path=`pwd`/temp
schema=$1
db=$2
. ${path}/setup.env
table_list=tables_syb.lst
data_dir_new=${data_dir}/${schema}_${db}
if [ ! -d ${data_dir_new} ];then mkdir ${data_dir_new};chmod -R 777 ${data_dir_new};fi
#if [[ -Z ${schema} ]];then echo "Usage : bcp_tables.ksh schema";exit 1;fi
###Creating the BCP standard file
if [ -e ${path}/bcp_standard.txt ];then rm -f ${path}/bcp_standard.txt;fi
if [ ${server} != DELTA2PMCYC1 ]
then
echo "bcp \"[${schema}]..[TABLE]\" out \"${data_dir_new}/LTAB.dat\" -b 50000 -c -t \"<EOFD>\"
-r \"<EORD>\" -U ${uid} -P \"${passwd}\" -S${server} -I ${interfaces_file} >$
{data_dir_new}/bcp_TABLE.log" > ${path}/bcp_standard.txt
else
echo "bcp \"[${schema}]..[TABLE]\" out \"${data_dir_new}/LTAB.dat\" -b 50000 -c -t \"<EOFD>\"
-r \"<EORD>\" -U ${uid} -P \"${passwd}\" >${data_dir_new}/bcp_TABLE.log">$
{path}/bcp_standard.txt
fi
chmod 777 ${path}/bcp_standard.txt

####Create individual files for tables for running them paralllely

if [ -d ${path}/temp_syb ];then rm -rf ${path}/temp_syb;fi
if [ ! -d ${path}/temp_syb ];then mkdir ${path}/temp_syb;chmod -R 777 ${path}/temp_syb;fi
for tab in `cat ${path}/${table_list}`
do
##Remove existing error or ready files for these tables
if [ -e ${data_dir_new}/${tab}.ready ];then rm -f ${data_dir_new}/${tab}.ready;fi
if [ -e ${data_dir_new}/${tab}.error ];then rm -f ${data_dir_new}/${tab}.error;fi
if [ -e ${data_dir_new}/${tab}.done ];then rm -f ${data_dir_new}/${tab}.done;fi

Data Loading: Sybase Database to Oracle Database

20

Problem Definition

if [ `echo ${tab}|wc -c` -gt 29 ]

then
echo "Table name greater than 28 characters not supported by BCP">${data_dir_new}/${tab}.error
touch ${path}/temp_syb/bcp_${tab}.ksh
chmod 777 ${path}/temp_syb/bcp_${tab}.ksh
chmod 777 ${data_dir_new}/${tab}.error
else
ltab_tmp=`echo ${tab}|tr '[A-Z]' '[a-z]'`
if [ `cat ${path}/keywords.lst|grep -iw ${ltab_tmp}|wc -l` -eq 1 ]
then
ltab=${ltab_tmp}_1
else
ltab=${ltab_tmp}
fi
cat ${path}/bcp_standard.txt|sed "s/TABLE/${tab}/g;s/LTAB/${ltab}/g" > ${path}/temp_syb/bcp_$
{tab}.ksh
chmod 777 ${path}/temp_syb/bcp_${tab}.ksh
echo "if [ \$? -eq 0 ];then mv ${data_dir_new}/bcp_${tab}.log ${data_dir_new}/${ltab}.ready;else mv
${data_dir_new}/bcp_${tab}.log ${data_dir_new}/${ltab}.error;fi">>${path}/temp_syb/bcp_${tab}.ksh
echo "chmod -R 777 ${data_dir_new}/*${tab}*">>${path}/temp_syb/bcp_${tab}.ksh
echo "chmod -R 777 ${data_dir_new}/*${ltab}*">>${path}/temp_syb/bcp_${tab}.ksh
fi
done

####Run the BCP commands for the tables paralelly

for tab in `cat ${path}/${table_list}`
do
${path}/temp_syb/bcp_${tab}.ksh &
x=1
while [[ $x -eq 1 ]]
do
proc=`ps -elf|grep bcp_*.ksh|wc -l`
if [ ${proc} -gt ${max_proc_syb} ]
then
sleep 20
echo "sleeping"
else
x=0
fi
done
done

Data Loading: Sybase Database to Oracle Database

21

Problem Definition

Appendix E: keywords.lst

access

commit

external

like4

numeric

committe
d

externall
y

likec

object

add

compile

fetch

limit

ocicoll

admin

compiled

file

limited

ocidate

after

compres
s

final

link

ocidatetim
e

agent

connect

fixed

lists

ociduration

aggregat
e

constant

float

local

ociinterval

all

constrain
t

flush

lock

ociloblocat
or

allocate

constrain
ts

for

logfile

ocinumber

alter

construct
or

forall

long

ociraw

analyze

contents

force

loop

ociref

and

context

foreign

manage

ocirefcurso
r

any

continue

fortran

manual

ocirowid

archive

controlfile

found

map

ocistring

archivelo
g

convert

freelist

max

ocitype

array

count

freelists

maxdatafiles

of

as

crash

from

maxextents

off

asc

create

function

maxinstance
s

offline

at

create,

general

maxlen

old

attribute

current

go

maxlogfiles

on

audit

cursor

goto

maxloghistor
y

online

authid

customd
atum

grant

maxlogmem
bers

only

authoriz
ation

cycle

group

maxtrans

opaque

avg

dangling

groups

maxvalue

open

backup

data

hash

member

operator

Data Loading: Sybase Database to Oracle Database

22

Problem Definition

become

database

having

merge

optimal

before

datafile

heap

min

option

begin

date

hidden

minextents

or

between

date_bas
e

hour

minus

oracle

bfile_bas
e

day

identified

minute

oradata

binary

dba

if

minvalue

order

blob_ba
se

dec

immedia
te

mlslabel

organizatio
n

block

decimal

in

mod

orlany

body

declare

including

mode

orlvary

both

default

increme
nt

modify

others

bound

define

index

module

out

bulk

delete

indexes

month

overlaps

by

desc

indicator

mount

overriding

byte

determini
stic

indices

multiset

own

disable

infinite

name

package

cache

dismount

initial

nan

parallel

call

distinct

initrans

national

parallel_en
able

calling

double

insert

native

parameter

cancel

drop

instance

nchar

parameters

cascade

dump

instantia
ble

new

parent

case

duration

int

next

partition

change

each

integer

nnect

pascal

char

element

interface

noarchivelog

pctfree

char_ba
se

else

intersect

noaudit

pctincreas
e

characte
r

elsif

interval

nocache

pctused

charset

empty

into

nocompress

pipe

charsetf
orm

enable

invalidat
e

nocopy

pipelined

charseti
d

end

is

nocycle

plan

check

escape

isolation

nomaxvalue

pli

checkpoi
nt

events

java

nominvalue

pragma

clob_bas
e

except

key

none

precision

Data Loading: Sybase Database to Oracle Database

23

Problem Definition

close

exception

languag
e

noorder

primary

cluster

exception
s

large

noresetlogs

prior

clusters

exclusive

layer

normal

private

cobol

exec

leading

nosort

privileges

colauth

execute

length

not

procedure

collect

exists

level

nowait

profile

column

exit

library

null

public

columns

explain

like

number

quota

commen
t

extent

like2

number_bas
e

raise

range

rows

sql

tables

unlimited

raw

rsor

sqlcode

tablespace

unsigned

read

sample

sqldata

tdo

until

real

save

sqlerror

temporary

untrusted

record

savepoint

sqlname

the

update

recover

sb1

sqlstate

then

use

ref

sb2

standard

thread

user

referenc
e

sb4

start

time

using

referenc
es

schema

stateme
nt_id

timestamp

validate

referenci
ng

scn

static

timezone_ab
br

valist

relies_o
n

second

statistics

timezone_ho
ur

value

rem

section

stddev

timezone_mi
nute

values

remaind
er

segment

stop

timezone_re
gion

varchar

rename

select

storage

to

varchar2

resetlog
s

self

stored

tracing

variable

resource

separate

string

trailing

variance

restricte
d

sequenc
e

struct

transaction

varray

result

serializab
le

style

transactional

varying

result_c
ache

session

submulti
set

trigger

view

return

set

subpartit
ion

triggers

views

returning

share

substitut
able

truncate

void

Data Loading: Sybase Database to Oracle Database

24

Problem Definition

reuse

shared

subtype

trusted

when

reverse

short

successf
ul

type

whenever

revoke

size

sum

ub1

where

role

size_t

switch

ub2

while

roles

smallint

synonym

ub4

with

rollback

snapshot

sysdate

uid

work

row

some

system

under

wrapped

rowid

sort

tabauth

union

write

rownum

sparse

table

unique

year

zone

Data Loading: Sybase Database to Oracle Database

25

Problem Definition

Appendix F:
sqlldr_tables_trim.ksh script
#!/bin/ksh
###############################################################################
######################
### sqlldr_tables.ksh -s schema - To create the control files and run sqlldr
###
### sqlldr_tables.ksh -s schema -L - To avoid creation of Control Files again
###
### sqlldr_tables.ksh -s schema -C - To just create the control files
###
###############################################################################
######################
path=`pwd`
. ${path}/setup.env
table_list=tables_ora.lst
### Making sure that the tables are in lower case
cat ${path}/${table_list}|tr '[A-Z]' '[a-z]'>${path}/${table_list}_temp
mv ${path}/${table_list}_temp ${path}/${table_list}
###CHECK FOR CORRECTNESS OF ARGUMENTS
while getopts s:LC name
do
case $name in
s) schema="$OPTARG" ;;
L) load="Y" ;;
C) create="Y";;
*) echo "Usage sqlldr_tables.ksh -s schema OR sqlldr_tables.ksh -s schema -C";exit 1 ;;
esac
done
if [[ -z ${create} ]];then create="N";fi
if [[ -z ${load} ]];then load="N";fi
#echo ${create}
#echo ${load}
sc=`echo ${schema}|gawk '{gsub("_own","",$0);print}'`
ORA_DATA_DIR_NEW=${ORA_DATA_DIR}/${sc}
CTL_FILE_DIR_NEW=${CTL_FILE_DIR}/${sc}
LOG_FILE_DIR_NEW=${LOG_FILE_DIR}/${sc}
BAD_FILE_DIR_NEW=${BAD_FILE_DIR}/${sc}
if [ ! -d ${ORA_DATA_DIR_NEW} ];then echo "${ORA_DATA_DIR_NEW} missing" ; exit 1;fi
Data Loading: Sybase Database to Oracle Database

26

Problem Definition

if [ ! -d ${CTL_FILE_DIR_NEW} ];then mkdir ${CTL_FILE_DIR_NEW};chmod -R 777 $

{CTL_FILE_DIR_NEW};fi
if [ ! -d ${LOG_FILE_DIR_NEW} ];then mkdir ${LOG_FILE_DIR_NEW};chmod -R 777 $
{LOG_FILE_DIR_NEW};fi
if [ ! -d ${BAD_FILE_DIR_NEW} ];then mkdir ${BAD_FILE_DIR_NEW};chmod -R 777 $
{BAD_FILE_DIR_NEW};fi

###CREATE CONTROL FILE

#if [ ${ORA_SERVER} == dv01pimi_am_con ]
#then
#SQ_CMD=${ORA_USR}/${ORA_PWD}@${ORA_SERVER}
#else
SQ_CMD=${ORA_USR}/${ORA_PWD}@${ORA_SERVER}
#fi
if [[ ${load} == "N" || ${create} == "Y" ]]
then
for tab in `cat ${path}/${table_list}`
do
sqlplus -s ${SQ_CMD}<<EOF >test.log
SET HEADING OFF
set pagesize 0
set linesize 9999
set trimspool on
set trim on
set tab off
set echo off
set feedback off
set recsep off
spool ${CTL_FILE_DIR_NEW}/${tab}.ctl
select 'OPTIONS(DIRECT=TRUE, ROWS=1000000, SKIP_INDEX_MAINTENANCE=TRUE)'||
chr(10)||'unrecoverable'||chr(10)||'load data ' ||chr(10)|| 'infile '''||'${ORA_DATA_DIR_NEW}/'||
LOWER('${tab}')||'.dat'''||CHR(10)||
'"str ''<EORD>''"' ||CHR(10)|| 'into table '|| '${schema}' ||'.'||'${tab}'|| CHR(10) ||'REENABLE
DISABLED_CONSTRAINTS'||chr(10)||
'fields terminated by ''<EOFD>'''||CHR(10)||
'trailing nullcols' ||CHR(10) ||'(' FROM DUAL
UNION ALL
select cols from
(SELECT LOWER(COLUMN_NAME)||' '||
CASE WHEN DATA_TYPE = 'TIMESTAMP(3)' AND column_id=b.cid THEN 'TIMESTAMP "Mon dd
yyyy HH12:MI:SS:FF3AM"'
WHEN DATA_TYPE = 'TIMESTAMP(3)' AND column_id<>b.cid THEN 'TIMESTAMP "Mon dd
yyyy HH12:MI:SS:FF3AM",'
WHEN DATA_TYPE = 'NUMBER' AND column_id=b.cid THEN '"TRANSLATE ('||':'||
lower(COLUMN_NAME)||',''1,'',''1'''||')"'
WHEN DATA_TYPE = 'NUMBER' AND column_id<>b.cid THEN '"TRANSLATE ('||':'||
lower(COLUMN_NAME)||',''1,'',''1'''||')",'

Data Loading: Sybase Database to Oracle Database

27

Problem Definition

WHEN DATA_TYPE='CLOB' AND column_id=b.cid THEN 'CHAR(2147483647) TERMINATED

BY ''<EORD>'''
WHEN DATA_TYPE='CLOB' AND column_id<>b.cid THEN 'CHAR(2147483647) TERMINATED
BY ''<EOFD>'','
WHEN DATA_TYPE='DATE' AND column_id=b.cid THEN 'DATE "Mon dd yyyy HH12:MIAM"'
WHEN DATA_TYPE='DATE' AND column_id<>b.cid THEN 'DATE "Mon dd yyyy HH12:MIAM",'
WHEN DATA_TYPE='VARCHAR2' AND column_id=b.cid THEN '"case when rtrim(:'||
column_name||') is null and :'||column_name||' is not null then '||''''||' '||''''||' else rtrim(:'||
column_name||') end"'
WHEN DATA_TYPE='VARCHAR2' AND column_id<>b.cid THEN '"case when rtrim(:'||
column_name||') is null and :'||column_name||' is not null then '||''''||' '||''''||' else rtrim(:'||
column_name||') end",'
WHEN DATA_TYPE='CHAR' AND column_id=b.cid THEN 'PRESERVE BLANKS'
WHEN DATA_TYPE='CHAR' AND column_id<>b.cid THEN 'PRESERVE BLANKS,'
WHEN DATA_TYPE NOT IN ('TIMESTAMP(3)','NUMBER','CLOB','DATE','VARCHAR2','CHAR')
AND column_id<>b.cid THEN ',' end as cols
from user_tab_columns a,
(select max(column_id) cid from user_tab_columns where lower(table_name) = '${tab}'
-- and lower(owner)='${schema}'
)b
where lower(table_name) = '${tab}'
-- and lower(owner)='${schema}'
order by column_id)a
union all
select ')' from dual;
spool off
quit
EOF
chmod 777 ${CTL_FILE_DIR_NEW}/${tab}.ctl 2>/dev/null
done
fi
###CREATE FILES FOR SQLLOADER
if [ -d ${path}/temp_ora ];then rm -rf ${path}/temp_ora;fi
if [ ! -d ${path}/temp_ora ];then mkdir ${path}/temp_ora;chmod -R 777 ${path}/temp_ora;fi
for tab in `cat ${path}/${table_list}`
do
echo "#!/bin/ksh">${path}/temp_ora/sqlldr_${tab}.ksh
echo "mv ${ORA_DATA_DIR_NEW}/${tab}.next ${ORA_DATA_DIR_NEW}/${tab}.running">>$
{path}/temp_ora/sqlldr_${tab}.ksh
echo "if [ -e ${LOG_FILE_DIR_NEW}/${tab}.error ];then rm -f ${LOG_FILE_DIR_NEW}/$
{tab}.error;fi">>${path}/temp_ora/sqlldr_${tab}.ksh
echo "if [ -e ${LOG_FILE_DIR_NEW}/${tab}.done ];then rm -f ${LOG_FILE_DIR_NEW}/$
{tab}.done;fi">>${path}/temp_ora/sqlldr_${tab}.ksh
echo "if [ -e ${LOG_FILE_DIR_NEW}/${tab}.log ];then rm -f ${LOG_FILE_DIR_NEW}/$
{tab}.log;fi">>${path}/temp_ora/sqlldr_${tab}.ksh
echo "sqlldr ${SQ_CMD} control=${CTL_FILE_DIR_NEW}/${tab}.ctl log=${LOG_FILE_DIR_NEW}/$
{tab}.log bad=${BAD_FILE_DIR_NEW}/${tab}.bad" >>${path}/temp_ora/sqlldr_${tab}.ksh
chmod 777 ${path}/temp_ora/sqlldr_${tab}.ksh
Data Loading: Sybase Database to Oracle Database

28

Problem Definition

echo "if [ \$? -eq 0 ];then mv ${LOG_FILE_DIR_NEW}/${tab}.log ${LOG_FILE_DIR_NEW}/$

{tab}.done;mv ${ORA_DATA_DIR_NEW}/${tab}.running ${ORA_DATA_DIR_NEW}/${tab}.done;else
mv ${LOG_FILE_DIR_NEW}/${tab}.log ${LOG_FILE_DIR_NEW}/${tab}.error;mv $
{ORA_DATA_DIR_NEW}/${tab}.running ${ORA_DATA_DIR_NEW}/${tab}.skip;fi">>$
{path}/temp_ora/sqlldr_${tab}.ksh
echo "chmod -R 777 ${LOG_FILE_DIR_NEW}/*${tab}*">>${path}/temp_ora/sqlldr_${tab}.ksh
done

####Run the SQL LOADER commands for the tables paralelly

if [[ ${create} == "N" || ${load} == "Y" ]]
then
total=0
while [[ `cat ${path}/${table_list}|wc -l` -ne ${total} && ! -e ${ORA_DATA_DIR_NEW}/stop ]]
#while [[ `cat ${path}/${table_list}|wc -l` -ne `ls ${ORA_DATA_DIR_NEW}/*.done 2>/dev/null|wc -l`
&& ! -e ${ORA_DATA_DIR_NEW}/stop ]]
do
nd=`ls ${ORA_DATA_DIR_NEW}/*.done 2>/dev/null|wc -l`
ns=`ls ${ORA_DATA_DIR_NEW}/*.skip 2>/dev/null|wc -l`
ne=`ls ${ORA_DATA_DIR_NEW}/*.error 2>/dev/null|wc -l`
total=`expr ${nd} + ${ns} + ${ne}`
while [[ `ls ${ORA_DATA_DIR_NEW}/*.ready 2>/dev/null|wc -l` -ne 0 || `ls $
{ORA_DATA_DIR_NEW}/*.next 2>/dev/null|wc -l` -ne 0 && ! -e ${ORA_DATA_DIR_NEW}/stop ]]
do
if [ `ls ${ORA_DATA_DIR_NEW}/*.next 2>/dev/null|wc -l` -ne 0 ]
then
file=`ls ${ORA_DATA_DIR_NEW}/*.next 2>/dev/null|head -1|cut -d"." -f1`
tab=`echo ${file}|gawk -F"/" '{print $NF}'`
else
if [ `ls ${ORA_DATA_DIR_NEW}/*.ready 2>/dev/null|wc -l` -ne 0 ]
then
file=`ls ${ORA_DATA_DIR_NEW}/*.ready 2>/dev/null|head -1|cut -d"." -f1`
mv ${file}.ready ${file}.next
tab=`echo ${file}|gawk -F"/" '{print $NF}'`
#cat ${file}.dat|tr "\222" "'"|tr "\221" "'"|tr "\363" "\363" |iconv -f cp1252 -t UTF-8 >${file}.dat_conv
#mv ${file}.dat_conv ${file}.dat
#chmod 777 ${file}.dat
fi
fi
if [ ! -z ${tab} ]
then
nohup ${path}/temp_ora/sqlldr_${tab}.ksh &
sleep 5
fi
x=1
while [ $x -eq 1 ]
do
proc=`ps -elf|grep sqlldr*|wc -l`
if [ ${proc} -gt ${max_proc_ora} ]
then
sleep 20
Data Loading: Sybase Database to Oracle Database

29

Problem Definition

echo "sleeping"
else
x=0
fi
done
echo "Looping external"
done
done

fi

Data Loading: Sybase Database to Oracle Database

30