SUM DMO With System Move
SUM DMO With System Move
one-step upgrade and migration approach, performance optimization and downtime minimization for
migration of ECC on oracle database to HANA database with EHP upgrade via SUM DMO with SYSTEM
MOVE option.
We have recently migrated from ECC6.0 EHP4 on oracle database to ECC6.0 EHP8.0 on HANA database
with hardware migration (DMO with SYSTEM MOVE) for a customer.
Main objective for the project was to achieve a downtime between 8-12 hours for a source database of
size 5TB. The setting was a bit complex such that DOWNTIME optimized DMO route couldn’t have been
opted for and multiple downtimes were not affordable at all.
Now, the challenge was to make this a successful execution within the expected downtime. It was
unachievable in the first look but “UNPRECEDENTED TIMES LEAD TO UNEXPECTED LEARNING” and it
indeed did. In this blog post I will share details on following :-
1. Project planning and approach
2. Landscape strategy
3. Downtime optimization strategy
1. The project planning was done for 3 DMO cycles till production migration as mentioned below
except that the number of MOCK runs increased in the actual project.
In below table a DMO cycle denotes 1 full export/import SUM run for SUM DMO with SYSTEM MOVE.
DMO2 is for migration of a MOCK environment copied from production to the HANA server, here we
faced few more issues which were known issues and SAP documentation proved quite helpful. The
downtime came out to be around 2 days and 11 hours.
We built QAS system with DB restore method and tested DR at this stage.
DMO3 is another MOCK run with latest production data, the SUM run got smoother this time however,
the downtime requirement could not be fulfilled, it took about 1 day and 9 hours. So we ran a couple of
more MOCK runs to improvise the downtime.
2. Landscape strategy
It’s a dual system landscape where old systems (source SAP systems) on old hardware (SOURCE) run
simultaneously un-till production Go-LIVE. The dual maintenance strategy has an advantage of
minimizing change freeze i.e. no impact on ongoing developments, however it comes with it’s own
restrictions.
To quote one of them – any change that is made in source is done in target and vise-versa and the 2
landscapes must be kept in sync. A change freeze is applied when cutover starts.
3. Downtime optimization
A comparison of 2 DMO runs can also be done as shown below to analyze the differences and
improvements in downtime.
2. Data cleansing is another important area of any migration project which no body dares to dig
into because business users are quite reluctant while perceiving this idea. In our experience, the
data cleaning contributed a lot in reducing the data export time and hence reducing the
downtime. Here’s how.
a. At the end of DMO run, “DMO POST analysis” provides the durations of all tables, R3load graph, table
split order etc. for you to analyze where optimization can be done. for example in below graph we can
see that there’s a tail towards the end of export.
Hence, the target of my next simulation would be to reduce this tail.
b. Now to optimize the downtime, we must look at the problem tables or tables that took the largest time
in processing. In our case, tables like DBTABLOG, JHAGA, SOFFCONT1 etc. took the most. Therefore, we
started with SAP reorg for tables like DBTABLOG, RFBLG etc.
After data cleaning, a database reorg must be performed, we performed a database reorg for all
problem tables, even the ones from which data wasn’t deleted and it worked well thereafter.
c. Table splits – As SAP claims, DMO is an optimized tool for HANA migrations in every possible way,
you just have to learn how to make the best use of it. Make sure that you know the mechanism of table
splits that happens automatically during preparation phases of SUM DMO, I found the
blog https://blogs.sap.com/2015/06/11/dmo-background-on-table-split-mechanism/ very helpful in
achieving our project goals.
The R3load process count which was provided while defining SUM parameters for UPTIME should be
kept same as DOWNTIME (this doesn’t refer to actual uptime or downtime, this is specific to R3load and
SQL parameters). Once an R3load and SQL count is established by various benchmark runs, it should
fixed during the actual run and must not be altered while SUM is running.
The table split mechanism is based upon R3load (UPTIME) processes and DURATIONS file.
2. SUM benchmarking tool, before execution of MOCK, run the benchmarking as mentioned
in https://blogs.sap.com/2015/03/17/dmo-optimizing-system-downtime/ to determine the
right count of processes for your environment. The CPU count and calculations as per SAP
documentation is good in theory for estimation, but to see what is the most efficient number and
what doesn’t break the system at the same time must be established through this simulated run.
We ran multiple iterations and established a count which could help us achieve an optimum R3load
count which left no tail. Here’s the final result.
To summarize, if you want to make the best use of SUM DMO, run the benchmark iterations as many
times as required to establish the best suitable settings for your environment.
Always use the latest SUM patch level even if you are in the mid of your project, SAP keeps on
improvising each second.
Last but not the least, if you read the SAP documentation thoroughly, you’d always know what SUM is
doing in the foreground and what’s going on in the background.
I will cover below points in Part – 2 which contributed the most in optimizing SUM downtime.
During the migration of application tables, the migration of big tables might dominate the overall
runtime. That is why SAPup considers table splitting to reduce the downtime of the DMO procedure.
Table splitting shall prevent the case that all tables but a few have been migrated, and that only a small
portion of all R3load processes these remaining (big) tables processes. The other R3load processes
would be idle (to be more precise: would not run), and the long tail processing of the big tables would
increase the downtime unnecessarily. See figure 1 below for a schematic view.
SAPup uses the following approach to define the tail: if the overall usage of R3load pairs (for export and
import) drops below 90 %, SAPup handles all tables that are processed afterwards as being part of the
tail (see figure 2 below).
During the configuration of the DMO procedure, you will configure a number of R3load processes, which
determines the number of R3loads that may run in parallel. This explanation will talk about R3load pairs
that are either active of idle, which is rather a virtual view. If an R3load pair did execute one job, it will
not wait in status idle, but end. SAPup may then start another R3load pair. Still for the discussion of table
split, we consider a fixed number of (potential) R3load pairs, which are either active or idle. The
following figure 3 illustrates this view.
Automatic table splitting
SAPup will automatically determine the table split conditions, and there is no need and no
recommendation to influence the table splitting. Your task is to find the optimal number of R3load
processes during a test run, and provide the table duration files for the next run. (SAPup will use the
table duration files to calculate the table splitting based on the real migration duration instead of the
table size; see DMO guide, section 2.2 “Performance Optimization: Table Migration Durations”).
You may still want to learn more on the split logic, so this blog introduces some background on table
splitting. Note that SAPup will not use R3ta for the table split.
Typically, you will expect table splitting to happen for big tables only, but as we will see, the attempt to
optimize the usage of all available (configured) R3load processes may result in splitting other tables as
well. Still, splitting a table into too many pieces may result in a bad export performance: lots of parallel
fragmented table segments will decrease read performance, and increase the load on the database
server. A table may be big, but as long as has been completely processed before the tail processing starts,
there is no reason to split that table. That is why the tool will calculate a minimum of table splits to
balance all requirements.
The logic comprises four steps: table size determination, table sequence shuffling, table split
determination, and assignment to buckets. A detailed explanation of the steps will follow below. During
the migration execution, SAPup will organize tables and table segments in buckets, which are a kind of
work packages for the R3load pair to export and import. During the migration phase, each R3load pair
will typically work on several buckets, one after the other.
SAPup will determine the individual table sizes, and then sort all tables descending by size.
In case you provide the table duration file from a previous run in the download folder, SAPup will use
the table migration duration instead of the table size.
Assuming we only had sixteen tables, figure 4 above shows the sorted table list. The table number shall
indicate the respective initial positioning in the table list.
Migrating the tables in sequence of their size is not optimal, so the table sequence is reordered
(“shuffled”) to achieve a good mixture of bigger and smaller application tables. Figure 5 below tries to
illustrate an example.
SAPup uses an internal algorithm to shuffle the table sequence, so that table sizes alternate between
bigger and smaller.
SAPup will now simulate table splitting, based on the number of configured R3load processes. Note that
changing the number of configured R3load processes later during the migration phase will affect the
runtime of the procedure.
For the simulation, SAPup will work on “slots” that represent the R3load pairs, and will distribute the
tables from the shuffled table list into these slots. Note that these R3load “slots” are not identical to the
buckets. SAPup will use buckets only at a later step. A slot is a kind of sum of all buckets, which are
processed by an R3load pair.
Initially, the simulation will assign one table from the shuffled table list into each slot until all slots are
filled with one table. In an example with only eight R3load pairs, this means that after the first eight
tables, all slots have a table assigned, as shown in figure 6 below.
In our example, SAPup has filled all slots with one table, and the second slot from above has the smallest
table, so it has the fewest degree of filling.
Now for all following assignments, SAPup will always assign the next table from the list to the slot that
has the fewest degree of filling. In our example, SAPup would assign the next table (T7) to the second
slot from top. After that, SAPup will most probably assign the next table (T9) to the first slot, see figure 7
below (sounds like Tetris, doesn’t it?).
Finally, SAPup has assigned all tables from the shuffled table list to the slots, as shown in figure 8 below.
Note that the figures are not precise in reflecting the table sizes introduced in figure 4 and 5.
As last part of this simulation run, SAPup will now determine which tables to split. The goal is to avoid a
long tail, so SAPup will determine the tail, and split all tables that are part of the tail.
SAPup determines the tail by the following calculation: SAPup sorts the slots by filling degree, and the
tail begins at the point in time at which the usage of all R3load pairs is below 90%. All tables that are
part of the tail – either completely or partially – are candidates for a split, as shown in figure 9 below. As
an example, table T2 is shown as being part of the tail.
SAPup determines the number of segments into which the table will be split by the degree by which the
table belongs to the tail. The portion of the table that does not belong to the tail is the scale for the table
segments to be created. For the example of table T2, this may result in three segments T2/1, T2/2, and
T2/3.
SAPup will now extend the shuffled table list by replacing the detected tables by its table segments.
Figure 10 shows the example with three segments for table T2.
SAPup starts the next iteration of the simulation, based on the shuffled table list with table segments.
If the calculated tail is negligible (lower than a specific threshold) or if the third simulation has finished,
SAPup will continue with step 4.
The result of step 3 is a list of tables and table segments whose sequence does not correlated to the table
size, and which was optimized to fill all R3load slots with a small tail. Now SAPup will work with buckets
(work packages for R3load pairs) instead of slots. This is a slightly different approach, but as the filling
of the buckets will use the same sequence of tables before, the assumption is that it has the same result.
SAPup will assign the tables of this list to the buckets in the sequence of the list. The rules for this
assignment are
1. A bucket will get another table or table segment assigned from the list as long as the bucket size
is lower than 10 GB.
2. If the next table or table segment is bigger than 10 GB, the current bucket is closed, and SAPup
will assign the table or table segment to next bucket.
3. SAPup will put segments of a split table into different buckets– otherwise two table segments
would reside in one bucket, which neutralizes the desired table split.
The first rule results in the effect that a bucket may have more table content than 10 GB. If a table of e.g.
30 GB was not determined for a split, the respective bucket will have this size.The second rule may
result in the effect that a bucket is only filled to a low degree, if the following table / table segment was
bigger than 10 GB so that it was put into the following bucket.The third rule results in the effect that e.g.
for a table with four segments of size 5 GB each, several buckets will have a size of 5 GB. Figure 11 below
tries to illustrate this with some examples.
Now SAPup has defined the distribution of tables and table segments into buckets, which in turn are
part of a bucket list. All this happens during the phase EU_CLONE_MIG_DT_PRP for the application tables
(and during phase EU_CLONE_MIG_UT_PRP for the repository). Note that the DT or UT part of the phase
name is no indication whether or not the phase runs in uptime (UT) or downtime (DT):
EU_CLONE_MIG_DT_PRP runs in uptime. The migration of application tables happens in downtime
during phase EU_CLONE_MIG_DT_RUN. During the migration phase, SAPup will start the R3load pairs
and assign the next bucket from the bucket list. As soon as an R3load pair is finished (and closes), SAPup
will start another R3load pair and assign the next bucket to this pair, as shown in the following figure 12.
Relevant log files are
Typically, each R3load pair will execute more than one bucket. Exceptions may happen for small
database sizes. As an example, for a total database size of 9992.3 MB and 20 R3load pairs (so 40
configured R3load processes), the tool would reduce the bucket size to put equal load to all R3load pairs.
The log will contain a line such as “Decreasing bucket size from 10240 to 256 MB to make use of 160
processes”. Below you see the respective log entry in EUMIGRATEUTPRP.LOG:
1 ETQ000 ==================================================
1 ETQ000 ==================================================
1 ETQ000 ==================================================
DMO: optimizing system downtime is
timeless
Increasing the DMO performance
You can provide the Software Update Manager with the information about table migration durations
from a previous DMO run. SUM uses this data to optimize the performance of subsequent DMO runs on the
same system. Although SUM does consider the table sizes for the migration sequence, other factors can
influence the migration duration. In other words, more criteria than just the table size have to be considered
for the duration of a table migration.
The real durations are the best criteria, but they are only known after the migration of the tables.
this can improve the downtime up to 50%, due to the more “aggressive” table splitting after the first
run and the results in the XML files. During a migration, SUM creates text files with the extension XML which
contain the information about the migration duration for each migrated table. The files are created in
directory SUM/abap/htdoc/ and are called (create them beforehand, if you want to use the setting from
the beginning)
Copy the above-mentioned XML-files directly to the download folder. The creation of the specific file
location in the file SAPup_add.par in directory SUM\abap\bin become obsolete with the current
Versions of the SUM Versions 1.0/2.0 (September 2020).
whatever you will find out: You can trust the DMO optimization when it comes to the table splitting.
the Algorithm behind is smarter than you think. Overruling this via the manual setup of the
file EUCLONEDEFS_ADD.LST is not possible.
In the SAP First Guidance – Using the new DMO to migrate to BW on HANA is also mentioned, how to use the
file SUM/abap/htdoc/UPGANA.XML to optimize the runtime further.
don´t forget the BW Housekeeping Task before you start the DMO procedure, and don´t
underestimate the importance saving time and space! the Blog SAP BW-Performance Tuning through
System Tables gives you a good overview about the “waste” you have collected in your BW System.
Together with the manual table splitting option, you can take over the tables without content.
SAP First Guidance – BW Housekeeping and BW-PCA
Introducing the DMO Benchmark tool
http://server.wdf.sap.corp:1128/lmsl/sumabap/<SID>/set/procpar
Note 1616401 – Parallelism in the Upgrades, EhPs and Support Packages implementations
Do not use more than 32 R3trans Processes and 24 SQL Processes as maximum for the Start. The
R3load Processes can be increased/decreased as you have the resources available. Consider their
Values divided by 2: e.g. 96 Export and 96 Import Processes on the AppServer will lead to 192
Processes. If you want to increase the migration performance, only the downtime processes are relevant.
As the ABAP DDIC is the almost same size despite the DB size, SQL PROCESSES and PARALLEL
PHASES can be even on the default values.
For the configuration phase you can use higher values for the uptime/downtime R3load process to
allow a higher split value of the tables. Before the downtime process is starting, use the much lower value
for R3load to ensure that you always in a optimal range of CPU/RAM/Network usage. Increasing the
R3load step by step (e.g. with the value 20) is more effective, than starting with a high value and reducing
them. The little Monitoring tool called “nmon” (e.g. AIX, Linux64, etc.) can be very useful in monitoring
Hardware and Network resources during the migration process.
– Phases behind DMO R3load parallel export/import during UPTIME and DOWNTIME to target HANA DB
– SUM: introduction to shadow system
For the detailed monitoring of the HANA resources you can use the following SAP Notes and enable
the HANA Configuration Check Tool
Once the files EUMIGRATEUTRUN.LOG and EUMIGRATEDTRUN.LOG are created, you cannot only
improve the next DMO run, you can also use these files as input into a graphical representation of the
DMO run. With the Release of SUM 1.0 SP14 the extended UI allows you to do realtime monitoring and
detailed analysis of existing DMO run´s. See also the Document – SAP First Guidance – Using the new
DMO to Migra… | SCN
EU_CLONE_DT_SIZES/EU_CLONE_UT_SIZES
During the update with DMO, the following phases can be long-running:
– EU_CLONE_DT_SIZES
– EU_CLONE_UT_SIZES
In the course of these phases, the system updates the database statistics regarding the usage of space
that the tables need on the database. The aim is a better distribution of the tables during the system
cloning. Before you start the update, you have the option to suppress these long-running phases using the
following procedure:
1. Log on to the host where the Oracle database instance is running. Use ora<dbsid> for UNIX system,
or user <sapsid>adm for Windows.
Especially Oracle based RDBMS (which is still the largest SAP customer implementation) need
special attention for the DB statistics despite which version you are running. “An old statistic is a dead
statistic”. Old could mean 10 seconds or an empty table as well. You can always see in transaction SM50,
which table is busy and run an updated statistic with transaction DB20. This can help already a lot, but
of course can be time consuming. so have a look to the following SAP Notes as well. Oracle is the RDBMS
which need the most attention before you start the DMO process.
Also manually created additional Indexes on the source database can lead to errors in the HANA
import described in SAP Note 2007272.There is an Update for the current 7.42 R3load available – See SAP
Note 2144285
Don´t go for the SAP Notes title and don´t mix up with manual heterogeneous system copy
recommendations. DMO is highly optimized is a way, a custom build migration script or monitor would
never reach DMO performance and is not supported in this context anyway.
init.ora/spfile
filesystemio_options = setall
disk_asynch_io = true
log_buffer = 1048576
parallel_execution_message_size = 16384
parallel_threads_per_cpu = 1
parallel_max_servers = number of CPU’s * number of
concurrent R3load processes * 2
processes = processes + parallel_max_servers
SQL*Net configuration
Increase SDU_SIZE in listener.ora to 64KB
Increase SDU_SIZE in tnsnames.ora to 64KB
so for the most of the systems this example of the file SUM/abap/bin/SAPup_add.par would
increase the performance a lot.
*/clonepar/imp/procenv = HDB_MASSIMPORT=YES
*/clonepar/indexcreation = after_load
*/clonepar/clonedurations =
<absolute_path>/MIGRATE_UT_DUR.LST,<absolute_path>/MIGRATE_DT_DUR.LST
/ORA/update_spacestat = 0
* retired values/now default values with the current R3load versions (Note 2118195) and the latest
SUM Versions (see below).
SAP Kernel handling – always use the latest version of the R3* tools and LibDBSL
During the migration to SAP HANA DMO of SUM has to deal with 3 kernel versions. Those are in detail:
• Kernel currently used by the system for the source DB (e.g. 7.20 EXT for Oracle)
• Kernel for the target release and source DB (e.g. 740 for Oracle – used for shadow system)
• Kernel for the target release and SAP HANA
The kernel currently used by the system can usually be found in /sapmnt/<SID>/exe/…
The other two target kernel versions (for AnyDB and SAP HANA) can be found in the SUM directory.
At the beginning of the migration process those directories look like this:
• SUM/abap/exe < contains the target kernel for AnyDB
• SUM/abap/exe_2nd < contains the target kernel for SAP HANA
During downtime (phase MIG2NDDB_SWITCH) the directories will be switched. After the switch it looks like
this:
together with the R3* tools you always exchange the LibDBSL as well for source and target DB.
Currently for Kernel 7.42 (which is needed for SAP BW 7.40 SP08 and higher) these latest patches are
needed:
Note 2054965 – R3load: TOC for logical table is incomplete in declustering mode after restart
Note 2124912 – R3load sporadically produces empty task files
Note 2118195 – R3load aborts during unicode conversion and declustering
Note 2144285 – R3load can produce duplicates at export if bulk fetch mode is active
Note 2130541 – SAP HANA: Deactivate fast data access in a non-Unicode system
Note 2144274 – R3load migration to SAP HANA database Revision 85.02 can lead to a disk full event
Attention: The Software Update Manager 2.0 SP01 is part the Software Logistics Toolset 1.0
SP stack 22 and exists in parallel to the Software Update Manager 1.0 SP21
* Attention: The Software Update Manager 2.0 SP02 is part the Software Logistics Toolset 1.0 SP
stack 23 and exists in parallel to the Software Update Manager 1.0 SP22.
** Attention: The Software Update Manager 2.0 SP03 is part the Software Logistics Toolset 1.0 SP
stack 24 and exists in parallel to the Software Update Manager 1.0 SP23.
*** Attention: The Software Update Manager 2.0 SP04 is part the Software Logistics Toolset 1.0 SP
stack 25 and exists in parallel to the Software Update Manager 1.0 SP24
Which SUM/DMO Version for which scenario?
Note 2223738 – Central Note – Software Update Manager 1.0 SP17 [lmt_008]
Note 2277058 – Central Note – Software Update Manager 1.0 SP18 [lmt_004]
Note 2328500 – Central Note – Software Update Manager 1.0 SP19 [lmt_005]
Note 2418924 – Central Note – Software Update Manager 1.0 SP21 [lmt_007]
Note 2472928 – Central Note – Software Update Manager 1.0 SP22 [lmt_004]
Note 2428168 – Central Note – Software Update Manager 2.0 SP00 [lmt_020]
Note 2472850 – Central Note – Software Update Manager 2.0 SP01 [lmt_021]
Note 2529257 – Central Note – Software Update Manager 2.0 SP02 [lmt_022]
To keep the additional complexity to minimum, try to stick with the current HANA 1.0 SP12 Revisions,
when running the DMO procedure. To find a list of the available SAP HANA Revisions have a look to the
Note 1912332 – SQL array DML operations fail with error code “-10709”
dbs/hdb/connect_property = CONNECTTIMEOUT=0
Note 2105761 – High memory consumption by RANGE-partitioned column store tables due to missing
optimize compression
with the release of SUM 1.0 SP14 and higher, there will also a new and functionality improved and
consolidated UI5 for all SUM/DMO procedures (except Dual Stacks) be available.