Data Archiving and Archive Development Kit: October 2001
Data Archiving and Archive Development Kit: October 2001
SAP AG
October 2001
Page 2 of 2
Introduction to Data Archiving for mySAP Technology
When you implement mySAP.com e-business solutions, data archiving is an important aspect to be
taken into account from the beginning. The amount of transaction and master data in your production
database can expand rapidly during operation, often faster than technology can handle it (for
example, database limits, hard-disk sizes, time available for backup and recovery or statistics update,
and so on). With data archiving, you have everything you need to manage database size and system
performance while also ensuring that data can continue to be accessed in the long term. Data from
completed business operations that is no longer required for daily operations or accessed less
frequently is copied from the database to archive files and then deleted from the database. To take
full advantage of what data archiving has to offer, you have to plan and schedule the process on a
regular basis.
Comprehensive archivability checks at the application level guarantee that the data archived is
consistent and complete and that only data from completed business processes is archived. You
simply select the business objects to be archived, such as accounting and material documents,
production orders, or billing documents. These objects are stored in archive files outside the
production database and can then be read from these files at a later date if required.
The following graph shows how data archiving can keep database growth under control. Note that
with regular data archiving process execution even though the database continues to grow, it grows at
a slower and more controlled rate. Data growth, which indicates healthy business growth, is managed
and controlled:
700.00
Expected size
without Archiving
600.00
500.00
400.00
300.00
'Without' Initial With regular archiving
Archiving Archiving
200.00
100.00
DB growth: Reduction: DB growth: ~7 GB/month
~15 GB/month ~60GB
0.00
MAR 99
MAR 00
MAY 00
MAY 99
AUG 99
AUG 00
OCT 99
NOV 99
APR 99
DEC 99
APR 00
JUN 00
JUN 99
SEP 99
JAN 00
FEB 00
SEP 00
JUL 99
JUL 00
Page 3 of 3
• Remote-enabled administration functions for creating your own Web applications
• Archive browsing using the Archive Information System (AS)
Ø Better performance
• Reduced hardware costs (fewer hard disks, less CPU, and memory capacity needed)
• Reduced administration costs
Archiving Object
SAP’s archiving concept is based on the use of archiving objects. An archiving object is a logical unit
that describes which database objects are required to obtain a complete business object.
Archiving Objects
Data
Customizing
Programs
Figure 2: Archiving Object
Page 4 of 4
The archiving object includes:
§ A definition of the logical data units in business processes
§ All programs required for archiving, for example preprocessing, write, delete, and read programs
§ Customizing settings
Optional programs for creating and deleting an ADK index and for postprocessing may also be
included.
The data archiving solution developer uses transaction AOBJ to define archiving objects and
customize the settings.
Archiving Process
You can archive data simultaneously with normal online operations. You schedule the archiving
programs as background jobs in transaction SARA and do not have to backup the database before
you start the archiving process. However, we strongly recommend you backup the archive files after
archiving.
The data archiving process consists of two basic steps and an optional third step. This two-step
process guarantees data protection if problems occur during the archiving process; for an overview,
see figure 3.
Business
Transactions/
Read Archived Files
Reporting
Database
Application Data
Data Objects
Archive
Files
During archiving, data is automatically compressed on average by a factor of five. However, if the
data to be archived is stored in cluster tables, it is not compressed any further.
Page 5 of 5
The delete program reads the data from the archive files and, if the archive file is successfully
read (that is the archive file is intact), deletes the corresponding data in the database.
The delete program only deletes data from the database that has been successfully read from an
archive file. If an error occurs, such as a data transfer error, you can restart the archiving process
because the data is still either in the database or in an archive file.
To increase data security during archiving even further, verification information can be written to the
archive files along with the data. The system uses this verification information to check archive files
during delete, read, and reload sessions. This allows the system to recognize corrupt files in time and
notify the user. For example, no delete operation is executed in the database for a corrupt archive file.
As of SAP R/3 4.6C, you can specify when to move the archive files to permanent storage:
Archive Administration (transaction SARA) is the central starting point for all data archiving activities.
From here you can:
• Schedule write and delete jobs
Copyright 2001 SAP AG
Page 6 of 6
• Interrupt and continue archiving sessions (new in SAP Web AS 6.10)
• Manage archiving sessions and check the status of both archiving sessions and archive files
• Display detailed data archiving statistics (new in SAP Web AS 6.10)
• Store and retrieve archive files
• Read archive files
• Build or delete an archive file index
• Reload archiving sessions
Data Archiving for mySAP Technology is based on the Archive Development Kit (ADK), a service of
mySAP Technology. ADK is a software layer between mySAP.com components, the database, and
an archive. In archiving programs, it separates the technical aspects from the application logic. This
means that archiving programs do not need to deal with file management or job scheduling. ADK
provides an application programming interface (API), also used by SAP, that customers and partners
can use for developing their own archiving solutions. This API includes all the functions needed for
developing archiving programs, including write, delete, and read programs. With SAP Web
Application Server 6.10, ADK supports the archiving of Unicode data.
The following figure clarifies the concept of ADK and archive management in the context of
mySAP.com:
Page 7 of 7
mySAP.com Legend
Data Flow
Archive Administration
Control Flow
File System
Archiving
program
Archiving
Archive Files
Writeobject A
for
program
Program A
for object
for Object
XYZ
Archive
Development ArchiveLink Hierarchical
Kit and Storage
(ADK) Content Tape Management
Management (HSM)
Infrastructure
Database
Application Data
Administration
Data
Storage System
ADK also automatically makes the required conversions due to any hardware and software changes
when archived data is read. When archived files from SAP R/3 3.0 on are accessed, ADK takes
database structural changes into account. ADK also handles different hardware-dependent storage
formats. This is only done temporarily during read access and does not change the data in the archive
file.
The following items are changed (if necessary) during an online conversion:
• Database table structure: new and deleted columns
• Data type of a column
• Column length
• Character encoding (ASCII, EBCDIC, or specific code pages)
• Number format (such as the use of the integer format on various hardware platforms)
If database structures have changed more than ADK can handle, for example, fields have been
moved from one table to another or a table has been divided into several others, the application will
usually take care of these changes. If not, the application will provide a program you can use to
permanently convert existing archive files for standard SAP archiving objects.
Page 8 of 8
• Demonstrate the effectiveness of data archiving
• Better plan data archiving projects
• Recognize bottlenecks in time
From SAP Web AS 6.10, data archiving statistics are stored permanently and can be displayed as
needed using the Statistics function in transaction SARA.
In write programs, ADK automatically collects statistics. In delete, read, and reload programs, you
must use ARCHIVE_GIVE_STATISTICS to pass statistical information to ADK. The statistics
collected by ARCHIVE_GIVE_STATISTICS can be displayed from transaction SARA.
ARCHIVE_GIVE_STATISTICS can accept data for any of the six ABAP Dictionary object types:
• TRANSP: transparent table
• POOL: pooled table
• VIEW: database or projection view
• CLUSTER: cluster table
• INTTAB: structure, help view, maintenance view, or structure view
• APPEND: append structure
ADK uses the function module DDIF_TABL_GET to get the width of the relevant table or index. The
statistics are then calculated by multiplying the number of records by the returned ABAP Dictionary
width.
After calculating the database space, ADK assigns each result to one of four database space
categories (DB Table Space, DB Index Space, DB Cluster Space, and DB Structure Space) . These
correspond to the categories in the Statistics function in Archive Administration. DB Space is the sum
of these four categories.
Data container: The buffer in which the various table entries for a logical business object are
collected during a program run, before they are written as a complete data object to the archive file.
Archive handle: A temporary identifier to group the data objects and related archive files processed
by the ADK function modules in a set order in a program.
Page 9 of 9
Structure of a Write Program
Data Flow
Database ADK
Definition of Archiving Object XYZ
A Table
B Table
C Table
Select
Data
1 ARCHIVE_OPEN_FOR_WRITE
ABAP Program with
Internal Tables
2 ARCHIVE_NEW_OBJECT
Request Data Container Data Container with
Current Data Object
3
A ARCHIVE_PUT_RECORD
_________ A
_________ Write Data Record
to Data Container B 4
3 B ARCHIVE_SAVE _OBJECT
B ARCHIVE_PUT_RECORD
B Write Data Object Archive File
_________
_________ B to Archive File
3 C
C ARCHIVE_PUT_RECORD
C
_________
_________ C
5 ARCHIVE_CLOSE_FILE
Structure
Declaration part
DATABASE TABLES (tabA, tabB, tabC)
INTERNAL TABLES (itabA, itabB, itabC)
PARAMETERS
Select data from database
SELECT * FROM tab INTO itab
Open a new archiving session
ARCHIVE_OPEN_FOR_WRITE
Loop (n = number of data objects/records in header table)
LOOP n TIMES
Initialize data container
ARCHIVE_NEW_OBJECT
Put data records from itabA into data container
ARCHIVE_PUT_RECORD
Loop (m = number of records in itabB)
LOOP m TIMES
Put data records from itabB into data container
ARCHIVE_PUT_RECORD
ENDLOOP
Loop (p = number of records in itabC)
LOOP p TIMES
Put data records from itabC into data container
ARCHIVE_PUT_RECORD
ENDLOOP
Write data objects to archive file
ARCHIVE_SAVE_OBJECT
ENDLOOP
Create statistics
ARCHIVE_WRITE_STATISTICS
Close archiving session
2001 SAP AG
CopyrightARCHIVE_CLOSE_FILE
Page 10 of 10
Structure of a Delete Program
Data Flow
Database ADK
Definition of Archiving Object XYZ
A Table
B Table
C Table
Delete
Data
C 3 C
ARCHIVE_GET_NEXT_RECORD
C
__ _ _ _ _ _ _ _
__ _ _ _ _ _ _ _ C
4 ARCHIVE_CLOSE_FILE
Structure
Declaration part
DATABASE TABLES (tabA, tabB, tabC)
INTERNAL TABLES (itabA, itabB, itabC)
PARAMETERS
Open a new archiving session to delete data
ARCHIVE_OPEN_FOR_DELETE
Get the commit counter belonging to the archive object
ARCHIVE_GET_CUSTOMIZING_DATA
Loop (n = number of data objects in archive file)
LOOP n TIMES
Get next object from the archive files into data container
ARCHIVE_GET_NEXT_OBJECT
Loop (r = number of records in data container)
LOOP r TIMES
Get next data records from data container into ABAP report
ARCHIVE_GET_NEXT_RECORD or ARCHIVE_GET_TABLE (better)
ENDLOOP
Delete data (array delete according to commit counter)
DELETE tab FROM itab
ARCHIVE_GIVE_STATISTICS
ENDLOOP
Close the archiving session
ARCHIVE_CLOSE_FILE
Page 11 of 11
Structure of a Read Program
Data Flow
ADK
Definition of Archiving Object XYZ
A Table
B Table
C Table
Read
Data
Data Container
A 3 With Current
ARCHIVE_GET_NEXT_RECORD Data Object
__ _ _ _ _ _ _ _ A
__ _ _ _ _ _ _ _ Read Data Record
from Data Container 2
3 B ARCHIVE_GET_NEXT _OBJECT
B
ARCHIVE_GET_NEXT_RECORD B Archive File
Read Data Object
__ _ _ _ _ _ _ _ B from Archive file
__ _ _ _ _ _ _ _
B
C 3 C
ARCHIVE_GET_NEXT_RECORD
C
__ _ _ _ _ _ _ _
__ _ _ _ _ _ _ _ C
4 ARCHIVE_CLOSE_FILE
Structure
Declaration part
DATABASE TABLES (tabA, tabB, tabC)
INTERNAL TABLES (itabA, itabB, itabC)
PARAMETERS
Open a new archiving session to delete data
ARCHIVE_OPEN_FOR_READ
Loop (n = number of data objects in archive file)
LOOP n TIMES
Get next object from the archive files into data container
ARCHIVE_GET_NEXT_OBJECT
Loop (r = number of records in data container)
LOOP r TIMES
Get next data records from data container into ABAP report
ARCHIVE_GET_NEXT_RECORD
ENDLOOP
Read data
WRITE ...
ENDLOOP
Close the archiving session
ARCHIVE_CLOSE_FILE
Page 12 of 12
Archiving Function Modules in Function Group ARCH
This list is not a complete list of all archiving function modules. It only reflects those function modules
mentioned in this white paper. For information about all the function modules, see the online function
module documentation.
ARCHIVE_CLOSE_FILE
This function module closes all archive files that belong to one handle, independently of whether they
were opened for reading, writing, deleting, or reloading.
ARCHIVE_GET_CUSTOMIZING_DATA
This function module delivers values from archiving object Customizing, for example, the commit
counter, which determines after how many data objects a COMMIT WORK is issued.
ARCHIVE_GET_NEXT_OBJECT
This function module reads the next data object for an archive handle into the ADK data container
from an archive open for reading or deleting. This call is a prerequisite for function module
ARCHIVE_GET_NEXT_RECORD and ARCHIVE_GET_TABLE.
ARCHIVE_GET_NEXT_RECORD
This function module sequentially reads the next record of a data object determined by
ARCHIVE_GET_NEXT_OBJECT. The first call automatically reads the first record from the data
container.
ARCHIVE_GIVE_STATISTICS
This function module is used to pass statistical data to ADK and must be used in delete programs, but
must not be used in write programs.
ARCHIVE_NEW_OBJECT
This function module prepares a new data object to be written to an archive, that is, it requests a new
data container for an archive handle. There can only be one current data object per handle. When
calling this function module all data of the current data object is discarded. Because it may be
necessary to undo any changes to the data object, ADK does not check whether the current data
object was saved using function module ARCHIVE_SAVE_OBJECT.
ARCHIVE_OPEN_FOR_DELETE
This function module opens archives in order to delete their data from the database. Each delete job
processes one archive file for an archive handle created by ARCHIVE_OPEN_FOR_DELETE. The
status of the archive files is adjusted accordingly in archive management.
ARCHIVE_OPEN_FOR_READ
This function module opens archives for reading. An archive handle is created, through which an
archive file can be read. You can also open several archive files at the same time; they all share one
archive handle. Function modules that read using this archive handle treat all the files for this archive
handle as a single file. Every call of the function module generates a new archive handle. This
enables the simultaneous processing of several archives (even from different archiving objects).
Copyright 2001 SAP AG
Page 13 of 13
ARCHIVE_OPEN_FOR_WRITE
This function module creates a new archive file and an archive handle with which you have write
access to this file.
If you have not specified a logical file name in Customizing, the platform-independent logical file name
ARCHIVE_DATA_FILE is automatically used to determine a platform-dependent physical file name.
You can use control parameters to specify whether the delete program for the archived data should
be called automatically after writing.
ARCHIVE_PUT_RECORD
This function module passes data records to the data container that was previously requested by
function module ARCHIVE_NEW_OBJECT. All records you transfer to the data container are written
to the archive file when function module ARCHIVE_SAVE_OBJECT is called.
ARCHIVE_ROLLBACK_WORK
If a ROLLBACK WORK has to be carried out in a delete or reload program, it must be done by calling
this function module, not by the ABAP statement ROLLBACK WORK.
ARCHIVE_SAVE_OBJECT
This function module writes a data object to an archive file. The data passed by
ARCHIVE_PUT_RECORD is saved.
If the data object no longer fits in the current archive file, the file is closed and a new archive file is
automatically opened. The delete program is also called automatically if requested by the
Customizing setting.
ARCHIVE_WRITE_STATISTICS
This function module creates a statistics print-out for the data objects you have written to the archive
files with ARCHIVE_SAVE_OBJECT.
Page 14 of 14
Sample Programs
Write Program
For an additional example of a write program, see the RSARCH04 program.
********************************************************************************************
* This is an example of how to use the ADK function modules. *
* This demo WRITE program is part of the archiving object called *
* BC_SBOOK. It creates a new archiving session based on the data model *
* BC_TRAVEL. Actually, only the database table SBOOK is archived. *
* For production use, you have to include proper exception handling. *
********************************************************************************************
REPORT sbooka.
* data declaration
TABLES: sbook.
DATA: button(1) TYPE c,
create(1) TYPE c VALUE 'X',
no_delete(1) TYPE c VALUE 'X',
handle LIKE sy-tabix,
sbook_itab TYPE TABLE OF sbook WITH HEADER LINE,
data_object_id LIKE arch_idx_s-obj_id,
session_interrupted TYPE c VALUE ' '.
Page 15 of 15
IMPORTING
archive_handle = handle.
LOOP AT sbook_itab.
CONCATENATE sy-mandt sbook_itab-carrid sbook_itab-bookid
INTO data_object_id.
IF session_interrupted = 'X'.
* This means the last data object has not been saved.
* -> Clean-up and/or update status specific to interrupted sessions,
* but also write a detailed or standard log + close session as usual:
ENDIF.
Page 16 of 16
Delete Program
For an additional example of a delete program, see the RSARCH06 program.
*-------------------------------------------------------------------------------------------------------- *
* This is an example of how to use the ADK function modules. *
* This demo DELETE program removes data from the database table SBOOK. *
* At the same time, an index can be built to support random access. *
* The related write pgm in archiving object BC_SBOOK is SBOOKA. *
* For production use, you have to include proper exception handling. *
* *
* UNICODE-ENABLED *
*-------------------------------------------------------------------------------------------------------- *
REPORT sbookd.
* data declaration
CONSTANTS:
object LIKE arch_obj-object VALUE 'BC_SBOOK'.
DATA:
handle LIKE sy-tabix,
sel_files_itab TYPE TABLE OF arch_files,
record_type LIKE arc_buffer-rname,
buffer_ref TYPE REF TO data, " Unicode
sbook_itab TYPE TABLE OF sbook,
commit_cnt LIKE arch_usr-arch_comit,
object_cnt TYPE i,
arkey LIKE arch_idx_s-archivekey,
object_id LIKE arch_idx_s-obj_id,
offset LIKE arch_idx_s-obj_offset,
index_tab TYPE TABLE OF arch_idx_s WITH HEADER LINE,
index_flag LIKE arch_usr-arch_index,
sum_objects TYPE i VALUE 0,
stat_itab TYPE TABLE OF arch_stat,
tabfield LIKE arch_stat.
Page 17 of 17
* loop to get the next data object from the archive file(s)
CLEAR object_cnt.
DO.
CALL FUNCTION 'ARCHIVE_GET_NEXT_OBJECT'
EXPORTING
archive_handle = handle
IMPORTING
object_id = object_id
object_offset = offset
archive_name = arkey
EXCEPTIONS
end_of_file = 01.
IF sy-subrc = 1.
EXIT.
ENDIF.
ADD 1 TO object_cnt.
IF index_flag IS INITIAL.
* no index is to be built according to Customizing
ELSEIF testrun IS INITIAL.
MOVE: object_id TO index_tab-obj_id,
arkey TO index_tab-archivekey,
offset TO index_tab-obj_offset.
APPEND index_tab.
PERFORM save_index_tab ON COMMIT.
ENDIF.
Page 18 of 18
ENDIF.
*---------------------------------------------------------------------*
* FORM DELETE_FROM_TABLE *
*---------------------------------------------------------------------*
* global: sbook_itab, stat_itab
*---------------------------------------------------------------------*
FORM delete_from_table.
REFRESH stat_itab.
DELETE sbook FROM TABLE sbook_itab. " delete job restartable
tabfield-tabname = ' '.
tabfield-count = object_cnt.
APPEND tabfield TO stat_itab.
tabfield-tabname = 'SBOOK'.
tabfield-count = sy-dbcnt.
APPEND tabfield TO stat_itab.
CALL FUNCTION 'ARCHIVE_GIVE_STATISTICS'
EXPORTING
archive_handle = handle
TABLES
table = stat_itab.
COMMIT WORK.
REFRESH sbook_itab.
ENDFORM. " DELETE_FROM_TABLE
*---------------------------------------------------------------------*
* Form SAVE_INDEX_TAB *
*---------------------------------------------------------------------*
* global: index_tab *
*---------------------------------------------------------------------*
FORM save_index_tab.
DELETE arch_idx_s FROM TABLE index_tab.
Copyright 2001 SAP AG
Page 19 of 19
INSERT arch_idx_s FROM TABLE index_tab.
REFRESH index_tab.
ENDFORM. " SAVE_INDEX_TAB
Read Program
For additional examples of read programs, see the RSARCH19, SBOOKR_2, and SBOOKR_3
programs.
*-------------------------------------------------------------------------------------------------------- *
* This is an example of how to use the ADK function modules. *
* This READ program demonstrates reporting from/analyzing archives. *
* The only output is the number of records and the last record read. *
* It belongs to the archiving object BC_SBOOK based on BC_TRAVEL. *
* *
* THIS PROGRAM ALSO DEMONSTRATES HOW TO ENABLE A *
* READ PROGRAM FOR U N I C O D E. *
* *
* Here we use TYPED field symbols to get the data back from ADK. *
*-------------------------------------------------------------------------------------------------------- *
REPORT sbookr.
* data declaration
DATA: handle LIKE sy-tabix,
record_type LIKE arc_buffer-rname,
* buffer LIKE arc_buffer-segment, " non-Unicode
buffer_ref TYPE REF TO data, " Unicode
* sbook_wa LIKE sbook, " no longer needed
number_of_records_read TYPE i.
IF sy-subrc <> 0.
WRITE: / 'No file can be accessed'(001).
EXIT.
ENDIF.
CLEAR number_of_records_read.
* loop to get the next data object from the archive file(s)
DO.
CALL FUNCTION 'ARCHIVE_GET_NEXT_OBJECT'
EXPORTING
archive_handle = handle
EXCEPTIONS
end_of_file = 1
OTHERS = 2.
Copyright 2001 SAP AG
Page 20 of 20
IF sy-subrc <> 0.
EXIT.
ENDIF.
ADD 1 TO number_of_records_read.
ENDDO.
ENDDO.
Page 21 of 21