LCM262 Making Programs Unicode Enabled

Contributing Speaker(s) Markus Eble NW Foundation Internationalization, SAP AG Martin Schmidt NW Foundation Internationalization, SAP AG

SAP AG 2007, SAP TechEd 07 / LCM262 / 2

Learning Objectives As a result of this workshop, you will be able to:

Do the Unicode enabling for your own ABAP programs Understand how to adapt your file and RFC interfaces

SAP AG 2007, SAP TechEd 07 / LCM262 / 3

Overview ABAP Language Enhancements File Interfaces Communication via RFC Unicode enabling tools

Unicode, one Code Page for all Scripts

Japanese Hebrew Greek Taiwanese Russian Ukrainian And more languages can be supported easily without the need for new code pages or other new methods

Chinese Korean

English
Thai Croatian Czech Hungarian Polish Rumanian Slovakian Slovene

Danish Dutch, German Finnish French, Italian Norwegian Portuguese Spanish Swedish Turkish

SAP AG 2007, SAP TechEd 07 / LCM262 / 5

Icela

ndic

Unicode characters
ASCII General Scripts Symbols

CJK Ideographs 65,000 characters

Hangul

Compatibility Surrogate Area Additional 1,000,000 characters (e.g. Hong Kong Chinese)

SAP AG 2007, SAP TechEd 07 / LCM262 / 6

Representation of Unicode Characters UTF-16 Unicode Transformation Format, 16 bit encoding

Fixed length, 1 character = 2 bytes (surrogate pairs = 2 + 2 bytes) Platform dependent byte order

UTF-8 Unicode Transformation Format, 8 bit encoding

Variable length, 1 character = 1...4 bytes Platform independent 7 bit US ASCII compatible
Character Unicode scalar value U+0061 U+00E4 U+03B1 U+3479 U+2007B
SAP AG 2007, SAP TechEd 07 / LCM262 / 7

UTF-16 big endian 00 61 00 E4 03 B1 34 79 DA00 DC7B

UTF-16 little endian 61 00 E4 00 B1 03 79 34 00DA7BDC

UTF-8 61 C3 A4 CE B1 E3 91 B9 F0A081BB

Representation of Unicode characters in ABAP

Keep existing coding as far as possible Use existing character type for Unicode data Use UTF-16 to keep buffer sizes

SAP AG 2007, SAP TechEd 07 / LCM262 / 8

Transparent Unicode Enabling of R/3 Character Expansion Model

Separate Unicode and non-Unicode versions of R/3
1 character = 1 byte (types C, N, D, T, STRING) Non-Unicode kernel Non-Unicode database

ABAP ABAP source source

NonUnicode R/3

Unicode R/3

1 character = 2 bytes (UTF16), (types C, N, D, T, STRING) Unicode kernel Unicode database

No explicit Unicode data type in ABAP Single ABAP source for Unicode and non-Unicode systems

SAP AG 2007, SAP TechEd 07 / LCM262 / 9

ABAP Unicode Enabling Golden Rule

Distinguish character and byte based data 1 Character 1 Byte

SAP AG 2007, SAP TechEd 07 / LCM262 / 10

Overview ABAP Language Enhancements File Interfaces Communication via RFC Unicode enabling tools

Program Attribute Unicode checks active

SAP AG 2007, SAP TechEd 07 / LCM262 / 12

Unicode-Enabled ABAP Programs Program attribute Unicode checks active

Required to run on a Unicode system
Non-Unicode system Attribute set (Unicode enabled) Attribute not set (not Unicode enabled) ok Unicode system ok

ok

not allowed

If attribute is set, additional restrictions:

apply at compile and at run time apply in Unicode systems and in non-Unicode systems ensure that program will run on non-Unicode and Unicode systems with (almost) identical behavior

SAP AG 2007, SAP TechEd 07 / LCM262 / 13

Unicode Restrictions String Processing Character Processing

CONCATENATE cf1 cf2 TO cf3. IF cf1 CS cf2. ... String operations are only allowed for character-like operands
ABAP types C, N, D, and T, STRING Structures consisting only of characters (C, N, D, T) X and XSTRING are no longer considered character-like types

Byte Processing
CONCATENATE xf1 xf2 TO xf3 IN BYTE MODE. IF xf1 BYTE-CS xf2. ... Variants of string operations for byte processing
Addition IN BYTE MODE for statements Prefix BYTE- for comparison operations

Only operands of type X or XSTRING allowed

SAP AG 2007, SAP TechEd 07 / LCM262 / 14

Unicode Restrictions Length and Distance Determining the Length and Distance
Counted in bytes or in characters? Specify! DESCRIBE FIELD...LENGTH... IN (BYTE | CHARACTER) MODE DESCRIBE DISTANCE BETWEEN ... AND ... INTO ... IN (BYTE | CHARACTER) MODE.

Example
FORM write3 USING fld TYPE c. DATA: fldlen TYPE i. DESCRIBE FIELD fld LENGTH fldlen IN CHARACTER MODE. IF fldlen >= 3. WRITE: / fld(3). ENDIF. ENDFORM.

SAP AG 2007, SAP TechEd 07 / LCM262 / 15

Unicode Restrictions Access with Offset or Length Access To Structures With Offset/Length
Structure must begin with characters Offset/length counted in characters Access only allowed within the character type prefix of a structure

N(6) +off(len)

C(4)

X(3)

C(5)

ASSIGN fld+off(len) TO ...

Access must not exceed field boundaries If ASSIGN fails, field-symbol is set to unassigned New ... RANGE addition allows the permissible boundaries to be expanded
SAP AG 2007, SAP TechEd 07 / LCM262 / 16

New ABAP Features Includes with Group Names Symbolic Access to Includes of Structures
TYPES: BEGIN OF t_key, k1(2) TYPE x, k2(2) TYPE c, END OF t_key. TYPES: BEGIN OF t_rest, r1(10) TYPE c, r2(10) TYPE c, END OF t_rest. stru

DATA: BEGIN OF stru. INCLUDE TYPE t_key as key. INCLUDE TYPE t_rest as rest. DATA: END OF stru. DATA: skey TYPE t_key, srest TYPE t_rest. Pre-Unicode skey = stru(4). srest = stru+4(20). WRITE: stru-r2.

k1 k2 key

r1 r2 rest

Unicode enabled with group names skey = stru-key. srest = stru-rest. WRITE: stru-r2.

SAP AG 2007, SAP TechEd 07 / LCM262 / 17

New ABAP Features Golden Rule

Think in types Think in semantics Tell the system what you want to do

SAP AG 2007, SAP TechEd 07 / LCM262 / 18

Unicode Restrictions - MOVE MOVE Between Incompatible Structures

Matching data layout (fragment views) required
struc1 C(6) N(4) X(3) N(4) I P(8)

struc2

C(4)

C(3) C(10)

C(3)

X(3) X(3)

C(4) C(4)

I I P(8) fragments

Example
DATA: BEGIN OF cstru, first(10) TYPE c, tab(1) TYPE c, last(10) TYPE c, END OF cstru. cstru = xstru.
SAP AG 2007, SAP TechEd 07 / LCM262 / 19

DATA: BEGIN OF xstru, first(10) TYPE c, tab(1) TYPE x VALUE '09', last(10) TYPE c, END OF xstru. "Unicode error!

New ABAP Features Import/Export Data Buffer Using fields of type xstring as data containers
Writing data to an xstring. DATA: my_buffer TYPE xstring. data1 TYPE some_type. ... EXPORT id = data1 TO DATA BUFFER my_buffer.
Data is stored in a platform-independent format Contents of xstring can be exchanged with any other 6.10-system (Unicode and non-Unicode)

Reading data from an xstring FORM read_buffer USING buffer TYPE xstring. DATA: fld2 TYPE some_type. IMPORT id = fld2 FROM DATA BUFFER buffer. ... ENDFORM.
Automatic conversion of data during import
SAP AG 2007, SAP TechEd 07 / LCM262 / 20

New ABAP Features Dynamic Programming Support Creating Data Objects Dynamically
Creating and accessing data objects on the heap DATA: dref TYPE REF TO data. CREATE CREATE CREATE CREATE DATA DATA DATA DATA dref dref dref dref TYPE TYPE TYPE TYPE sometype. (typename). c LENGTH len. STANDARD TABLE OF (typename) "access data object

ASSIGN dref->* TO <f>.

Casting to User Defined Types

Look at the contents of a field as a value of another type FIELD-SYMBOLS: <f> TYPE any. ASSIGN fld TO <f> CASTING TYPE sometype. ASSIGN fld TO <f> CASTING TYPE (typename). fld must provide sufficient alignment and length for the given type
SAP AG 2007, SAP TechEd 07 / LCM262 / 21

New ABAP Features - Generic Types New generic types for parameters and field-symbols
Eliminate untyped parameters or field-symbols for improved security and performance

SIMPLE SIMPLE

CLIKE CLIKE

CSEQUENCE CSEQUENCE

XSEQUENCE XSEQUENCE

NUMERIC NUMERIC

C C

STRING STRING

N N

D D

T T

character-like character-like structures structures (C,N,D,T) (C,N,D,T)

X X

XSTRING XSTRING

II

F F

P P

SAP AG 2007, SAP TechEd 07 / LCM262 / 22

New ABAP Features Enhancement categorization If you are writing software for others you may have the following Problem
Enhancements on structures or tables may affect your coding:
Syntax-/runtime errors Changed behavior (e.g. damaged or changed data)

Solution
Maintaining the enhancement category in the DDIC: SE11 (Extras -> Enhancement Category)
Can not be enhanced Can be enhanced - character like Can be enhanced character and numerical type Can be arbitrarily enhanced

Additional checks are done on your ABAP programs (SLIN) and show possible problems in allowed enhancement situations

SAP AG 2007, SAP TechEd 07 / LCM262 / 23

New ABAP Features ABAP list programming ABAP lists: Difference between memory and display length

Character units in the memory

Display columns

Non-Unicode Unicode

2 1
1 Display Column

2 2

1 Character

SAP AG 2007, SAP TechEd 07 / LCM262 / 24

New ABAP Features Different list types

Half width (Default)

Dynamic

Full width

SAP AG 2007, SAP TechEd 07 / LCM262 / 25

ABAP list programming Golden Rules

Use ALV Grid and ALV List Explicitely define display length class CL_ABAP_LIST_UTILITIES helps in complex cases

SAP AG 2007, SAP TechEd 07 / LCM262 / 26

New ABAP Features Character Utilities Class CL_ABAP_CHAR_UTILITIES

Constant attributes with system specific values
charsize length of 1 character in bytes newline cr_lf form_feed horizontal_tab vertical_tab backspace minchar X00 in non-Unicode systems, U+0000 in Unicode systems maxchar XFF in non-Unicode systems, U+FFFD in Unicode systems

Example

CLASS cl_abap_char_utilities DEFINITION LOAD. DATA: text TYPE string. REPLACE cl_abap_char_utilites=>horizontal_tab WITH space INTO text.

SAP AG 2007, SAP TechEd 07 / LCM262 / 27

New ABAP Features Conversion Classes Conversion classes

Code page conversion
Unicode / non-Unicode code pages

Endian conversion
little endian / big endian byte order

Character conversion
Unicode codepoint / ABAP character

ABAP Class CL_ABAP_CONV_IN_CE CL_ABAP_CONV_OUT_CE CL_ABAP_CONV_X2X_CE

Conversion
any code page system code page any code page any code page

system code page any code page

SAP AG 2007, SAP TechEd 07 / LCM262 / 28

Overview ABAP Language Enhancements File Interfaces Communication via RFC Unicode enabling tools

File transfer Golden Rule

Know the code page used by the other side or ask the user

SAP AG 2007, SAP TechEd 07 / LCM262 / 30

File transfer: Application server Pattern for writing/reading files on the application server:

OPEN DATASET IN <mode> MODE TRANSFER/READ CLOSE DATASET

SAP AG 2007, SAP TechEd 07 / LCM262 / 31

File transfer: TEXT MODE ENCODING NON-UNICODE TEXT MODE ENCODING NON-UNICODE
Allowed types character like Behavior convert text data between system code page and non-Unicode encoding matching to current system language (sy-langu) Usage backward compatible exchange of text data with systems that cannot support UTF-8

SAP AG 2007, SAP TechEd 07 / LCM262 / 32

File transfer: TEXT MODE ENCODING NON-UNICODE Example: TEXT MODE NON-UNICODE

TEXT MODE NON-UNICODE SY-LANGU

1100 8000

TEXT MODE NON-UNICODE

R/3 Enterprise
TEXT MODE NON-UNICODE SY-LANGU
1100 8000

R/3
TEXT MODE NON-UNICODE
ISO8859-1 SJIS

Only part of UC charset supported (possible data loss in the file) Structured data as a whole
SAP AG 2007, SAP TechEd 07 / LCM262 / 33

write field by field =

File transfer: ENCODING UTF-8 TEXT MODE ENCODING UTF-8

Allowed types character like Behavior convert text data between system code page and UTF-8 encoding Usage loss free exchange of text data

SAP AG 2007, SAP TechEd 07 / LCM262 / 34

File transfer: TEXT MODE ENCODING UTF-8 Example: TEXT MODE UTF-8

TEXT MODE UTF-8

TEXT MODE UTF-8 SY-LANGU

R/3 Enterprise
TEXT MODE UTF-8 TEXT MODE UTF-8 SY-LANGU

R/3
ISO8859-1 SJIS

Full charset supported (no data loss in the file) Structured data as a whole
SAP AG 2007, SAP TechEd 07 / LCM262 / 35

write field by field =

File transfer: BINARY MODE BINARY MODE

Allowed types binary and character like Behavior no conversion, copy bytes between memory and file Usage fix format with defined encoding, endian and data structure

SAP AG 2007, SAP TechEd 07 / LCM262 / 36

File transfer: Using XML Using XML as transport format

Use CALL TRANSFORMATION with target data type XSTRING to create an UTF-8 based XML representation of your data. Structure information (no layout / alignment problems) UTF-8 based (no data loss) Transport in binary form

SAP AG 2007, SAP TechEd 07 / LCM262 / 37

File transfer: BINARY + Using XML Example: UTF-8 based XML + BINARY MODE
CALL TRANSFORMATION + BINARY MODE BINARY MODE + CALL TRANSFORMATION SY-LANGU

R/3 Enterprise
BINARY MODE + CALL TRANSFORMATION CALL TRANSFORMATION + BINARY MODE SY-LANGU

R/3
ISO8859-1 SJIS

Full charset supported (no data loss in the file) Structured data
SAP AG 2007, SAP TechEd 07 / LCM262 / 38

File transfer: LEGACY MODE LEGACY TEXT/BINARY MODE

Allowed types binary and character like Behavior Produce in Unicode system format of non-Unicode convert text data to non-Unicode read/transfer binary data as is Usage backward compatibility with systems that mix binary and character data within a file Difference between TEXT and BINARY LEGACY TEXT MODE adds end of line marker

SAP AG 2007, SAP TechEd 07 / LCM262 / 39

Overview ABAP Language Enhancements File Interfaces Communication via RFC Unicode enabling tools

RFC Golden Rule

Everything Unicode or ensure data fit to receiver

SAP AG 2007, SAP TechEd 07 / LCM262 / 41

RFC Unicode

Unicode

R/3 Enterprise

R/3 Enterprise

In case of an Unicode Unicode combination RFC passes all character data without code page conversion or merely with adaption of the endianness. UTF-16 big endian UTF-16 little endian = SAP code page 4102 = SAP code page 4103

Information about the destination is maintained in SM59 special options character width in target system 1 Byte = non-Unicode 2 Byte = Unicode
SAP AG 2007, SAP TechEd 07 / LCM262 / 42

RFC Unicode

non-Unicode single code page

R/3 Enterprise

R/3 4.6C
ISO8859-1

In case of an Unicode non-Unicode single code page combination, RFC passes all character data with code page conversion between Unicode and the old code page. As Unicode is a true superset of any old standard codepage not all Unicode characters can be transfered to the non-Unicode system: # # # #
SAP AG 2007, SAP TechEd 07 / LCM262 / 43

RFC Unicode

non-Unicode MDMP (table with lang key)

R/3 Enterprise

R/3 4.6C
ISO8859-1 SJIS

In case of an Unicode non-Unicode MDMP combination RFC passes all character data with code page conversion between Unicode and the different old code pages. Which of the MDMP code pages is chosen depends on the language: DE DE JA JA This mechanism is only available for flat tables with language key

SAP AG 2007, SAP TechEd 07 / LCM262 / 44

Overview ABAP Language Enhancements File Interfaces Communication via RFC Unicode enabling tools

Migrating to Unicode Enabled ABAP

Step 1
In non-Unicode system Adapt all ABAP programs to Unicode syntax and runtime restrictions Set attribute "Unicode enabled" for all programs

Step 2
Set up a Unicode system
Unicode kernel + Unicode database Only ABAP programs with the Unicode attribute are executable

Do runtime tests in Unicode system

Check for runtime errors Look for sematic errors Check ABAP list layout with former double byte characters

SAP AG 2007, SAP TechEd 07 / LCM262 / 46

Transaction UCCHECK

SAP AG 2007, SAP TechEd 07 / LCM262 / 47

Step 1 Unicode Enabling with UCCHECK Use UCCHECK to analyze your applications:
Remove errors Inspect statically not analyzable places (optional)
Untyped field symbols Offset with variable length Generic access to database tables

Set unicode program attribute using UCCHECK or SE38 / SE24 / ... Do additional checks with SLIN (e.g. matching of actual and formal parameters in function modules)

SAP AG 2007, SAP TechEd 07 / LCM262 / 48

UCCHECK Setting Unicode Flag

SAP AG 2007, SAP TechEd 07 / LCM262 / 49

UCCHECK Statically Non-Analyzable Places What to do with the places that can only be checked at runtime?
Reduce their number
In many cases you can specify the type of parameters and field-symbols Use generic ABAP types where neccessary Mark those places that really need untyped parameters due to some kind of generic programming with #EC * as OK after you did revise them.

Do

Runtime tests

SAP AG 2007, SAP TechEd 07 / LCM262 / 50

Step 2 Testing Your Application Final tests in the Unicode system Runtime tests, Runtime tests, Runtime tests, ...
Because the amount of warnings due to statically not analyzable places may be very large, you cannot type everything. In this case you have to rely on run-time tests. Some semantic problems may be seen only in the Unicode system (e.g. byte or character length) ABAP list layout can be checked only manually

Monitoring of runtime tests: Having test plans is good, knowing the coverage of the test is better: Use the ABAP Coverage Analyzer to monitor runtime tests

SAP AG 2007, SAP TechEd 07 / LCM262 / 51

Summary Distinguish characters and bytes Distinguish characters and display cells Think in types, think in semantics, tell the system what you want Define code page for each communication You can start ABAP Unicode enabling today

SAP AG 2007, SAP TechEd 07 / LCM262 / 52

Further Information

SAP Public Web: SAP Developer Network (SDN): www.sdn.sap.com SAP Service Marketplace: www.service.sap.com/unicode@sap Related SAP Education and Certification Opportunities http://www.sap.com/education/ Related Workshops/Lectures at SAP TechEd 2007 LCM211, Conversion of MDMP Systems to Unicode, Lecture LCM212, Conversion of Single Code Page Systems to Unicode, Lecture

SAP AG 2007, SAP TechEd 07 / LCM262 / 53

SDN Subscriptions Program The SDN Subscriptions Program introduces the SAP NetWeaver, Development Subscription for individual developers. Available for purchase in Germany and the United States. Subscription gives you one year access to
SAP NetWeaver platform software, patches, and updates Development license for SAP NetWeaver to evaluate, develop and test Standard software maintenance Online sessions from SAP TechEd Access to SAP Enterprise Services Workplace for testing Premium presence in forums

Purchase the SAP NetWeaver, Development Subscription today at the SAP Community Clubhouse, or online at https://www.sdn.sap.com/irj/sdn/devsub Visit us at the Community Clubhouse, show us you are a subscriber, and get a gift!
SAP AG 2007, SAP TechEd 07 / LCM262 / 54

Q&A

THANK YOU FOR YOUR ATTENTION !

QUESTIONS SUGGESTIONS DISCUSSION

SAP AG 2007, SAP TechEd 07 / LCM262 / 55

Exercises
SAP AG 2007, SAP TechEd 07 / LCM262 / 56

Feedback
Please complete your session evaluation. Be courteous deposit your trash, and do not take the handouts for the following session.

Thank You !

SAP AG 2007, SAP TechEd 07 / LCM262 / 57

Disclaimer
This presentation outlines our general product direction and should not be relied on in making a purchase decision. This presentation is not subject to your license agreement or any other agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to develop or release any functionality mentioned in this presentation. This presentation and SAP's strategy and possible future developments are subject to change and may be changed by SAP at any time for any reason without notice. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or noninfringement. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.

SAP AG 2007, SAP TechEd 07 / LCM262 / 58

Copyright 2007 SAP AG. All Rights Reserved

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, OS/2, Parallel Sysplex, MVS/ESA, AIX, S/390, AS/400, OS/390, OS/400, iSeries, pSeries, xSeries, zSeries, System i, System i5, System p, System p5, System x, System z, System z9, z/OS, AFP, Intelligent Miner, WebSphere, Netfinity, Tivoli, Informix, i5/OS, POWER, POWER5, POWER5+, OpenPower and PowerPC are trademarks or registered trademarks of IBM Corporation. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries. Oracle is a registered trademark of Oracle Corporation. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc. HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C , World Wide Web Consortium, Massachusetts Institute of Technology. Java is a registered trademark of Sun Microsystems, Inc. JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. MaxDB is a trademark of MySQL AB, Sweden. SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. Please note that this document is subject to change and may be changed by SAP at any time without notice. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence. The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages.

SAP AG 2007, SAP TechEd 07 / LCM262 / 59