GG Troubleshooting v104
GG Troubleshooting v104
October 2009
Troubleshooting and Tuning Guide, version 10.4 Copyright 1995, 2009 Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle USA, Inc., 500 Oracle Parkway, Redwood City, CA 94065. This software is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications which may create a risk of personal injury. If you use this software in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure the safe use of this software. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software in dangerous applications. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. This software and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services.
Contents
...............
Preface About the GoldenGate Guides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Typographic conventions used in this manual. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 New in this version of GoldenGate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Database-specific enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Other enhancements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Documentation enhancements and corrections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Chapter 1 GoldenGate performance and troubleshooting tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 Overview of the GoldenGate performance tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 Using the information commands in GGSCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 Positioning processes using GGSCI commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12 Using the process report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 Scheduling runtime statistics in the process report . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Viewing record counts in the process report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Managing process reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Using the error log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 Using the discard file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 Using GoldenGate Director . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Using the system logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 Using SHOWSYNTAX to view Replicat SQL statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 Using tracing parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 Using DUMPDDL to view DDL statements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 Using GoldenGate error documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 Using the GoldenGate Knowledge Base. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 Using the Logdump utility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 Chapter 2 Tuning GoldenGates Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 Overview of performance tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 Timing process startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 Balancing the load across parallel process groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 Considerations for using parallel process groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
........................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 1
Contents
Using parallel Replicat groups on a target system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Using parallel Extract groups with parallel Replicat groups . . . . . . . . . . . . . . . . . . . . . 25 Splitting large tables into row ranges across process groups . . . . . . . . . . . . . . . . . . . 27 Tuning a data pump for pass-through efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 Configuring GoldenGate to use the network efficiently. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 Eliminating disk I/O bottlenecks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31 Optimizing data filtering and conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32 Applying similar SQL statements in arrays. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32 Preventing full table scans in the absence of keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32 Splitting large transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33 Tuning Replicat transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33 Chapter 3 Troubleshooting GoldenGate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 Overview of GoldenGate troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 Troubleshooting example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Troubleshooting process startup failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36 Configuration problems that cause startup failure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36 Operating system errors that cause startup failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38 Database errors that cause statup failure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39 Troubleshooting slow startup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39 Troubleshooting Extract problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40 Extract freezes during recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40 Extract is too slow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40 Extract stalls or pauses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42 Extract abends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42 Troubleshooting Replicat problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45 Replicat is too slow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45 Replicat abends. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46 Cannot delete a Replicat group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51 Troubleshooting data-synchronization problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51 Testing for out-of-sync rows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51 Causes of out-of-sync data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52 Discrepancies caused by DML on the target. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52 Discrepancies caused by tables without keys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52 Discrepancies caused by character sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52 Discrepancies caused by non-matching structures or keys. . . . . . . . . . . . . . . . . . . . . . . . . . .53 Resynchronizing an out-of-sync table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53 Troubleshooting data-mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54 Table and column names. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55
........................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 2
Contents
Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55 Column-missing error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55 Fetch failures on Oracle source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56 Replicat errors on Sybase targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57 Escaped characters not converting correctly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57 Troubleshooting SQLEXEC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57 Troubleshooting missed transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58 Troubleshooting problems with Oracle DDL replication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60 Extract abends with OCI Error executing fetch ... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60 No DDL is captured by Extract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61 Specific DDL is not captured or excluded . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64 Data on new, altered, and renamed tables not captured . . . . . . . . . . . . . . . . . . . . . . . . . . . . .64 RENAMEs converted to ALTER TABLE RENAME. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .65 RENAME operations are failing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .65 Comments are in the wrong place . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66 Derived object names are not being converted correctly . . . . . . . . . . . . . . . . . . . . . . . . . . . .66 String substitution not working . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66 DDL processing steps not in the process report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66 Replicat abends on ggs_ table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .67 The database returns ORA-04021 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .67 The DDL objects tablespace filled up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .67 Troubleshooting file-maintenance problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .68 Process report not generated. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .68 Discard file not created. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .68 Discard file too big . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .68 Trail file is not emptying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69 Trail not rolling over. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69 Trail files not being purged . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69 Trail files purge too soon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70 Report file too big. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70 Troubleshooting other problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71 Too many open cursors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71 Time differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71 Oracle ADD TRANDATA problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71 Other Oracle-specific problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72 Sybase missing server messages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72 DB2 z/OS-specific problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72 SQL Server-specific problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .74
........................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 3
Contents
Teradata module not found . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .75 Chapter 4 Using the Logdump Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76 Getting started with Logdump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76 Viewing the first record . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76 Executing basic Logdump tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .78 Evaluating transaction size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81 Maintaining command history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82 Logdump command reference guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82 Logdump command summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82 ASCIIDATA | EBCDICDATA. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85 ASCIIDUMP | EBCDICDUMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85 ASCIIHEADER | EBCDICHEADER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85 CD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86 COMPUTETIMESTAMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86 COUNT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86 CTIME . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89 DEBUG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89 DECRYPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90 DETAIL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90 DUMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90 ENCRYPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .91 ENV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .91 EXIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92 FC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92 Using the editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 FILEHEADER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93 FILES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 FILTER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Comparison operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 GHDR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 HEADERTOKEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 HELP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 HISTORY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 INTERPRETINTERVAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 INTERPRETTIMESTAMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 LOG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 NEXT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 NEXTTRAIL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
........................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 4
Contents
NOTIFY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 OBEY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 OPEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 POSITION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 RECLEN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 SAVE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 SCANFORENDTRANSACTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 SCANFORHEADER. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 SCANFORRBA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 SCANFORTIME . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 SCANFORTYPE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 SCANSCROLLING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 SHOW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 SKIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 TIME. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 TIMEOFFSET . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 TRAILFORMAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 TRANSBYTELIMIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 TRANSHIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 TRANSRECLIMIT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 USERTOKEN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 WRITELOG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 X . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Chapter 5 Event Error and Warning Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
........................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 5
PREFACE
GoldenGate for HP NonStop Administrator Guide: Introduces GoldenGate components and explains how to plan for, configure, and implement GoldenGate on the NonStop platform. GoldenGate for HP NonStop Reference Guide: Provides detailed information about GoldenGate parameters, commands, and functions for the NonStop platform.
Installation and Setup guides: There is one such guide for each database that is supported by GoldenGate. GoldenGate for Windows and UNIX Administrator Guide: Introduces GoldenGate components and explains how to plan for, configure, and implement GoldenGate on the Windows and UNIX platforms. GoldenGate for Windows and UNIX Reference Guide: Provides detailed information about GoldenGate parameters, commands, and functions for the Windows and UNIX platforms. Troubleshooting and Tuning Guide: Provides suggestions for improving the performance of GoldenGate in different situations, and provides solutions to common problems.
Parameter and command arguments are shown in upper case, for example:
CHECKPARAMS
File names, table names, and other names are shown in lower case unless they are case-sensitive to the operating system or software application they are associated with, for example:
account_tab GLOBALS
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 6
When one of multiple mutually-exclusive arguments must be selected, the selection is enclosed within braces and separated with pipe characters, for example:
VIEW PARAMS {MGR | <group> | <file name>}
When there are numerous multiple optional arguments, a placeholder such as [<option>] may be used, and the options are listed and described separately, for example:
TRANLOGOPTIONS [<option>]
When an argument is accepted more than once, an ellipsis character (...) is used, for example:
PARAMS ([<requirement rule>] <param spec> [, <param spec>] [, ...])
GoldenGate now supports Multi Dimensional Clustered Tables (MDC) for DB2 LUW 9.5 and later. GoldenGate now supports Materialized Query Tables. GoldenGate does not replicate the MQT itself, but only the base tables. The target database automatically maintains the content of the MQT based on the changes that are applied to the base tables by Replicat.
DB2 z/OS A new SQLID option was added to DBLOGIN that issues the SQL command SET CURRENT SQLID = 'sqlid' after the USERID login (with PASSWORD, if applicable) is completed. If the SET command fails, the entire DBLOGIN command fails as a unit. Oracle
GoldenGate now supports Oracle 10.2 running on zLinux SUSE on S/390. A new LOGSOURCE target of S390 has been added to TRANLOGOPTIONS. For Oracle versions 10g and later, you can now use a simple parameter statement instead of a trace table to prevent Replicat transactions from being captured in a bidirectional configuration. This enhancement eliminates the overhead of creating and writing to the trace table. The new EXCLUDEUSER and EXCLUDEUSERID options of TRANLOGOPTIONS support this enhancement. These options also can be used to exclude the work of other database users, and multiple instances of either parameter can be used. For more information, see the GoldenGate for Windows and UNIX Reference Guide. For Oracle versions 9i and earlier, the trace table is still required.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 7
Concurrent with the EXCLUDEUSER and EXCLUDEUSERID enhancements, @GETENV was enhanced with two new functions: @GETENV ("TRANSACTION", "USERID") returns the user-id for the Oracle user that issued the last committed transaction. @GETENV ("TRANSACTION", "USERNAME") returns the name of the Oracle user that issued the last committed transaction. GoldenGate now supports Oracle Spatial objects, object tables, Oracle Multimedia ORDDicom (DICOM), and XMLType stored as an object. A new XMLBUFSIZE option to DBOPTIONS sets the size of the memory buffer that stores XML data that was extracted from the sys.xmltype attribute of a SDO_GEORASTER object type. GoldenGate now supports cluster tables. GoldenGate now supports the capture and replication of Oracle DDL statements of up to 2 MB in length (including some internal GoldenGate maintenance information). Extract will skip statements that are greater than the supported length, but a new ddl_ddl2file.sql script can be used to save the skipped DDL to a text file in the USER_DUMP_DEST directory of Oracle.
NOTE To use the new support, the DDL trigger must be reinstalled in INITIALSETUP mode, which removes all of the DDL history. Follow the procedure for restoring an existing DLL environment to a clean state in the Oracle Installation and Setup Guide.
A new INSERTAPPEND and NOINSERTAPPEND parameter set controls whether or not Replicat uses an APPEND hint when it applies inserts to Oracle target tables. INSERTAPPEND is appropriate for use as a performance improvement when the replicated transactions are large and contain multiple inserts into the same table. These parameters can be used in two ways: When used as standalone parameters at the root of the parameter file, one remains in effect for all subsequent TABLE or MAP statements, until the other is encountered. When used within a MAP statement, they override any standalone INSERTAPPEND or NOINSERTAPPEND entry that precedes the MAP statement. Schema name translation for DDL operations on objects of UNMAPPED scope was revised. When Oracle DDL is of UNMAPPED scope in the Replicat configuration, Replicat will set the current owner of the Replicat session to the owner of the source DDL object, then execute the DDL as that owner, then restore Replicat as the current owner of the Replicat session.
SQL Server GoldenGate now supports delivery to a SQL Server 2008 target. See the GoldenGate for SQL Server Installation and Setup Guide for more information. Teradata Database name translation for DDL operations on objects of UNMAPPED scope was revised. When Teradata DDL is of UNMAPPED scope in the Replicat configuration, it is applied to the target in one of these ways:
If the required Replicat connection parameter TARGETDB contains just a DSN (as in tdtarg), but not a database name, it is applied to the target object with the same owner (database name) and object name as in the source DDL. If a specific database name is used in TARGETDB (as in db@tdtarg), all of the DDL operations are applied to the target with the owner from TARGETDB.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 8
Other enhancements
General
GoldenGate now supports the replication of DDL statements that contain a space before or after the owner and object names (or both before and after). For example, fin . customers.
A new OSVARIABLE option of @GETENV returns the string value of a specified operatingsystem environment variable. For example, @GETENV ("OSVARIABLE", "HOME") would return the value of the HOME variable. Because the function only returns an exact match, other variables that contain HOME, such as ORACLE_HOME, would not be returned. This option is valid for Extract and Replicat.
Parameters
A new TRANSACTIONTIMEOUT parameter for Replicat limits the amount of time that Replicat will hold a target transaction open if it has not received the end-of-transaction record for the last source transaction in that transaction. TRANSACTIONTIMEOUT helps prevent an uncommitted Replicat target transaction from holding locks on the target database and consuming its resources unnecessarily. You can change the value of this parameter so that Replicat can work within existing application timeouts and other database requirements on the target. A new GLOBALS parameter OUTPUTFILEUMASK specifies an octal umask that will be used by GoldenGate processes to create trail files and discard files. This parameter is not valid for WIN32 systems. The default for CACHESIZE is now 8GB for 64-bit systems and 2GB for 32-bit systems.
Commands
The SEND EXTRACT <group> STATUS command now makes it clear when Extract is going through a recovery after an abend event. This feature is especially useful when a very long-running transaction was open at the point of failure and Extract must search far back into the logs to find the begin-transaction record. The new status messages appear on the Current status line and look as follows:
In recovery[1] Extract is recovering to its input checkpoint. In recovery[2] Extract is recovering to its output checkpoint. Recovery complete The recovery is finished, and normal processing will resume.
To support TRANSACTIONTIMEOUT, the SEND REPLICAT command with STATUS has two more status conditions:
Performing transaction timeout recovery Aborting current incomplete transaction and repositioning to start new one (see TRANSACTIONTIMEOUT parameter). Waiting for data at logical EOF after transaction timeout recovery Waiting to receive remainder of incomplete source transaction after a TRANSACTIONTIMEOUT
termination.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 9
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 10
CHAPTER 1
GGSCI commands GoldenGate tracing parameters GoldenGate reports and logs GoldenGate Director graphical user interface System logs GoldenGate Logdump utility GoldenGate error documentation GoldenGate Knowledge Base
Run status, checkpoints, approximate lag, and environmental information Run status and port number
INFO output for all GoldenGate processes on the system
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 11
Table 1
Command STATS {EXTRACT | REPLICAT} <group> STATUS {EXTRACT | REPLICAT} <group> STATUS MANAGER LAG {EXTRACT | REPLICAT} <group>
Statistics for operations processed Run status (starting, running, stopped, abended) Run status Latency between last record processed and timestamp in the data source Name of associated process, position of last data processed, maximum file size Run status, information about child processes, port information, trail purge settings Depending on the process, returns information about memory pool, lag, TCP statistics, longrunning transactions, process status, recovery progress, and more. Contents of the process report Contents of the GoldenGate error log Information dependent on the <command> type:
INFO LAG SEND STATS STATUS <wildcard> is a wildcard specification for the process groups to be affected, for example: INFO ER ext* STATS ER *
SEND MANAGER
ADD EXTRACT and ALTER EXTRACT for a primary Extract or a data pump have options to start at a given timestamp or physical location in the data source. In addition, ALTER EXTRACT enables you to start a new file within a trail. (SEND EXTRACT also provides this capability.)
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 12
SEND EXTRACT provides the option to skip the oldest transaction, allowing the management of long-running transactions. ADD REPLICAT and ALTER REPLICAT have options to start at a given timestamp or at a
For full details and syntax, see the GoldenGate for Windows and UNIX Reference Guide.
parameters in use table and column mapping database information runtime messages and errors runtime statistics for the number of operations processed
Every Extract, Replicat, and Manager process generates a report file at the end of each run. The report can help you diagnose problems that occurred during the run, such as invalid mapping syntax, SQL errors, and connection errors.
Figure 1
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 13
Syntax
<group> shows an Extract or Replicat report that has the default name, which is the
Report names are in upper case if the operating system is case-sensitive. By default, reports have a file extension of .rpt, for example EXTORA.rpt. The default location is the dirrpt sub-directory of the GoldenGate directory.
To determine the name and location of a process report
Run the process from the command shell of the operating system (not GGSCI) to send the information to the terminal.
Syntax <process> paramfile <path name>.prm Where:
<process> is either Extract or Replicat. paramfile <path name>.prm is the fully qualified name of the parameter file.
Example
Use the REPORT parameter in the Extract or Replicat parameter file to specify a day and time to generate runtime statistics in the report.
To send runtime statistics to the report on demand
Use the SEND EXTRACT or SEND REPLICAT command with the REPORT option to view current runtime statistics when needed.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 14
1011 Sep 27 14:10 3184 Sep 27 14:10 2655 Sep 27 14:06 2655 Sep 27 14:04 2744 Sep 27 13:56 3571 Aug 29 14:27
Use the REPORTROLLOVER parameter to force report files to age on a regular schedule, instead of when a process starts. For long or continuous runs, setting an aging schedule controls the size of the active report file and provides a more predictable set of archives that can be included in your archiving routine.
To prevent SQL errors from filling up the Replicat report
Use the WARNRATE parameter to set a threshold for the number of SQL errors that can be tolerated on any target table before being reported to the process report and to the error log. The errors are reported as a warning. If your environment can tolerate a large number of these errors, increasing WARNRATE helps to minimize the size of those files.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 15
a history of GGSCI commands GoldenGate processes that started and stopped processing that was performed errors that occurred informational and warning messages
Because the error log shows events as they occurred in sequence, it is a good tool for detecting the cause (or causes) of an error. For example, you might discover that:
someone stopped a process a process failed to make a TCP/IP or database connection a process could not open a file
Figure 3
Standard shell command to view the ggserr.log file within the root GoldenGate directory GoldenGate Director
VIEW GGSEVT command in GGSCI
Syntax
The error log can become very large, but you can filter it based on a keyword. For example, this filter show only errors:
$ more ggserr.log | grep ERROR
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 16
Because the error log will continue to grow as you use GoldenGate, consider archiving and deleting the oldest entries in the file.
NOTE The Collector process might stop reporting to the log on UNIX systems after the log has been cleaned up. To get reporting started again, restart the Collector process after the cleanup.
The database error message The sequence number of the data source or trail file The relative byte address of the record in the data source or trail file The details of the discarded operation, such as column values of a DML statement or the text of a DDL statement.
A discard file can be used for Extract or Replicat, but it is most useful for Replicat to log operations that could not be reconstructed or applied.
Figure 4
Include the DISCARDFILE parameter in the Extract or Replicat parameter file. You must supply a name for the file. The parameter has options that control the maximum file size, after which the process abends, and whether new content overwrites or appends to existing content.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 17
Syntax
DISCARDFILE <file name> [, APPEND | PURGE] [, MAXBYTES <n> | MEGABYTES <n>] NOTE A process does not create a discard file until the first time that it cannot process a record from its data source.
Syntax
Use the DISCARDROLLOVER parameter to set a schedule for aging discard files. For long or continuous runs, setting an aging schedule prevents the discard file from filling up and causing the process to abend, and it provides a predictable set of archives that can be included in your archiving routine.
Syntax DISCARDROLLOVER {AT <hh:mi> | ON <day of week> | AT <hh:mi> ON <day of week>}
GoldenGate messaging to the syslog is enabled by default on UNIX. On Windows, GoldenGate messaging to the Event Viewer is disabled by default.
To prevent GoldenGate from logging messages to the syslog, exclude the user.info or *.info from the syslog file. Contact your Systems Administrator for assistance with this procedure.
To enable GoldenGate messaging on Windows
1. 2.
Run the install program with the addevents option. This enables generic messages to be logged. (Optional) To get more specific Windows messages, copy the category.dll and ggsmsg.dll libraries from the GoldenGate directory to the SYSTEM32 directory, either before or after running install. The detailed messages contain the GoldenGate user name and process, the name of the parameter file, and the error text.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 18
NOTE
Windows event messaging might have been installed when GoldenGate was installed. For more information on running install, see the GoldenGate installation guide for your database.
Figure 5
GoldenGate events in the UNIX system log are similar to those in the Windows log. See your system documentation for more information about the system logs. GoldenGate errors that appear in the system logs also appear in the GoldenGate error log.
Use the TLTRACE parameter to trace the activity of the database transaction log. Two levels of tracing can show either basic or detailed information about DML and DDL operations being processed.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 19
Use the TRACE and TRACE2 parameters to capture Extract or Replicat processing information to help reveal processing bottlenecks.
TRACE provides step-by-step processing information. TRACE2 identifies the code segments on which Extract or Replicat is spending the most
time.
Go to http://support.goldengate.com.
To get an ID and password to access the support site
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 20
CHAPTER 2
To address these questions, you can tune the performance of the following components in a GoldenGate environment:
Manager Primary Extract process Data pump Extract process Network Replicat
NOTE Performance tuning and troubleshooting often are done at the same time. See Troubleshooting GoldenGate on page 35 if you need additional help.
Use the AUTOSTART parameter to start one or more Extract and Replicat processes automatically when Manager starts. AUTOSTART ensures that no process groups are overlooked and that synchronization activities start immediately.
To restart processes automatically after recovery
Use the AUTORESTART parameter to start one or more Extract and Replicat processes automatically after they fail. AUTORESTART provides fault tolerance when something temporary interferes with a process, such as intermittent network outages or programs that interrupt access to transaction logs.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 21
Use the BOOTDELAYMINUTES parameter on a Windows system to delay the activities that Manager performs when it starts, such as executing parameters. For example, BOOTDELAYMINUTES can be used to delay AUTOSTART parameters until database services are started.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 22
depends on how much system memory is available. Each Extract and Replicat process needs approximately 25-55 MB of memory, or more depending on the size of the transactions and the number of concurrent transactions. The GoldenGate GGSCI command interface fully supports up to 300 concurrent Extract and Replicat groups per instance of GoldenGate Manager. At the supported level, all groups can be controlled and viewed in full with GGSCI commands such as the INFO and STATUS commands. Beyond the supported level, group information is not displayed and errors can occur. GoldenGate recommends keeping the number of Extract and Replicat groups (combined) at 300 or below in order to manage your environment effectively.
NOTE When creating the groups, keep tables that have relational constraints to each other in the same group.
Isolating processing-intensive tables Use dedicated process groups for certain kinds of tables that tend to interfere with normal processing and cause latency to build on the target. For example:
Extract fetches data for large-object types, for columns that are not logged to the transaction log, and for data that must be fetched because of parameter specifications or SQL procedures. When data must be fetched from the database, it affects the performance of Extract because the data must be fetched using native calls. You can get fetch statistics from the STATS EXTRACT command if you include the STATOPTIONS REPORTFETCH parameter in the Extract parameter file. The Replicat process is most often a source of performance bottlenecks because, in its normal mode of operation, it is a single-threaded process that applies operations one at a time by using regular SQL. Replicat takes longer to process tables that have large or long-running transactions, heavy volume, a very large number of columns that change, and LOB data.
NOTE You can use the Logdump TRANSHIST command to view the size of the transactions that are contained in a trail file. See Chapter 4. Depending on the types of transactions your applications generate, you might be able to gain processing efficiencies by using BATCHSQL instead of the normal Replicat mode. See the GoldenGate for Windows and UNIX Reference Guide.
NOTE
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 23
table1 and table2 and any child tables through one trail-group pair, and process table3 and table4 and their child tables through another.
Refer to the GoldenGate for Windows and UNIX Reference Guide for complete command and parameter syntax. For detailed instructions on configuring change synchronization, see the GoldenGate for Windows and UNIX Administrator Guide.
To create the Extract group NOTE If needed, one or more Extract data-pumps can be added to this configuration.
1.
On the source, use the ADD EXTRACT command to create an Extract group, specifying TRANLOG as the data source.
ADD EXTRACT <group>, TRANLOG, BEGIN <time> [, THREADS <n>]
2.
On the source, use the ADD RMTTRAIL command to specify the full path name of a trail to be created by GoldenGate on the target system.
ADD RMTTRAIL <trail_1>, EXTRACT <group>
Repeat this step to create a trail for each Replicat group that you will be using. For the EXTRACT argument of the command, specify the same Extract group.
ADD RMTTRAIL <trail_2>, EXTRACT <group>
3.
On the source, use the EDIT PARAMS command to create an Extract parameter file that contains the following required parameters plus any others that are appropriate for your database environment.
EXTRACT <group> [SOURCEDB <dsn>,][USERID <user>[, PASSWORD <pw>]] RMTHOST <hostname>, MGRPORT <portnumber> RMTTRAIL <trail_1> TABLE <owner>.<table_group_1>; RMTTRAIL <trail_2> TABLE <owner>.<table_group_2>;
Divide the source tables among different TABLE parameters, as many as you need. This example shows two. Link each TABLE statement to a different trail that is specified with the Extract parameter RMTTRAIL. If needed, you link multiple TABLE statements to any given trail.
4.
On the target, create a Replicat checkpoint table. This is a best practice. For instructions, see the GoldenGate for Windows and UNIX Administrator Guide. All Replicat groups can use the same checkpoint table. On the target, use the ADD REPLICAT command to create a Replicat group for each trail that you created. Use the EXTTRAIL argument of that command to link the group to a trail.
ADD REPLICAT <group_1>, EXTTRAIL <trail_1> ADD REPLICAT <group_2>, EXTTRAIL <trail_2>
5.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 24
6.
On the target, use the EDIT PARAMS command to create a Replicat parameter file for each Replicat group. In each file, use the following required parameters plus any others that are appropriatge for your database environment. Replicat group 1
REPLICAT <group_1> {SOURCEDEFS <full_pathname>} | ASSUMETARGETDEFS [TARGETDB <dsn>,] [USERID <user id>[, PASSWORD <pw>]] MAP <owner>.<table_group_1>, TARGET <owner>.<table_group_1>;
Replicat group 2
REPLICAT <group_2> {SOURCEDEFS <full_pathname>} | ASSUMETARGETDEFS [TARGETDB <dsn>,] [USERID <user id>[, PASSWORD <pw>]] MAP <owner>.<table_group_2>, TARGET <owner>.<table_group_2>;
You can use any number of MAP statements for any given Replicat group. All MAP statements for a given Replicat group must specify the same objects that are contained in the trail that is linked to the group.
7.
In the Manager parameter file on the target system, use the PURGEOLDEXTRACTS parameter to control the purging of files from the trails.
NOTE You can take this configuration a step further by dividing the rows of an individual table among multiple Replicat processes. This might be necessary, for example, when table1 receives one million inserts and updates per day, while the rest of the tables receive 100,000 or less. See Splitting large tables into row ranges across process groups on page 27.
Refer to the GoldenGate for Windows and UNIX Reference Guide for complete command and parameter syntax. For detailed instructions on configuring change synchronization, see GoldenGate for Windows and UNIX Administrator Guide.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 25
To create the Extract groups NOTE If needed, an Extract data-pump can be added to each Extract-Replicat pair.
1.
On the source, use the ADD EXTRACT command to create the Extract groups, specifying TRANLOG as the data source.
ADD EXTRACT <group_1>, TRANLOG, BEGIN <time> [, THREADS <n>] ADD EXTRACT <group_2>, TRANLOG, BEGIN <time> [, THREADS <n>]
2.
On the source, use the ADD RMTTRAIL command to create a trail for each Extract group.
ADD RMTTRAIL <remote_trail_1>, EXTRACT <group_1> ADD RMTTRAIL <remote_trail_2>, EXTRACT <group_2>
3.
On the source, use the EDIT PARAMS command to create an Extract parameter file for each Extract group. Use the following required parameters plus any others that are appropriate for your database environment. Extract group 1
EXTRACT <group_1> [SOURCEDB <dsn>,][USERID <user>[, PASSWORD <pw>]] RMTHOST <hostname>, MGRPORT <portnumber> RMTTRAIL <remote_trail_1> TABLE <owner>.<table_group_1>;
Extract group 2
EXTRACT <group_2> [SOURCEDB <dsn>,][USERID <user>[, PASSWORD <pw>]] RMTHOST <hostname>, MGRPORT <portnumber> RMTTRAIL <remote_trail_2> TABLE <owner>.<table_group_2>;
Divide the source tables among the Extract groups by using the TABLE parameter in each parameter file. This example shows two Extract parameter files. You can use any number of TABLE statements for any given Extract group. Use the Extract parameter RMTTRAIL to link each Extract group to a different trail.
4.
On the target, create a Replicat checkpoint table. This is a best practice. For instructions, see the GoldenGate for Windows and UNIX Administrator Guide. All Replicat groups can use the same checkpoint table. On the target, use the ADD REPLICAT command to create a Replicat group for each trail. Use the EXTTRAIL argument of that command to link the group to the trail.
ADD REPLICAT <group_1>, EXTTRAIL <remote_trail_1>, BEGIN <time> ADD REPLICAT <group_2>, EXTTRAIL <remote_trail_2>, BEGIN <time>
5.
6.
On the target, use the EDIT PARAMS command to create a Replicat parameter file for each Replicat group. In each file, use the following required parameters plus any others that are appropriatge for your database environment.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 26
Replicat group 1
REPLICAT <group_1> {SOURCEDEFS <full_pathname>} | ASSUMETARGETDEFS [TARGETDB <dsn>,] [USERID <user id>[, PASSWORD <pw>]] MAP <owner>.<table_group_1>, TARGET <owner>.<table_group_1>;
Replicat group 2
REPLICAT <group_2> {SOURCEDEFS <full_pathname>} | ASSUMETARGETDEFS [TARGETDB <dsn>,] [USERID <user id>[, PASSWORD <pw>]] MAP <owner>.<table_group_2>, TARGET <owner>.<table_group_2>;
You can use any number of MAP statements for any given Replicat group. All MAP statements for a given Replicat group must specify the same objects that are contained in the trail that is linked to the group.
7.
In the Manager parameter files on the source system and the target system, use the PURGEOLDEXTRACTS parameter to control the purging of files from the trails.
NOTE You can take this configuration a step further by dividing the rows of an individual table among multiple Extract or Replicat processes. This might be necessary, for example, when table1 receives one million inserts and updates per day, while the rest of the tables receive 100,000 or less. See Splitting large tables into row ranges across process groups.
will always be processed by the same process group. It might be more efficient to use Extract to calculate the ranges than to use Replicat. To calculate ranges, Replicat must filter through the entire trail to find data that meets the range specification. However, your business case should determine where this filtering is performed.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 27
Figure 7
Figure 8
See the GoldenGate for Windows and UNIX Administrator Guide for more information about using parallel processing groups. Also see the GoldenGate installation guide for your database to see if there are any limitations that apply to how GoldenGate can be configured in your environment. For example, verify that the database can support multiple GoldenGate processes.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 28
When you first start a new GoldenGate configuration: 1. Establish benchmarks for what you consider to be acceptable lag and throughput volume for Extract and for Replicat. Keep in mind that Extract will normally be faster than Replicat because of the kind of tasks that each one performs. Over time you will know whether the difference is normal or one that requires tuning or troubleshooting. Set a regular schedule to monitor those processes for lag and volume, as compared to the benchmarks. Look for lag that remains constant or is growing, as opposed to occasional spikes. Continuous, excess lag indicates a bottleneck somewhere in the GoldenGate configuration. It is a critical first indicator that GoldenGate needs tuning or that there is an error condition.
2.
To view volume statistics Syntax STATS {EXTRACT | REPLICAT | ER} {<group | wildcard>} [TABLE {<name | wildcard>}]
Or...
Syntax SEND {EXTRACT | REPLICAT} {<group | wildcard>}, GETLAG To detect a network bottleneck
1.
Issue the following command to view the ten most recent Extract checkpoints. If you are using a data-pump Extract on the source system, issue the command for the primary Extract and also for the data pump.
INFO EXTRACT <group>, SHOWCH 10
2.
Look for the Write Checkpoint statistic. This is the place where Extract is writing to the trail.
Write Checkpoint #1 GGS Log Trail Current Checkpoint (current write position): Sequence #: 2 RBA: 2142224 Timestamp: 2006-06-09 14:16:50.567638 Extract Trail: ./dirdat/eh
3.
Determine whether there are more than one or two checkpoints. There can be up to ten.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 29
Find the Write Checkpoint <n> heading that has the highest increment number (for example, Write Checkpoint #8) and make a note of the Sequence, RBA, and Timestamp values. This is the most recent checkpoint.
4.
Refer to the information that you noted, and make this validation:
Is the primary Extract generating a series of checkpoints, or just the initial checkpoint? If a data pump is in use, is it generating a series of checkpoints, or just one?
5.
Issue INFO EXTRACT for the primary and data pump Extract processes again.
Has the most recent write checkpoint increased? Look at the most recent Sequence, RBA, and Timestamp values to see if their values incremented forward since the previous INFO EXTRACT command.
6.
Issue the following command to view the status of the Replicat process.
SEND REPLICAT <group>, STATUS
The status indicates whether Replicat is delaying (waiting for data to process), processing data, or at the end of the trail (EOF).
7.
There is a network bottleneck if: You are only using a primary Extract and its write checkpoint is not increasing or is increasing too slowly. Because this Extract process is responsible for sending data across the network, it will eventually run out of memory to contain the backlog of extracted data and abend. Or... You are using a data pump, and its write checkpoint is not increasing, but the write checkpoint of the primary Extract is increasing. In this case, the primary Extract can write to its local trail, but the data pump cannot write to the remote trail. The data pump will abend when it runs out of memory to contain the backlog of extracted data. The primary Extract will run until it reaches the last file in the trail sequence and will abend because it cannot make a checkpoint. And... The status of Replicat is either in delay mode or at the end of the trail file. Even when there is a network outage, Replicat will process in a normal manner until it applies all of the remaining data from the trail to the target. Eventually, it will report that it reached the end of the trail file.
Use parallel data pumps to overcome any per-process bandwidth limitations of your network. Data pumps also remove TCP/IP responsibilities from the primary Extract, and their local trails provide fault tolerance. For more information, see the GoldenGate for Windows and UNIX Administrator Guide.
To reduce bandwidth requirements
Use the compression options of the RMTHOST parameter to compress data before it is sent across the network. Weigh the benefits of compression against the CPU resources that are
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 30
Use the TCPBUFSIZE option of the RMTHOST parameter to increase the size of the TCP socket buffer that Extract maintains. By increasing the size of the buffer, you can send larger packets to the target system.
A regular Extract generates disk writes to a trail and disk reads from a data source. A data pump and Replicat generate disk reads from a local trail. Each process writes a recovery checkpoint to its checkpoint file on a regular schedule.
Look for I/O waits on the disk subsystems that contain the trail files. Put the trails on the fastest disk controller possible. Check the RAID configuration. Because GoldenGate writes data sequentially, RAID 0+1 (striping and mirroring) is a better choice than RAID 5, which uses checksums that slow down I/O and are not necessary for these types of files.
To improve I/O within the GoldenGate configuration
Try increasing the values of the following parameters: Use the CHECKPOINTSECS parameter to control how often Extract and Replicat make their routine checkpoints. Use the GROUPTRANSOPS parameter to control the number of SQL operations that are contained in a Replicat transaction when operating in its normal mode. Increasing the number of operations in a Replicat transaction improves GoldenGates performance by:
Reducing the number of transactions executed by Replicat. Reducing I/O activity to the checkpoint file and the checkpoint table, if used. Replicat issues a checkpoint whenever it applies a transaction to the target, in addition to its scheduled checkpoints.
Use the EOFDELAY or EOFDELAYCSECS parameter to control how often Extract, a data pump, or Replicat checks for new data after it has reached the end of the current data in its data source. You can reduce the system I/O overhead of these reads by increasing the value of this parameter.
NOTE Increasing the values of these parameters will improve performance, but it also increases the amount of data that must be reprocessed if the process fails. This has an effect on overall latency between source and target. Some testing will help you determine the optimal balance between recovery and performance.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 31
Avoid using the primary Extract to filter and convert data. Keep it dedicated to data capture. It will perform better and is less vulnerable to any process failures that result from those activities. Use Replicat or a data-pump to perform filtering and conversion. Consider any of the following configurations:
Use a data pump on the source if the system can tolerate the overhead. This configuration works well when there is a high volume of data to be filtered, because it uses less network bandwidth. Only filtered data gets sent to the target, which also can help with security considerations. Use a data pump on an intermediate system. This configuration keeps the source and target systems free of the overhead, but uses more network bandwidth because unfiltered data is sent from the source. Use a data pump or Replicat on the target if the system can tolerate the overhead, and if there is adequate network bandwidth for sending large amounts of unfiltered data.
Consider using parallel data pumps or parallel Replicat processes to balance the load of filtering and conversion. Parallel processes work well when sending data to different target systems, because you can dedicate one or more pumps to each target. Keep tables that have relational constraints to each other in the same process group. If you have limited system resources, a least-best option is to divide the filtering and conversion work between Extract and Replicat.
For more information about configuring processes, see the GoldenGate for Windows and UNIX Administrator Guide.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 32
to use as a key. The columns must contain unique values. On the target system, create an index on those columns.
TABLE <source_table>, KEYCOLS (<column> [, <column>, <column>, ...]);
and...
MAP <source_table>, TARGET <target_table>, KEYCOLS (<column> [, <column>, <column>, ...]); Example TABLE hr.emp, KEYCOLS (FIRST_NAME, LAST_NAME, DOB, ID_NO); MAP hr.emp, TARGET hr.emp2, KEYCOLS (FIRST_NAME, LAST_NAME, DOB, ID_NO);
The Replicat process maintains cursors for cached SQL statements and for SQLEXEC operations. Without enough cursors, Replicat must age more statements. By default, Replicat maintains as many cursors as allowed by the MAXSQLSTATEMENTS parameter. You might find that the value of this parameter needs to be increased. If so, you might also need to adjust the maximum number of open cursors that are permitted by the database.
Improve update speed
Excessive block fragmentation causes Replicat to apply SQL statements at a slower than normal speed. Reorganize heavily fragmented tables, and then stop and start Replicat to register the new object ID.
Ensure effective execution plans
Stale database statistics can reduce the performance of Replicat as it adds rows to tables. Generate new statistics to ensure the use of appropriate execution plans.
Set a Replicat transaction timeout
Use the TRANSACTIONTIMEOUT parameter to prevent an uncommitted Replicat target transaction from holding locks on the target database and consuming its resources unnecessarily. You can change the value of this parameter so that Replicat can work within existing application timeouts and other database requirements on the target.
TRANSACTIONTIMEOUT limits the amount of time that Replicat will hold a target transaction
open if it has not received the end-of-transaction record for the last source transaction in that transaction. By default, Replicat groups multiple source transactions into one target transaction to improve performance, but it will not commit a partial source transaction and will wait indefinitely for that last record. The Replicat parameter GROUPTRANSOPS controls
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 33
the minimum size of a grouped target transaction. The following events could last long enough to trigger TRANSACTIONTIMEOUT:
Network problems prevent trail data from being delivered to the target system. Running out of disk space on any system, preventing trail data from being written. Collector abends (a rare event). Extract abends or is terminated in the middle of writing records for a transaction. An Extract data pump abends or is terminated. There is a source system failure, such as a power outage or system crash.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 34
CHAPTER 3
Troubleshooting GoldenGate
...............
Overview of GoldenGate troubleshooting
One of the challenges of solving problems in a replication environment is in deciding which component of the environment is at the root of the problem:
the system or network? the database or applications? the GoldenGate installation? a specific GoldenGate process? the way that GoldenGate is configured? SQL or procedures?
The GoldenGate troubleshooting documentation provides help with many common issues that can arise during GoldenGate startup and the processing of data. You can find help for other known issues and frequently asked questions by going to the GoldenGate Technical Support website at http://support.goldengate.com/ and viewing the Knowledge Base. Troubleshooting tools described in this chapter are documented in Chapter 1.
Important: The more invasive of the troubleshooting solutions, such as those affecting data
records and processing continuity, should be performed by individuals who have authority to resolve problems and who understand the GoldenGate components, configuration structure, and processing methodology. Less experienced users should undertake such solutions in conjunction with GoldenGate Technical Support staff or a more experienced user.
Troubleshooting example
Here is a simple example of how you can use GGSCI commands and the error log to determine why a process (in this case Extract group ggext) failed. 1. The STATUS EXTRACT command returns the following:
GGSCI (sysa) 20> status extract ggext EXTRACT GGEXT: ABENDED
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 35
Troubleshooting GoldenGate
Configuration problems that cause startup failure
2.
Viewing the error log with VIEW GGSEVT shows the following:
GGSCI (sysa) 22> view ggsevt 2006-04-27 16:38:14 GGS INFO 399 GoldenGate Command Interpreter for Oracle: GGSCI command (admin): start extract ggext. 2006-04-27 16:38:15 GGS INFO 301 GoldenGate Manager for Oracle, mgr.prm: Command received from GGSCI on host 100.000.000.000 (START EXTRACT GGEXT). 2006-04-27 16:38:15 GGS INFO 310 GoldenGate Capture for Oracle, ggext.prm: EXTRACT GGEXT starting. 2006-04-27 16:38:17 GGS ERROR 501 GoldenGate Capture for Oracle, ggext.prm: Extract read, error 13 (Permission denied) opening redo log /newrdbms/oracle/ora920/arch/0001_0000000758.arc for sequence 758. 2006-04-27 16:38:17 GGS ERROR 190 GoldenGate Capture for Oracle, ggext.prm: PROCESS ABENDING.
The error message 501 indicates the Extract user does not have permission to read the redo logs. Solution: Contact the system administrator to grant the user running the extract program read permission on those files. Then do the following: 1. 2. 3. 4. Stop Manager. Exit GGSCI. Exit the terminal session. Start the processes again from a new session.
GoldenGate provides a unique build of the software for each type and version of the operating systems and databases that it supports. The build name contains the operating system version, database version, GoldenGate release number, and GoldenGate build number, as shown in the following example for Oracle on AIX:
aix51_ora92_v8020_023.tar.gz aix51_ora817_v8020_023.tar.gz aix52_ora102_v8020_023.tar.gz aix53_ora92_v8020_023.tar.gz aix53_ora101_v8020_023.tar.gz aix53_ora102_v8020_023.tar.gz
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 36
Troubleshooting GoldenGate
Configuration problems that cause startup failure
To find out the GoldenGate build and version that is installed, go to the GoldenGate home directory and issue the following command from the operating system:
ggsci -v
Did you use the CHECKPARAMS parameter and forget to remove it?
This parameter causes the process to stop after it verifies parameter syntax. View the process report for syntax errors, fix them, and then remove CHECKPARAMS.
Does the process group that you are trying to start exist?
Use INFO ALL to view all processes and groups on the system. Maybe there was a spelling error when the group was created or when the START command was issued.
If not, Extract cannot write its initial checkpoint, and Replicat has no data source to read. Use INFO EXTRACT <group> or INFO REPLICAT <group> with the DETAIL option to see if a trail or file exists.
Check for the GLOBALS file (if being used) in the root GoldenGate installation directory. It should not have a file extension. Check for the Manager parameter file and any other parameter files in the GoldenGate dirprm sub-directory. The Manager file must be named MGR.prm, and all other parameter files should have the name of the associated process group. If a parameter file is not there, did you store it elsewhere? Use the INFO EXTRACT <group>, DETAIL command to see where else it might be. To store a parameter file elsewhere (not recommended), use the PARAMS argument with ADD EXTRACT or ALTER EXTRACT.
Check the permissions for the system user assigned to the process. Grant read permission, if needed.
Requirements vary by database and configuration. Review the requirements for your database in the GoldenGate for Windows and UNIX Administrator Guide and in the GoldenGate installation guide that is specific to the database type.
GoldenGate reports syntax problems in the process report, usually as a bad parameter error.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 37
Troubleshooting GoldenGate
Operating system errors that cause startup failure
Use the CHECKPARAMS parameter to verify syntax. It verifies syntax when the process starts, writes results to the report file, and then stops the process. After using CHECKPARAMS, remove it or comment it out; otherwise the process will stop again. Common syntax errors:
Comma not followed by a space. Missing commas, quotes, or parentheses in nested clauses, such as a COLMAP.
See the GoldenGate for Windows and UNIX Reference Guide for complete parameter syntax information.
Parameters are processed in the order listed; some must precede others. Parameter ordering examples:
RMTHOST must precede RMTTRAIL: A host name must be qualified before a file name can
be qualified.
SOURCEDB or TARGETDB and USERID must precede TABLE: GoldenGate must log into the
Issue the env command and make sure LD_LIBRARY_PATH and PATH are correct. Set the paths correctly in the .profile startup file in the GoldenGate home directory.
The memory that stores arguments supplied to and from GoldenGate functions has been exceeded. To increase the memory allocated, use the FUNCTIONSTACKSIZE parameter. This parameter can adversely affect GoldenGates performance.
read and write for files in the GoldenGate directory. read for the transaction and archive logs.
The process cannot open the checkpoint file. Issue INFO * in GGSCI to view the group name, then issue VIEW PARAMS <group>. Make sure the group name from INFO * matches the one in the EXTRACT or REPLICAT parameter. Check the file permissions.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 38
Troubleshooting GoldenGate
Database errors that cause statup failure
On NonStop systems, look for access problems related to Safeguard and Enform reports.
Is the database running? Was a database user created for the process and does it have the correct permissions?
Extract and Replicat require a database user with certain permissions. See the GoldenGate installation guide that is specific for your database type.
If a data source name is required as part of the login, is it specified with SOURCEDB or TARGETDB? If database authentication is required, are the correct database user and password specified with the USERID and PASSWORD parameters?
Connect to the database through its native interface by using the same user and password that is contained in the parameter file of the failed process. The login that a process uses is specified with the USERID and PASSWORD parameters. Connecting from outside GoldenGate helps to identify whether the authentication itself is valid. If you cannot connect from outside GoldenGate, check the database configuration to make certain the user and password are still valid. Perhaps a change was made to the user or password. After you identify the correct login, update the USERID and PASSWORD parameters in the parameter file by using the EDIT PARAMS command. Stop and start the process when you are finished. (Oracle) If database passwords change frequently, you can specify operating-system authentication with the / option of the USERID parameter.
(Oracle) Are the ORACLE_SID and ORACLE_HOME system variables set to the correct instance name in the GoldenGate user profile?
This filters out all the other information that would be shown if you were to use the basic env command.
Make certain that the WILDCARDRESOLVE parameter is set to the default of DYNAMIC, so that tables satisfying wildcard definitions are resolved whenever the wildcard rule is satisfied, instead of all at once at startup.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 39
Troubleshooting GoldenGate
Extract freezes during recovery
Did you create the group with the BEGIN NOW option of the ADD EXTRACT command? If an Extract is created that way and also configured for ALO mode, the Extract might not capture data immediately upon startup. The start time of NOW corresponds to the time of the current online redo log, but an ALO Extract cannot read the online logs, so it waits for that log to be archived when Oracle switches logs. The timing of the switch depends on the size of the redo logs and the volume of database activity. This can happen in both regular and RAC database configurations. On RAC systems this effect is compounded by the fact that Extract needs to read data from all RAC instances before it can proceed. Extract will wait for Oracle to archive all of the online logs that correspond to the NOW time. To force capture to begin sooner, you can switch the logs manually with the following command:
ALTER SYSTEM SWITCH LOGFILE
In recovery[1] Extract is recovering to its input checkpoint. In recovery[2] Extract is recovering to its output checkpoint. Recovery complete The recovery is finished, and normal processing will resume.
Is tracing enabled?
Check the parameter file for TLTRACE and TRACE. Tracing slows down Extract. Disable those parameters unless required by a support case.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 40
Troubleshooting GoldenGate
Extract is too slow
See To detect a network bottleneck on page 29 for instructions that help you detect whether a network problem is affecting GoldenGate. Work with your network administrator to resolve network problems as soon as possible. If Extract lags too far behind the speed at which transaction logs are created, latency on the target will increase to an intolerable level, and Extract could eventually lose access to data that it still needs if the archives are moved off the system. Major network delays can also cause eventual process failure. If the network is functional, check to see if the per-process bandwidth has been used up for the Extract process. You might need to split the data load among some data pumps to gain more bandwidth. The use of data pumps also reduces the chance of process failures and data loss if the network fails intermittently. To configure data pumps, see the GoldenGate for Windows and UNIX Administrator Guide.
The more columns that are changed in a table, the longer it takes for Extract to process them. This is normal and cannot be avoided. You can isolate tables like this that cause slowdowns by using parallel processes. See page 22 for more information.
Extract waits if it cannot find the transaction log that it needs, which means it probably was archived off the system or was moved to a location that is unknown to GoldenGate. This can happen if there is high Extract lag and the archives are moved off the system, or if there are long-running open transactions, because Extract keeps a pointer to that log in the expectation of eventually receiving a commit operation.
NOTE This could take hours or days of searching, and Extract will eventually abend if it cannot locate the log.
To determine which log is needed, use the VIEW REPORT <group> command. Restore that log to the directory where it was before, and also restore subsequent ones until Extract is finished with them. (Oracle) If Oracle archives are not stored in Oracles default location, use the Extract parameter ALTARCHIVELOGDEST <path name> to specify the alternate location. (Oracle) You can check for and manage open transactions against Oracle tables from within GoldenGate by using options of the SEND EXTRACT command. You can use the WARNLONGTRANS parameter to set a threshold after which a transaction is considered long-running. To check for open transactions from the database interface, query the V$Transaction table. If the transaction logs that GoldenGate requires are no longer available, resynchronize the source and target data. See Resynchronizing an out-of-sync table on page 53.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 41
Troubleshooting GoldenGate
Extract stalls or pauses
Is this an Oracle RAC system, and does Extract pause across all RAC nodes?
On RAC, if an Extract thread reaches the end of a nodes last archived log, and the next log is not immediately available, the entire Extract process will stop, even though other Extract threads have archive logs available. This is to maintain transactional integrity. ALO mode does not provide enough information for Extract to determine if there is other, older transaction data somewhere that must be processed before it processes the data captured by the other threads.
Is this an Oracle RAC system, and did one of the instances fail?
If one or more of the RAC instances fails, archive log generation will stop for those instances. In this situation, an Extract operating in ALO mode will stop. To allow Extract to continue, set the THREADOPTIONS paramter with the PROCESSTHREADS {SELECT | EXCEPT} option to skip processing on the failed nodes. This is an undocumented parameter. Please contact GoldenGate Technical Support before using this parameter.
Extract abends
Check to see if the disk is full. Verify that the Extract user has read permissions on the log files. If you have to grant permissions, do the following afterward: 1. Stop Manager. 2. Exit GGSCI. 3. Close the terminal session. 4. Start processes again from a new session.
Extract reads the archived logs when an online log is not available. If you started Extract after it was stopped for a long time, the records it needs to process probably were archived away. Restore the archive log that is specified in the error message in the report file, and also restore all subsequent logs.
NOTE Extract should not be stopped for a long time. If you must stop Extract, make certain that archives are available all the way back to the time that you stopped Extract. If Extract remains stopped for an exceptionally long time, it might be more practical to resynchronize the table(s) rather than wait for Extract to go through the archives and catch up.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 42
Troubleshooting GoldenGate
Extract abends
A typical TCP/IP error is 4127 connection refused. It indicates that the target Manager or Collector process is not running, or that Extract is pointing to the wrong TCP/IP address or Manager port number. On the target, use INFO MGR in GGSCI to view the port number that is specified for the target Manager. On the source, check the Extract parameter RMTHOST for the following: Verify that MGRPORT is the same as the port shown by INFO MGR. If a host name was given for RMTHOST, make sure the domain name server (DNS) can resolve it. If an IP address was used, verify that it is correct. To verify an IP address, use the following commands from the OS command shell: UNIX:
ifconfig
Windows:
ipconfig
Connection errors can indicate Collector security violations, a full file system, or errors relating to the system or to the GoldenGate configuration. Use VIEW GGSEVT to check for those errors.
The target Manager was unable to obtain a port on which to communicate with the source Manager. Manager looks for a port in the list specified with the DYNAMICPORTLIST parameter in its parameter file or, if that parameter is not in use, the next available port higher than the one it is running on, as specified with the PORT parameter. If you are using DYNAMICPORTLIST, it is possible that there are not enough numbers to use, or that the numbers are being used by an active or orphan process. Try adding some port numbers to the list. Or, contact your systems administrator to find out which ports can be used. If you are using GoldenGate Director to connect to the target server, you will need to increase the number of ports in your dynamic port list, because Director uses ports from the dynamic port list.
If you are using the PASSTHRU parameter for the data pump, but are trying to use data filtering, conversion, or other manipulation, this configuration will not work. In passthrough mode, the source and target table names and structures must be identical, and no filtering or manipulation can be done to the data. Does the parameter file that includes PASSTHRU also include USERID or SOURCEDB parameters? If the system does not have a database, these parameters should not be used.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 43
Troubleshooting GoldenGate
Extract abends
Are you trying to combine pass-through for some tables with normal processing for other tables? For the tables being processed normally, the system must have a database. The database login parameters of SOURCEDB and USERID must be used, and a source definitions file must be used if filtering is to be performed. A target definitions file must be used if column mapping or conversion is to be done.
Is the error for a Sybase source that states Failed to reserve LTM context?
Check to see what other processes are using the LTM. Extract uses the LTM to read the transaction log, and only one process at a time can reserve a context. Therefore, no other Extract or RepServer processes can be run against the same database. Errors similar to the following in the error log indicate that some other process is running.
2005-01-18 22:54:01 GGS INFO 631 SYBLTM - Server message:. 2005-01-18 22:54:01 GGS INFO 631 number(9121) severity(16) state(2) line(1). 2005-01-18 22:54:01 GGS INFO 631 Server name: SysA. 2005-01-18 22:54:01 GGS INFO 631 The log transfer context for the current database is already reserved by SQL Server process 16. The log transfer context for the current database is not reserved. 2005-01-18 22:54:01 GGS ERROR 630 Failed to reserve LTM context during log reader initialization. 2005-01-18 22:54:01 GGS ERROR 190 PROCESS ABENDING.
Is the database configured to retain the logs? Either USEREXIT or LOGRETAIN must be enabled. Is the DB2 parameter OVERFLOWLOGPATH set to the archive log directory? This enables GoldenGate to get archived data stored in a non-default location. Are you using wildcards to specify tables? Check to see if DATA CAPTURE CHANGES was specified for all of the tables that satisfy the wildcard conditions. If not, use the TRANLOGOPTIONS parameter with the IGNOREDATACAPTURECHANGES option. This configuration directs Extract to ignore tables for which change capture is not enabled. A warning will be issued to the error log that those tables were skipped. Have columns been added with the ALTER command to a table that is already in the DB2 LUW configuration? And, have any of those columns been defined as key columns? If so, check to see if there were updates on those columns. A primary key update requires a before image for GoldenGate to compare to the after image, but the DB2 API used by GoldenGate does not provide the before image for the added columns. This will cause Extract to abend. Columns added to existing tables cannot be part of a key in the data definitions or as part of a KEYCOLS clause in the GoldenGate parameters TABLE or MAP. Is the error similar to the following example? If so, Extract does not have the correct permissions to read the catalog tables and log files.
[SC=-1224:SQL1224N A database agent could not be started to service a request, or was terminated as a result of a database system shutdown or a force command. SQL STATE 55032: The CONNECT statement is invalid, because the database manager was stopped after this application was started].
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 44
Troubleshooting GoldenGate
Replicat is too slow
Are the correct permissions set? The API (GoldenGate name of IFI IFCID 306) requires Extract to call the API in Supervisor State with Storage Key 0 and allocate a return area in Extended Common Storage Area (ECSA) memory subpool 241 with Storage Key 7. For more information about required privileges, see the installation guide for the DB2 for z/OS database.
Does Extract return an error on LOB length when extracting from a SQL Server 2005 source?
Msg 7139, Level 16, State 1, Line 1 Length of LOB data (<n_bytes>) to be replicated exceeds configured maximum 65536. The statement has been terminated.
This means that the data for LOB or VARCHAR (MAX) columns exceeds the SQL Server default size. To resolve this issue, extend the maximum data size with the max text repl size parameter by using sp_configure. For example:
max text repl size, 2147483647
Use sp_configure to find the current default value of max text repl size.
Is the DEFERAPPLYINTERVAL being used in the Replicat parameter file? To find out, you can examine the parameter file or use the following GGSCI command and look for a status of Waiting on deferred apply.
SEND REPLICAT <group> STATUS
This parameter causes an intentional, specific delay in the timing of Replicats transactions. Replicat will always wait for the specified amount of time after reading data from the trail, before applying that data to the target table. If your target applications do not require this delay, you can remove DEFERAPPLYINTERVAL. You must stop Replicat and then start it again for the change to take effect.
Issue the following command in GGSCI a few times to see whether or not the checkpoint is moving.
INFO REPLICAT <group>, SHOWCH
Look for the Read checkpoint. If the number of Read checkpoints is not increasing, Replicat is stalled.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 45
Troubleshooting GoldenGate
Replicat abends
Do the following: 1. Note the trail name that Replicat is reading from, as shown with INFO REPLICAT. 2. Issue INFO EXTRACT <group>. Note the trail name that Extract is writing to. 3. Make certain that Replicat is reading the same trail that Extract is writing to. If not, issue ALTER REPLICAT <group>, EXTTRAIL <trail> to specify the correct trail. 4. Issue INFO RMTTRAIL * (or INFO EXTTRAIL * if local) and make sure the trail was created. Create the trail, if necessary, with the following command:
ADD {EXTTRAIL | RMTTRAIL} <trail>, EXTRACT <group>.
5. Make certain that the RMTTRAIL or EXTTRAIL Extract parameter is also set to the correct trail name.
If the checkpoint is moving, Replicat is processing data, and other issues are causing the slowdown. See Tuning GoldenGates Performance on page 21 for conditions that might be affecting Replicats performance.
Problems with the network, loss of disk space, process failures, and system failures all can interrupt the transfer of source operations to the trail. If Replicat cannot commit its current transaction because it is waiting for the rest of the last source transaction in the transaction group to arrive, the lock will be held indefinitely. If you continue to see that Replicat is holding locks, and you cannot resolve the reasons that are delaying source transactions, consider using the Replicat TRANSACTIONTIMEOUT parameter to allow Replicat to handle the condition.
Replicat abends
Or...
2006-09-25 13:22:16 GGS ERROR 516 Extract read, Checkpoint table HR.REPCHECK does not exist. Please create the table or recreate the REPORA group using the correct table.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 46
Troubleshooting GoldenGate
Replicat abends
The checkpoint table may be corrupted, deleted, or missing a row. Do the following: 1. Issue INFO REPLICAT <group> to verify that the checkpoint file on disk is intact. If the command fails, the file is missing or corrupt. If the file is intact, continue with these steps. 2. If the table was deleted, recreate it with same schema and name as before. If the table is corrupt or missing a row, truncate it. 3. From the GoldenGate installation directory, run the convchk utility, specifying the checkpoint table as shown:
convchk <group name> <schema>.<table>
4. Start Replicat. 5. SELECT from the checkpoint table to verify that checkpoints are being added again.
NOTE Running convchk causes the checkpoint file to become the master source for the groups checkpoints. When started, Replicat will populate the new or truncated checkpoint table with the data from the checkpoint file, in effect resynchronizing them.
Do the following: 1. Issue the following command, and record the trail name to which Extract is writing.
INFO EXTRACT <group>
4. Issue the following command to link that trail to the Replicat process.
ALTER REPLICAT <group>, EXTTRAIL <trail>
5. Make certain that the RMTTRAIL or EXTTRAIL Extract parameter is also set to the correct trail name.
Verify that the Replicat user has read and write permissions on trail files.
If you specified ENCRYPTTRAIL for Extract, make certain that DECRYPTTRAIL is specified for Replicat. Otherwise, Replicat will return GoldenGate error 160. The encrypted data will be interpreted as a bad record.
If the target cannot handle large transactions, Replicat will abend with a memory error.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 47
Troubleshooting GoldenGate
Replicat abends
Try using the MAXTRANSOPS parameter to split Replicats transactions into smaller ones.
For example, were columns added to a source table? If so, did you create a new source-definitions file for that table? Use DEFGEN and then transfer the new file to the target.
Are you moving a LOB to a Sybase CHAR, VARCHAR, BINARY, or VARBINARY column? If so, the size of the LOB must be smaller than, or equal to, the size of the target column. You can set Replicat to truncate the LOB and post it without error by using the DBOPTIONS parameter with the ALLOWLOBDATATRUNCATE option in the Replicat parameter file. Are you replicating empty source LOB columns to a Sybase target? Replicat converts empty LOB columns to NULL, but Sybase does not permit null LOB columns. Use the DBOPTIONS parameter with the EMPTYLOBSTRING option to substitute a string value in place of NULL so that the operation is accepted.
Either the data is corrupted or written in a format other than the GoldenGate internal format. The following are the recommended steps to take.
First step Did you recently run an initial load? See if FORMATASCII was used for Extract during the load. Other Extract parameters to look for are FORMATSQL and FORMATXML. All of these parameters write data in a format that is incompatible with Replicat. Next step Is Extract configured to write over a trail that was used by an old process? Or, are two Extract processes configured to write to the same trail? Possible causes of these problems are:
A previously active group was dropped and recreated to link to the same trail name as before. This causes the new Extract to start writing to the beginning of the first file in the trail again, overwriting data that was written by the old group. The parameter file for one Extract group was copied as the basis for the second group, but trail names (and possibly other names) were not changed for the second group. One group is writing over the others data in the same file.
What happens to cause the Replicat error is this: A complete record is followed by an incomplete record that was overlayed by a different write. Instead of the incomplete record starting with the beginning of a header, as Replicat expects, it starts somewhere in the middle.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 48
Troubleshooting GoldenGate
Replicat abends
1. 2.
Issue OPEN <path name> to open the sequence number shown in the error, for example OPEN C:\GOLDENGATE802\DIRDAT\JD000000. Referring to the RBA in the error, use the POS command to look at the timestamps of records before and after that position. If later records have earlier timestamps, those records overwrote existing ones in the trail. The best solution is to start over, because data was probably lost or corrupted.
1. 2. 3. 4. 5.
Move the old trail files out to a temporary location, in case you need them later, or in case a GoldenGate support engineer needs to see them. Drop and re-create the Extract groups. Create a new trail for each Extract group, and give each a unique name. Create new Replicat groups. Resynchronize the data. If the source data must remain active while you are resynchronizing the data, refer to the instructions for performing an initial load in the GoldenGate for Windows and UNIX Administrator Guide.
NOTE When a trail is corrupted this way, a debug509.txt file is created in the GoldenGate installation directory. Contact GoldenGate Technical Support and send them the file, or see GoldenGate Knowledge Base public solution 4897 for help with resolving this yourself.
Next step
Did a data-pump Extract abend? If a data pump abends before it can update its checkpoint file, it processes data from the previous checkpoint when it starts again. This causes it to write the same records over again in the trail. However, those records probably will not match the previous records exactly, and the file will get corrupted. If this is the case, Replicat is probably stuck in the middle of an incomplete or corrupted record.
To reposition Replicat in the trail
1.
Using Logdumps POS and SCANFORHEADER commands, try to determine the time and RBA when Extract started to overwrite the data. Usually a time gap will reveal this position. If you cannot find the gap, position Replicat to the RBA of a record that has a commit timestamp of a few minutes before the timestamp of the record nearest to the RBA where Replicat abended. To do this, issue the following command in GGSCI.
ALTER REPLICAT <group>, EXTSEQNO <number>, EXTRBA <RBA>
2.
3.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 49
Troubleshooting GoldenGate
Replicat abends
4.
Run Replicat until it is past the RBA where it abended. Use the INFO REPLICAT command to view the current RBA.
INFO REPLICAT <group>
5.
If there is a record in the trail that you want to skip, use the following procedure. 1. Open the Replicat parameter file by issuing the following command in GGSCI.
EDIT PARAMS <group>
2. 3. 4. 5.
In the file, add the RESTARTCOLLISIONS parameter. This parameter handles duplicate or missing operations. In the file, add the GROUPTRANSOPS or MAXTRANSOPS parameter with a value of 1. This sets Replicat to commit after one operation. Save and close the parameter file. Stop and start Replicat and let it fail on that transaction.
STOP REPLICAT <group> START REPLICAT <group>
6. 7.
Change directories to the GoldenGate directory, and run Logdump. Open the trail file that contains the record.
OPEN <file>
8.
9.
Use the Logdump SCANFORHEADER command to determine the RBA of the next good record.
SCANFORHEADER
11. Edit the Replicat parameter file and comment out, or remove, GROUPTRANSOPS or MAXTRANSOPS.
EDIT PARAMS <group>
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 50
Troubleshooting GoldenGate
Cannot delete a Replicat group
For more information, see GoldenGate Knowledge Base public solution 4897.
To prevent Replicat from abending on errors
You can specify error-handling rules with the REPERROR parameter. You can use one REPERROR statement to handle most errors in a default way, while using one or more other REPERROR statements to handle specific errors differently. For example, Replicat can ignore duplicate-record errors but abend in other cases. REPERROR can be used as a global parameter or within a MAP statement. For more information, see the GoldenGate for Windows and UNIX Reference Guide.
Are you trying to delete a Replicat group but the DELETE REPLICAT command fails?
Does the error state checkpoint table does not exist? Either the group was created without the NODBCHECKPOINT argument, or the checkpoint table was deleted. In either case, use the ! argument with DELETE REPLICAT. The ! tells GoldenGate to ignore the checkpoint table. GoldenGate will generate a warning that the database checkpoint could not be deleted, but the group will be removed successfully. The syntax is:
DELETE REPLICAT <group> !
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 51
Troubleshooting GoldenGate
Causes of out-of-sync data
DML on target tables (other than replication) tables without keys character sets differences between source and target structure or primary keys
Triggered operations are replicated from the source. If target triggers reproduce those same operations, they cause errors. Disable triggers, alter them to ignore the Replicat user, or use SQLEXEC to disable them on demand by issuing the appropriate database command.
NOTE Triggers affecting tables outside the GoldenGate replication environment are permissible.
Are there target tables that lack a primary key or unique constraints?
Without a primary key or unique constraints, GoldenGate uses all of the columns as a key. Multiple rows could be updated if they contain identical data. Use the KEYCOLS option of TABLE and MAP, or use the DBOPTIONS parameter with the LIMITROWS option to prevent multiple rows from being updated (supported databases only).
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 52
Troubleshooting GoldenGate
Discrepancies caused by non-matching structures or keys
(Oracle) Check the NLS_LANG environment variable on the source and target databases. The source NLS_LANG must be set to the character set of the target, and the target character set must be a superset of the source.
If a source structure changes, including changes in a key definition, you must run DEFGEN to create a new source-definitions file. Otherwise, the same structure changes must be applied to the target. If source and target key columns are different, use the KEYCOLS option of TABLE and MAP to define other columns as keys.
The original group, known as groupA A new group that you will create, known as groupB
To resynchronize a table
1.
Comment out the out-of-sync table in the original Replicat parameter file.
EDIT PARAMS <groupA>
2.
Stop the original Replicat and start it again so it continues processing the tables that are not out-of-sync.
STOP REPLICAT <groupA> START REPLICAT <groupA>
3. 4.
Write down the timestamp on the source system. Try to resolve any long-running transactions on the source out-of-sync table. You will be copying the source data to the target, and if transactions are open while the copy is being made, they will not be captured. Start a copy of the source data. Import the copy to the target table. Create a new Replicat group for the out-of-sync table, using BEGIN to start at the source timestamp that you recorded step 3, and using the existing trail for EXTTRAIL.
ADD REPLICAT <groupB>, EXTTRAIL <existing trail>, BEGIN <yyyy-mm-dd hh:mm:ss>
5. 6. 7.
8.
Create a parameter file for the new group and include HANDLECOLLISIONS.
EDIT PARAMS <groupB>
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 53
Troubleshooting GoldenGate
Resynchronizing an out-of-sync table
9.
10. View the lag of the new group until it shows At EOF, no more records to process.
SEND REPLICAT <groupB>, GETLAG
12. Edit the parameter file of the new group to comment out or remove HANDLECOLLISIONS.
EDIT PARAMS <groupB>
The next steps merge the table back with the others so that only one Replicat group is needed again. 14. Stop Extract.
STOP EXTRACT <extract_group>
15. View the lag of each Replicat group until it shows At EOF, no more records to process.
SEND REPLICAT <groupA>, GETLAG SEND REPLICAT <groupB>, GETLAG
17. Uncomment the newly synchronized table in the original Replicat parameter file.
EDIT PARAMS <groupA>
Troubleshooting data-mapping
Because table and column mapping can be complex, it is a common cause of processing
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 54
Troubleshooting GoldenGate
Table and column names
The process report shows where a data-mapping error occurred. Issue VIEW REPORT <group> or use GoldenGate Director to view the report.
Do the source and target tables exist in the databases? Are they specified correctly in the TABLE or MAP statement?
(Sybase and SQL Server) Check spelling and case. (Oracle) Make sure names are fully qualified (schema.table).
Do table and column names contain only supported characters? Are any column names within quotes? (They will be interpreted as literals).
See the TABLE and MAP documentation in the GoldenGate for Windows and UNIX Reference Guide for more details about requirements for table and column names.
Definitions
Was a source definitions file created and placed on the target system? Is the file referenced correctly with the Replicat parameter SOURCEDEFS? Are source columns mapped to target columns of a similar data type?
If types differ, use a column-conversion function in the mapping statement to prevent mismatched data type errors.
GoldenGate truncates or rounds target data if the target column is smaller than the source column.
Column-missing error
(Oracle and SQL Server only) Are the specified columns being logged to the transaction log? Enable logging of the columns. For an Oracle table, you can use ADD TRANDATA with the COLS option. For SQL Server, use the ADD TRANDATA command to enable the additional log information that GoldenGate needs to reconstruct update operations.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 55
Troubleshooting GoldenGate
Fetch failures on Oracle source
Are you using KEYCOLS for both the source and target tables?
Did you include the columns of an existing primary key or unique index in the KEYCOLS definition?
This indicates that data is missing from the transaction record. Try one of the following:
Filter on key columns Enable logging of required columns Use FETCHCOLS or FETCHCOLSEXCEPT in the TABLE statement
For example, FILTER takes a numeric value, and WHERE takes a string but not arithmetic operators or floating-point numbers.
Was the source row that contained the data deleted? (Oracle 9i and later) Has the undo retention expired?
If so, then the read-consistent image that Extract needs is gone, and a snapshot-too-old message will be returned. Add the FETCHOPTIONS parameter with the NOUSESNAPSHOT option to Extract. This parameter tells Extract to fetch from the table, not the undo segments. Once GoldenGate is past the affected record, remove the parameter so that Extract resumes fetching from the undo segments. If possible, increase the amount of time that Oracle keeps information in the undo segments.
NOTE Certain DDL operations that alter the structure of a table, such as a table truncate, can cause fetch errors too.
If so, GoldenGate cannot default to fetching from the table if the undo record is not available. Unless error handling options are set with that parameter, Extract abends. If fetched data is available but cannot be merged with the redo data, then Extract abends.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 56
Troubleshooting GoldenGate
Replicat errors on Sybase targets
TIMESTAMP columns must be excluded from GoldenGate extraction. These data types are populated automatically by the database, which generates errors if Replicat attempts to post a timestamp value that was replicated from the source record. To exclude timestamp columns from being captured by GoldenGate, use the COLSEXCEPT option of the TABLE parameter. Note: Because the system generates these data types, the source and target values will be different.
Are you using an octal or hexadecimal escape sequence to represent native encoding or Unicode characters in a parameter file?
An octal sequence must contain exactly three digits. A hexadecimal sequence must contain exactly two digits. Any more than the expected number of digits causes the sequence to be interpreted as multiple characters. For more information about using escape sequences, see TABLE or MAP in the GoldenGate for Windows and UNIX Reference Guide.
Troubleshooting SQLEXEC
Does the Extract or Replicat user have permissions to call and execute database procedures and commands? Are you trying to use input or output parameters for a SQLEXEC statement at the root of a parameter file?
Parameters can only be used in SQLEXEC statements within a TABLE or MAP statement.
(Oracle and SQL Server) Is the referenced column being logged to the transaction log? Enable logging with the ADD TRANDATA command. For an Oracle table, you also can use ADD TRANDATA with the COLS option. Is the stored procedure supposed to execute more than once, for example to specify the same stored procedure for two different column maps? In that case, use the ID <logical name> option of SQLEXEC to specify a logical name for each execution, similar to that shown in bold face here:
TABLE sales.srctab, TARGET sales.targtab, SQLEXEC (SPNAME lookup, ID lookup1, PARAMS (param1 = srccol)), COLMAP (targcol1 = lookup1.param2), SQLEXEC (SPNAME lookup, ID lookup2, PARAMS (param1 = srccol)), COLMAP, (targcol2 = lookup2.param2);
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 57
Troubleshooting GoldenGate
Troubleshooting missed transactions
Is the stored procedure supposed to synchronize a source row with multiple target rows? Set the EXEC option of SQLEXEC to SOURCEROW so that it executes for each sourcerow operation. To prevent a process from abending on SQLEXEC errors, specify error handling with the ERROR option.
Use the TRACE option of SQLEXEC. This option has two modes:
ALL writes all input and output parameters to the discard file. This is the default. ERROR writes the input and output parameters to the discard file only after a SQL error occurs.
The discard file shows the parameters that are being passed, and the report file shows the number of attempts that were made to pass them. Sample parameter file that includes tracing:
MAP MASTER.INTERACTION_ATTR_VALUES , & TARGET MASTER.INTERACTION_ATTR_VALUES, & SQLEXEC ( SPNAME ggs_conflict_rule, ID ggs_interaction_attr_values, & PARAMS ( LMS_TABLE = "INTERACTION_ATTR_VALUES", & KEY1 = interaction_attribute_value_id, & KEY2 = attribute_value_seq), & ERROR RAISE ,TRACE ALL ) ;
Verify that the tables are included in TABLE and MAP statements. Check syntax: If fully qualified names (owner.table) are used for TABLE, then fully qualified names must be used for MAP. The name in TABLE, an Extract parameter, is written to the trail record header, and Replicat expects an exact match in the source 58
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide
Troubleshooting GoldenGate
Troubleshooting missed transactions
portion of the Replicat MAP statement. Otherwise, nothing is replicated. Qualifying a table name by including the owner is recommended for GoldenGate versions 8.0 and later, and is required for Oracle tables regardless of the GoldenGate version.
Did you recently run an initial load and use the same parameter file for your load and online groups?
Check the parameters. After a load, some parameters from the load configuration must be altered to suit the online configuration. A common mistake: Using RMTFILE for the load, but forgetting to change it to RMTTRAIL for the online group.
1. 2. 3. 4.
Stop Extract. Edit the parameter file to fix or add parameters. Add the remote trail with ADD RMTTRAIL in GGSCI, if you have not done so already. Start Extract.
(Oracle 8i) For LOBs to be captured in Oracle 8i, other columns in the row must be modified in the same transaction. Solution: Write a stored procedure to set a column value to itself.
GoldenGate manages the secondary truncation point. If Extract is stopped for an extended period of time, the secondary truncation point does not move, which prevents transaction data from being purged. Ultimately, the log fills up, and the database halts. Solution: The only thing that you can do if the log fills up is to disable the secondary truncation point and finish purging the log. The purge will remove entries that were not yet read by Extract, so you must resynchronize the source and target data. Afterward, initialize the secondary truncation point by issuing the following database command before starting Extract.
dbcc settrunc( 'ltm', valid)
(Applies to most supported databases) Was ADD TRANDATA issued for each table? This GGSCI command makes modifications to database settings so that GoldenGate can obtain the information it needs from the transaction logs.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 59
Troubleshooting GoldenGate
Extract abends with OCI Error executing fetch ...
For more information about ADD TRANDATA, see the GoldenGate installation guide for your database and the GoldenGate for Windows and UNIX Reference Guide.
To include missed transactions in the target
To update the target with missed operations after running ADD TRANDATA or setting data capture changes, you need to resynchronize the affected tables.
To skip missed transactions
To skip the transactions that were missed before running ADD TRANDATA, follow this procedure. The target will not reflect accurate data in this case. 1. Stop Replicat.
STOP REPLICAT <group>
2.
3.
4.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 60
Troubleshooting GoldenGate
No DDL is captured by Extract
This error happens if key columns are changed before GoldenGate can process updates or deletes that were issued on the old column structure. In this context, the key columns are whichever columns are being used by GoldenGate as the key. This could be a primary key, a unique key or index, a substitute key defined with a KEYCOLS clause, or all of the columns in the absence of those other identifiers.
Does the DDL trace file (default name of ggs_ddl_trace.log) show the following message?
DDL operation failed because RECYCLEBIN is turned on. Recycle bin must be turned off by setting in parameter file: RECYCLEBIN=OFF (10gr2 and up) or _RECYCLEBIN=FALSE (10gr1), and restarting database.
This indicates that the Oracle recycle bin is on, so the DDL trigger failed. The recycle bin must be disabled for GoldenGate DDL processing. Follow the instructions in the error message and then enable the trigger by running the ddl_enable script from the GoldenGate directory.
Does the value of the GGSCHEMA parameter in the GLOBALS file match the name of the schema that was used to install the GoldenGate DDL objects? Are you still unable to determine why DDL is not being captured? Use the following procedure to verify that the DDL objects are installed and are operating correctly.
To diagnose failure to capture DDL
1. 2. 3. 4.
In the root GoldenGate directory, open the GLOBALS file. Make a note of the GoldenGate DDL user name and, if applicable, any non-default names specified with the DDLTABLE and MARKERTABLE parameters. Connect to the source database as SYSDBA. Run the marker_status.sql script. This script verifies that the GoldenGate marker table is installed. It should show the following results.
Please enter the name of a schema for the GoldenGate database objects: ggsuser Setting schema name to GGSUSER MARKER TABLE ------------------------------OK MARKER SEQUENCE ------------------------------OK
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 61
Troubleshooting GoldenGate
No DDL is captured by Extract
5. 6.
Run the ddl_status.sql script. This script verifies that the GoldenGate DDL objects are installed. In the script output, look for the location of the DDL trace file, as shown in the following example, and make a note of it.
LOCATION OF DDL TRACE FILE --------------------------C:\ORACLE\ADMIN\TEST10G1\UDUMP/ggs_ddl_trace.log
7.
At the end of the output, verify that the following shows successful installation of the DDL objects.
SUCCESSFUL installation of DDL Replication software components
8. 9.
10. Edit the ddl_tracelevel.sql script that resides in the root GoldenGate directory, and set the trace level to 1. 11. Start Extract again.
START EXTRACT <group>
12. On the source, issue a DDL statement as a test. 13. Open the trace file in a text editor. 14. At the bottom of the trace file, search for a line that contains the date, time and DDL text, for example:
... 08-10 11:08:16 : DDL : DDL operation [create table qatest1.testTable (<n> Number) ], sequence [17047] ...
If this line exists, it means that the DDL trigger captured the DDL. 15. Make a note of the sequence number. 16. Proceed to the next step to confirm that the record exists in the DDL history table. 17. Connect to the database as SYS. 18. Issue the following query in SQL*Plus, using the sequence number you recorded in place of the one shown here.
select metadata_text from <ggsuser>.<ggs_ddl_hist> where seqno=17047; Where:
<ggsuser> is the GoldenGate DDL user name that is listed in the GLOBALS file that resides in the GoldenGate root directory. <ggs_ddl_hist> is either the name listed with DDLTABLE in the GLOBALS file or else the default DDL history table name of ggs_ddl_hist.
19. In the output, look for a record that contains the DDL statement that you issued.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 62
Troubleshooting GoldenGate
No DDL is captured by Extract
20. If the record exists, keep the query output open, and then proceed to the next step to determine the marker sequence number. 21. Starting from the line you read in the previous step, search for the first occurrence of MD_TAB_MARKERSEQNO. You should see something like the following, which shows the marker sequence number.
... MD_TAB_MARKERSEQNO(key1 = [] key2 = []) = [16549] ....
Record the marker sequence number. Copy the query statement and paste it into a text file for future use. Proceed to the next step to verify that the marker record exists in the marker table.
23. In SQL*Plus, issue the following query, using the sequence number you recorded in the previous step.
select metadata_text from <ggsuser>.<ggs_marker> where seqno=16549; Where:
<ggsuser> is the GoldenGate DDL user name that is listed in the GLOBALS file that resides in the GoldenGate root directory. <ggs_marker> is either the name listed with MARKERTABLE in the GLOBALS file or the default DDL marker table name of ggs_marker.
24. In the output, look for the DDL statement that you issued. If the statement is there, a marker was written to the redo logs as expected. Do the following:
Copy the query statement and paste it into the text file with the other query. Proceed to the next step to verify that Extract can read the marker.
25. In SQL*Plus connect to the database as the GoldenGate DDL user, the one listed in the GLOBALS file. 26. Issue the following queries, which are the two that you copied and pasted to the text file.
select metadata_text from <ggsuser>.<ggs_ddl_hist> where seqno=17047; select metadata_text from <ggsuser>.<ggs_marker> where seqno=16549;
27. Verify that these queries returned the same results as the ones you issued as the SYS user. Do one of the following:
If you do not get output, but instead get a database error, this indicates that the ddl_rolesetup.sql script was not executed properly to grant the GoldenGate DDL user the correct privileges. Run the role_setup script as SYSDBA and then grant the role that you just created to all GoldenGate users under which the following GoldenGate processes run: Extract, Replicat, GGSCI, and Manager. If you receive valid results from the query, proceed to the next step.
28. In GGSCI, start Extract. Start the existing group, not a new or recreated one. Otherwise, the test DDL statement will not be captured.
START EXTRACT <group>
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 63
Troubleshooting GoldenGate
Specific DDL is not captured or excluded
30. If you do not see this statement, or if you get an error, contact GoldenGate Technical Support. Or, reinstall the DDL environment, understanding that you will lose any DDL operations that have not been captured.
This is normal. These operations are not considered to be DDL by the database and are ignored.
Are other specific operations that are supported by GoldenGate not being captured?
First, make certain that the operation is listed in an INCLUDE OPTYPE statement or else is not listed in an EXCLUDE OPTYPE statement, and verify that the missing operation does not apply to an object whose name was in an EXCLUDE OBJNAME statement. Perhaps you have the same operation type or object name in both an EXCLUDE and INCLUDE statement. If so, the EXCLUDE takes priority over the INCLUDEs, and this might be why some operations are not being captured or applied to the target. Or, perhaps you are only replicating objects that are MAPPED in scope, and the missing DDL was for objects that are UNMAPPED in scope.
The filter criteria may not be inclusive or exclusive enough. For example, if you filter on INSTR supplemental log, it will filter both ADD SUPPLEMENTAL LOG and DROP SUPPLEMENTAL LOG statements. Check for spaces in the source DDL. If the source includes spaces, then you must include those spaces in your filter specification.
Did you forget to use the DDLOPTIONS ADDTRANDATA parameter statement or issue the ADD TRANDATA command in GGSCI for the new source table name?
The ADD TRANDATA functionality of GoldenGate creates a supplemental log group for a new table that is to be part of the GoldenGate configuration. The supplemental logging provides information that GoldenGate needs to capture transactional data from the logs. Unless you either enable supplemental logging for a new table through the database interface, or else use one of the ADD TRANDATA options described here, the data is not captured by the Extract process.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 64
Troubleshooting GoldenGate
RENAMEs converted to ALTER TABLE RENAME
When DDL support is enabled, it is recommended that you use the DDLOPTIONS ADDTRANDATA statement, instead of the ADD TRANDATA command in GGSCI. By using the parameter, you can ensure that supplemental logging is created or updated automatically for all CREATE, ALTER, and RENAME operations that affect tables in the DDL configuration. It also deletes the old supplemental log groups when a table is renamed, preventing the need for a manual procedure.
Did you use a DDLOPTIONS ADDTRANDATA statement and data still is not captured?
There might have been a lag between when the table was created or altered, and when the ADD TRANDATA took effect. If there was DML issued during this interval, it was not captured because the supplemental logging was not yet added or updated.
Are you using TABLEEXCLUDE with the NORENAME option or DDLOPTIONS with the NOCROSSRENAME option?
If so, were the table names previously of UNMAPPED scope, but the new names are of MAPPED scope? You might see one of the following messages in the Extract report:
Object [FIN.ACCT] was renamed even though it's marked excluded with NORENAME.
or
Object that is not replicated [FIN.ACCT] was renamed into object that is [FIN.SALES].
Those renames will succeed, but data in those tables might not be replicated to the target.
This is normal. GoldenGate converts RENAME TABLE to ALTER TABLE RENAME to work around permissions problems that can occur if the source and target GoldenGate database user names are different from each other.
Oracle only allows 16 characters plus the double quote marks (18 total) for RENAME and ALTER TABLE RENAME, because of the ANSI limit for identifiers.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 65
Troubleshooting GoldenGate
Comments are in the wrong place
Were the comments in the middle of the name of the source object in the source DDL statement?
If so, it is expected behavior for GoldenGate to move them to the end of the object name on the target. This does not affect the integrity of the DDL replication.
If so, then regardless of whether you explicitly or implicitly mapped the derived object, NOMAPDERIVED overrides the mapping and prevents the conversion.
Check your mappings. For RENAME and ALTER TABLE RENAME, the new table name is considered the base object name, and the old name is the derived object name.
Does your DDL INCLUDE statement include the object for which you want to use string substitution? Or, is that object excluded somehow in an EXCLUDE statement?
The DDL statement always executes first, regardless of the ordering of the DDL and DDLSUBST parameters. Did you perhaps create a DDLSUBST statement that affects a nonincluded object, thinking it would execute independently of the INCLUDE or EXCLUDE in the DDL parameter?
This parameter removes comments before the point in processing where string substitution occurs. Use DDLOPTIONS REMOVECOMMENTS AFTER if you want comments removed on the target after string substitution occurs. To prevent the removal of comments, do not use a REMOVECOMMENTS option.
Is reporting enabled?
By default, DDL runtime information is not included in the report because it expands the size of the file. To enable DDL runtime reporting, use a DDLOPTIONS REPORT statement in the parameter file of the process for which you want the reporting to be generated (Extract, Replicat, or both).
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 66
Troubleshooting GoldenGate
Replicat abends on ggs_ table
This indicates that one or more GoldenGate DDL objects are included in the replication configuration. These objects are:
Do not include the GoldenGate DDL objects in any DDL, TABLE or MAP parameter statements, and make certain that wildcard specifications in those parameters do not include them. These objects must be excluded from the GoldenGate replication configuration.
This error can occur if any changes were made to the GoldenGate DDL objects while sessions that performed DDL operations were still open. Close all sessions in the database that performed DDL. If that does not help, close all database sessions. Then reinstall the GoldenGate DDL trigger, using the NORMAL option. If this procedure does not help, you need to reinstall the DDL objects. Refer to the GoldenGate for Windows and UNIX Administrator Guide for procedures.
The tablespace that contains the GoldenGate DDL objects is full. When the tablespace is full, users cannot perform DDL across the whole database system, even for DDL that is executed in other tablespaces.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 67
Troubleshooting GoldenGate
Process report not generated
1. 2. 3. 4. 5.
Stop all DDL. Disable the GoldenGate DDL trigger. Add more storage to the tablespace. Enable the DDL trigger. Start the GoldenGate process.
For more information about how to enable and disable the DDL trigger, see the DDL chapter in the GoldenGate for Windows and UNIX Administrator Guide. This chapter also contains information on how to purge the history and marker tables to control their size.
This happens occasionally. See Using the process report on page 13 for more information and a resolution.
Did you specify a location for the file with the DISCARDFILE parameter?
Perhaps the user that wrote the last discard file has a security level that won't let the new one be created.
First, try to resolve whatever problem is causing discarded records to accumulate in the file. The records themselves can provide information that helps you diagnose it. If you can tolerate the errors, you can control the size of the file with the following parameters:
DISCARDROLLOVER: specifies parameters for aging the file MAXDISCARDRECS: limits the number of errors written to the file
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 68
Troubleshooting GoldenGate
Trail file is not emptying
DISCARDFILE with the PURGE option: purges the file before writing new content DISCARDFILE with the MEGABYTES option: sets a different maximum file size (default is 1
MB).
Is it a local trail?
If a local trail is not emptying, check for TCP/IP issues that delay transfer of data across the network.
Is it a remote trail?
If a remote trail is not emptying, find out if Replicat is running and processing data. If yes, Replicat might be processing a large transaction. You can see which tables have large transactions by using the TRANSHIST command in Logdump. See Chapter 4. Consider partitioning tables that have large transactions into multiple trails and Extract or Replicat groups, or use data pumps. Use the FILTER option in the TABLE or MAP statements, and use the @RANGE function to divide the rows among the different groups. For more information about using multiple groups and row ranges, see page 22.
To check the maximum file size, use the INFO EXTTRAIL * or INFO RMTTRAIL * command and view the File Size field. Maybe the file size has not reached that value yet. To change the file size, use ALTER EXTTRAIL or ALTER RMTTRAIL with the MEGABYTES option.
If not, add PURGEOLDEXTRACTS to the Manager parameter file to prevent old files from accumulating. If you are using PURGEOLDEXTRACTS, make certain that the Manager user has the authority to purge trail files, and make certain that the PURGEOLDEXTRACTS options are used correctly. See the GoldenGate for Windows and UNIX Reference Guide.
A trail file will not be purged if another process has a checkpoint in it. Delete the obsolete group with the DELETE REPLICAT command, so that the checkpoint records are deleted.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 69
Troubleshooting GoldenGate
Trail files purge too soon
If a checkpoint table is being used for the group, log into the database with the DBLOGIN command first, so that the checkpoint will be removed from the table.
DBLOGIN [TARGETDB <dsn>,] [USERID <user>, PASSWORD <pw>] DELETE REPLICAT <group> NOTE Note: In the DBLOGIN command, TARGETDB or USERID might be required, or both will, depending on the database type and what kind of authentication is required.
Are multiple Replicat groups linked to the same trail? Or are a Replicat and a data pump linked to the same trail?
Are you using PURGEOLDEXTRACTS? If so, how are you using it? When more than one process reads a trail, PURGEOLDEXTRACTS should only be used as a Manager parameter, not as an Extract or Replicat parameter. When you use PURGEOLDEXTRACTS as a Manager parameter, there are better options to defer purging until all processes are done with a file.
If so, are you using CMDTRACE? This parameter causes inclusion of macro expansion steps in the report. The DETAIL option writes even more data. If your reporting requirements do not include macro tracing, disable CMDTRACE to reduce the report size.
This parameter lists the macros of a library in the report. If your reporting requirements do not include macro lists, remove LIST to reduce report size. Otherwise, you can control report size by setting an aging schedule with the REPORTROLLOVER parameter.
If DDL information is included in a process report, it increases the size of the file. If you do not need to see step-by-step processing history of DDL operations in the report file, remove the DDLOPTIONS REPORT parameter. The default is DDLOPTIONS NOREPORT.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 70
Troubleshooting GoldenGate
Too many open cursors
The Extract process maintains cursors for queries that fetch data. The Replicat process maintains cursors for caching SQL statements. These processes also might use other cursors, such as those needed for SQLEXEC operations. If the database will support additional cursors, you can increase the GoldenGate allocation with the Extract parameter MAXFETCHSTATEMENTS or the Replicat parameter MAXSQLSTATEMENTS. If the database will not support an increase, use those parameters to reduce the number of cursors. This will affect performance because GoldenGate will need to age more statements.
Time differences
Lag is always accurate in absolute terms, but may seem inaccurate because of differences in system time on source and target machines. Use the TCPSOURCETIMER parameter to reflect a more accurate lag result.
GoldenGate records are extracted in GMT (Greenwich Mean Time) format. However, source and target timestamps could differ for the same set of data if an application uses the timestamp recorded by GoldenGate as a column in the target record. Discrepancy occurs if:
the timestamp is converted to Local Civil Time (LCT) before being included in the database record, and that data is replicated to a location in another time zone. timestamps are derived locally and used in a key, which can create sequence differences if a table contains records created in two different time zones.
Solution: If using timestamps written by GoldenGate in application data, maintain all timestamps as GMT offsets. GMT is reliable in any time zone and through Daylight Saving time changes without sequence problems. If that is not possible, then use GoldenGate conversion functions to change the format back and forth as needed.
Did you type the table name correctly? Does it exist in the database? Did you use a fully qualified table name? When a schema name is not specified, GoldenGate uses the current user as the schema.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 71
Troubleshooting GoldenGate
Other Oracle-specific problems
Is the error SQL error (-942). ORA-00942: table or view does not exist.?
Was the command issued for an Oracle 8i database? For Oracle 8i tables, ADD TRANDATA creates a trigger on the table. The user running ADD TRANDATA might not have privileges to create the trigger. Grant permission, and the command should work.
Did you change storage for a SECUREFILE LOB? When changing a SECUREFILE LOB from one storage to another (such as from ENCRYPT to DECRYPT), Oracle updates the whole table, and Extract captures those updates from the log. Therefore, it will appear as though Oracle updated all of the data blocks that are associated with the table.
NOTE This also can happen if an ALTER TABLE command is issued to set a DEFAULT value to a column that has null values.
Are you replicating sequences with GoldenGate, but the target sequences are not in the expected range of values?
By default, Replicat checks to see if the target sequences are either higher or lower than those of the source, depending on the direction of the incrementation method that is in use. If there is a discrepancy, Replicat corrects it automatically unless NOCHECKSEQUENCEVALUE is used in the parameter file. If NOCHECKSEQUENCEVALUE is not in use, and you cannot find a possible cause outside the GoldenGate configuration, please contact GoldenGate Technical Support.
By default, certain information messages and messages with a severity level greater than 10 are not logged by GoldenGate to the error log. To show these messages in the log, use the Extract parameter DBOPTIONS with the SHOWINFOMESSAGES and SHOWWARNINGS options.
Is the number of operations that were executed on the database different from the number that GoldenGate shows were extracted?
These statistics will vary because of the way that messages are updated in the DB2 logs. DB2 does not log update statements if they do not physically change a row.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 72
Troubleshooting GoldenGate
DB2 z/OS-specific problems
This can happen if the DB2 parameter mvsattachtype is set to CAF (call attachment facility, the default). CAF does not support multiple connections, so GoldenGate issues commit locks on the system catalog tablespace until it receives the commit for its open connection. To avoid commit locks, set the attachment type to RRSAF (Resource Recovery Services Attachment Facility), if possible.
If the error is similar to the one following, it means that the location name of the local DB2 subsystem needs to be inserted into the DB2 remote server locations table,
SYSIBM.LOCATIONS. 2006-01-09 15:48:45 GGS WARNING 228 ODBC operation failed: Couldn't connect to cdnxdb2q. 2006-01-09 15:48:45 GGS WARNING 228 {DB2 FOR OS/390}{ODBC DRIVER} DSNT408I SQLCODE = -922, ERROR: AUTHORIZATION FAILURE: 00D31024 ERROR. REASON CONNECT DSNT418I SQLSTATE = 42505 SQLSTATE RETURN CODE DSNT415I SQLERRP = DSNLVINF SQL PROCEDURE DETECTING ERROR DSNT416I SQLERRD = 0 0 0 -1 0 0 SQL DIAGNOSTIC INFORMATION DSNT416I SQLERRD = X'00000000' X'00000000' X'00000000' X'FFFFFFFF' X'00000000' X'00000000' SQL DIAGNOSTIC INFORMATION ERRLOC=1:13:2. 2006-01-09 15:48:45 GGS ERROR 118 ODBC error 42505, native database error -922. 2006-01-09 15:48:45 GGS ERROR 190 PROCESS ABENDING.
A statement similar to the following adds the DB2 location name to the remote group:
INSERT INTO SYSIBM.LOCATIONS (LOCATION, PORT) VALUES ('DB2A', '446');
On a Sysplex, RRS is sensitive to system crashes. RRS might start normally after a crash, but GoldenGate might not be able to use RRSAF. You might see messages similar to:
ODBC error 58004, native database error -99999***** ABEND****** ABEND****** {DB2 FOR OS/390}{ODBC DRIVER} SQLSTATE=58004 ERRLOC=2:170:4; RRS IDENTIFY failed using DB2 system:DB7G, RC=0c and REASON=00f30006
To resolve this error, shut down RRS and then rebuild the RRS structures in the Coupling Facilities using your RRS installation job.
Did Extract fail and return DB2 reason codes 00C90063 and 00C90064?
This can be a symptom of mismatched compression dictionaries. You will need to resynchronize the affected data and start GoldenGate processing from the initialized state.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 73
Troubleshooting GoldenGate
SQL Server-specific problems
This message indicates that the Call Attachment Facility (CAF) is being used, instead of the Resource Recovery Services Attachment Facility (RRSAF). This message can mean one of the following:
The ODBC initialization file specified MVSATTACHTYPE=CAF or did not include an MVSATTACHTYPE specification. The ODBC initialization file does not contain a section for the DB2 subsystem. The ODBC initialization file is not valid because it the encoding is not IBM-1047. The ODBC initialization file is not readable (indicates a file permissions problem). No ODBC initialization file was specified. There are several ways to specify which ODBC initialization file should be used; GoldenGate sets the DSNAOINI environment variable to the path to the ODBC initialization file that it needs to use. For example, export DSNAOINI=/etc/odbcDB9A.ini.
Are you using the MANAGESECONDARYTRUNCATIONPOINT parameter so that SQL Server manages the secondary truncation point? If so, and if this is the configuration that you want, then did you also issue the ADD TRANDATA command for the tables? If so, you will get this message, because Extract expects the NOMANAGESECONDARYTRUNCATIONPOINT in that case. The message is generated as part of the truncation point management process.
Is Extract generating an error that states Updates are not supported on tables that do not have
TRANDATA added?
If the source database is SQL Server 2000, ADD TRANDATA sets a flag on the sysobjects table that tells SQL Server to log full before and after images. This flag is shared by update triggers and by replication. If an update trigger on a table in the Extract configuration is dropped, it will generate this error. Dropping the trigger drops the extended logging for that table. To enable capture of the table again, use the following procedure. This procedure assumes you can stop activity on the source table. If you cannot stop activity, see To resynchronize a table on page 53 for instructions. Take the table out of the Extract TABLE parameter. In GGSCI, issue the ADD TRANDATA command for the table. Stop all activity on the source table. Restart Extract and let it capture backlogged transaction data until it is current again. The SEND EXTRACT command with the STATUS option should return At EOF.
1. 2. 3. 4.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 74
Troubleshooting GoldenGate
Teradata module not found
5. 6. 7.
Stop Extract. Resynchronize the source and target tables. Restart Extract.
The error appears as follows only for Teradata on a Windows machine running Windows XP:
2006-08-22 16:48:15 GGS ERROR 520 VAMRDR-520 DLL LoadLibrary [Error 126:The specified module could not be found.:occurred loading C:\GGS\TERA\TAM.DLL]. 2006-08-22 16:48:15 GGS ERROR 190 PROCESS ABENDING.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 75
CHAPTER 4
Run the logdump program from the GoldenGate installation location. Logdump command lines are numbered so that you can use edit and history commands.
To set up the view
The following commands set up a Logdump environment that shows the information most commonly used when analyzing GoldenGate trail records. 1. To view the record header with the data:
Logdump 1> GHDR ON
The record header contains information about the transaction. 2. To add column information:
Logdump 2> DETAIL ON
Column information includes the number and length in hex and ASCII.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 76
3.
4.
User tokens are custom user-defined information that is specified in a TABLE mapping statement and stored in the trail file for specific purposes. 5. To control how much record data is displayed:
Logdump 5> RECLEN <length> To open a trail file
1.
2.
A GoldenGate trail record looks similar to the one in the following illustration, depending on what views are activated and what type of record it is. In this case, commands were issued to show the header portion of the record, to show column-level detail, and to show user tokens.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 77
Figure 9
1.
2.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 78
3.
View the TransInd field in the record header. The following tells you where the record is in relation to the transaction.
TransInd TransInd TransInd TransInd : . : . : . : . (x00) (x01) (x02) (x03)
First statement in transaction Statement in middle of transaction Last statement in transaction Sole statement in transaction
4.
Move through subsequent records by pressing N, and refer to the TransInd field to determine where each one is within the transaction. When TransInd is either x02 or x03, the TransInd of the next record should be x00, starting a new transaction.
(or just type SFET) The record shown will be the first one in the next transaction. To confirm, the TransInd field should be x00.
To go to a specific RBA in the file
or...
Logdump 37> POS 0 To filter based on a table name
Now, when you use the N command, you will only see records that satisfy this filter.
Conversely, to filter out records containing a specific table name, but show everything else, use the EXCLUDE option instead of INCLUDE.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 79
To filter on multiple conditions Logdump 65> FILTER INCLUDE FILENAME <schema>.<table>; FILTER RECTYPE <record_type>; FILTER MATCH ALL
Use MATCH ANY or MATCH ALL depending on whether you want the search to match any or all of the filter conditions, respectively, when multiple conditions are specified. The preceding example filters on a name and record type, typically an operation type such as INSERT.
To count the records in a trail file Logdump 67> COUNT
To close the current file and open the next one in the trail Logdump 70> NEXTTRAIL To keep a log of your session
To start logging:
Logdump 71> LOG TO <filename>.txt
To stop logging:
Logdump 73> LOG STOP
This shows which features are enabled, such as filtering and header views, and it shows environment information such as the current trail and position.
To get online command help Logdump 75> HELP
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 80
Or
Logdump 100> QUIT
2.
Use either the TRANSRECLIMIT or TRANSBYTELIMIT command to set a lower boundary for what is considered a normal sized transaction. These commands prevent normal-sized transactions from being tracked. Eliminating normal-sized transactions reduces the amount of data that must be reviewed.
{TRANSBYTELIMIT <n bytes> | TRANSRECLIMIT <n records>}
3.
Use Logdumps COUNT command to display the statistics on transaction size, which appear at the end of the output and look like the following excerpt:
Transactions with at least 100 records or 100000 bytes 2006/02/01 09:31:24.000.000 00:00:00.000, Seq 0, RBA 13101 Bytes/Trans ..... 1168167 Records/Trans ... 1001 Files/Trans ..... 1 2006/02/01 09:31:35.000.000 00:00:11.000, Seq 0, RBA 1205292 Bytes/Trans ..... 1168167 Records/Trans ... 1001 Files/Trans ..... 1
Logdump scans the file(s) and reports the information. 4. Use Logdumps POSITION <RBA> command to go to each RBA listed in the COUNT output to find out the name of the table that generated the transaction. You can group these tables into their own processing group so that they do not affect processing of other tables that generate normal sized transactions.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 81
If the file exists, Logdump loads the command history into a buffer. The command history buffer holds 400 commands. Upon termination of the Logdump session, the sessions history is appended to the file.
Sets the default directory. Writes a session log. Closes the current file and opens the next file in the trail sequence. Opens a trail file or extract file. Sets the read position in the file. Writes record data to another file. Writes text to a session log.
Table 3
Displays record count information. Displays file names in the current directory.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 82
Table 3
Displays current Logdump settings. Displays file header information. Displays the number of records scanned, the trail position, and the record timestamp at specified intervals when using COUNT and records are being suppressed from display through filtering options. Displays internal information such as the current Logdump environment, a list of GoldenGate record types, and current filter settings. Displays the current time in local and GMT formats.
SHOW
TIME
Table 4
Command DUMP
Displays the specified number of bytes of data from the current position in the file. Filters the display of records. Displays the next record(s) in the file. Finds a record that is the last record of, or the only record in, a transaction, and then displays the first record of the next transaction. Finds the start of the next record header. Finds a specific relative byte address. Finds the next record with a specific timestamp. Finds the next record of a specific type. Skips a specified number of records.
Table 5
Command COMPUTETIMESTAMP
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 83
Table 5
Converts a C timestamp to an ASCII timestamp. Decrypts data before displaying it in Logdump. Encrypts file data. Displays a 64-bit Julian interval as days-hh:mm:ss:ms:us. Displays a 64-bit Julian timestamp in ASCII format.
Table 6
Controls whether data is displayed in ASCII or EBCDIC format on an IBM mainframe. Sets the character set on an IBM mainframe. Controls whether the table name is displayed in ASCII or EBCDIC format on an IBM mainframe. Controls the display of detailed record information. Controls the display of header information. Controls the display of header token indicators. Sets the maximum data output length. Controls whether a count notification displays on one line or multiple lines. Sets the time offset from GMT. Sets the trail format to the old version (pre-GoldenGate 6.0) or the new version. Sets a byte-count threshold for what is defined as a normal-sized transaction. Sets the size of the transaction history table that is used for tracking transaction size. Sets a record-count threshold for what is defined as a normal-sized transaction.
TIMEOFFSET TRAILFORMAT
TRANSBYTELIMIT
TRANSHIST
TRANSRECLIMIT
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 84
Table 6
Command USERTOKEN
Table 7
Turns on Logdump debugging. Exits Logdump. Edits a previous command. Shows syntax for Logdump commands. Lists previously issued commands. Executes a series of commands stored in a file. Executes a program from within Logdump.
ASCIIDATA | EBCDICDATA
Use ASCIIDATA or EBCDICDATA to control whether record data is displayed in ASCII or EBCDIC format on an IBM mainframe. You may need to use the ASCIIDUMP command to set the character set first.
Default Syntax ASCIIDATA ON (same as EBCDICDATA OFF) ASCIIDATA {ON | OFF} | EBCDICDATA {ON | OFF}
ASCIIDUMP | EBCDICDUMP
Use ASCIIDUMP or EBCDICDUMP to set the character set for the output of the DUMP command on an IBM mainframe. The hex part of the dump display is not affected by this command.
Default Syntax ASCIIDUMP ON (same as EBCDICDUMP OFF) ASCIIDUMP {ON | OFF} | EBCDICDUMP {ON | OFF}
ASCIIHEADER | EBCDICHEADER
Use ASCIIHEADER or EBCDICHEADER to control whether or not the table name in the record header is in ASCII or EBCDIC format on an IBM mainframe. You may need to use the ASCIIDUMP command to set the character set first.
Default ASCIIHEADER ON (same as EBCDICHEADER OFF)
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 85
Syntax
CD
Use CD to set the default directory. An alias for this command is VOLUME.
Default Syntax
None
CD <directory>
COMPUTETIMESTAMP
Use COMPUTETIMESTAMP to convert a datetime string to Julian format.
Default Syntax Argument <datetime string>
None
COMPUTETIMESTAMP <datetime string> Description
Example
COUNT
Use COUNT to produce a record count summary and other information related to the amount of data in the file. The basic output, without options, shows the following:
The RBA where the count began The number of records in the file The total data bytes and average bytes per record Information about the operation types Information about the transactions
When the DETAIL command is issued prior to issuing COUNT, the information includes a count for each table. COUNT options allow you to show table detail without using the DETAIL command first, set a start and end time for the count, filter the count for a table, trail file, or extract file, and specify a time interval for counts. For arguments that take a time string, use the following format:
[[yy]yy-mm-dd] [hh[:mm][:ss]] Default
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 86
Syntax
COUNT [, DETAIL] [, END[TIME] <time_string>] [, FILE <specification>] [, INT[ERVAL] <minutes>] [, LOG] <wildcard>] [, START[TIME] <time_string>] Description
Argument DETAIL
Adds a count for each table that was processed by Extract to the summary count. The information includes the total and average number of data bytes and information about the operations that were performed. This data can also be obtained by using the DETAIL command before issuing COUNT. Stops the count with the last record written at the specified time. Specifies the count to be generated for the specified table or group of names designated with a wildcard (*). Displays statistics for total bytes, average bytes, and number of each type of operation that occurred within a specified interval of time, in minutes. Then it displays the totals for those statistics. Produces a count for multiple trail or extract files specified with a wildcard. Begins the count with the first record written at the specified time.
INT[ERVAL] <minutes>
LOG <wildcard>
START[TIME] <time_string>
Example 1 Example 2
This displays something similar to the following (individual table count has been truncated due to space constraints):
Interval from 2005/02/28 11:51:18.000 to 2005/02/28 12:13:16.000, Recs 3 Total Data Bytes 120 Avg Bytes/Record 40 Delete 3 Before Images 3 LogTrail /home/ggs/dirdat/rt000000 has 304 records Total Data Bytes 12120 Avg Bytes/Record 39 Delete 3 Insert 300 FieldComp 1 Before Images 3 After Images 301 Average of 303 Transactions
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 87
Bytes/Trans ..... Records/Trans ... Files/Trans ..... HR.JOBS Total Data Bytes Avg Bytes/Record Insert After Images HR.REGIONS Total Data Bytes Avg Bytes/Record Insert After Images Example 3 COUNT LOG ls*
This produces a count for all files whose names begin with LS. (Individual table count has been truncated due to space constraints.)
Current LogTrail is c:\goldengate802\dirdat\ls000000 Bad record found at RBA 5287, format 5.50) 2A56 623F | *Vb? LogTrail c:\goldengate802\dirdat\ls000000 has 33 records LogTrail c:\goldengate802\dirdat\ls000000 closed Current LogTrail is c:\goldengate802\dirdat\ls000001 LogTrail c:\goldengate802\dirdat\ls000001 has 99 records LogTrail c:\goldengate802\dirdat\ls000001 closed Current LogTrail is c:\goldengate802\dirdat\ls000002 LogTrail c:\goldengate802\dirdat\ls000002 has 0 records LogTrail c:\goldengate802\dirdat\ls000002 closed Current LogTrail is c:\goldengate802\dirdat\ls000003 LogTrail c:\goldengate802\dirdat\ls000003 has 0 records LogTrail c:\goldengate802\dirdat\ls000003 closed LogTrail c:\goldengate802\dirdat\ls* has 132 records Total Data Bytes Avg Bytes/Record Insert After Images Average of 4 Transactions Bytes/Trans ..... Records/Trans ... Files/Trans ..... HR.JOBS Total Data Bytes Avg Bytes/Record Insert After Images Example 4 COUNT DETAIL 9468 71 132 132
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 88
0, Chain
CTIME
Use CTIME to convert a C timestamp to an ASCII timestamp.
Default Syntax Example
None
CTIME <C timestamp string> CTIME 1109823330
DEBUG
Use DEBUG to run debugging for Logdump. Use this command with the guidance of a GoldenGate support analyst.
Default Syntax
Disabled
DEBUG
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 89
DECRYPT
Use DECRYPT to decrypt data that was encrypted with GoldenGate trail encryption, so that it can be viewed with Logdump.
Default Syntax OFF DECRYPT {ON | OFF}
DETAIL
Use DETAIL to include additional information in the Logdump output. By default, Logdump only shows the hex and ASCII representation of the record. Without options, DETAIL displays the status of record detail (ON or OFF). Options do the following:
DETAIL ON displays a list of columns that includes the column ID, length, and value in
Shows detailed column information. Suppresses detailed column information. Adds the hex and ASCII data values to the column information.
DUMP
Use DUMP to display a HEX/ASCII or HEX/EBCDIC dump of the specified number of bytes from the open trail or extract file, starting at the current RBA.
Default Syntax Argument <bytes>
256
DUMP <bytes> Description
The number of bytes forward to display. Valid values are from 1 through 28672.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 90
Example
DUMP 300
ENCRYPT
Use ENCRYPT to encrypt text supplied as an argument. The encryption method is 256-key byte substitution. The results are printed to screen.
Default Syntax Example
None
ENCRYPT <text> ENCRYPT 123456789
ENV
Use ENV to show current Logdump settings.
Default Syntax
None
ENV
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 91
Example
EXIT
Use EXIT to exit Logdump and terminate the process. An alias for EXIT is QUIT.
Default Syntax
None
EXIT
FC
Use FC to edit a previously issued Logdump command and then execute it again. Previous commands are stored in the memory buffer and can be displayed by issuing the HISTORY command (see page 110). Issuing FC without arguments executes the most recently used command. By using options, you can retrieve a specific command by specifying its line number or a text substring.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 92
Argument r <text>
Description
Deletes a character. To delete multiple characters, enter a d for each one. For example:
Logdump 26> fc 11 > scanforrrbba .. dd scanforrba
<replacement text>
Replaces the displayed command with the text that you enter on a one-forone basis. For example:
Logdump 26> fc 10 > scanforrba 107 .. 127 scanforrba 127
To execute the command, press Enter twice, once to exit the editor and once to issue the command. To cancel an edit, type a forward slash (/) twice.
Default Syntax Argument <n>
Returns the specified command line. Each Logdump command line is sequenced, beginning with 1 at the start of the session. Returns the command that was issued <n> lines before the current line. Returns the last command that starts with the specified text string.
FC 9 FC -3 FC sca
-<n> <string>
FILEHEADER
Use FILEHEADER to display the contents of the header of the currently open trail file. The file header is stored as a record at the beginning of a trail file preceding the data records. The information that is stored in the trail header provides enough information about the records to enable a GoldenGate process to determine whether the records are in a format that the current version of GoldenGate supports.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 93
The trail header fields are stored as tokens, where the token format remains the same across all versions of GoldenGate. If a version of GoldenGate does not support any given token, that token is ignored. Depracated tokens are assigned a default value to preserve compatibility with previous versions of GoldenGate. The current FILEHEADER command applies globally to the Logdump session, until a different FILEHEADER command is issued.
To view the file header
1.
Position to the beginning of the trail file with the following Logdump command.
pos 0
2.
Issue the following Logdump command to see the first record of the file, the one that contains the file header.
next
To retrieve file header values as input parameters, use the @GETENV function with the GGFILEHEADER option. See the GoldenGate for Windows and UNIX Reference Guide.
NOTE The Logdump command HEADERTOKEN also shows trail tokens, but it shows a brief summary of each one. FILEHEADER shows actual token values.
Table 8
GoldenGate file header tokens Data Type Description Information about the trail file. UINT32 UINT16
Internal use. The version of the trail. The compatibility level of the GoldenGate software must be greater than, or equal to, that of the trail file for a process to be able to read the trail file. Current valid values are 0 or 1.
CharSet
INT32
The global character set of the trail file, as defined in the parameter file or the default value. For example:
WCP1252-1
The time that the trail file was created, in local GMT Julian time, INT 64.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 94
Table 8
Token/subtoken URI
The universal resource identifier of the process that created the trail file, in the format of:
<host_name>:<dir>:[:<dir>][:<dir_n>]<group_name> Where:
host_name is the name of the server that hosts the process dir is a subdirectory of the GoldenGate installation path. group_name is the name of the process group that is linked with the process.
Example: sys1:home:oracle:v9.5:extora
Shows where the trail was processed and by which process. This includes a history of previous runs.
URIHistory UString Vector
List of the URIs of processes that wrote to the trail file before the current process.
For a primary Extract, this field is empty. For a data pump, this field is URIHistory + URI of the input trail file.
FileName
UString
Name of the trail file. Can be absolute or relative path, with forward or backward slash depending on the filesystem.
True/false flag indicating whether the trail file is a single file (such as one created for a batch run) or a sequentially numbered file that is part of a trail for online, continuous processing. If false, the SeqNum subtoken is not valid.
MultiPart
Boolean
SeqNum
UINT32
The sequence number of the file in the trail, if MultiPart is true. Invalid if multipart is false. The value is the numerical sequence number, without any zero padding. Size of the trail file. Value is NULL until the trail file is completed. Non-NULL values are in bytes. The commit sequence number (CSN) of the first record in the trail file.Value is NULL until the trail file is completed. The commit sequence number (CSN) of the last record in the trail file.Value is NULL until the trail file is completed. The time that the first record in the trail file was written.Value is NULL until the trail file is completed. The time that the last record in the trail file was written.Value is NULL until the trail file is completed.
FileSize
UINT642
FirstRecordCSN
CSN
LastRecordCSN
CSN
FirstRecordIOTime
Timestamp
LastRecordIOTime
Timestamp
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 95
Table 8
GoldenGate file header tokens Data Type Description Information about the local host of the trail file. UString
NodeName Release
UString UString
The name of the machine, for example sys1. The release level of the operating system, for example:
5.10 2.6.9-11.ELsmp 2000 Advanced Server
Version
UString
Hardware
UString
Information about the database that produced the data in the trail file.
Name Instance
UString UString
The name of the database, for example findb. The name of the database instance, if applicable to the database type, for example ORA1022A.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 96
Table 8
Token/subtoken Charset
The character set of the database. Currently, the valid value is -1 (unknown). (For some databases, this will be empty.) The major version of the database. The minor version of the database. The maintenance (patch) level of the database. The character set of the database client. Currently, the valid value is -1 (unknown). (For some databases, this will be empty.) The maintenance (patch) level of the database client. (For some databases, this will be empty.)
Information about the GoldenGate process that created the trail file.
ClientVerString
UString
The group name that is associated with the process. The data source that was read by the process. Can be one of:
trigger-based extraction)
DS_DATABASE (source was a direct select from database table written to a trail, used for SOURCEISTABLE-driven
initial load)
DS_TRAN_LOGS (source was the database transaction log) DS_INITIAL_DATA_LOAD (source was Extract; data taken directly from source tables) DS_VAM_EXTRACT (source was a vendor access module) DS_VAM_TWO_PHASE_COMMIT (source was a VAM trail)
The major version of the process (xx). The minor version of the process (xx.xx). The maintenance version of the process (xx.xx.xx). The patch version of the process (xx.xx.xx.xx). The build number of the process. The version string of the process. For example:
9.5.1.17A not for production
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 97
Table 8
GoldenGate file header tokens Data Type Description Contains recovery information that is carried over from the previous trail file in the sequence. UINT16 CSN Xid CSN Xid
Token/subtoken ContinuityInfo
Internal use Internal use Internal use Internal use Internal use
Enables the display of the file header, showing the main header tokens. Disables the display of the file header. Provides detailed information that includes the sub-tokens.
FILEHEADER ON TokenID TokenID TokenID TokenID TokenID TokenID TokenID x46 x30 x31 x32 x33 x34 x5a 'F' '0' '1' '2' '3' '4' 'Z' Record Header TrailInfo MachineInfo DatabaseInfo ProducerInfo ContinunityInfo Record Trailer Info Info Info Info Info Info Info x00 x00 x00 x00 x00 x00 x00 Length Length Length Length Length Length Length 587 303 103 88 85 4 587 587 RBA 0 0../0...f..q1.....2. ......3.....x4...?4. .7.1uri:tellurian::h ome:mccargar:ggs:ggs Oracle:source6...... /dirdat/er0000007... .8......@9..........
Example 1
2008/07/18 13:39:18.951.346 FileHeader Name: *FileHeader* 3000 012f 3000 0008 660d 0a71 3100 0006 0008 0000 0016 3300 000c 02f1 7834 eac7 0037 0031 7572 693a 7465 6c6c 7572 6961 6f6d 653a 6d63 6361 7267 6172 3a67 6773 4f72 6163 6c65 3a73 6f75 7263 6536 0000 2f64 6972 6461 742f 6572 3030 3030 3030 0138 0000 0800 01e2 4039 0000 0c00 0000
Len 0001 7f3f 6e3a 3a67 1700 3700 0000 3200 3400 3a68 6773 112e 0005 001d | | | | | | |
GroupID x30 '0' TrailInfo Info x00 Length 303 3000 012f 3000 0008 660d 0a71 3100 0006 0001 3200 | 0../0...f..q1.....2. 0008 0000 0016 3300 000c 02f1 7834 eac7 7f3f 3400 | ......3.....x4...?4.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 98
0037 6f6d 4f72 2f64 0138 a33b 0000 0000 0000 0000 0000 0000 0000 0000
0031 653a 6163 6972 0000 0000 0000 0000 0000 0000 0000 0000 0c02 00
7572 6d63 6c65 6461 0800 450a 0000 0000 0000 0000 0000 0000 f178
693a 6361 3a73 742f 01e2 3634 0000 0000 0000 0000 0000 0000 34eb
7465 7267 6f75 6572 4039 3136 0000 0000 0000 0000 0000 0000 556a
6c6c 6172 7263 3030 0000 3138 0000 0000 3aff 0000 0000 0000 403c
7572 3a67 6536 3030 0c00 3936 0000 0000 0045 0000 0000 0000 ff00
6961 6773 0000 3030 0000 3932 0000 0000 0000 0000 0000 0000 0c00
6e3a 3a67 1700 3700 0000 0000 0000 0000 0000 0000 0000 0000 0000
3a68 6773 112e 0005 001d 0000 0000 0000 0000 0000 0000 003d 0000
| | | | | | | | | | | | | |
.7.1uri:tellurian::h ome:mccargar:ggs:ggs Oracle:source6...... /dirdat/er0000007... .8......@9.......... .;..E.6416189692.... .................... .................... ..........:..E...... .................... .................... ...................= .....x4.Uj@<........ ...
GroupID x31 '1' 3100 0067 3000 0974 656c 6c75 392d 3131 2e45 4d50 2046 7269 3330 2045 4454 5f36 34 GroupID x32 '2' 3200 0058 3000 3032 3241 3200 0008 ffff ffff 0006 0000 3700 2e32 2e30 2e32 GroupID x33 '3' 3300 0055 3000 3200 0006 0000 0006 0000 3600 696f 6e20 5374 6420 3032 33
MachineInfo 000b 0005 4c69 7269 616e 3200 4c73 6d70 3300 204d 6179 2032 2032 3030 3534
Info x00 Length 6e75 7831 0000 0f00 0014 000e 322e 362e 0029 0023 2331 2053 3020 3138 3a32 353a 0000 0c00 0678 3836
103 | 1..g0.....Linux1.... | .tellurian2.....2.6. | 9-11.ELsmp3..).##1 S | MP Fri May 2018:25: | 30 EDT 20054.....x86 | _64 88 | 2..X0.....1.....ORA1 | 022A2.....ora1022a3. | ......4.....5.....6. | ....7.......8.....10 | .2.0.2.0 85 | 3..U0.....EFAR1..... | 2.....3.....4.....5. | ....6.....7..#..Vers | ion St. Andrews Buil | d 023 4 | 4...
DatabaseInfo 0006 0007 3100 000e 0008 6f72 3400 0006 0000 0008 ffff ffff 2e30 ProducerInfo 000a 0004 4546 3300 0006 0000 0006 0017 3700 2e20 416e 6472
Info x00 Length 000e 0008 4f52 4131 6131 3032 3261 3300 3500 0006 0000 3600 3800 0010 000a 3130
Info x00 Length 4152 3100 0006 0003 3400 0006 0000 3500 0023 001d 5665 7273 6577 7320 4275 696c
GroupID x34 '4' ContinunityInfo 3400 0004 Example 2 FILEHEADER DETAIL TokenID TokenID TokenID TokenID TokenID TokenID TokenID x46 x30 x31 x32 x33 x34 x5a 'F' '0' '1' '2' '3' '4' 'Z' Record Header TrailInfo MachineInfo DatabaseInfo ProducerInfo ContinunityInfo Record Trailer
Info x00
Length
Len
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 99
| | | | | | |
GroupID x30 '0' 3000 012f 3000 0008 0000 0016 0037 0031 7572 6f6d 653a 6d63 4f72 6163 6c65 2f64 6972 6461 0138 0000 0800 a33b 0000 450a 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0c02 f178 0000 00 TokenID x30 '0' 660d 0a71 TokenID x31 '1' 0001 TokenID x32 '2' 0000 0016 TokenID x33 '3' 02f1 7834 eac7 TokenID x34 '4' 0031 7572 693a 653a 6d63 6361 6163 6c65 3a73 TokenID x36 '6' 0011 2e2f 6469 TokenID x37 '7' 01 TokenID x38 '8' 0001 e240 TokenID x39 '9' 0000 0000 0000 TokenID x3b ';' 0a36 3431 3631 0000 0000 0000 0000 0000 0000 0000 0000 00 TokenID x3a ':' 0000 0000 0000
TrailInfo 0008 660d 3300 000c 693a 7465 6361 7267 3a73 6f75 742f 6572 01e2 4039 3634 3136 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 34eb 556a Signature
0a71 02f1 6c6c 6172 7263 3030 0000 3138 0000 0000 3aff 0000 0000 0000 403c
Info x00 Length 3100 0006 0001 3200 7834 eac7 7f3f 3400 7572 6961 6e3a 3a68 3a67 6773 3a67 6773 6536 0000 1700 112e 3030 3030 3700 0005 0c00 0000 0000 001d 3936 3932 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0045 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 003d ff00 0c00 0000 0000 Info x00 Info x00 Info x00 Length Length Length
Compatibility Charset
CreationTime Info x00 Length 7f3f URI Info x00 Length 7465 6c6c 7572 6961 6e3a 3a68 6f6d 7267 6172 3a67 6773 3a67 6773 4f72 6f75 7263 65 Filename Info x00 Length 7264 6174 2f65 7230 3030 3030 30 MultiPart Info x00 Length Seqno Info x00 Length
FileSize Info x00 Length 1da3 LastCSN Info x00 Length 3839 3639 3200 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 FirstCSN Info xff Length 0000 0000 0000 0000 0000 0000 0000
303 | 0../0...f..q1.....2. | ......3.....x4...?4. | .7.1uri:tellurian::h | ome:mccargar:ggs:ggs | Oracle:source6...... | /dirdat/er0000007... | .8......@9.......... | .;..E.6416189692.... | .................... | .................... | ..........:..E...... | .................... | .................... | ...................= | .....x4.Uj@<........ | ... 8 | f..q 6 | .. 8 | .... 12 | ..x4...? 55 | .1uri:tellurian::hom | e:mccargar:ggs:ggsOr | acle:source 23 | .../dirdat/er000000 5 | . 8 | ...@ 12 | ........ 69 | .6416189692......... | .................... | .................... | ..... 69 | ....................
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 100
0000 0000 0000 0000 0000 0000 0000 0000 00 TokenID x3d '=' 02f1 7834 eb55 TokenID x3c '<' 0000 0000 0000 GroupID x31 '1' 3100 0067 3000 0974 656c 6c75 392d 3131 2e45 4d50 2046 7269 3330 2045 4454 5f36 34 TokenID x30 '0' 0005 4c69 6e75 TokenID x31 '1' 0009 7465 6c6c TokenID x32 '2' 000e 322e 362e TokenID x33 '3' 0023 2331 2053 3138 3a32 353a TokenID x34 '4' 0006 7838 365f GroupID x32 '2' 3200 0058 3000 3032 3241 3200 0008 ffff ffff 0006 0000 3700 2e32 2e30 2e32 TokenID x30 '0' 0007 TokenID x31 '1' 0008 4f52 4131 TokenID x32 '2' 0008 6f72 6131 TokenID x33 '3' ffff ffff TokenID x34 '4' 0000 TokenID x35 '5' 0000 TokenID x36 '6' 0000 TokenID x37 '7' ffff ffff TokenID x38 '8' 000a 3130 2e32
0000 0000 0000 0000 0000 0000 0000 | .................... 0000 0000 0000 0000 0000 0000 0000 | .................... | ..... LastIOTime Info x00 Length 12 6a40 | ..x4.Uj@ FirstIOTime Info xff Length 12 0000 | ........ MachineInfo 000b 0005 4c69 7269 616e 3200 4c73 6d70 3300 204d 6179 2032 2032 3030 3534 103 | 1..g0.....Linux1.... | .tellurian2.....2.6. | 9-11.ELsmp3..).##1 S | MP Fri May 20 18:25: | 30 EDT 20054.....x86 | _64 Sysname Info x00 Length 11 78 | ..Linux Nodename Info x00 Length 15 7572 6961 6e | ..tellurian Release Info x00 Length 20 392d 3131 2e45 4c73 6d70 | ..2.6.9-11.ELsmp Version Info x00 Length 41 4d50 2046 7269 204d 6179 2032 3020 | .##1 SMP Fri May 20 3330 2045 4454 2032 3030 35 | 18:25:30 EDT 2005 Hardware Info x00 Length 12 3634 | ..x86_64 DatabaseInfo 0006 0007 3100 000e 0008 6f72 3400 0006 0000 0008 ffff ffff 2e30 Vendor Name 3032 3241 Instance 3032 3261 Charset MajorVersion MinorVersion VerString ClientCharset ClientVerString 2e30 2e32 2e30 Info x00 Length 000e 0008 4f52 4131 6131 3032 3261 3300 3500 0006 0000 3600 3800 0010 000a 3130 Info x00 Info x00 Info x00 Info x00 Info x00 Info x00 Info x00 Info x00 Info x00 Length Length Length Length Length Length Length Length Length 88 |2..X0.....1.....ORA1 |022A2.....ora1022a3. |......4.....5.....6. |....7.......8.....10 | .2.0.2.0 6 | .. 14 | ..ORA1022A 14 | ..ora1022a 8 | .... 6 | .. 6 | .. 6 | .. 8 | .... 16 | ..10.2.0.2.0 Info x00 Length 6e75 7831 0000 0f00 0014 000e 322e 362e 0029 0023 2331 2053 3020 3138 3a32 353a 0000 0c00 0678 3836
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 101
GroupID x33 '3' 3300 0055 3000 3200 0006 0000 0006 0000 3600 696f 6e20 5374 6420 3032 33 TokenID x30 '0' 0004 4546 4152 TokenID x31 '1' 0003 TokenID x32 '2' 0000 TokenID x33 '3' 0000 TokenID x34 '4' 0000 TokenID x35 '5' 0000 TokenID x36 '6' 0017 TokenID x37 '7' 001d 5665 7273 7320 4275 696c
ProducerInfo 000a 0004 4546 3300 0006 0000 0006 0017 3700 2e20 416e 6472
85 | 3..U0.....EFAR1..... | 2.....3.....4.....5. | ....6.....7..#..Vers | ion St. Andrews Buil | d 023 Name Info x00 Length 10 | ..EFAR DataSource Info x00 Length 6 | .. MajorVersion Info x00 Length 6 | .. MinorVersion Info x00 Length 6 | .. MaintLevel Info x00 Length 6 | .. BugFixLevel Info x00 Length 6 | .. BuildNumber Info x00 Length 6 | .. VerString Info x00 Length 35 696f 6e20 5374 2e20 416e 6472 6577 | ..Version St.Andrew 6420 3032 33 | s Build 023 Info x00 Length 4 | 4...
Info x00 Length 4152 3100 0006 0003 3400 0006 0000 3500 0023 001d 5665 7273 6577 7320 4275 696c
FILES
Use FILES to display summary file information for files on the local system. The default command displays all files in the curent directory. To constrain the display to specific files, you can supply a wildcarded name. This command can be shortened to FI. An alias for this command is DIR or FILEINFO.
Default Syntax Argument <directory>
The name of a directory or a wildcard for specific files. Note: On a Windows system, if any file or directory in the specified path contains spaces, the entire path must be enclosed within double quotation marks.
Example
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 102
FILTER
Use FILTER to filter the display based on one or more criteria.
You can string multiple FILTER commands together, separating each one with a semicolon, as in:
FILTER INCLUDE FILENAME fin.act*; FILTER RECTYPE 5; FILTER MATCH ALL
To avoid unexpected results, avoid stringing filter options together with one FILTER command. For example, the following would be incorrect:
FILTER INCLUDE FILENAME fin.act*; RECTYPE 5; MATCH ALL
Without arguments, FILTER displays the current filter status (ON or OFF) and any filter criteria that are in effect.
Comparison operators
For options that take comparison operators, standard operators may be used. These are:
Table 9 Operator Filter option comparison operators Example = EQ == < LT <= LE > GT >= GE <> NE !=
Equal
Less than Less than or equal Greater than Greater than or equal Not equal
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 103
Syntax
FILTER [INCLUDE] [EXCLUDE] <filter option> Where: <filter option> can be one of: { AUDITRBA <rba> [<comparison operator>] | CLEAR {<filter_spec> | ALL} | CSN | LogCSN [<comparison operator>] [<value>] ENDTIME <time_string> | FILENAME <name> [, <name>] | GGSTOKEN <token name> [<comparison operator>] [<token value>] | HEX <hex_string> [<byte_range>][, <hex_string> [<byte_range>]] [...] | INT16 <16-bit_integer> | INT32 <32-bit_integer> | IOTYPE <operation type> [, <operation type>] | MATCH {ANY | ALL} | OFF | ON | PROCESS <process_name> | RBA <byte address> [<comparison operator>] [...] | RECLEN <length> [<comparison operator>] | RECTYPE {<type_number> | <type_name>} | SHOW | STARTTIME <time_string> | STRING [BOTH] [B],<text> [<column_range>] [[B],<text> [<column_range>]] [...] | SYSKEY <system key> [<comparison operator>] [...] | TRANSIND <indicator> [<comparison operator>] | TYPE <type> | UNDOFLAG <type> [<comparison operator>] | USERTOKEN <token name> [<comparison operator>] [<token value>] }
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 104
Description
Filters based on the relative byte address of a commit record. For <comparison operator>, see Comparison operators on page 103. Removes filtering criteria.
ALL removes all filter criteria. <filter_spec> removes only the specified criterion. Specify any FILTER option, but not the value. The following example
is valid:
FILTER CLEAR STRING
Filters based on a commit sequence value. For <comparison operator>, see Comparison operators on page 103. Ends the filter at the last record written at the specified time. For the time string, use the format of:
[[yy]yy-mm-dd] [hh[:mm][:ss]]
ENDTIME <time_string>
Example:
ENDTIME 2005-01-12 00:00:00
Filters based on the name of a SQL table, or a group of names, with the name format being:
[catalog.]<owner>.<table> [catalog.]<owner>.<string>*
Also filters on the name of a NonStop data file, or a group of names, with the name format being:
<volume>.<subvolume>.<file> <volume>.<subvolume>.<string>* FILENAME is case-sensitive. FILENAME can be shortened to FILE or FI.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 105
Description
Filters based on a hex string and, optionally, a range of columns. To specify a range of columns, use the format of:
<start_column>:<end_column>
Example:
10:35
This option allows up to eight hex string and column arguments. Hex strings must be enclosed within quotes.
INCLUDE
Specifies that the filter will include the information specified with other options in the current FILTER statement. Can be shortened to INC. Specifies that the filter will exclude the information specified with other options in the current FILTER statement. Can be shortened to EXC. Filters based on a 16-bit integer. Use with 16-bit processors. Filters based on a 32-bit integer. Use with 32-bit processors. Filters based on the type of operation. A list of record types can be viewed with the SHOW RECTYPE command in Logdump. Up to 32 operation types can be specified with IOTYPE. Controls filtering response when multiple filters have been specified. Can be shortened to MAT or MA.
EXCLUDE
OFF
Disables record filtering. By default, filtering is disabled. An alias for this option is DISABLE. Enables record filtering. An alias for this option is ENABLE. Filters based on a relative byte address. Accepts either a 32bit or 64-bit value. Up to 32 specifications can be supplied. Filters based on a record length, in bytes. For <comparison operator>, see Comparison operators on page 103.
ON RBA <byte address> [<comparison operator>] [...] RECLEN <length> [<comparison operator>]
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 106
Description
To view the record type names and numbers, issue the SHOW RECTYPE command. (See page 118.)
SHOW
Displays filter settings. Same as using FILTER without any options. Starts the filter with the first record written at the specified time. For the time string, use the format of:
[[yy]yy-mm-dd] [hh[:mm][:ss]]
STARTTIME <time_string>
Example:
STARTTIME 2005-01-11 00:00:00
<text> filters based on a string. Enclose the string within quotes. <column_range> filters based on a range of columns. Use the
format of:
<start_column>:<end_column>
Example:
10:35
BOTH filters on both a string and a column range. [B] specifies a case-insensitive match. You can match up to
If the trail data is EBCDIC, issue the EBCDICDATA ON or ASCIIDATA OFF command before using FILTER STRING to ensure the correct matching.
STRING can be shortened to STR. SYSKEY <system key> [<comparison operator>] [...]
Filters based on a NonStop source key. Accepts either a 32-bit or 64-bit value. Up to 32 specifications can be supplied.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 107
Description
Filters based on the TransInd field of the record header. Valid values:
0 = start of transaction 1 = middle of transaction 2 = end of transaction 3 = only record in transaction
For example, to filter for the end of a transaction, use the following command, including the spaces in the syntax:
FILTER INCLUDE TransInd > = 2
Filters based on the NonStop undo flag. The undo flag is set for records that are undone when a TMF transaction is aborted. Normally, UndoFlag is set to zero, but if the record is the backout of a previously successful operation, then UndoFlag will be set to 1. An undo that is performed by the disc process because of a constraint violation is not marked as an undo. For <comparison operator>, see Comparison operators on page 103.
<token name> is the name of any token that is defined with the TOKENS clause of a TABLE statement of the Extract parameter file. It is not case-sensitive. <token value> is either a constant that is enclosed within double quotes or the result of a GoldenGate columnconversion function, depending on what was specified in the TOKENS clause for <token name>.
Example 1
Example 2
The following filters for a data file name and for a relative key 19446, which has a hex value of 00004bf6. Because MATCH ALL is used, a record must meet all of the filter specifications to be included in the filter.
FILTER INCLUDE FILENAME $QA01.QAESRC.ACCT* FILTER INCLUDE HEX 00004bf6 0:3 FILTER MATCH ALL
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 108
Example 3
The following filters for tables that start with ACC, except for the ACCDET table, and for records that contain a timestamp between the specified start and stop times. By default, if a record matches any of the INCLUDE specifications, it is included in the filter.
FILTER FILTER FILTER FILTER INCLUDE EXCLUDE INCLUDE INCLUDE FILENAME SALES.ACC* FILENAME SALES.ACCDET STARTTIME 2005-01-11 17:30:15 ENDTIME 2005-01-11 19:00:00
Example 4
The following shows filter options with multiple specifications. By default, a record that matches any of these specifications will be included in the filter. Note that in the STRING filter, two of the criteria are not case-sensitive, while one is, and the filter is confined to a column range.
FILTER INCLUDE IOTYPE insert,update,delete FILTER INCLUDE STRING bString1 string2 bString3 25:50 FILTER INCLUDE FILENAME $QA01.QAESRC.ACCT1, $QA01.QAESRC.ACCT2, $QA01.QAESRC.ACCT3
GHDR
Use GHDR to control whether or not the record header is displayed with each record. Each record contains a header that includes information about the transaction environment. Without arguments, GHDR displays the status of header display (ON or OFF).
Default Syntax OFF GHDR {ON | OFF}
HEADERTOKEN
Use HEADERTOKEN to control whether or not header token indicators are displayed with each record. The header token indicators are the following:
G record header (begin of record) H header area D data area T GoldenGate internal token U user token area (does not display if user tokens are not in use) Z end of record
Without arguments, HEADERTOKEN displays the status of header token indicators (ON or OFF).
Default OFF
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 109
Enables the display of header tokens. Disables the display of header tokens. Provides detailed token values.
HEADERTOKEN, without DETAIL
Example 1
Example 2
HELP
Use HELP to view the syntax of Logdump commands.
Default Syntax
None
HELP
HISTORY
Use HISTORY to view the most recently issued Logdump commands since the session started, or to reset the command count starting at line 1 again. HISTORY can be shortened to HIST.
NOTE Default Syntax You can use the FC command to re-execute a command in the list. See page 92.
Argument <n>
Description
Returns the specified number of previously issued commands, where <n> is any positive number.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 110
Description
Deletes the command history buffer and reverts the command line to 1.
HISTORY 3
INTERPRETINTERVAL
Use INTERPRETINTERVAL to display a 64-bit Julian time interval in the format of dayshh:mm:ss.ms.us.
Default Syntax Argument <interval string>
None
INTERPRETINTERVAL <interval string> Description
Example
INTERPRETINTERVAL 1234567
INTERPRETTIMESTAMP
Use INTERPRETTIMESTAMP to display a 64-bit Julian timestamp as an ASCII value.
Default Syntax Argument <timestamp>
None
INTERPRETTIMESTAMP <timestamp> Description
A JULIANTIMESTAMP value.
Example
INTERPRETTIMESTAMP 211976584185800569
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 111
LOG
Use LOG to start and stop the logging of Logdump sessions. When enabled, logging remains in effect for all sessions of Logdump until disabled with the LOG STOP command. Without arguments, LOG displays the status of logging (ON or OFF). An alias for LOG is OUT.
Default Syntax Argument <file_name>
Disabled
LOG <file_name> | STOP} Description
Specifies the name of the log file. Specify a full path name to store the file in a directory other than the current working directory. Stops logging.
STOP
Example
LOG /home/ggs/dirrpt/logdumpout.txt
NEXT
Use NEXT to display the next record or records in the file. The default displays only the next record. NEXT can be shortened to N. An alias for NEXT is RECORD.
Default Syntax Argument <n>
Example
NEXTTRAIL
Use NEXTTRAIL to close an open trail file and open the next one in the sequence. An alias for NEXTTRAIL is NT.
Default Syntax
None
NEXTTRAIL
NOTIFY
Use NOTIFY to display the number of records scanned, the trail position, and the record timestamp at specified intervals when using COUNT and records are being suppressed from display through filtering options. An alias for NOTIFY is NOTIFYINTERVAL.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 112
Instead of displaying each notify interval on a separate line, you can configure Logdump to simply update a single line with each new scan result. See SCANSCROLLING on page 118.
Default Syntax Argument <interval>
None
NOTIFY <interval> Description
Example
OBEY
Use OBEY to process a file that contains a list of Logdump commands. OBEY is useful for executing commands that are frequently used in sequence.
OBEY can be shortened to O. An alias for OBEY is SOURCE. Default Syntax Argument <file name>
None
OBEY <file name> Description
The fully qualified name of the file containing the list of commands.
Example
OBEY ./ldcommands.txt
The preceding command executes a file that might look something like this:
ghdr on usertoken on detail filter enable filter clear filter match all
OPEN
Use OPEN to open a trail file or extract file in Logdump. Without arguments, the command displays the name of the file that is currently open. Aliases for OPEN are FROM and LOGTRAIL.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 113
None
OPEN <file_name> Description
<file_name>
The fully qualified path name of the trail file or extract file to be opened. To specify a trail file, specify the trail name (a two-character prefix) and the sequence number, for example jd000000.
Example
OPEN /home/ggs/dirdat/jd000000
POSITION
Use POSITION to set the read position in the file. The position of a record in the file is noted in the record header in the AuditPos field. Without options, POSITION displays the current read position. Options let you specify an exact position. After you set the position, issue the NEXT command to view the record at that position.
POSITION can be shortened to POS. Default Syntax Argument <bytes>
None
POSITION [<bytes> | {0 | FIRST}] Description
Specifies the number of bytes into the file at which to read. Use the NEXT command to view the specified record. Positions Logdump at the beginning of the file.
0 | FIRST
Example
POS 77580548
RECLEN
Use RECLEN to control how much of the record data is displayed. You can use RECLEN to control the amount of scrolling that must be done when records are large, while still showing enough data to evaluate the record. Data beyond the specified length is truncated.
Default Syntax Argument <n>
140 bytes
RECLEN <n> Description
Example
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 114
SAVE
Use SAVE to write a subset of the records to a new trail or extract file. By saving a subset to a new file, you can work with a smaller file that is easier to debug. Saving to another file also enables you to extract valid records that can be processed by GoldenGate, while excluding records that may be causing errors. To set the version of the trail or file (to old or new format), use the TRAILFORMAT command.
Default
None
SAVE <file_name> [!] {<n> records | <n> bytes}[NOCOMMENT] [OLDFORMAT | NEWFORMAT] [TRANSIND <indicator>] [TRUNCATE]
Argument <file_name>
Description
The name of the new file. To specify a trail file, specify the two-character trail name and a sequence number, for example rt000001. Overwrites the specified file, if the same file already exists. First a purge is done, and then the specified records are saved to the file. Specifies either a number of records or a number of data bytes to write to the new file. The <n> number of records or bytes are taken forward from the current position in the file. You can change the position with the POSITION command. See page 114. Suppresses the leading and trailing comment records that are placed by default in the new file. These records describe the context of the file. The begin comment record contains source trail information and the position where the save started. The end comment record identifies the end of the saved data. These headers are useful to separate different sets of records that are saved to the same file, but can be omitted. Writes the data in either the current trail format (NEWFORMAT, the default) or the format that was used for GoldenGate versions 6.0 and earlier (OLDFORMAT). Sets the TransInd header field in the records written to one of the following:
FIRST MIDDLE END ONLY
NOCOMMENT
OLDFORMAT | NEWFORMAT
TRANSIND <indicator>
This allows you to reorder records in a transaction. TRANSIND applies to all records written by a SAVE command.
TRUNCATE
Example
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 115
SCANFORENDTRANSACTION
Use SCANFORENDTRANSACTION to scan for a record that has a transaction indicator of 2 or 3, as shown in the TransInd field of the header. When one of those indicators is found, Logdump displays the first record of the next transaction. The indicators represent the following:
None
SCANFORENDTRANSACTION
SCANFORHEADER
Use SCANFORHEADER to go to the next record header. Before using this command, use the GHDR ON command to show record headers (see page 109). SCANFORHEADER can be shortened to SFH.
Default Syntax Argument PREV
None
SCANFORHEADER [PREV] Description
SCANFORRBA
Use SCANFORRBA to scan for the record at a relative byte address specified by the AuditRBA field of the record header. Before using this command, use the GHDR command to show record headers (see page 109). SCANFORRBA can be shortened to SFR.
Default
None
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 116
Syntax Argument
Specifies the relative byte address to find. Constrains the search to an Enscribe or SQL data file. A file name is required even if you are searching a file that is open in Logdump.
Example
SCANFORTIME
Use SCANFORTIME to scan for a record that contains a specific timestamp. The timestamp is contained in the IO Time field of the record header. Before using this command, use the GHDR command to show record headers (see page 109). SCANFORTIME can be shortened to SFTS.
Default Syntax Argument <time_string>
None
SCANFORTIME <time_string> [, <name>] Description
Scans for a specific timestamp. For the time string, use the format of:
[[yy]yy-mm-dd] [hh[:mm][:ss]]
<name>
Constrains the search to a specific table name, or a group of names specified with a wildcard. SCANFORTIME 2005-10-27 14:33:57
Example
SCANFORTYPE
Use SCANFORTYPE to scan for the next record of the specified type. SCANFORTYPE can be shortened to SFT.
Default Syntax Argument <type_name> | <type_number>
None
SCANFORTYPE {<type_name> | <type_number>} Description
Specifies the type of record to search for, either by type name or type number. To view a list of record types and their associated numbers, use the SHOW RECTYPE command (see page 118).
Example
Both of the following commands return the same result: They display commit records.
SCANFORTYPE Commit SFT 2
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 117
SCANSCROLLING
Use SCANSCROLLING to configure Logdump to update a single line after COUNT scans when NOTIFY is enabled. Otherwise, each scan notification appears on a different line. See NOTIFY on page 112 for more information.
Default Syntax Argument ON OFF OFF SCANSCROLLING {ON | OFF} Description
Enables the use of a single line for count notification results. Disables the use of a single line, causing a separate line to be used for each notification.
SHOW
Use SHOW to display internal Logdump information, including the current Logdump environment, a list of GoldenGate record types, and current filter settings. SHOW can be shortened to SH or SHO.
Default Syntax
None
SHOW [ENV] [FILTER] [RECTYPE] Description
Argument ENV
Displays the current Logdump environment. Same as the ENV command (see page 91). Displays current filter settings. Displays a list of GoldenGate record types that can be displayed with Logdump.
SHOW FILTER
FILTER RECTYPE
Example 1
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 118
This shows results similar to the following. (This list might not reflect all possible record types. New types are added when needed to support new functionality.)
LogTrail record types 1 - Abort 3 - Delete 5 - Insert 7 - TMF-Shutdown 9 - TransRelease 11 - UpdateComp 13 - FileCreate 15 - FieldComp 17 - AuxPointer 19 - NetworkAbort 89 - SQL/MX DDL OP 100 - GGSPurgedata 102 - GGSCreateFile 104 - GGSRenameFile 107 - GGSControl 160 - DDL OP 117 - GGSKeyFieldComp32 116 - LargeObject 133 - GGSAlterSequence 150 - RestartAbend 152 - RecoveryEnd 201 - GGSFileClose 203 - GGSExtractTS 205 - GGSComment 251 - LibCloseTrace 253 - LoggerCloseTrace 249 - LoggerAddedStats
2 - Commit 4 - EndRollBack 6 - Prepared 8 - TransBegin 10 - Update 12 - FileAlter 14 - FilePurge 16 - FileRename 18 - NetworkCommit 20 - CurrentPos 90 - GGSSQLCol 101 - GGSPurgeFile 103 - GGSAlterFile 105 - GGSSetmode 106 - GGSChangeLabel 115 - GGSKeyFieldComp 161 - RecordFragment 132 - GGSCreateSequence 134 - GGSDropSequence 151 - RestartOK 200 - GGSBulkio 202 - GGSLoggerTS 204 - GGSCollectTS 250 - LibOpenTrace 252 - LoggerOpenTrace 254 - LoggerAddedInfo
SKIP
Use SKIP to skip the specified number of records.
Default Syntax Argument <n>
None
SKIP <n> Description
Example
TIME
Use TIME to display the current time in local and GMT formats.
Default
None
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 119
Syntax
TIME
TIMEOFFSET
Use TIMEOFFSET to set the Logdump time format. Without arguments, TIMEOFFSET displays the current time offset. Options enable you to set the time to the local time, Greenwich Mean Time (GMT), or a specific offset from GMT. The specified time format applies to the timestamps shown in records as well as any Logdump commands that accept a time string argument.
Default Syntax Argument LOCAL GMT GMT + <hh[:mm]> LOCAL TIMEOFFSET {LOCAL | GMT | GMT + <hh[:mm]> | GMT - <hh[:mm]>} Description
Sets the time to that of the local system. Sets the time to Greenwich Mean Time (GMT). Sets the time ahead of GMT by the specified number of hours and, optionally, minutes. Sets the time behind GMT by the specified number of hours and, optionally, minutes.
GMT - <hh[:mm]>
Example
TRAILFORMAT
Use TRAILFORMAT to set the version of the GoldenGate trail or extract file that is being saved when using the SAVE command.
Default Syntax Argument NEW OLD NEW TRAILFORMAT {NEW | OLD} Description
Sets the format to that used by GoldenGate version 6.0 and later. Sets the format to that used by GoldenGate versions earlier than 6.0.
TRANSBYTELIMIT
Use TRANSBYTELIMIT to prevent normal-sized transactions from being tracked in the transaction table specified with the TRANSHIST command. It sets a lower boundary for the number of bytes in a transaction and should be set to represent a normal-sized transaction for the environment being evaluated with Logdump. Setting a boundary reduces the
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 120
amount of data that is stored and, consequently, the amount that must be reviewed when troubleshooting.
Default Syntax Argument <n>
10000 bytes
TRANSBYTELIMIT <n> Description
Example
TRANSHIST
Use TRANSHIST to keep track of the size of transactions in a trail or file. Logdump tracks the transactions in an internal history table in descending order according to the number of bytes of data in each one. When the history table is full, the smallest transaction is removed to allow a larger transaction to be added to the list. Use TRANSHIST in conjunction with other Logdump commands to determine whether or not your applications generate large transactions and to identify their relative size. This information can be used when deciding how to group tables into different processing groups for faster throughput. For more information, see page 81.
NOTE You can use the SEND EXTRACT command with the SHOWTRANS option to view a list of long-running transactions. Other options enable you to control whether those transactions are ignored or processed by GoldenGate.
Sets the size of the history table, in bytes. Valid values are 0 through 200 bytes. A value of 0 means that no transaction history is maintained.
TRANSHIST 150
Example
TRANSRECLIMIT
Use TRANSRECLIMIT to prevent normal-sized transactions from being tracked in the transaction table specified with the TRANSHIST command. It sets a lower boundary for the number of records in a transaction and should be set to represent a normal-sized transaction for the environment being evaluated with Logdump. Setting a boundary reduces the amount of data that is stored and, consequently, the amount that must be reviewed when troubleshooting.
Default
100 operations
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 121
Example
USERTOKEN
Use USERTOKEN to control whether or not user token data is displayed with each record. A user token is data specified by a GoldenGate user that is stored in the record header and can be mapped to a target column or used for some other purpose during GoldenGate processing. Without arguments, USERTOKEN displays the status of user token display (ON or OFF). With the ON option, the name of the token and its length are displayed. The DETAIL option shows the actual token data.
Figure 10
Enables the display of user tokens. Disables the display of user tokens. Displays the token data.
WRITELOG
Use WRITELOG to write text to the session log. Before using this command, start logging with the LOG command (see page 112).
Default Syntax Argument <text>
None
WRITELOG <text> Description
Example
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 122
X
Use X to execute a program from within Logdump. When you exit the program, the Logdump prompt returns.
Default Syntax Argument <command> <string>
None
X <program> [<string>] Description
Example
The following series of commands and output shows how you can exit Logdump, issue other commands from the shell or within GGSCI, and then return to the Logdump command line.
Logdump 696 >x ggsci GoldenGate Command Interpreter Version ..... GGSCI (sysa) 1> GGSCI (sysa) 2> GGSCI (sysa) 3> GGSCI (sysa) 4> Logdump 697 > status er * start er * info er * exit
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 123
CHAPTER 5
ERR-STARTUP-PARAM 101
Cause Recovery
An unrecognized or missing parameter was encountered during program startup. Correct the parameter file and restart.
ERR-FILE 103
Cause Recovery
A file error was encountered during an operation on the file or process. Examine the error number to determine the exact nature of the error (for example, a security violation or disk is full). Attempt to correct the error.
ERR-MEM-ALLOC 105
Cause Recovery
There was an internal problem allocating memory. Restart the process. If the problem persists, contact GoldenGate Support.
ERR_BAD_SECURITY 108
Cause Recovery
An error occurred processing the CMDSEC file, or there was unauthorized access to a GoldenGate program. Modify security settings in the CMDSEC file as necessary.
ERR-SQL 109
Cause Recovery
An error was encountered during a SQL operation. Dependent on the type of error. If the error occurred during a replicated operation, error detail can be found in the Replicat discard file. Error detail indicating the exact nature of the database error can generally be found in the report file for the process as well (using the VIEW REPORT command).
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 124
ERR-COMPILE-WHERE 111
Cause Recovery
Incorrect syntax was specified for a WHERE, COLMAP, or KEYCOLS statement in the parameter file. Correct the syntax and restart the process.
ERR-EXTRACT-FILE 112
Cause Recovery
An error was encountered relating to processing a trail file. Further information is included about the cause in the error message. Dependent on the error message. If this is a file error (such as a security violation), correct the problem and restart the process.
ERR_USER_EXIT 117
Cause Recovery
GoldenGate failed to load or execute a user exit. Make certain that the correct shared library and function name are specified in the parameter file
ERR-DATABASE 118
Cause Recovery
An error was encountered during a database operation (possibly an ODBC operation or a native database error). Dependent on the type of error. If the error occurred during a replicated operation, more error detail can be found in the Replicat discard file. Error detail indicating the exact nature of the database error can generally be found in the report file for the process as well (using the VIEW REPORT command).
ERR_DATA_TYPE 120
Cause Recovery
Unrecognized data type. Make certain that tables being processed only contain supported data types.
ERR_ASCII_FORMATTING 121
Cause Recovery
An error occurred converting a record to ASCII format. Examine the error detail to determine the exact nature of the problem. Use this information to modify the ASCII formatting options in the parameter file.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 125
ERR-PROCESS-CREATE 130
Cause Recovery
Manager encountered an error trying to create another process. The error message provides more detail as to the cause. Fix the underlying cause of the problem. The cause may be a security violation, a missing program file or other reason.
ERR-TCPIP 150
Cause Recovery
A TCP/IP error occurred while attempting to write extracted data to a remote system. If the error is Connection Refused, make sure that the Manager process has been started on the remote system. Also, make sure the PORT argument of the RMTHOST parameter matches the remote Manager PORT parameter. TCP/IP errors usually are retried automatically for a period of time.
ERR-CHECKPOINT 151
Cause Recovery
A file error was encountered during an operation on the checkpoint file. Examine the error number to determine the exact nature of the error (for example, a security violation or disk is full). Attempt to correct the error and restart the process.
ERR_TRACE 152
Cause Recovery
An error occurred opening or writing to a trace file. Evaluate the error detail to determine the exact nature of the problem. For example, there may be a permissions problem, or the disk may be full.
ERR-BAD-RECS 160
Cause
An error was encountered replicating one or more records, and the record was discarded. By default, this message is generated only once per every 100 records discarded. Examine the discard file to determine the exact cause of the error. The cause of this error is either a data mapping problem or a database error (such as a duplicate or missing record). The discard file also contains the contents of the record, which can be used to manually recover the data, if desired. Recovery is application-specific and action is not always required.
Recovery
ERR-MAX-DISCARD-RECS 161
Cause Recovery
The discard file has reached its maximum size and cannot hold any more discard records. Several actions can be taken:
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 126
Ignore the condition and allow processing to continue. Increase the MEGABYTES option of the DISCARDFILE parameter, and restart the process.
Specify DISCARDROLLOVER (which schedules the creation of new discard files on a periodic basis), and restart the process.
ERR-MAPPING-TO-TANDEM 162
Cause Recovery
The Extract process was unable to map data from a row in the source database into the corresponding table or file format on the NonStop system. The discard file contains the contents of the source record. Recovery procedures can include:
Specifying compatible columns in the map and restarting replication. Manually fixing the data in the source database. Ignoring the problem.
ERR-NT-INTERNAL 170
Cause Recovery
A Windows error was encountered by Manager. Fix the error condition indicated by the reported message and restart Manager.
ERR_BAD_COMMAND 171
Cause Recovery
An invalid command was received during inter-process communications. Make certain that source and target versions of GoldenGate are the same.
ERR_INTERNAL_RESOURCE 172
Cause Recovery
ERR_EXCEEDED_BUFFER_SIZE 173
Cause Recovery
The size of a record exceeded the maximum internal buffer size. Remove the table with the large record from replication or use a parameter to warn only.
ERR_THREAD_ERROR 180
Cause Recovery
An error occurred using threads. Make certain that threads are supported by the platform.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 127
ERR_TRANSACTION_ERROR 181
Cause Recovery
ERR-PROCESS-ABEND 190
Cause Recovery
The Extract or Replicat process is terminating abnormally (abending). Examine the previous error messages and the report file (using VIEW REPORT) to determine the exact cause.
ERR_LIBRARY_ABEND 191
Cause Recovery
WRN-RECORD-VERSION 208
Cause Recovery
A record being processed by the Collector process does not match the definition of the record specified in the definitions file (the d parameter to Collector). If the definition of a source table or file has changed, run the DEFGEN program on the NonStop Server again, export the new definitions to the target platform, and restart Extract and/or Collector.
WRN_FILE_NO_AUDIT 210
Cause Recovery
Transactions for this table are not being captured in the transaction log. Modify the table or database configuration to capture transactions for the table.
WRN-MISSING-INFO 216
Cause Recovery
Replicat could not evaluate a WHERE clause because one or more columns was missing from the source record. Specify a WHERE clause that includes only columns that are always present in the record (such as primary key columns). Otherwise, you may need to specify uncompressed updates on the source platform.
WRN-FILE 217
Cause Recovery
A non-fatal file error was encountered during an operation on a file. Examine the error number to determine the exact nature of the error (for example, a security violation or disk is full). Attempt to correct the error.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 128
WRN-SQL 218
Cause Recovery
A non-fatal SQL error occurred. Dependent on the type of error. Evaluate the error detail. If the error occurred during a replicated operation, error detail can be found in the Replicat discard file. Error detail indicating the exact nature of the database error can generally be found in the report file for the process as well (using the VIEW REPORT command).
WRN-MISSING-TRANS-BEGIN 221
Cause
Replicat cannot process the current transaction because the beginning of the transaction is missing. Processing resumes at the next transaction. Possibly, a previous trail file was purged. Examine the discard file for the rejected data and use manual procedures to recover the data.
Recovery
WRN-MAPPING-PROBLEM 222
Cause
A problem was encountered while mapping a source record to a target format. Usually this means that the source record has unexpected or incorrect data in it. The record was discarded. Depending on how error handling is configured in the parameter file, the process continues, aborts the transaction, or ends abnormally. Examine the discard record to determine what may have caused the mapping problem. Use manual procedures to extract and/or replicate the necessary data, or alter the mapping specification and restart.
Recovery
WRN_TRANS_ABORT 223
Cause Recovery
A database recovery was detected and all in-progress transactions are aborted. None required.
INF-ROLLOVER 224
Cause Recovery
A new trail file was opened by Extract. None. Used to trace processing activity.
WRN-PURGE-EXTRACT 226
Cause Recovery
The Manager process purged a trail file that is no longer required by any replication processes. None required. Informational.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 129
WRN-DATABASE 228
Cause Recovery
An error was encountered during a database operation (possibly an ODBC operation or a native database error). Dependent on the type of error. If the error occurred during a replicated operation, error detail can be found in the Replicat discard file. Error detail indicating the exact nature of the database error can generally be found in the report file for the process as well (using the VIEW REPORT command).
WRN-START-EXTRACT-FAILED 230
Cause Recovery
Manager failed to start Extract for the reason included as part of this message (usually a missing file or security violation). Correct the condition and restart Extract.
WRN-START-REPLICAT-FAILED 231
Cause Recovery
Manager failed to start Replicat for the reasons included as part of this message (usually a missing file or security violation). Correct the condition and restart Replicat.
WRN_MGR_START_BATCH_FAILED 232
Cause Recovery
The Manager process was unable to start a child process to perform background tasks. Specify NOUSETHREADS in the Manager parameter file to force the parent process to perform background tasks.
WRN-STMT-MAX 240
Cause Recovery
Replicat for Oracle failed because an internal Oracle statement limit was reached. Several actions can be taken:
Consult the Oracle user documentation for instructions on how to increase the number of open cursors held by a program at any given time. Specify the MAXSQLSTATEMENTS parameter in the Replicat parameter file (down to a value of 20 or lower). Specify the NODYNSQL parameter (which allows replication to continue at a slower rate).
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 130
WRN-PROCESS-ABENDED 250
Cause Recovery
This message is reported periodically by Manager when an Extract or Replicat process has ended abnormally. Review the cause of the abnormal termination in the process report file (using VIEW REPORT) and the error log. Correct the error condition and restart processes as appropriate.
WRN-FORCE-STOP 251
Cause
This message is generated by Extract and Replicat whenever the STOP command is issued with the ! (exclamation point) option, which causes Extract and Replicat to abort the current transaction, if one exists. None. Used to trace processing activity.
Recovery
WRN_FORCE_STOP 251
Cause Recovery
WRN-MGR-LAG-THRESHOLD 260
Cause
This message is generated by Manager whenever lag for the specified process exceeds a configured threshold. It is a warning that a problem may exist in the system. Investigate the cause of possible performance problems (such as a process running at low priority or a faulty network).
Recovery
WRN-COLL-OPEN-FILE-ERR 261
Cause Recovery
A file error was encountered when Collector opened a file. Examine the error number to determine the exact nature of the error (for example, a security violation or disk is full). Attempt to correct the error and restart Extract on the source system.
WRN_MGR_LAG_THRESHOLD 260
Cause Recovery
The process lag time has exceeded the warning threshold. Evaluate the process in question to determine the cause of the lag.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 131
WRN-SERVICE-RQST-STOP 270
Cause Recovery
Manager has received a request from the Windows Control Services Manager to stop. None. Used to trace processing activity.
WRN_SERVICE_RQST_PAUSE 271
Cause Recovery
INF_SERVICE_RQST_CONT 272
Cause Recovery
WRN-SERVICE-RQST-SHUTDOWN 273
Cause Recovery
Manager has received a request from the Windows Control Services Manager to stop because of a general system shutdown. None. Used to trace processing activity.
WRN_WIN32_API_CALL_ERR 274
Cause Recovery
WRN_COLUMN_VALUE_TRUNCATED 275
Cause Recovery
A column value was truncated because of a limitation on the maximum column length. Increase the maximum column length by using the MAXLONGLEN parameter.
WRN_MEM_THRESHOLD 276
Cause Recovery
GoldenGate is approaching memory threshold for processing stored procedures. Reduce the number of parameters passed by the stored procedure.
WRN_FILTER_NOT_PASSED 280
Cause Recovery
The FILTER clause did not pass, so the record was ignored. Modify the FILTER clause.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 132
INF-PROCESSED-MARKER 300
Cause Recovery
An audit marker record created by GGSCI was processed by Replicat. None. Markers are used to mark application-specific points of interest within the audit trails on NonStop Server.
INF_MGR_STARTING_PROCESS 302
Cause Recovery
INF-USER-CMD 301
Cause Recovery
A command was received by the Extract or Replicat process from GGSCI. None. Used to trace processing activity.
INF-TCP-ERR-RECOVERED 305
Cause Recovery
A TCP/IP operation that previously encountered an error was retried successfully. None. Indicates that processing has resumed normally after problem correction.
INF-EXTRACT-STARTED 310
Cause Recovery
INF-EXTRACT-RESTARTED 311
Cause Recovery
Extract was restarted by Manager after ending abnormally. None. Used to trace processing activity.
INF-EXTRACT-STOPPED 312
Cause Recovery
Extract stopped normally (at the request of GGSCI or by parameter specification). None. Used to trace processing activity.
INF-REPLICAT-STARTED 320
Cause Recovery
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 133
INF-REPLICAT-RESTARTED 321
Cause Recovery
Replicat was restarted by Manager after ending abnormally. None. Used to trace processing activity.
INF-REPLICAT-STOPPED 322
Cause Recovery
Replicat stopped normally (at the request of GGSCI or by parameter specification). None. Used to trace processing activity.
INF-MGR-STARTED 330
Cause Recovery
INF-MGR-STOPPED 331
Cause Recovery
Manager stopped normally (at the request of GGSCI or by parameter specification). None. Used to trace processing activity.
INF-PURGE-OLD-ROWS 340
Cause Recovery
Manager deleted unneeded rows from the GoldenGate log table (as the result of the PURGEOLDEXTRACTS parameter). None. Informational.
INF-PROCESS-RUNNING 350
Cause Recovery
Manager is reporting that a process is alive. None required. Used to trace processing activity.
INF-MGR-PROCESS-DOWN-OK 351
Cause Recovery
Manager is reporting that a process is down and terminated normally. None required. Used to trace processing activity. If this message is marked critical, restarting the process may be required.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 134
INF-MGR-LAG-THRESHOLD 360
Cause Recovery
This message is generated by Manager whenever lag for the specified process exceeds a configured threshold. This message is most often generated for status only and usually does not require recovery.
INF-MGR-AUTOSTART 361
Cause
This message is generated by Manager at startup if the AUTOSTART parameter was configured. (AUTOSTART automatically starts specified GoldenGate processes.) None. Informational.
Recovery
INF-MGR-BOOTDELAY 362
Cause
This message is generated by Manager at startup if the BOOTDELAYMINUTES parameter was configured. (BOOTDELAYMINUTES delays further Manager activities on Windows until the system has been running for a specified number of minutes.) None. Informational.
Recovery
INF_RETRY_REP_OP 363
Cause Recovery
INF_FILE_ERR_RECOVERED 364
Cause Recovery
GGS_EVT_COLL_OPENING_FILE 370
Cause Recovery
GGS_EVT_COLL_CLOSING_FILE 371
Cause Recovery
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 135
GGS_EVT_COLL_TERMINATING 372
Cause Recovery
GGS_EVT_COLL_LISTENING 373
Cause Recovery
INF_CTRL_EVENT_RECEIVED 374
Cause Recovery
GoldenGate received a CLOSE, SHUTDOWN, or LOGOFF event from Windows. No action required.
INF_TRACE_STARTED 375
Cause Recovery
INF_TRACE_STOPPED 376
Cause Recovery
GGS_EVT_COLL_TRUNCATED_FILE 377
Cause Recovery
GGS_EVT_COLL_CLOSE_ACTION 378
Cause Recovery
The predefined close action has been executed for a batch file on the IBM z/OS system. No action required.
INF_GGSCI_CMD 399
Cause Recovery
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 136
Extract was unable to open the redo log file. Indicates that Extract does not have read permission on the file. Grant operating system permission to the user running Extract to read the redo log files. This user also must have read permission to the archive files.
EXTRACT READ, KEY COLUMN(S) MISSING FROM UPDATE FOR ... 509
Cause Recovery
Key-column values for updates are not available. Enable replication or supplemental logging (depending on the database). See GoldenGate for Windows and UNIX Administrator Guide for database-specific requirements for making key values available to GoldenGate.
ERR_THREAD 990
Cause Recovery
An error occurred using threads. Make sure threads are supported by the platform.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 137
Glossary
...............
The following explains terminology contained in this manual.
Term Definition
abend
Abnormal end. The failure or unexpected termination of a process running on a computer system. The values of a row in a database after an insert or update is performed. An Extract group that operates on a target system that resides within a more secure network zone than the source system. The purpose of the alias Extract is to initiate TCP/IP connections from the target to the less-trusted source. Once a connection is established, data is processed and transferred across the network in the usual manner by a passive Extract group that operates on the source system. The default method of writing to the trail, whereby Extract appends reread data to the trail file after a failure, instead of overwriting the old data. A mode of operation for Extract, where the process is configured to read exclusively from the archived transaction logs on a production or standby database system. A file on a NonStop Server system that stores modifications made to a database for the purpose of replication and recovery. A one-time processing run that has a distinct beginning and an end, as opposed to continuous processing that does not have a specific end point, such as online change synchronization. The values that exist in a row in a database before a SQL operation is performed on that row. Permits load distribution across multiple databases and servers where, in most cases, different users can change the same sets of data and those changes are synchronized by GoldenGate. See LOB. The GoldenGate process that executes a user exit routine.
append mode
audit trail
batch run
before image
bidirectional synchronization
BLOB caller
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 138
Glossary
cascading synchronization
Term
Definition
cascading synchronization
A GoldenGate configuration in which data is sent from a source system to one or more intermediary systems and, from those systems, to one or more other systems in a synchronized state. The process of synchronizing data changes made to a database on one system with a similar set of data on one or more other systems. Internal indicators that record the current read and write position of a GoldenGate process. Checkpoints are used by the Extract and Replicat processes for online change synchronization to ensure data accuracy and fault tolerance. A file on disk that stores the checkpoint generated by GoldenGate processes. A table created in the target database that maintains Replicat checkpoints, used optionally in conjunction with a standard checkpoint file on disk. See LOB. A GoldenGate file that stores rules for GGSCI command permissions. The process that receives data from the Extract process over TCP/IP and writes it to a trail or extract file on the target system. Errors that occur when data changes that are replicated by GoldenGate are applied to a target table, but the target row is either missing or is a duplicate. One among a set of attributes assigned to an entity that is described by a database table. For example, there can be columns for the name, address, and phone number of the entity called employees. Built-in GoldenGate processing functions that perform comparisons, tests, calculations, and other processing for the purpose of selecting and manipulating data. See map. A method of logging SQL update operations by which only column values that changed as the result of the update are logged to the transaction log. Instructions used in bidirectional synchronization that provide processing and error-handling rules in the event that the same SQL operation is applied to the same row in two or more databases at (or about) the same time.
change synchronization
checkpoints
checkpoint file
checkpoint table
collisions
column
column-conversion functions
conflict resolution
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 139
Glossary
consolidated synchronization
Term
Definition
The process of replicating different data from two or more databases to one central database, such as in data warehousing. See transformation. See source definitions file and target definitions file. A secondary Extract process that reads from an extract file or trail. The trail is populated by a primary Extract process that reads from the data source. The container of the data changes that are to be processed by GoldenGate. A data source can be:
data source
A DSN defines an ODBC connection to a database. A DSN consists of a database name, the database directory, the database ODBC driver name, database authentication information, and other information depending on the database. External applications, such as GoldenGate, require a DSN, because a DSN enables an application to connect to a database without having to encode the required information within the application program. The three types of DSN are
A system DSN can be used by any entity that has access to the machine. It is stored within the system configuration. A user DSN can only be used by a specific user. It is stored within the system configuration. A file DSN is stored in a text file with a .dsn extension. It can be shared among different systems where the required ODBC driver is installed.
data type
An attribute of a piece of data that identifies what kind of data it is and what kinds of operations can be performed on it. For example, an integer data type is a number, and a character data type contains letters. Data Definition Language. Data that defines the structure of a database, including rows, columns, tables, indexes, and database specifics such as file locations, users, privileges, and storage parameters. A GoldenGate utility that generates table-creation instructions based on the source tables. These instructions are used to create target tables. A GoldenGate utility that generates a data definitions file.
DDL
DDLGEN
DEFGEN
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 140
Glossary
Director
Term
Definition
Director
Also known as GoldenGate Director, a graphical user interface that enables users to monitor and manage GoldenGate processes. The components of Director are:
Director Administrator: A utility used by administrators to define users and instances of GoldenGate. Director Server: A software module that gathers data about the GoldenGate processes. Director Client: Software installed on a users system as an interface to
Director.
Director Web: A browser-based user interface to Director (requires no software to be installed).
discard file
A GoldenGate file containing information about SQL operations that failed. This file is created when a record cannot be processed, but only if the DISCARDFILE parameter exists in the parameter file to specify the location for the file. Data Manipulation Language. Retrieves and manipulates data in a database. In the case of SQL, the actions are select, insert, update, and delete. See data source name (DSN). A Collector process that the Manager process starts automatically, as opposed to a static Collector. A GoldenGate utility that distributes GoldenGate system error messages that originate on Windows and other supported operating systems to the EMS (Event Management Subsystem) server on the NonStop Server. A GoldenGate lookup file that stores encryption keys. A method of encoding data into a format that is unreadable to anyone except those who posses a password or decryption code to decipher it. A file that shows processing events, messages, errors, and warnings generated by GoldenGate. Its name is ggserr.log and it is located in the root GoldenGate directory. A system that customizes GoldenGate to take a specific action during processing based on a record that qualifies for filtering criteria. For example, you can skip the record or stop the GoldenGate process when the record is encountered. See also event record.
DML
EMSCLNT
error log
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 141
Glossary
event record
Term
Definition
event record
A record in the transaction log that satisfies specific filter criteria and is used to trigger a specific action during processing. See also event marker system. A special MAP parameter used specifically for error handling, which executes only after an error and sends error data to an exceptions table. A database table to which information about failed SQL operations is written as the result of an exceptions map. Used for error handling. The GoldenGate program that reads data either from a data source, from source tables, or from a local trail or file. Extract processes the data for delivery to the target system. A primary Extract reads the data source or database tables, and a data-pump Extract reads a local trail that is populated by a primary Extract. A file written by GoldenGate where data is stored temporarily awaiting further processing during a batch run or initial load. The processing of reading data from database tables or from a data source in preparation for further processing and/or transmission to a target database. A query to the database issued by the Extract process when processing a record from the transaction log. A fetch is required if the data values that are needed to complete the SQL operation are not present in the record. See header. The use of rules to select and exclude data for extraction or replication. A segment of code that can be executed within an application or routine. See also column-conversion functions. GoldenGate Software Command Interface. The primary interface for issuing commands that configure, control, and monitor GoldenGate. A text file in the root GoldenGate directory that contains parameters which apply to the GoldenGate instance as a whole, as opposed to runtime parameters that are specific to a process such as Extract or Replicat. A utility that uses before images to undo changes made to a database. Also known as process group. A group consists of a GoldenGate process (either Extract or Replicat) and the parameter file, the checkpoint file, and any other files associated with that process.
exceptions map
exceptions table
Extract
extract file
extraction
fetch
GGSCI
GLOBALS file
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 142
Glossary
header
Term
Definition
header
trail file that contains information about the transaction environment for that record.
A file header: an area at the beginning of each file in a trail, or at the
beginning of an extract file. This header contains information about the file itself, such as the GoldenGate version.
heterogeneous
A data environment where data is being exchanged among different types of applications, different types of databases, or different operating systems, or among a combination of those things. A data environment where data is being exchanged among identical types of applications, databases, and operating systems. The duplication of source data into a target database to make the two databases identical. A system on the network that serves as a transfer station between the source and target systems. This system can be host to additional processing activities, such as transformation. A column or columns in a table that are being used as a unique identifier for the rows in that table. GoldenGate uses the key to find the correct row in the target database and for fetches from the source database. For GoldenGate, a key can be the primary key, a unique key, a substitute key, or all of the columns of a table in the absence of a defined identifier. A clause in a TABLE or MAP statement that defines a column or columns for GoldenGate to use as a unique identifier to locate any given row in a table. A GoldenGate utility that generates encryption keys. Extract lag is the difference between the time that a record was processed by Extract and the timestamp of that record in the data source. Replicat lag is the difference between the time that the last record in a trail was processed by Replicat and the timestamp of the record in the trail.
homogeneous
initial load
intermediary system
key
KEYCOLS
KEYGEN lag
latency
The difference in time between when a change is made to source data and when that change is reflected in the target data.
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 143
Glossary
LOB
Term
Definition
LOB
Large Object. A data type in a database that represents an unstructured object that is too large to fit into a character field, such as a Microsoft Word document or a video or sound file. Subsets of LOB are CLOB (Character Large Object) and BLOB (Binary Large Object), which contain character data and binary data, respectively. A method of extracting data changes from the database transaction log. A name for a stored procedure that represents an instance of the execution of the procedure, as opposed to its actual name. For example, logical names for a procedure named lookup might be lookup1, lookup2, and so forth. Linux, UNIX, Windows. An acronym that describes an application that runs on any of these platforms, such as DB2 LUW. A computer program that automates a task, such as the implementation of parameters and commands. An association between a set of source data and a set of target data. A map can include data selection and conversion criteria. These maps are specified in a Replicat MAP parameter. A Replicat parameter that specifies the relationship between a source table and a target table and the processing rules for those tables. The control program for GoldenGate processing. A record that is inserted into the audit trail on a NonStop Server to identify application-specific events in the context of Extract and Replicat processing. See also event marker system. For the purpose of this documentation, the term object refers to any logical component of a database that is visible to, and can be created by, its users for the purpose of storing data (for example, tables), defining ownership and permissions (for example, roles), executing an action on another object (for example, triggers), and so forth. A file containing attributes of the tables and other database objects that are configured for processing by GoldenGate, such as column IDs and data types. Open Database Connectivity. Acronym for a standard interface that enables applications to connect to different types of databases in a uniform manner. The goal of ODBC is to make the process of connecting to a database independent of programming languages, database systems, and operating systems. An Extract group that is configured for online change synchronization.
LUW
macro
map
MAP statement
Manager marker
object
object record
ODBC
online Extract
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 144
Glossary
online change synchronization
Term
Definition
A GoldenGate processing method in which Extract and Replicat processes run continuously to synchronize data changes unless they are stopped by a GoldenGate user. Online processes maintain checkpoints in the trail. See online change synchronization. A Replicat group that is configured for online change synchronization. A single unit of work. This typically refers to a SQL change made to data or a change made to the structure of an object in the database, but can also refer to any work done by a computer process. A logical namespace in a database to which database objects are assigned as part of the organizational hierarchy. Because the ownership of database objects is managed differently by different database types, the term owner is used in this documentation to denote whichever entity is recognized by the database as the qualifier of an object name, typically a user or schema name. For example, in a qualified Oracle table name of scott.emp, the owner is scott. A method of writing data to the trail that was used in GoldenGate versions prior to version 10.0. In this mode, Extract overwrites existing data upon recovery, instead of appending it to the end of the trail file. An input or output value for a computer program, such as the code of an application like GoldenGate, a stored procedure, a macro, script, or other processing instructions. A file containing parameters that control the behavior of a GoldenGate process. The default location for parameter files is the dirprm directory in the GoldenGate installation directory. An Extract process that operates on the source system when an alias Extract is being used on the target. This GoldenGate configuration is required when security rules do not permit TCP/IP connections to be initiated from the source system (as a typical Extract would do) because the target is inside a more secure network zone. The passive Extract is the data pump, when one is being used; otherwise, it is the primary Extract. A data pump that is configured with the PASSTHRU parameter to bypass the need to look up data definitions. This enables faster processing and enables a pump to be used on an intermediary system that has no database. See pass-through data pump.
owner
overwrite mode
parameter
parameter file
passive Extract
pass-through Extract
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 145
Glossary
primary Extract
Term
Definition
primary Extract
An Extract group that reads from the data source or directly from the database tables. A primary Extract can write to a local trail, which is then read by a data pump Extract, or it can send the data across TCP/IP to the target system. An integrity constraint consisting of a column or columns that uniquely identify all possible rows that exist in a table, current and future. There can be only one primary key for a table. A primary key contains an implicit NOT NULL constraint. A report generated for Extract, Replicat, and Manager that provides information about the process configuration and runtime statistics and events. The default location for process reports is the dirrpt directory of the GoldenGate installation directory. A unit of information in a transaction log or trail that contains information about a single SQL operation performed on a row in a database. The term record is also used to describe the information contained in a specific row of a table. See header. An extract file on a remote system. A trail on a remote system. The GoldenGate process that applies data to target tables or moves it to another application or destination. See process report. The process of recreating source database operations and applying them to a target database. See process report. The closing of one file in a sequence of files, such as a trail, and the opening of a new file in the sequence. A segment of code that is executed within an application such as GoldenGate, which calls functions that retrieve and return values and provide responses. See also user exit. Information about a single instance of an entity, such as an employee, that is stored within a database table. For example, a row stores information about John Doe in relation to the broader collection of rows that stores information about John and the other employees in a company. Also commonly known as a record.
primary key
process report
record
report replication
routine
row
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 146
Glossary
source
Term
Definition
source
The location of the original data that GoldenGate will be extracting, as in source database and source system. A file containing the definitions of the source tables, which is transferred to the target system. This file is used by the Replicat process for data conversion when the source and target tables are dissimilar. See batch run. An elementary instruction in a computer programming language, for example a SQL statement, parameter statement, or command statement. A Collector process that is started manually by a GoldenGate user, instead of being started automatically by the Manager process. A group of SQL, PL/SQL, or Java statements that are stored in the database and called on demand by a process or application to enforce business rules, supplement application logic, or perform other work as needed. A unique identifier that consists of any columns in a table that can uniquely identify the rows in that table. A substitute key is not defined in the definition of a table; it is created by creating a KEYCOLS clause in a TABLE or MAP statement. The process of making or keeping two or more sets of data consistent with one another. To be consistent, one set might be identical to the other, or one set might be a reorganized, reformatted, or expanded version of the other, while retaining the essence of the information itself. A logical unit of storage in a database that consists of rows and columns, which together identify the instances of a particular entity (for example, employees) and the attributes of that entity, such as name, address, and so forth. An Extract parameter that specifies a source table or tables whose data is to be extracted from the database. An interface between the Change Data Capture (CDC) component of a Teradata database and the Extract process. It allows GoldenGate to communicate with the Teradata replication components. The destination for the data that is processed by GoldenGate, as in target database and target system.
static Collector
stored procedure
substitute key
synchronization
table
TABLE statement
target
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 147
Glossary
target definitions file
Term
Definition
A file containing the definitions of the target tables. This file is transferred to the source system and is used by the Extract process for data conversion when the source and target tables are dissimilar. A special type of batch run in which the Extract process communicates directly with the Replicat process over TCP/IP instead of using a Collector process or trail. A user-defined piece of information that is stored in the header portion of a record in the GoldenGate trail file. Token data can be used to customize the way that GoldenGate delivers information. A special table created for use by GoldenGate in an Oracle database. The table is used in conjunction with parameter settings to prevent replicated data from being sent back to the source in a bidirectional synchronization configuration. A series of files on disk where GoldenGate stores data temporarily in preparation for further processing. GoldenGate records checkpoints in the trail for online change synchronization. A group of one or more SQL operations executed within a begin and end statement by a single user on a database. A set of files that records all of the SQL change operations performed on a database for the purpose of data recovery or replication. Also called conversion. The process of manipulating source data to the format required by target tables or applications, for example converting dates or performing arithmetic calculations. You can do transformation by means of the GoldenGate column-conversion functions. A configuration where data changes are replicated in one direction, source-to-target. Changes cannot be made to that same data and then sent back to the source, as is the case in a bidirectional configuration. An integrity constraint consisting of a column or columns that uniquely identify all possible rows that exist in a table, current and future. Differs from a primary key in that it does not have an implicit NOT NULL constraint. There can be more than one unique key on a table. A proprietary data format that GoldenGate uses to store data in a trail or extract file. Universal data format allows data to be exchanged rapidly and accurately among heterogeneous databases. A user-created program written in C programming code that is called during GoldenGate processing to perform custom processing such as to convert data, to respond to database events, and to repair invalid data.
task
token
trace table
trail
transaction
transaction log
transformation
unidirectional synchronization
unique key
user exit
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 148
Glossary
VAM (Vendor Access Module)
Term
Definition
An API interface that is used by a GoldenGate process module, such as Extract or Replicat, to communicate with certain kinds of databases. A series of files, similar to a transaction log, that are created automatically and aged as needed. Data operations from concurrent transactions are recorded in time sequence, as they occur, but not necessarily in transaction order. Used to support the Teradata maximum protection commit protocol. A placeholder for an unknown or unspecified character or set of characters. A wildcard is a means of specifying multiple names in a parameter or command statement. GoldenGate supports the asterisk (*) wildcard, which represents any number of unknown characters.
wildcard
..............................................................................
Oracle GoldenGate Troubleshooting and Tuning Guide 149
Index
...............
Numerics
509 error 49
B
bad parameter error 37, 38 BOOTDELAYMINUTES parameter 22 build problems 36
A
ADD TRANDATA error 00942 72 error on table specification 71 errors on 71 in DDL support 64 not issued 59 to enable logging 55, 57 ADDTRANDATA option, DDLOPTIONS 64 ALLOWLOBDATATRUNCATE option, DBOPTIONS 48 ALO mode, Extract not capturing data 40 ALTARCHIVELOGDEST parameter 41 ALTER DATABASE, excluded 64 ALTER SYSTEM, excluded 64 archive log, not found 41, 42, 44 ASCII Logdump display 85 table name in Logdump 85 ASCIIDATA command, Logdump 85 ASCIIDUMP command, Logdump 85 ASCIIHEADER command, Logdump 85 authentication database 39, 45 operating system 38, 39 authorization failure 00D31024 73 AUTORESTART parameter 21 AUTOSTART parameter 21 auto-starting processes 21
C
CD command, Logdump 86 character set different on source and target 52 Logdump 85 CHECKPARAMS parameter 37, 38 checkpoint frequency, tuning 31 read, not increasing 45 checkpoint file cannot open 38 converting to master 47 validity check 47 checkpoint table errors 46 CMDTRACE parameter 70 COLS option, ADD TRANDATA 55, 57 COLSEXCEPT option, TABLE 57 columns adding with ALTER command 44 missing 55, 57 numerous in tables 41 problems with mapping 55 commands GGSCI info 11 Logdump 76 comments, wrong place in DDL 66 commit locks 73 COMPUTETIMESTAMP command, Logdump 86
..............................................................................
Oracle GoldenGateTroubleshooting and Tuning Guide 150
Index
D
connection-refused errors 43 convchk utility 47 conversion in pass-through mode 43 optimizing 32 COUNT command, Logdump 86 CTIME command, Logdump 89 cursors, open 33, 71
E
EBCDICDATA command, Logdump 85 EBCDICDUMP command, Logdump 85 EBCDICHEADER command, Logdump 85 EMPTYLOBSTRING option, DBOPTIONS 48 ENCRYPT command, Logdump 91 encryption, trail 47 ENV command, Logdump 91 errors GoldenGate, list of 124 process 13 SQL 17 startup 36 troubleshooting 35 escape characters not converting 57 Event Viewer, GoldenGate messages in 18 EXIT command, Logdump 92 Extract process problem solving 40 stopping for long time 42 writing over a trail 48
D
data displaying in Logdump utility 76 missing 58 out-of-sync 51 segregating 22 DATA CAPTURE CHANGES 44 data pump abending 49 in pass-through mode 43 tuning 28 data source name, in login 39 database login failure 39 tuning for GoldenGate 33 DBOPTIONS parameter 48 DDL troubleshooting 60 DDLOPTIONS parameter 64, 66, 70 DDLTABLE parameter 61 DEBUG command, Logdump 89 debug509.txt file 49 DECRYPT command, Logdump 90 DECRYPTTRAIL parameter 47 DELETE REPLICAT fails 51 derived name not converted 66 DETAIL command, Logdump 90 DIR command, Logdump 102 discard file 17, 68 DISCARDFILE parameter 17, 69 DISCARDROLLOVER parameter 18, 68 disk full 42 DNS server, host name on 43 DUMP command, Logdump 90
F
FC command, Logdump 92 FETCHCOLS options, TABLE 56 fetching failures 56 optimizing 23 FETCHOPTIONS parameter 56 file discard 17 ggserr.log 16 file header, viewing 93 file, creating from Logdump 115 FILEHEADER command 93 FILEINFO command, Logdump 102 FILES command, Logdump 102
..............................................................................
Oracle GoldenGateTroubleshooting and Tuning Guide 151
Index
G
FILTER command, Logdump 103 filters, optimizing 32 FORMATASCII parameter 48 FORMATSQL parameter 48 FORMATXML parameter 48 FROM command, Logdump 113 FUNCTIONSTACKSIZE parameter 38
K
key adding columns with ALTER 44 changing during lag period 61 tables without 52 KEYCOLS clause 44, 51, 55, 56
L G
GGSCHEMA parameter 61 GHDR command, Logdump 109 group cannot delete 51 invalid name 38 missing 37 GROUPTRANSOPS parameter 50 lag Extract 40 inaccurate report of 71 Replicat 45 LAG command 12 large transactions, managing 33 latency troubleshooting 45 viewing 12 LD_LIBRARY_PATH 38 library, missing 38 LIST parameter 70 load balancing 22 LOB length error 45 not captured 59 replicating to Sybase target 48 log error 16 process 13 system 18 LOG command, Logdump 112 log, transaction cannot open 42 failure to retain 44 Logdump commands 76 login failure 39 LOGRETAIN parameter 44 LOGTRAIL command, Logdump 113 LTM context, failure to reserve 44
H
HANDLECOLLISIONS parameter 49, 53 header scanning for 116 tokens, displaying 109 viewing in Logdump 109 HEADERTOKEN command, Logdump 109 HELP command, Logdump 110 HISTORY command 110 host name in MGRPORT 43
I
I/O bottlenecks, eliminating 31 IGNOREDATACAPTURECHANGES option, TRANLOGOPTIONS 44 INFO commands 11 INSTR option, DDL 64 INTERPRETINTERVAL command, Logdump 111 INTERPRETTIMESTAMP command, Logdump 111 invalid identifier error 60 IP address in MGRPORT 43
..............................................................................
Oracle GoldenGateTroubleshooting and Tuning Guide 152
Index
M
M
Manager statistics, viewing 12 tuning 21 mapping, problems with 54 marker_status.sql script 61 MARKERTABLE parameter 61 max text repl size parameter 45 MAXDISCARDRECS parameter 68 MAXFETCHSTATEMENTS parameter 71 MAXSQLSTATEMENTS parameter 33, 71 MAXTRANSOPS parameter 33, 48, 50 MD_TAB_MARKERSEQNO 63 memory Replicat error 47 to process functions 38 used up after network delays 30 z/OS subpool 241 45 messages, viewing 16 MGRPORT parameter 43 module not found error 75 Multi Dimensional Clustered Tables (MDC) 7
O
OBEY command 113 OPEN command, Logdump 113 open cursors, number of 33 operations, SQL fetch failures 56 inaccurate count of 72 missed 58 procedure failures 57 statistics, viewing 12 viewing syntax of 19 OPTYPE option, DDL 64 ORA-04021 error 67 ORACLE_HOME 39 ORACLE_SID 39 OUT command, Logdump 112 OVERFLOWLOGPATH parameter 44
P
parallel processing diagram 22 Replicat 22 parallel processing, with GoldenGate 25 parameter file permissions on 37 reuse problems 59 wrong place 37 parameters missing or wrong 37 SQLEXEC 57 syntax 37 wrong order 38 pass-through data pump 28 PASSTHRU parameter 28, 43 password, database 39 PATH 38 performance, tuning 21
N
network instability 28, 41 not enough bandwidth 41 transfer problems 41 NEXT command, Logdump 112 NEXTTRAIL command, Logdump 112 NOCROSSRENAME option, DDLOPTIONS 65 NODBCHECKPOINT option, ADD REPLICAT 51 NOMAPDERIVED option, DDLOPTIONS 66 NORENAME option, TABLEEXCLUDE 65 NOTIFY(INTERVAL) command, Logdump 112 NOUSELATESTVERSION option, FETCHOPTIONS 56 NOUSESNAPSHOT option, FETCHOPTIONS 56
..............................................................................
Oracle GoldenGateTroubleshooting and Tuning Guide 153
Index
Q
permissions checkpoint file 38 database 39, 45 GoldenGate files 38 on trail files 47 parameter files 37 read transaction log 42 SQLEXEC 57 to create update trigger 72 port number, errors on 43 POSITION command, Logdump 114 prepared statements, too many 71 problem solving procedures 35 procedures, errors on 57 process report DDL missing 66 not generated 68 PURGEOLDEXTRACTS parameter 69, 70
Replicat process abends on DDL table 67 preventing 51 cannot delete 51 troubleshooting 45 tuning 33 replication not enabled 74 REPORT option DDLOPTIONS 66 SEND commands 14 REPORT option, DDLOPTIONS 70 REPORT parameter 14 report, process missing DDL 66 not generated 68 too big 70 using 13 REPORTFILE option, ADD/SEND commands 14 reporting DDL steps 66 process events and errors 13, 16 REPORTROLLOVER parameter 15, 70 RESETMINUTES option, AUTORESTART 10 restart processes automatically 21 RESTARTCOLLISIONS parameter 50 resynchronizing out-of-sync tables 53 RMTHOST parameter 43 rows dividing into ranges 27 out-of-sync 51
Q
QUIT command, Logdump 92
R
RAID configuration 31 RANGE function 27, 69 ranges, calculating 27 RBA, finding with Logdump 116 RECLEN command, Logdump 114 RECORD command, Logdump 112 record type, scanning for 117 record, trail incompatible 48 skipping 50 unreadable format 48 viewing with Logdump 76 Recycle bin error 61 REMOVECOMMENTS option, DDLOPTIONS 66 RENAME converting to ALTER TABLE RENAME 65 failures 65 REPERROR parameter 51
S
SAVE command, Logdump 115 SCANFORENDTRANSACTION command, Logdump 116 SCANFORHEADER command, Logdump 116 SCANFORRBA command, Logdump 116 SCANFORTIME command, Logdump 117 SCANFORTYPE command, Logdump 117 SCANSCROLLING command, Logdump 118 secondary truncation point, problems with 59 SEND commands 12
..............................................................................
Oracle GoldenGateTroubleshooting and Tuning Guide 154
Index
T
server messages, missing 72 SHOW command 118 SHOWINFOMESSAGES option, DBOPTIONS 72 SHOWWARNINGS option, DBOPTIONS 72 SKIP command, Logdump 119 snapshot-too-old message 56 SOURCE command, Logdump 113 source definitions, changes to 48 SOURCEDB parameter 39, 43 sp_configure 45 SQL STATE 55032 44 SQL statements, prepared 33 SQLEXEC failures 57 startup failures 36 speed, improving 39 statistics operations processed 12 viewing from Director 18 STATS command 12 STATUS command 12 stored procedures, errors on 57 string substitution not working 66 syslogs, GoldenGate messages in 18
timestamps, inaccurate 71 TLTRACE parameter 40 tokens header, displaying 109 user, displaying in Logdump 122 TRACE option, SQLEXEC 58 TRACE parameter 40 tracing, GoldenGate 40 trail files cannot be opened 47 encrypted 47 filling up 69 missing 37, 47 not rolling over 69 purging too soon 70 unreadable format 48 viewing in Logdump 76 TRAILFORMAT command, Logdump 120 TRANLOGOPTIONS parameter 44 transaction log cannot open 42 not found 41, 42, 44 transactions, missed 58 TRANSBYTELIMIT command, Logdump 120 TRANSHIST command, Logdump 81, 121 TRANSRECLIMIT command, Logdump 121 trigger DDL 61 on target table 52 triggers on target tables 52 troubleshooting tips 35
T
tables default to fetching from 56 missed in transactions 58 problems with mapping 55 resynchronizing 53 TARGETDB parameter 39 TCP/IP bottlenecks 28 connection refused error 43 not enough bandwidth 41 TCPBUFSIZE option, RMTHOST 31 TCPSOURCETIMER parameter 71 text, writing to Logdump log 122 TIME command, Logdump 119 TIMEOFFSET command, Logdump 120 TIMESTAMP columns, mapping errors 57
U
undo retention, expired 56 user, database 39 USEREXIT parameter 44 USERID parameter 39, 43 USERTOKEN command, Logdump 122
V
VIEW GGSEVT command 16
..............................................................................
Oracle GoldenGateTroubleshooting and Tuning Guide 155
Index
W
W
warnings long-running transactions 41 viewing 16 WARNRATE parameter 15
X
X command, Logdump 123
..............................................................................
Oracle GoldenGateTroubleshooting and Tuning Guide 156