How to Recreate the Controlfile on 10gR2 RAC with ASM and

Data Guard

Recently, I had the pleasure of having to rebuild the control file on a three-
node RAC cluster with two, single, stand-alone standby databases on
different servers in different locations and all of them are using ASM. Sound
like a handful? You bet! The most difficult portion was finding the
information on how to do it.

I've done hundreds of control file rebuilds dating back to Oracle7. I've
created standby databases since Oracle8i. I am experienced with Oracle8i's
Oracle Parallel Server (OPS) and Oracle 9i, Oracle 10gR1, and Oracle 10gR2
Real Application Clusters installing, tuning, configuring with and without
ASM. Try finding all of these components in a step-by-step document is
impossible.

With my frustration, I researched and used over a dozen documents, how-

to's, Bulletins, and other Oracle documentation to put the steps together.
After all my research, I had the chance to test the scenario. It worked
successfully.

In case you are ever experience the need to Recreate a Control file on
Oracle 10gR2 RAC with ASM and have physical standby databases involved,
here are the steps to do it.

Step 1: Backup your RAC Environment.

Before you make any changes to anything, BACK UP YOU DATABASE,
CONTROLFILES, ARCHIVELOGS, and SPFILE. Make copies of your
TNSNAMES.ORA and LISTENER.ORA file just in case. I use RMAN for my
backups so I took a hot backup.

Since this is a RAC database, create an initialization file from your current
SPFILE to be used later.
SQL> create pfile='/db/app/oracle/wkdir/init_new.ora' from
spfile;

If you have lost all copies of the control file, a new one will need to be
generated using SQL. The syntax is available in the SQL Reference manual
for all versions.

Step 2: Recreating the Existing Controlfile

If your RAC database is up and running or in a mounted state, connect to
your RAC instance via SQL*PLUS as sysdba, issue the following command to
dump a trace file that contains a create controlfile script. The file will be
generated in the user_dump_dest on the local instance. If you are not sure
where your user_dump_dest is, at the SQL prompt type:

SQL> show parameter user_dump_dest

NAME TYPE VALUE
-------------------------------------------------------------
user_dump_dest string /db/app/oracle/admin/rcmp/udump

Now create the backup controlfile script by issuing the following:

SQL> alter database backup controlfile to trace;

Database altered.

SQL> exit

Quickly change directories to what your user_dump_dest settings are and

list the contents in the directory ordering by date.

cd /db/app/oracle/admin/dgvd/udump
Find the trace file using "ls -ltr" in the user_dump_dest, it will probably be
the last or one of the last files listed as it will be very recent.

/db/app/oracle/admin/dgvd/udump> ls -ltr

-rw-r--r-- 1 oracle dba 926982 Aug 7 21:28 dgvd1_ora_7718.trc

-rw-r--r-- 1 oracle dba 2141 Aug 7 21:30 dgvd1_ora_9416.trc
-rw-r--r-- 1 oracle dba 2060 Aug 7 21:37 dgvd1_ora_27649.trc
-rw-r----- 1 oracle dba 2061 Aug 7 21:52 dgvd1_ora_16924.trc
drwxr-xr-x 2 oracle dba 128000 Aug 7 21:54 ./
-rw-r----- 1 oracle dba 19407 Aug 7 21:54 dgvd1_ora_18404.trc

Copy, move, or rename the last file to a name you will remember. In my
case, I called it ctrl_dgvd.sql. I also moved it to a working directory I will
use later on.

oracle@rmodbp03:/db/app/oracle/admin/dgvd/udump > mv
dgvd1_ora_18404.trc /db/app/oracle/wkdir/ctrl_dgvd1.sql

Change to the working directory or wherever you placed you new file. I
make a couple of copies of the same file so I can edit one and keep one as
the standard.

Step 3. Editing the New Controlfile Script

Once the file is opened, remove all of the header information before the
"STARTUP NOMOUNT" below Set #2. RESETLOGS case statement.

Here is my example of what you should be seeing:

-- Set #2. RESETLOGS case

--
-- The following commands will create a new control file and use
it
-- to open the database.
-- Data used by Recovery Manager will be lost.
-- The contents of online logs will be lost and all backups will
-- be invalidated. Use this only if online logs are damaged.
-- After mounting the created controlfile, the following SQL
-- statement will place the database in the appropriate
-- protection mode:
-- ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PERFORMANCE
STARTUP NOMOUNT
CREATE CONTROLFILE REUSE DATABASE "DGVD" RESETLOGS FORCE LOGGING
ARCHIVELOG
MAXLOGFILES 192
MAXLOGMEMBERS 6
MAXDATAFILES 1024
MAXINSTANCES 6
MAXLOGHISTORY 65535
LOGFILE
GROUP 1 (
'+DGV_DATA_DG/dgvd/onlinelog/group_1.264.662159045',
'+DGV_FLASH_DG/dgvd/onlinelog/group_1.646.662159047'
) SIZE 100M,
GROUP 2 (
'+DGV_DATA_DG/dgvd/onlinelog/group_2.260.662159049',
'+DGV_FLASH_DG/dgvd/onlinelog/group_2.635.662159049'
) SIZE 100M,
GROUP 3 (
'+DGV_DATA_DG/dgvd/onlinelog/group_3.266.662159051',
'+DGV_FLASH_DG/dgvd/onlinelog/group_3.479.662159053'
) SIZE 100M
-- STANDBY LOGFILE
-- GROUP 10 (
-- '+DGV_DATA_DG/dgvd/onlinelog/group_10.263.658495127',
-- '+DGV_FLASH_DG/dgvd/onlinelog/group_10.764.658495129'
-- ) SIZE 100M,
-- GROUP 11 (
-- '+DGV_DATA_DG/dgvd/onlinelog/group_11.262.658495135',
-- '+DGV_FLASH_DG/dgvd/onlinelog/group_11.765.658495137'
-- ) SIZE 100M,
-- GROUP 12 (
-- '+DGV_DATA_DG/dgvd/onlinelog/group_12.259.658495141',
-- '+DGV_FLASH_DG/dgvd/onlinelog/group_12.766.658495143'
-- ) SIZE 100M
DATAFILE
'+DGV_DATA_DG/dgvd/datafile/system.280.657902505',
'+DGV_DATA_DG/dgvd/datafile/undotbs1.279.657902513',
'+DGV_DATA_DG/dgvd/datafile/sysaux.278.657902515',
'+DGV_DATA_DG/dgvd/datafile/crwd.270.657917921',
'+DGV_DATA_DG/dgvd/datafile/undotbs2.267.657908869',
'+DGV_DATA_DG/dgvd/datafile/undotbs3.261.657908875'
CHARACTER SET WE8ISO8859P1
;
-- Configure RMAN configuration record 1
VARIABLE RECNO NUMBER;
EXECUTE :RECNO := SYS.DBMS_BACKUP_RESTORE.SETCONFIG('ARCHIVELOG
DELETION POLICY','TO
-- Commands to re-create incarnation table
-- Below log names MUST be changed to existing filenames on
-- disk. Any one log file from each branch can be used to
-- re-create incarnation records.
-- ALTER DATABASE REGISTER LOGFILE '+DGV_FLASH_DG';
-- ALTER DATABASE REGISTER LOGFILE '+DGV_FLASH_DG';
-- Recovery is required if any of the datafiles are restored
backups,
-- or if the last shutdown was not normal or immediate.
RECOVER DATABASE USING BACKUP CONTROLFILE
-- Create log files for threads other than thread one.
ALTER DATABASE ADD LOGFILE THREAD 2
GROUP 4 (
'+DGV_DATA_DG/dgvd/onlinelog/group_4.274.662159057',
'+DGV_FLASH_DG/dgvd/onlinelog/group_4.461.662159065'
) SIZE 100M REUSE,
GROUP 5 (
'+DGV_DATA_DG/dgvd/onlinelog/group_5.277.662159067',
'+DGV_FLASH_DG/dgvd/onlinelog/group_5.459.662159069'
) SIZE 100M REUSE,
GROUP 6 (
'+DGV_DATA_DG/dgvd/onlinelog/group_6.272.662159071',
'+DGV_FLASH_DG/dgvd/onlinelog/group_6.457.662159073'
) SIZE 100M REUSE;
ALTER DATABASE ADD LOGFILE THREAD 3
GROUP 7 (
'+DGV_DATA_DG/dgvd/onlinelog/group_7.271.662159083',
'+DGV_FLASH_DG/dgvd/onlinelog/group_7.453.662159091'
) SIZE 100M REUSE,
GROUP 8 (
'+DGV_DATA_DG/dgvd/onlinelog/group_8.269.662159095',
'+DGV_FLASH_DG/dgvd/onlinelog/group_8.449.662159097'
) SIZE 100M REUSE,
GROUP 9 (
'+DGV_DATA_DG/dgvd/onlinelog/group_9.268.662159099',
'+DGV_FLASH_DG/dgvd/onlinelog/group_9.447.662159101'
) SIZE 100M REUSE;
-- Database can now be opened zeroing the online logs.
ALTER DATABASE OPEN RESETLOGS;
-- Commands to add tempfiles to temporary tablespaces.
-- Online tempfiles have complete space information.
-- Other tempfiles may require adjustment.
ALTER TABLESPACE TEMP ADD TEMPFILE
'+DGV_DATA_DG/dgvd/tempfile/temp.273.657902517'
SIZE 104857600 REUSE AUTOEXTEND ON NEXT 104857600 MAXSIZE 32767M;
-- End of tempfile additions.

Step 4. Edit the create controlfile script for RAC.

First Script Copy: Edit the two lines by including the path of the initialization
file you backed up in the beginning and adding the word "set" before the
word database.

STARTUP NOMOUNT pfile='/db/app/oracle/wkdir/init_new.ora'

CREATE CONTROLFILE REUSE set DATABASE "DGVD" RESETLOGS ARCHIVELOG

Notice that in the script all of the logfiles are listed together. We will need to
separate these out by thread. This is a great time to start using your backup
copies of your new controlfile script. I use four scripts, one for each step in
the process. I'll keep the original untouched. I'll use the First copy of the
script for these steps. First remove all logfiles listed that are not from the
first instance or Thread 1. In this case, I removed Groups 4, 5, and 6 from
Thread 2 and Groups 7, 8, and 9 from Thread 3. Your first script should look
like this:

STARTUP NOMOUNT pfile='/db/app/oracle/wkdir/init_new.ora'

CREATE CONTROLFILE REUSE set DATABASE "DGVD" RESETLOGS ARCHIVELOG
MAXLOGFILES 192
MAXLOGMEMBERS 3
MAXDATAFILES 1024
MAXINSTANCES 3
MAXLOGHISTORY 65535
LOGFILE
GROUP 1 (
'+DGV_DATA_DG',
'+DGV_FLASH_DG') SIZE 100M,
GROUP 2 (
'+DGV_DATA_DG',
'+DGV_FLASH_DG') SIZE 100M,
GROUP 3 (
'+DGV_DATA_DG,
'+DGV_FLASH_DG') SIZE 100M
DATAFILE
'+DGV_DATA_DG/dgvd/datafile/system.280.657902505',
'+DGV_DATA_DG/dgvd/datafile/undotbs1.279.657902513',
'+DGV_DATA_DG/dgvd/datafile/sysaux.278.657902515',
'+DGV_DATA_DG/dgvd/datafile/crwd.270.657917921',
'+DGV_DATA_DG/dgvd/datafile/undotbs2.267.657908869',
'+DGV_DATA_DG/dgvd/datafile/undotbs3.261.657908875'
CHARACTER SET WE8ISO8859P1;

Second Script Copy: The second script is one and only one line. It is the most
crucial of all the steps. This script will use the backup controlfile to recover
the database and make sure all the log files, datafiles, and controlfiles will be
in sync.

RECOVER DATABASE USING BACKUP CONTROLFILE UNTIL CANCEL;

When this is finished you should get a "Media Recovery Complete" message.

Third Script Copy: Because not all of the logfiles are listed before the
controlfile is created, such as additional threads or standby log files, you will
need to use the RESETLOGS option on the create controlfile statement. With
RAC, this is absolutely necessary. By Opening with the RESETLOGS Option,
all of the SCN's will be reset back to zero. At the same time, I will add the
additional log groups listed by thread#. My third script looks like this:

ALTER DATABASE ADD LOGFILE THREAD 2

GROUP 4 (
'+DGV_DATA_DG, '+DGV_FLASH_DG) SIZE 100M,
GROUP 5 (
'+DGV_DATA_DG','+DGV_FLASH_DG') SIZE 100M,
GROUP 6 (
'+DGV_DATA_DG','+DGV_FLASH_DG') SIZE 100M;

ALTER DATABASE ADD LOGFILE THREAD 3

GROUP 7 (
'+DGV_DATA_DG','+DGV_FLASH_DG') SIZE 100M,
GROUP 8 (
'+DGV_DATA_DG','+DGV_FLASH_DG') SIZE 100M,
GROUP 9 (
'+DGV_DATA_DG','+DGV_FLASH_DG') SIZE 100M;

-- Database can now be opened zeroing the online logs.

ALTER DATABASE OPEN RESETLOGS;

ALTER DATABASE ENABLE PUBLIC THREAD 2;
ALTER DATABASE ENABLE PUBLIC THREAD 3;

After the alter database open command, add an "alter database enable
public thread #" command(s). Do this for each additional thread to be
added.

NOTE: It is highly recommended to take a full backup of the database after completing
this procedure.

Final Script Copy: The last step in rebuilding the control file requires rebuilding the
temp tablespace. For example:

ALTER TABLESPACE TEMP ADD TEMPFILE

'+DGV_DATA_DG/dgvd/tempfile/temp.273.657902517' SIZE 104857600
REUSE AUTOEXTEND ON NEXT 104857600 MAXSIZE 32767M;

Step 5: Ready? Not quite yet. Change your parameter settings.

At this point you are confident with your scripts to rebuild your controlfile for
your RAC cluster. Before you begin executing your scripts, your environment
needs to be prepared.

First, edit your initialization parameter file with the following setting:

cluster_database=false

Stop all instances and database that need the controlfile built cleanly with
srvctl. DO NOT SHUT DOWN ASM or CRS or other databases.
srvctl -stop database -d dgvd

This is a good point to take cold backups or copying your existing controlfiles
to disk.

Step 6: Execute the scripts

Open a SQL*PLUS session as SYS and execute each script in order. Verify
that you are running from the instance designated for thread 1. This is
because any logfiles designated in the create controlfile statement will go
into thread 1. You will get errors if you are running from another instance.
For example

/db/app/oracle/wkdir > sqlplus / as sysdba

SQL*Plus: Release 10.2.0.3.0 - Production on Thu Aug 7 23:11:19

2008
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.

Connected to an idle instance.

SQL> @dgvd1_ctl

If you get the "Statement processed" message, the database will be opened
with a brand new control file.

At this time, create the spfile from the pfile;

SQL> create spfile='+DGV_DATA_DG/DGVD/spfiledgvd.ora' from

pfile='/db/app/oracle/wkdir/init_new.ora'

Make sure the cluster_database=true parameter is set in the init/spfile.

alter system set cluster_database=true scope=BOTH sid='*';

Show parameter control to determine the new controlfile settings.

SQL> show parameter control

NAME_COL_PLUS_SHOW_PARAM
-----------------------------------------------------------------
---------------
TYPE
-----------
VALUE_COL_PLUS_SHOW_PARAM
-----------------------------------------------------------------
---------------
control_files
string
+DGV_DATA_DG/dgvd/controlfile/current.285.662169519,
+DGV_FLASH_DG/dgvd/controlfile/current.824.662169521

Just like changing the cluster_database parameter, insert the control_files

values into the spfile as well. Shutdown the database and restart using the
spfile.

SQL> shutdown immediate

Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup

Using srvctl, bring up the remaining instances.

srvctl start instance -d dgvd -i dgvd2, dgvd3

Final Step: Backup

At the earliest convenience, take a full backup of the database. If you are
using RMAN, you will have a new incarnation of the database now. Add
standby logfiles to the primary RAC system. Once your RAC system has been
backed up successfully, each of your physical standby databases will need to
be rebuilt from a new standby controlfile. I'll have to right up that post for
another day.