The document provides steps for conducting a disaster recovery drill, which includes taking the primary systems offline, switching the primary and standby databases, and bringing the standby systems online as the new primary. Key steps include shutting down applications and the CI, checking database sync between primary and standby, switching database roles, starting the CI and applications on the new primary systems, and updating DNS, host entries, and other configurations.
The document provides steps for conducting a disaster recovery drill, which includes taking the primary systems offline, switching the primary and standby databases, and bringing the standby systems online as the new primary. Key steps include shutting down applications and the CI, checking database sync between primary and standby, switching database roles, starting the CI and applications on the new primary systems, and updating DNS, host entries, and other configurations.
The document provides steps for conducting a disaster recovery drill, which includes taking the primary systems offline, switching the primary and standby databases, and bringing the standby systems online as the new primary. Key steps include shutting down applications and the CI, checking database sync between primary and standby, switching database roles, starting the CI and applications on the new primary systems, and updating DNS, host entries, and other configurations.
The document provides steps for conducting a disaster recovery drill, which includes taking the primary systems offline, switching the primary and standby databases, and bringing the standby systems online as the new primary. Key steps include shutting down applications and the CI, checking database sync between primary and standby, switching database roles, starting the CI and applications on the new primary systems, and updating DNS, host entries, and other configurations.
Download as DOCX, PDF, TXT or read online from Scribd
Download as docx, pdf, or txt
You are on page 1/ 30
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
SBI – DR DRILL DOCUMENT
Prepared by Tushar Gaikwad Page 1
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
CONTENTS (Press ctrl + link to directly go the required page)
Contents
1. Prerequisite Checks for DR Drill......................................................................................................................3
2. Steps of DR Switch...........................................................................................................................................4 3. Application shutdown at DC............................................................................................................................5 4. Database Switch...........................................................................................................................................11 5. Application Start at DR..................................................................................................................................18 6. DNS changes...................................................................................................................................................20 7. Host Entry changes........................................................................................................................................20 8. Configuration changes...................................................................................................................................21 9. Sync AL11 directories in RP1 and transport filesystem................................................................................23 10. STMS configuration in FGP........................................................................................................................24 11. Sanity Test of Cloud Connector.................................................................................................................28 12. Mail connection in RP1 system..................................................................................................................29 13. License installation.....................................................................................................................................30
Prepared by Tushar Gaikwad Page 2
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
1. Prerequisite Checks for DR Drill
1. The Oracle& Kernel Patch Levels should be same of Primary & Standby Databases.
2. ORACLE_HOME directory should have sufficient space for switch over activities.
3. The Primary & Standby Database should be in Sync.
4. Server time should be sync before starting the DB at DR side.
5. Check with customer and make sure no critical and long running jobs scheduled during the DR drill.
6. Take a server reboot of all DR servers and make sure the servers are working fine.
7. Filesystem /oracle/ZPSG/ZPSG_EN/ should be empty at DR site or same as DC site. It should not
contain salary files older than 3 months.
Prepared by Tushar Gaikwad Page 3
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
2. Steps of DR Switch
1. Remove the application servers from load balancer
2. Stop the applications and CI 3. Make sure Primary & DR DB are in Sync 4. Switch Primary DB as STANDBY system 5. Switch New DR DB as PRIMARY system 6. Start CI & App Servers 7. Check connectivity in SM51, SM66, SICK 8. Change RFC FGPCLNT500 in RP1 system 9. Change the DNS of entries of RP1, FGP, EP1 and SMP systems (HRMS IT team) 10. Add the application servers in Load balancer 11. Sync all AL11 directories of RP1 system and transport directories 12. Check STMS in FGP, change RFC of TMSADM only for FGP connection 13. Check CHARM 14. Check SAP Cloud connector 15. Install SAP license (If required) 16. Check Mail connection 17. Host entry file in RP1 for Festival advance and Aadhar service
Prepared by Tushar Gaikwad Page 4
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
3. Application shutdown at DC 1. Login to load balancer and disable the application servers of RP1 and FGP systems
https://10.189.6.71:8888
Got to -> SLB->Real Services -> Search for the application server and click on Actions button and select disable selected real services
Prepared by Tushar Gaikwad Page 5
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
Prepared by Tushar Gaikwad Page 6
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
2. Log on to the application servers of systems as per IP address sheet and stop the sap applications and after that stop CI.
Login to PIMS using credentials : https://pims.onlinesbi.com/frmLoginACMO.aspx
Prepared by Tushar Gaikwad Page 7
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
3. Login to the server and run the below command to stop application server
Command : stopsap r3
After stopping the application check the instance number and run the command
Please refer to below excel sheet for the commands for all Prod systems
SAP stop command.xlsx
Command: sapcontrol -nr 08 -function StopService
Prepared by Tushar Gaikwad Page 8
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
After that run command cleanipc 08 remove twice till the output shows the number of IPC-objects 0
Command : cleanipc 08 remove
Prepared by Tushar Gaikwad Page 9
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
Run the command ipcs | grep <sid>adm to check if any shared memory objects are left or not. IF output shows no result it means everything is OK.
e.g. For RP1 system, use command : ipcs | grep rp1adm
For FGP system, use command : ipcs | grep fgpadm
After stopping the all the application servers, stop the CI in the same way after that start with DB switch activity
Prepared by Tushar Gaikwad Page 10
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
4. Database Switch
I) PRI & STDBY: SQL> select name, open_mode from V$Database;
The open_mode 'READ WRITE' describes the open database and 'MOUNTED' explains the standby database.
Both commands above give the current sequence of the archived logs.
STDBY: SQL> select max(sequence#) from v$archived_log where applied='YES';
Prepared by Tushar Gaikwad Page 11
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
III)After ensuring that PRI & STDBY are in sync issue the commands below.
PRI: SQL> alter system switch logfile;
This command creates archive log files.
Issue this commands for 4-5 times which creates archive log files & check the PR-DR Sync.
Prepared by Tushar Gaikwad Page 12
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
IV) Issue the below command to display the switch over status of Primary & Standby Databases.
PRI & STDBY SQL> select name,switchover_status from v$database;
The result would be 'To Standby' or 'session active' on PRI and on STDBY it would be 'session active'.
Prepared by Tushar Gaikwad Page 13
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
V) Issue the command below on Primary to switch the Database role from Primary to standby.
PRI: SQL> alter database commit to switchover to physical standby with session shutdown;
VI) Issue the command below on Standby Database to commit the role to Primary.
STDBY: SQL> alter database commit to switchover to primary;
Prepared by Tushar Gaikwad Page 14
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
VII) Issue the commands below on PRI to change the Archive Destination.
PRI: SQL> startup nomount;
PRI: SQL>alter system set log_archive_dest_state_2='defer'; PRI: SQL>shutdown immediate;
Now the PRI System is the DR System.
Startup the database as described below.
Prepared by Tushar Gaikwad Page 15
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
PRI (Current Standby Database)
PRI: SQL> startup nomount; PRI: SQL> alter database mount standby database; PRI: SQL> alter database recover managed standby database disconnect from session;
VIII) The Actual DR System is the PR System now. Shut it down and start the database in normal mode as described below.
STDBY (Current Primary Database)
Prepared by Tushar Gaikwad Page 16
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
STDBY: SQL>alter system set log_archive_dest_state_2='enable';
STDBY: SQL>shut immediate; STDBY: SQL> startup;
5. Application Start at DR Prepared by Tushar Gaikwad Page 17 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
1. After DB switch start the CI and after that application servers using below command
Command: startsap r3
2. After starting all the applications check the T-codes SM51 for all applications servers
Prepared by Tushar Gaikwad Page 18
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
3. Check T-code sick
4. Login to the load balancer of DR site and enable the application servers of RP1 and FGP
https://10.176.189.155:8888
6. DNS changes Prepared by Tushar Gaikwad Page 19 SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
Contact HRMS IT team for DNS changes for RP1, FGP, EP1 and SMP systems
7. Host Entry changes
Festival advance and Aadhar update services communicates with different SBI departments so need to do changes the host entry file in RP1 system. (Note: Do the changes only if SBI HRMS department is performing DR drill)
#DR-10.176.1.90 eispri.sbi.co.in EISPRI.sbi.co.in
DC-10.191.171.90 eispri.sbi.co.in EISPRI.sbi.co.in (enable this entry when only SBI HRMS department is performing DR drill)
9. Sync AL11 directories in RP1 and transport filesystem
Prepared by Tushar Gaikwad Page 22
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
1. Login to RP1 system and go to T-code AL11 and take list of filesystem and sync it (if any new filesystem is created)
Important Note: Make sure that while performing DR Drill the filesystem /oracle/ZPSG/ZPSG_EN should be empty at DR site or it should be same as DC site. Because DR drill happens in every 6 months so if old salary files are present then remove them.
Command to sync: rsync -avuzx /oracle/ZPSG/ [email protected]:/oracle/ZPSG/
1. Login to FGD, FGQ and FGP systems and go to T-code SM59 and change the RFC [email protected]_FGD
Change the hostname from fgpprdci to fgpdrcci and change the password of TMSADM
Prepared by Tushar Gaikwad Page 24
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
Prepared by Tushar Gaikwad Page 25
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
Check the connection test and authorization test it should be OK
Go to T-code STMS-> Overview-Systems and select all the systems and Go to Menu-> SAP systems- >Check-> Transport Tool
Prepared by Tushar Gaikwad Page 26
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
Click on Yes button
The output should be All Green
Prepared by Tushar Gaikwad Page 27
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
11. Sanity Test of Cloud Connector
Login to SAP cloud connector using https://10.1.147.172:8443/ link and check if the connections are active or not.
If the connections are in red color then click on connect button from Action column.
Prepared by Tushar Gaikwad Page 28
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
12. Mail connection in RP1 system
Change the IP address in T-code SCOT if mails are not getting delivered
PR: 10.189.213.228 Port 587
DR: 10.176.200.27 Port 587 (use this only if mail are not getting delivered)
Prepared by Tushar Gaikwad Page 29
SBI DR-DRILL COMPLETE DOCUMENT– VERSION 2
13. License installation
Download SAP license from SAP Market place for All DR systems and install it. By default temporary license get installed which is valid for 1 month but if we are going to stay at DR site for more than 1 month then install the license or after 20 days you will get message while login for license expire.
Go to T-code SLICENSE-> Menu Edit-> Install license and select the license file.
Alert Log Recommendation - "Increase Per Process Memlock (Soft) Limit To at Least MB To Lock % of SHARED GLOBAL AREA (SGA) Pages Into Physical Memory" (Doc ID 2049901.1)
Alert Log Recommendation - "Increase Per Process Memlock (Soft) Limit To at Least MB To Lock % of SHARED GLOBAL AREA (SGA) Pages Into Physical Memory" (Doc ID 2049901.1)
