Oracle RAC 19c: Best Practices and Secret Internals
21 likes•11,636 views
Oracle Real Application Clusters 19c provides best practices and new features for upgrading to Oracle 19c. It discusses upgrading Oracle RAC to Linux 7 with minimal downtime using node draining and relocation techniques. Oracle 19c allows for upgrading the Grid Infrastructure management repository and patching faster using a new Oracle home. The presentation also covers new resource modeling for PDBs in Oracle 19c and improved Clusterware diagnostics.
![• Environment changes,
incorrect permissions of
binaries can prevent stack
startup
• Oracle 19c Clusterware stack
attempts to auto-diagnose
unsuccessful startup issues
• Provides detailed logging in
case of failures
CRS-41053: checking Oracle Grid Infrastructure for file permission
issues
PRVG-2031 : Owner of file ”…gipcd.bin" did not match the
[Expected="grid(54320)" Found="oracle(54325)"]
Oracle Clusterware start Diagnostics
init
cssdAgent
cssdmonitor
ohasd
oraAgent
cssd
crsd
ctssd
HAIP
ACFS
mdnsd
gipcd
evmd
ASM
*NOT all daemons are shown in illustration above
oraRootAgent](https://image.slidesharecdn.com/oracleracinternals2019forexternaldistribution-190924023744/85/Oracle-RAC-19c-Best-Practices-and-Secret-Internals-30-320.jpg)
![Clusterware runtime Diagnostics
• Oracle 19c Clusterware processes maintains histograms, statistics
such as trace file rotation frequency, time taken for rotation
• Preserves critical information on very busy systems
• Severity tagging provides human readable criticality of messages
• 2019-08-20 08:36:13.142 : CSSD:1871161088: [ ERROR]
clssgmclienteventhndlr: (SENDCOMPLETE) No proc found for ClientID
• 2019-08-20 08:36:13.188 : CSSD:1871161088: [ INFO] clssgmDeadProc:
Removing clientID 2:43454:0 (0x7fda802df820), with GIPC
• New diagnostics monitor thread ensures in-memory logs (UTS) are
periodically written to ensure diagnostics are available in case of
process crash](https://image.slidesharecdn.com/oracleracinternals2019forexternaldistribution-190924023744/85/Oracle-RAC-19c-Best-Practices-and-Secret-Internals-31-320.jpg)
Oracle RAC 19c: Best Practices and Secret Internals
- 1. 1 Oracle Real Application Clusters 19c: Best Practices and Secret Internals Copyright © 2019 Oracle and/or its affiliates. Anil Nair Sr Principal Product Manager, Oracle Real Application Clusters (RAC) @RACMasterPM http://www.linkedin.com/in/anil-nair-01960b6 http://www.slideshare.net/AnilNair27/
- 2. Guest speaker: Paresh Patel Senior Member of Technical Staff, PayPal
- 3. The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.
- 4. Session Survey Help us make the content even better. Please complete the session survey in the Mobile App. Session id is TRN 4851 Copyright © 2019 Oracle and/or its affiliates.
- 5. Agenda Best Practices to Upgrade to Oracle 19c Oracle 19c Grid Infrastructure (GI) New Features What’s new with Cache Fusion? 1 2 5 3
- 6. Agenda Best Practices to Upgrade to Oracle 19c1 2 6 3
- 7. • Execution of ./gridSetup.sh on old OS releases may fail • Failure is reported as a perl error message • perl has hard dependency on glibc • Similar message reported by DB installer • Additional details in URL below https://www.linkedin.com/pulse/high-level- steps-upgrade-oracle-19c-rac-anil-nair/ Oracle 19c Upgrade requires Linux 7
- 8. Upgrade to Linux 7 with least downtime Drain delNode Linux7 addNode For each Node perform the following until last node Linux6 Linux6 Linux6 Linux6 Services $srvctl relocate service –drain_timeout *Inline Upgrade depends on initial configuration
- 9. Upgrade to Linux 7 with least downtime Drain delNode Linux7 addNode For each Node perform the following until last node Linux6 Linux6 Linux6 Linux6 Services $srvctl relocate service –drain_timeout *Inline Upgrade depends on initial configuration
- 10. Upgrade to Linux 7 with least downtime Drain delNode Linux7 addNode For each Node perform the following until last node Linux6 Linux6 Linux6 Linux6 Services $srvctl relocate service –drain_timeout $./delNode …. *Inline Upgrade depends on initial configuration
- 11. Upgrade to Linux 7 with least downtime Drain delNode Linux7 addNode For each Node perform the following until last node Linux6 Linux6 Linux6 Linux6Linux7 Services $srvctl relocate service –drain_timeout $./delNode …. Upgrade or Reinstall *Inline Upgrade depends on initial configuration
- 12. Upgrade to Linux 7 with least downtime Drain delNode Linux7 addNode For each Node perform the following until last node Linux6 Linux6 Linux6 Linux6Linux7 Services $srvctl relocate service –drain_timeout $./delNode …. Upgrade or Reinstall $./addNode …. *Inline Upgrade depends on initial configuration
- 13. • Download latest orachk and benefit from the latest checks • No need to individually download autoupgrade.jar or cluvfy • Single report with results from autoupgrade.jar, orachk and cluvfy checks • *orachk also includes other components like Application Continuity and Security related checks ORAchk=ORAchk+cluvfy+Autoupgrade.jar* Cluvfy PreUpgrade ORAchk ORAchk autoupgrade includes autoupgrade.jar checks and cluvfy pre-upgrade checks
- 14. One command for all Autoupgrade checks Report includes results from all components with appropriate options • orachk -preupgrade -targetversion 19.3.0.0.0 –showpass • cluvfy –stage pre/post • orachk includes preupgrade.jar and cluvfy pre-upgrade checks
- 15. One command for all Autoupgrade checks Report includes results from all components with appropriate options • orachk -preupgrade -targetversion 19.3.0.0.0 –showpass • cluvfy –stage pre/post • orachk includes preupgrade.jar and cluvfy pre-upgrade checks
- 16. One command for all Autoupgrade checks Report includes results from all components with appropriate options • orachk -preupgrade -targetversion 19.3.0.0.0 –showpass • cluvfy –stage pre/post • orachk includes preupgrade.jar and cluvfy pre-upgrade checks
- 17. One command for all Autoupgrade checks Report includes results from all components with appropriate options • orachk -preupgrade -targetversion 19.3.0.0.0 –showpass • cluvfy –stage pre/post • orachk includes preupgrade.jar and cluvfy pre-upgrade checks
- 18. • Grid Infrastructure Management Repository (GIMR) aka mgmtDB is NO longer mandatory with starting with Oracle 19c • Limited AHF functionality by utilizing filesystem without GIMR • No support for CHA GUI chactl • Trace File Analyzer (TFA) will provide limited graphical view State of the GIMR
- 19. • Grid Infrastructure Management Repository (GIMR) aka mgmtDB is NO longer mandatory with starting with Oracle 19c • Limited AHF functionality by utilizing filesystem without GIMR • No support for CHA GUI chactl • Trace File Analyzer (TFA) will provide limited graphical view State of the GIMR
- 20. Choose to install GIMR: Thanks to Your Feedback • Upgrades depend on Initial GIMR configuration • Choose to Install Grid Infrastructure Management Repository (GIMR) • Eventual goal is to move GIMR into its own new separate home 20 Only for New Installations
- 21. -dryRunForUpgrade: Thanks to Your Feedback 21 gridSetup now can be used with the –dryRunForUpgrade option for dry run testing of Oracle Grid Infrastructure upgrades $./gridSetup.sh –dryRunForUpgrade
- 22. GIMR state during upgrade From Version To Version GIMR state in source OH GIMR State in dest OH Comments Standalone Cluster Upgrade (Not Cluster Domain) Pre- 12.2 12.2 Jan 2019 RU No Yes/No Choice to select Yes/No to configure GIMR during upgrade Pre-18c 18.5 Yes Yes No Choice to change state of GIMR during upgrade Pre-19c 19.3 Yes Yes No Choice to change state of GIMR during upgrade Pre-19c 19.3 No No No Choice to change state of GIMR during upgrade Fresh Install 12.2 JAN 2019 RU Yes/No Choice to select Yes/No 18.5 Yes/No Choice to select Yes/No 19.3 - Yes/No To add GIMR post Installation use mgmtca
- 23. Read Only Oracle HOME (ROOH) • ROOH enabled Oracle Database homes store configuration files outside of the Oracle Home • Faster cloning of Oracle software home with ROOH as environment specific configuration files are stored outside of Oracle home • Improves security as running processes cannot create new files under Oracle Home • Oracle RAC DB Home is ROOH • $roohctl can be used in versions 18c and 19c for manual conversion • Only pertinent to Oracle Database Home (Not GI home) • Plan to remove configuration files used by application before converting to ROOH such as • tnsnames.ora
- 24. • Apply patch to a new grid home while stack continues to run from current home • Reduces downtime as stack is up and running during the copy process • Reduces errors caused by common issues such as “Out of space” • Easy fallback in case of issues Patch faster with -SwitchHome /u01/app/19.0/grid /u01/app/19.3/grid
- 25. Summary of Best Practices for Upgrade Always download the latest version of orachk/exachk from • https://support.oracle.com/epmos/faces/DocContentDisplay?id=1268927.2 • https://support.oracle.com/epmos/faces/DocContentDisplay?id=1070954.1 Consider storage requirements of GIMR Apply latest OS patches • orachk includes the DBSAT (Oracle Database security assessment tool) Add user defined checks to benefit from a single report Find environment specific files in ORACLE_HOME such as password, tnsnames.ora, pfile) and other files that may affect using ROOH
- 26. Agenda Oracle 19c Grid Infrastructure (GI) New Features 1 2 26 3
- 27. New GI Resource Modeling for PDBs • Optimize management of resources such as database instance, listener on nodes • Include the ability to startup, stop, prioritize, relocate resources • define pdb2 as more critical and therefore start pdb2 before other pdbs
- 28. • Utilizes Service(s) to drive workload placement • Services implicitly opens PDB Instance(s) • Order of PDB open based on service definition • Defined using Preferred, Available attributes • Default modeling after upgrades Resource Modeling Today Services trigger PDB open Services
- 29. • Utilizes Service(s) to drive workload placement • Services implicitly opens PDB Instance(s) • Order of PDB open based on service definition • Defined using Preferred, Available attributes • Default modeling after upgrades Resource Modeling Today Services trigger PDB open Services
- 30. • Environment changes, incorrect permissions of binaries can prevent stack startup • Oracle 19c Clusterware stack attempts to auto-diagnose unsuccessful startup issues • Provides detailed logging in case of failures CRS-41053: checking Oracle Grid Infrastructure for file permission issues PRVG-2031 : Owner of file ”…gipcd.bin" did not match the [Expected="grid(54320)" Found="oracle(54325)"] Oracle Clusterware start Diagnostics init cssdAgent cssdmonitor ohasd oraAgent cssd crsd ctssd HAIP ACFS mdnsd gipcd evmd ASM *NOT all daemons are shown in illustration above oraRootAgent
- 31. Clusterware runtime Diagnostics • Oracle 19c Clusterware processes maintains histograms, statistics such as trace file rotation frequency, time taken for rotation • Preserves critical information on very busy systems • Severity tagging provides human readable criticality of messages • 2019-08-20 08:36:13.142 : CSSD:1871161088: [ ERROR] clssgmclienteventhndlr: (SENDCOMPLETE) No proc found for ClientID • 2019-08-20 08:36:13.188 : CSSD:1871161088: [ INFO] clssgmDeadProc: Removing clientID 2:43454:0 (0x7fda802df820), with GIPC • New diagnostics monitor thread ensures in-memory logs (UTS) are periodically written to ensure diagnostics are available in case of process crash
- 32. Private Network Interface Check • Oracle 19c recommends using bonding mode 0 Balance-RR, 1- Active Backup when HAIP is not used for network redundancy • Recommendation is based on greater tolerance to network jitters with different combination of interface, switch, OS • Check also ensures every node of the cluster has same bonding mode • The check is a warning • It is possible but not recommended to use other modes
- 33. Private Network Interface Check • Oracle 19c recommends using bonding mode 0 Balance-RR, 1- Active Backup when HAIP is not used for network redundancy • Recommendation is based on greater tolerance to network jitters with different combination of interface, switch, OS • Check also ensures every node of the cluster has same bonding mode • The check is a warning • It is possible but not recommended to use other modes
- 34. • Clusterware processes communicate with each other using gIPC • gIPC today utilizes TLS (Transport Layer Security) • Easy configuration of any future secure communication protocol Oracle Clusterware Ciphers $crsctl get cluster security tls ON $crsctl get cluster tlsciphersuite enabled
- 35. Summary of Clusterware features Plan to utilize the New Resource modelling capabilities Clusterware start failure(s) will trigger environment checks with detailed logging to help pinpoint probable mis-configurations • Permission • Network/Storage Human readable messages in trace files during runtime failures for faster issue resolution Additional checks to ensure cluster interconnect is configured correctly
- 36. Agenda What’s new with Cache Fusion? 1 2 36 3
- 37. Cache Fusion: A long Journey BufferB B B Private Network Private Network B Private Network B Before Cache Fusion Cache Fusion • Multiple LMSs • Higher Priority • Auto Tune # of LMS • BOC Synchronization • Integration with DRF • Dynamic GRD resizing Oracle 8i Oracle 19c B 2 3 1 2 B 1
- 38. • During Startup • Resources are distributed across nodes • GRD maintains information on these resources • Resource Master may or may not be on same node as the resource • Steady State • DRM (Dynamic Re-Mastering) helps moves Resource Master to same node as Resource 38 Optimize Resource Master placement B BB B B BB B B BB B B BB B B BB B B BB B B BB B B BB B B BB B B BB B B BB B B BB B B BB B B BB B B BB B B BB B M MM M M MM M M MM M M MM M M MM M M MM M M MM M M MM M Goal is to reduce 3- way communication providing performance equal to Single Instance Global Resource Directory B Buffer M MasterB M
- 39. Oracle RAC Performance Automation • Note 1619155.1 Best Practices and Recommendations for RAC databases with SGA size over 100GB • Automatic configuration in 19c • Dynamic CR slaves to deal with changing workload • LMS CR slaves (_max_cr_rollbacks deprecated) (1630755.1) • Reduce “LMS process busy” event in AWR report • LMS CR Slaves • Dynamic DLM ticket adjustment to prevent hangs • remove _lm_tickets
- 40. • Subnet Manager for Fast Node Death detection • Network (Subnet Manager) • Disk (Diskmon) • Utilize low latency RDMA • Read/Write to remote memory without CPU • More details available at • https://www.slideshare.net/AnilNair2 7/oracle-rac-features-on-exadata Oracle RAC Exadata optimizations 0.8 30 0 10 20 30 40 Exadata Generic Systems Fast Node Death Detection Exafusion Smart Fusion Block Transfer
- 41. Oracle RAC at PayPal
- 42. At PayPal, we put people at the center of everything we do.
- 43. 150+ Oracle RAC Clusters Characteristics of Database platform 5M+ Execs/Sec 25% Y-o-Y DB Storage Growth 50+ PB Total DB Storage Across Databases 18M+ Logical Reads/Sec 4 x Oracle X7-8 (768 Cores, 24 TB memory) 200k Execs/Sec 300k GC/GE messages/Sec Extremely Busy OLTP RAC Cluster 2M+ IC packets sent/received < 0.250 µs Avg GC message latency < 4ms Avg SQL call latency 75K Execs/Sec on a table
- 44. Why PayPal adopted Oracle RAC Scalability limits Unpredictable availability Active/Passive configuration Single instance databases Does not meet business goals Capacity wastage
- 45. How did PayPal achieve HA and scalability? o Introduction of new technology and concepts Ø Oracle Real Application Clusters(RAC) on Oracle x86 server with IB for interconnect Ø Smart routing of read/write calls to instances on primary database Ø Shrinking buffer cache to reduce reconfiguration duration during maintenance Ø Oracle RAC based ADG/GG reader farms for read only and latency tolerant applications Ø Fail fast Read Only cluster to failover traffic to secondary cluster in < 10 seconds o Benefits Ø Elastic scalability Ø Availability improved by 10x Ø Eliminated Single Point Of Failure(SPOF) Ø Primary database Instance failover improved by 10x Ø Leverage all allocated capacity means reduced CAPEX
- 46. Oracle 19c RAC features addressing our needs/problems? o Availability improvement by 4x by introducing, Ø more LMS processes where enough compute capacity available Ø optimized algorithms reducing Oracle RAC reconfiguration duration Ø DBMS_CACHEUTIL helping with grab and dissolve resource affinity Ø Instance pairing for crash redo recovery o Improvements in scalability and Performance, Ø Oracle RAC scalable sequences Ø improved cache locality reducing interconnect traffic Ø Smart fusion block transfer eliminating redo log write latency on Exadata Ø commit cache for recent transactions reducing block transfers between nodes o Exadata Adoption Ø Benefit from Oracle RAC features on Exadata
- 47. Summary • Oracle RAC is the proven choice for Scalability and Availability without any application changes