Oracle Exadata Best Practices (Doc ID 757552.

1)

Do
 

c  

ID
Oracle Exadata Database Machine
Setup/Configuration Best Practices (Doc ID
1274318.1)

APPLIES TO:

Oracle Database Cloud Schema Service - Version N/A and later

Oracle Database Cloud Service - Version N/A and later
Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.2.0.1 [Release 12.1 to
12.2]
Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine) - Version
N/A and later
Oracle Database Backup Service - Version N/A and later
Information in this document applies to any platform.

PURPOSE

The goal of this document is to present the best practices for the deployment of 
Oracle Exadata Database Machine  in the area of Setup and Configuration. 

SCOPE

General audience working on Oracle Exadata

DETAILS

Primary and standby databases should NOT reside on the same IB Fabric
Use hostname and domain name in lower case
Verify ILOM Power Up Configuration
Verify Hardware and Firmware on Database and Storage Servers
Verify InfiniBand Cable Connection Quality
Verify Ethernet Cable Connection Quality
Verify InfiniBand Fabric Topology (verify-topology)
Verify key InfiniBand fabric error counters are not present
Verify InfiniBand switch software version is 1.3.3-2 or higher
Verify InfiniBand subnet manager is running on an InfiniBand switch
Disable Infiniband subnet manager service where subnet manager master should never
run
Verify key parameters in the InfiniBand switch /etc/opensm/opensm.conf file
Verify There Are No Memory (ECC) Errors
Verify celldisk configuration on disk drives
Verify celldisk configuration on flash memory devices
Verify there are no griddisks configured on flash memory devices
Verify griddisk count matches across all storage servers where a given prefix name
exists
Verify griddisk ASM status
Verify that griddisks are distributed as expected across celldisks
Verify the percent of available celldisk space used by the griddisks
Verify Database Server ZFS RAID Configuration
Verify InfiniBand is the Private Network for Oracle Clusterware Communication
Verify InfiniBand Address Resolution Protocol (ARP) Configuration on Database Servers
Verify Oracle RAC Databases use RDS Protocol over InfiniBand Network.
Verify Database and ASM instances use same SPFILE
Verify Berkeley Database location for Cloned GI homes
Configure Storage Server alerts to be sent via email
Configure NTP and Timezone on the InfiniBand switches
Configure NTP slew_always settings as SMF property for Solaris
Verify NUMA Configuration
Enable Xeon Turbo Boost 
Verify Exadata Smart Flash Log is Created
Verify Exadata Smart Flash Cache is Created
Verify Exadata Smart Flash Cache status is "normal"
Verify Master (Rack) Serial Number is Set
Verify Management Network Interface (eth0) is on a Separate Subnet
Verify RAID disk controller CacheVault capacitor condition
Verify RAID Disk Controller Battery Condition
Verify Ambient Air Temperature
Verify MaxStartups 100 in /etc/ssh/sshd_config on all database servers
Verify all datafiles have AUTOEXTEND attribute ON
Verify all BIGFILE tablespaces have non-default MAXBYTES values set
Ensure Temporary Tablespace is correctly defined
Enable portmap service if app requires it
Enable proper services on database nodes to use NFS
Be Careful when Combining the InfiniBand Network across Clusters and Database
Machines
Enable auditd on database servers
Verify AUD$ and FGA_LOG$ tables use Automatic Segment Space Management
Use dbca templates provided for current best practices
Updating database node OEL packages to match the cell
Disable cell level flash caching for grid disks that don't need it when using Write Back
Flash Cache
Gather system statistics in Exadata mode if needed
Verify Hidden Database Initialization Parameter Usage
Verify BDB location for Cloned GI homes
Verify Shared Servers do not perform serial full table scans
Verify Write Back Flash Cache minimum version requirements
Verify bundle patch version installed matches bundle patch version registered in
database
Verify database server file systems have "Maximum mount count" = "-1"
Verify database server file system have "Check interval" = "0"
Verify Automated Service Request (ASR) configuration
Verify ZFS File System User and Group Quotas are configured
Verify the file /.updfrm_exact does not exist
Verify the vm.min_free_kbytes configuration
Validate key sysctl.conf parameters on database servers
Remove "fix_control=32" from dbfs mount options
Set Linux kernel log buffer size to 1MB
Verify IP routing configuration on DB nodes
Set SQLNET.EXPIRE_TIME=10 in DB Home
Verify there are no .fuse_hidden files under the dbfs mount
Verify that the SDP over IB option "sdp_apm_enable(d)" is set to "0"
Verify /etc/oratab
Verify consistent software and configuration across nodes
Verify all database and storage servers time server configuration
Verify Sar files have read permissions for non-root user
Verify that the patch for bug 16618055 is applied
Verify kernels and initrd in /boot/grub/grub.conf are available on the system
Verify basic Logical Volume(LVM) system devices configuration
Ensure db_unique_name is unique across the enterprise
Verify average ping times to DNS nameserver
Verify Running-config and Startup-config are the same on the Cisco switch
Validate SSH is installed and configured on Cisco management switch
Verify Database Memory Allocation is not Greater than Physical Memory Installed on
Database node
Verify Cluster Verification Utility(CVU) Output Directory Contents Consume < 500MB of
Disk Space
Verify active system values match those defined in configuration file "cell.conf"
Verify that CRS_LIMIT_NPROC is greater than 65535 and not "UNLIMITED"
Verify TCP Segmentation Offload (TSO) is set to off
Check alerthistory for stateful alerts not cleared
Check alerthistory for non-test open stateless alerts
Verify clusterware state is "Normal"
Verify the grid Infrastructure management database (MGMTDB) does not use
hugepages
Verify the "localhost" alias is pingable
Verify bundle patch version installed matches bundle patch version registered in
database
Verify database is not in DST upgrade state
Verify there are no failed diskgroup rebalance operations
Verify the CRS_HOME is properly locked
Verify storage server data (non-system) disks have no partitions
Verify db_unique_name is used in I/O Resource Management (IORM) interdatabase
plans
Verify Datafiles are Placed on Diskgroups consisting of griddisks with cachingPolicy =
DEFAULT
Verify all datafiles are placed on griddisks that are cached on flash disks
Validate key sysctl.conf parameters on database servers
Detect duplicate files in /etc/*init* directories
Verify Database Server Quorum Disks configuration
Verify Oracle Clusterware files are placed appropriately
Verify "_reconnect_to_cell_attempts=9" on database servers which access X6 storage
servers
Verify passwordless SSH connectivity for Enterpise Manager (EM) agent owner userid
to target component userids
Check /EXAVMIMAGES on dom0s for possible over allocation by sparse files
Verify active kernel version matches expected version for installed Exadata Image
Verify Storage Server user "CELLDIAG" exists
Verify installed rpm(s) kernel type match the active kernel version
Verify Flex ASM Cardinality is set to "ALL"
Verify "downdelay" is set correctly for bonded client interfaces
Verify ExaWatcher is executing
Verify non-Default services are created for all Pluggable Databases
Verify Grid Infrastructure Management Database (MGMTDB) configuration
Verify Automatic Storage Management Cluster File System (ACFS) file systems do not
contain critical database files
Verify the ownership and permissions of the "oradism" file
Verify the SYSTEM, SYSAUX, USERS and TEMP tablespaces are of type bigfile
Verify the storage servers in use configuration matches across the cluster
Verify "asm_power_limit" is greater than zero
Verify the recommended patches for Adaptive features are installed
Verify initialization parameter cluster_database_instances is at the default value
Verify the database server NVME device configuration
Verify that Automatic Storage Management Cluster File System (ACFS) uses 4K
metadata block size
Evaluate Automated Maintenance Tasks configuration
Verify proper ACFS drivers are installed for Spectre v2 mitigation
Verify Exafusion Memory Lock Configuration
Verify there are no unhealthy InfiniBand switch sensors
Refer to MOS 1682501.1 if non-Exadata components are in use on the InfiniBand fabric
Verify the ib_sdp module is not loaded into the kernel
Verify all voting disks are online
Verify available ksplice fixes are installed
Verify the currently active image status
Verify that critical files reside on high redundancy disk groups
Verify that persistent memory (PMEM) is properly configured
Review Cluster Verification Utility (CVU) output
Verify diskgroup attributes are set to enable Hardware Assisted Resilient Data (HARD)
capability
Verify there are no unused Automatic Storage Management Cluster File System (ACFS)
file systems

Verify Cluster Synchronization Services (CSS) misscount value

Primary and standby databases should NOT reside on the same IB Fabric

Priority Added Machine Type OS Exadata Oracle

Type Version Version
Critical N/A X2-2(4170), X2- Linux 11.2.x + 11.2.x +
2, X2-8,X4-2
 

Benefit / Impact:

To properly protect the primary databases residing on the "primary" Exadata Database
Machine, the physical standby database requires fault isolation from IB switch
maintenance issues,
IB switch failures and software issues, RDS bugs and timeouts or any issue resulting
from a complete IB fabric failure. To protect the standby from these failures that
impact the primary's
availability, we highly recommend that at least one viable standby database resides on
a separate IB fabric.

 
Risk:

If the primary and standby resides on the same IB fabric, both primary and standby
systems can be unavailable due a bug causing an IB fabric failure.

Action / Repair:

The primary and at least one viable standby database must not reside on the same
inter-racked Exadata Database Machine. The communication between the primary and
standby
Exadata Database Machines must use GigE or 10GigE. The trade-off is lower network
bandwidth. The higher network bandwidth is desirable for standby database
instantiation
(should only be done first time) but that requirement is eliminated for post-failover
operations when flashback database is enabled.

Use hostname and domain name in lower case

Priority Added Machine Type OS Exadata Oracle

Type Version Version
Critical N/A X2-2(4170), X2- Linux, 11.2.x + 11.2.x +
2, X2-8, X4-2 Solari
 

Benefit / Impact:

Using lowercase will avoid known deployment time issues.

Risk:

OneCommand deployment will fail in step 16 if this is not done. This will abort the
installation with:

"ERROR: unable to locate file to check for string 'Configure Oracle Grid Infrastructure
for a Cluster ... succeeded' #Step 16#"

Action / Repair:

As a best practice, user lower case for hostnames and domain names

 
Verify ILOM Power Up Configuration

Priority Alert Date Owner Status Scope Bug(s)

Level
Critical FAIL 11/11/12 <Name>Production Exadata, SSC 14281920-
exachk
DB DB Engineered Exadata OS & Validation TBD
Version Role System Version Version Tool
Version
N/A N/A X2-2(4170), X2- 11.2.2.2.0+ Solaris - 11 exachk 2.2.2  
2, X2-8, X3-2, Linux x86-
X3-8,X4-2 64 UEK5.8
Benefit / Impact:

Verifying the ILOM power up configuration helps to ensure that a server (or more) are
booted up after a power interruption as quickly as possible.

Risk:

Not verifying the ILOM power up configuration may result in unexpected server boot
behavior after a power interruption.

Action / Repair:

To verify the ILOM power up configuration, as the root userid enter the following
command on each database and storage server:

if [ -x /usr/bin/ipmitool ]
then
#Linux
ipmitool sunoem cli force "show /SP/policy" | grep -i power
else
#Solaris
/opt/ipmitool/bin/ipmitool sunoem cli force "show /SP/policy" | grep -i power
fi;

The output varies by Exadata software version and should be similar to:

Exadata software version 11.2.3.2.1 or higher:

HOST_AUTO_POWER_ON=disabled
HOST_LAST_POWER_STATE=enabled

Exadata software version 11.2.3.2.0 or lower:

HOST_AUTO_POWER_ON=enabled
HOST_LAST_POWER_STATE=disabled

If the output is not as expected, as the root userid use the ipmitool "set /SP/policy"
command. For example:

# ipmitool sunoem cli force "set /SP/policy HOST_AUTO_POWER_ON=enabled"

Connected. Use ^D to exit.
-> set /SP/policy HOST_AUTO_POWER_ON=enabled
Set 'HOST_AUTO_POWER_ON' to 'enabled'
-> Session closed
Disconnected

 Verify Hardware and Firmware on Database and Storage Servers

Priority Added Machine Type OS Exadata Oracle

Type Version Version
Critical N/A X2-2(4170), X2-2, Linux 11.2.x + 11.2.x +
X2-8, X4-2
Benefit / Impact:

The Oracle Exadata Database Machine is tightly integrated, and verifying the hardware
and firmware before the Oracle Exadata Database Machine is placed into or returned to

production status can avoid problems related to the hardware or firmware

modifications.

The impact for these verification steps is minimal.

Risk:

If the hardware and firmware are not validated, inconsistencies between database and
storage servers can lead to problems and outages.

Action / Repair:

To verify the hardware and firmware configuration for a database server, execute the
following command as the "root" userid:

/opt/oracle.SupportTools/CheckHWnFWProfile

The output will contain a line similar to the following:

[SUCCESS] The hardware and firmware profile matches one of the supported
profile

If any result other than "SUCCESS" is returned, investigate and correct the condition.

To verify the hardware and firmware configuration for a storage server, execute the
following "cellcli" command as the "cellmonitor" userid:

CellCLI> alter cell validate configuration

The output will be similar to:

Cell <cell> successfully altered

If any result other than "successfully altered" is returned, investigate and correct the
condition.

NOTE: CheckHWnFWProfile is also executed at each boot of a database server.

NOTE: "alter cell validate configuration" is also executed once a day on a storage
server by the MS process and the result is written into the storage server alert history.

Verify InfiniBand Cable Connection Quality

Priority Added Machine Type OS Exadata Oracle

Type Version Version
Critical N/A X2-2(4170), X2-2, Linux 11.2.x + 11.2.x +
X2-8,X4-2
Benefit / Impact:

InfiniBand cables require proper connections for optimal efficiency. Verifying the
InfiniBand cable connection quality helps to ensure that the InfiniBand network
operates at optimal efficiency.

There is minimal impact to verify InfiniBand cable connection quality.

Risk:

InfiniBand cables that are not properly connected may negotiate to a lower speed,
work intermittently, or fail.

Action / Repair:
Execute the following command on all database and storage servers:

for ib_cable in `ls /sys/class/net | grep ^ib`; do printf "$ib_cable: ";

cat /sys/class/net/$ib_cable/carrier; done

The output should look similar to:

ib0: 1
ib1: 1

If anything other than "1" is reported, investigate that cable connection

Linux

Execute the following command as the "root" userid on all database and storage
servers:

for ib_cable in `ls /sys/class/net | grep ^ib`; do printf "$ib_cable: ";

cat /sys/class/net/$ib_cable/carrier; done

The output should look similar to:

ib0: 1 ib1: 1

If anything other than "1" is reported, investigate that cable connection.

Solaris

Execute the following command as the "root" userid on all database servers:

dladm show-ib | grep -v LINK | sed -e 's/ */ /g' -e 's/ *//' | awk '{print
$1":", $5}'| sort

The output should be similar to:

ib0: up
ib1: up

If anything other than "up" is reported, investigate that cable connection.

NOTE: Storage servers should report 2 connections. X2-2(4170) and X2-2 database
servers should report 2 connections. X2-8 database servers should report 8
connections.

Verify Ethernet Cable Connection Quality

Priority Added Machine Type OS Exadata Oracle

Type Version Version
Critical N/A X2-2(4170), X2-2, Linux 11.2.x + 11.2.x +
X2-8,X4-2
Benefit / Impact:

Ethernet cables require proper connections for optimal efficiency. Verifying the
Ethernet cable connection quality helps to ensure that the Ethernet network operates
at optimal efficiency.

There is minimal impact to verify Ethernet cable connection quality.

Risk:

Ethernet cables that are not properly connected may negotiate to a lower speed, work
intermittently, or fail.

Action / Repair:

Execute the following command as the root userid on all database and storage servers:

for cable in `ls /sys/class/net | grep ^eth`; do printf "$cable: "; cat
/sys/class/net/$cable/carrier; done

The output should look similar to:

eth0: 1
eth1: cat: /sys/class/net/eth1/carrier: Invalid argument
eth2: cat: /sys/class/net/eth2/carrier: Invalid argument
eth3: cat: /sys/class/net/eth3/carrier: Invalid argument
eth4: 1
eth5: 1

"Invalid argument" usually indicates the device has not been configured and is not in
use. If a device reports "0", investigate that cable connection.

NOTE: Within machine types, the output of this command will vary by customer
depending on how the customer chooses to configure the available ethernet cards.
 

Verify the InfiniBand Fabric Topology (verify-topology)

Priority Alert Date Owner StatusEngineered Engineered Bug(s)

Level System System
Platform
Critical  WARN  09/05/18  <Name>  Production  Exadata - ALL  20144798 -
Physical, exachk 
Exadata -
Management
Domain,
RA 
DB DB DB Role DB Exadata OS & Validation MAA
Versio Type Mode Version Version Tool Scorecard
n Version Section
N/A  N/A  N/A  N/A  ALL  Linux  exachk N/A 
18.4.0 
Benefit / Impact:

Verifying that the InfiniBand network is configured with the correct topology for an
Oracle Exadata Database Machine helps to ensure that the InfiniBand network
operates at maximum efficiency.

Risk:

An incorrect InfiniBand topology will cause the InfiniBand network to operate at

degraded efficiency, intermittently, or fail to operate.

Action / Repair:

To verify the InfiniBand Fabric Topology, execute the following code set as the "root"
userid on one database server in the Exadata environment:

unset VT_ERRORS
unset VT_WARNINGS
VT_OUTPUT=$(/opt/oracle.SupportTools/ibdiagtools/verify-topology)
VT_WARNINGS=$(echo "$VT_OUTPUT" | egrep WARNING)
VT_ERRORS=$(echo "$VT_OUTPUT" | egrep ERROR)
if [ -n "$VT_ERRORS" ]
then
echo -e "FAILURE: verify-topology returned one or more errors (and perhaps
warnings).\nDETAILS:\n$VT_OUTPUT"
elif [ -n "$VT_WARNINGS" ]
then
echo -e "WARNING: verify-topology returned one or more
warnings.\nDETAILS:\n$VT_OUTPUT"
else
echo -e "SUCCESS: verify-topology returned no errors or warnings."
fi

The expected output is:

SUCCESS: verify-topology returned no errors or warnings.

An example of a "FAILURE:" message:

FAILURE: verify-topology returned one or more errors (and perhaps warnings).

DETAILS:
[ DB Machine Infiniband Cabling Topology Verification Tool. ]
Every node is connected to two leaf switches in a single
rack.......................................................[FAILED]
[ERROR]
Node randomcel06 (Guid: 21280001f00464 ) is connected to just one leaf switch
randomsw-ib2(Guid: 2128f57723a0a0 )
Error found in following rack
<output truncated>

An example of a "WARNING:" message:

WARNING: verify-topology returned one or more warnings.

DETAILS:

[ DB Machine Infiniband Cabling Topology Verification Tool ]

[Version IBD VER 2.b ]

[WARNING] - Non-Exadata nodes detected! Please ensure this is OK

Approximating classification into cells and db hosts

Software UPGRADE required for the tool to be accurate

Looking at 1 rack(s).....
<output truncated>

If anything other than "SUCCESS:" is reported, investigate and correct the underlying
fault(s).

Verify key InfiniBand fabric error counters are not present

Priority Alert Date Owner Status Engineered Bug(s

Level System )
Critical WARN 09/28/16 <Name> Production Exadata-  
Management
Domain, Exadata-
 Physical, SSC,
Exalogic
DB DB Engineered Exadata OS & Validation Tool TBD
Version Role System Platform Version Version Version
N/A N/A X2-2(4170), X2-2, 11.2.x+ Linux x86- exachk 12.1.0.2.8  
X2-8, X3-2, X3-8, 64
X4-2, X4-8, X5-2, Solaris -
X6-2, X6-8 11
Benefit / Impact:

Verifying key InfiniBand fabric error counters are not present helps to maintain the
InfiniBand fabric at peak efficiency.

The impact of verifying key InfiniBand fabric error counters are not present is minimal.
The impact of correcting key InfiniBand fabric error counters varies depending upon
the root cause of the specific error counter present, and cannot be estimated here.

Risk:

If key InfiniBand fabric error counters are present, the fabric may be running in
degraded condition or lack redundancy.

NOTE: Uncorrected symbol errors increase the risk of node evictions and
application outages.

Action / Repair:

To verify key InfiniBand fabric error counters are not present, execute the following
command set as the "root" userid on one database server:

NOTE: This will not work in the user domain of a virtualized environment.

if [[ -d /proc/xen && ! -f /proc/xen/capabilities ]]

then
       echo -e "\nThis check will not run in a user domain of a virtualized
environment. Execute this check in the management domain.\n"
else
     RAW_DATA=$(ibqueryerrors | egrep 'SymbolError|LinkDowned|RcvErrors|
RcvRemotePhys|LinkIntegrityErrors');
     CRITICAL_DATA=$(echo "$RAW_DATA" | egrep 'SymbolError|RcvErrors');
      WARNING_DATA=$(echo "$RAW_DATA" | egrep -v 'SymbolError|RcvErrors');
     if [ -z "$RAW_DATA" ]
     then
              echo -e "SUCCESS: Key InfiniBand fabric error counters were not found"
     else
     if [ 'echo "$RAW_DATA" | egrep 'SymbolError|RcvErrors' | wc -l' -gt 0 ]
     then
           echo -e "FAILURE: receive errors or symbol errors or both were
found:\n\nCounters found:\n"
           echo -e "CRITICAL DATA:\n\n$CRITICAL_DATA\n\n\nWARNING
DATA:\n\n$WARNING_DATA"
    else
          echo -e "WARNING: Key InfiniBand fabric error counters were
found\n\nCounters Found:\n";
          echo -e "CRITICAL DATA:\n\n$CRITICAL_DATA\n\n\nWARNING
DATA:\n\n$WARNING_DATA"
    fi;
  fi;
fi;

The expected output should be:

SUCCESS: Key InfiniBand fabric error counters were not found

- OR -

This check will not run in a user domain of a virtualized environment. Execute this
check in the management domain.

Example of a FAILURE result:

FAILURE: receive errors or symbol errors or both were found:

Counters found:

CRITICAL DATA:

GUID 0x10e00001451161 port 1: [SymbolErrorCounter == 1367] [PortRcvErrors ==

1367]
GUID 0x10e08027b8a0a0 port ALL: [SymbolErrorCounter == 54679]
[LinkErrorRecoveryCounter == 76]
<output truncated>

WARNING DATA:

GUID 0x21280001fca219 port 1: [LinkDownedCounter == 1]

GUID 0x21280001fca21a port 2: [LinkDownedCounter == 1]
<output truncated>

Example of a WARNING result:

WARNING: Key InfiniBand fabric error counters were found

CRITICAL DATA:

WARNING DATA:

GUID 0x10e00001886289 port 1: [LinkDownedCounter == 1] [PortXmitDiscards ==

272] [PortXmitWait == 2021116]
GUID 0x10e0802617a0a0 port ALL: [LinkErrorRecoveryCounter == 63]
GUID 0x10e0802617a0a0 port 1: [LinkErrorRecoveryCounter == 10]
GUID 0x10e0802617a0a0 port 2: [LinkErrorRecoveryCounter == 11]
<output truncated>

In general, if the output is not "SUCCESS...", follow the diagnostic guidance in the
following documents:

InfiniBand Network Troubleshooting Guidelines and Methodologies.

"Gathering Troubleshooting Information for the Infiniband Network in Engineered
Systems (Doc ID 1538237.1)".
The "Exadata InfiniBand Issues" section of "Exadata Diagnostic Collection Guide
(Doc ID 1353073.2)".

Special Notes on Symbol errors:

Symbol errors create a much higher risk of node evictions if the error rate is too high.
On the InfiniBand switches, there is a mechanism that will automatically down a port if
the error rate becomes too high. On the database and storage servers, there is no
such mechanism at this time, so it is recommended to examine the Symbol error rate
manually, using ExaWatcher data.

 
NOTE: In the following example, all data pertaining to InfiniBand switches has been
filtered out for brevity.

As the "root" userid, the following example demonstrates how to examine the Symbol
error rate using ExaWatcher.

1) From the manual output, make note of the GUIDs with SymbolErrorCounter present:

FAILURE: receive errors or symbol errors or both were found:

  Counters found:   

   

GUID 0x10e00001451161 port 1: [SymbolErrorCounter == 1123] [PortRcvErrors

== 1123] [PortXmitWait == 230121020]   

<output turncated>

2) Use the following command to identify the server with the symbol errors present:

[root@randomadm01 ~]# ibqueryerrors -G 0x10e00001451161 | head -1 Errors

for "randomadm02 S 192.xxx.8.16,192.xxx.8.17 HCA-4"

3) Log onto the database server identified in the command above, randomadm02.

4) Change to the ExaWatcher directory for IB hca information (the default is in use
here):
 

# cd /opt/oracle.ExaWatcher/archive/IBCardInfo.ExaWatcher

5) Using the port identification provided in 1), use the following output to condense
(removes "0" entries) all relevant available ExaWatcher data:

[root@randomadm02 IBCardInfo.ExaWatcher]# cat <(bzcat *.bz2) <(cat *.dat) |

egrep "port 1" -A23 | egrep SymbolError | grep -v '0[[:blank:][:cntrl:]]*$' | sort
-k1.2,10 -k2.1,8

            

[09/13/2016 02:38:18] SymbolErrorCounter           999                    1

[09/13/2016 02:43:20] SymbolErrorCounter           1030                   31             

 <output truncated>           

[09/13/2016 17:28:56] SymbolErrorCounter           1062                   1

[09/13/2016 17:39:00] SymbolErrorCounter           1085                   23

[09/13/2016 17:59:10] SymbolErrorCounter           1100                   5

<output truncated>

6) Calculate the symbol error rate per minute. By default, ExaWatcher data intervals
are 5 minutes, but that can be changed. Using these two lines:

[09/13/2016 17:28:56] SymbolErrorCounter           1062                   1               

[09/13/2016 17:39:00] SymbolErrorCounter           1085                   23 

 
 The delta between 17:28 and 17:39 is "23". The time interval is 10 minutes, so
23 / 10 is 2.3 symbol errors per minute.

NOTE ESPECIALLY!! If the symbol error rate is consistently greater than 2

per minute, investigate for root cause and take corrective action!

NOTE: The InfiniBand fabric error counters should be cleared and validated after any
maintenance activity.

NOTE: The InfiniBand fabric error counters are cumulative and the errors may have
occurred at any time in the past. This check is the result at one point in time, and
cannot advise anything about history or an error rate.

NOTE: This check should not be considered complete validation of the InfiniBand
fabric. Even if this check indicates success, there may still be issues on the InfiniBand
fabric caused by other, more rare Infiinband fabric error counters being present. If
there are or appear to be issues with the InfiniBand fabric while this check passes,
perform a full evaluation of the "ibqueryerrors" command output and the output of
other commands such as "ibdiagnet".

NOTE: Depending upon the Exadata version, the key InfiniBand fabric error
counters have different names. In the following list, the older version of the
counter name is shown in square brackets.

Key Infiniband fabric error counters list:

SymbolErrorCounter [SymbolErrors]
LinkErrorRecoveryCounter [LinkRecovers]
LinkDownedCounter [LinkDowned]
PortRcvErrors [RcvErrors]
PortRcvRemotePhysicalErrors [RcvRemotePhysErrors]
LocalLinkIntegrityErrors [LinkIntegrityErrors]

NOTE: Some Infiinband fabric error counters (for example, "SymbolErrorCounter

[SymbolErrors]","PortRcvErrors [RcvErrors]") can increment when nodes are rebooted.
Small values for these Infiinband fabric error counters which are less than the
"LinkDownedCounter [LinkDowned]" counters are generally not a problem. The
"LinkDownedCounter [LinkDowned]" counters indicate the number of times the port
has gone down (usually for valid reasons, such as a node reboot) and are not typically
an error indicator by themselves.

NOTE: Links reporting high, persistent error rates (especially "SymbolErrorCounter

[SymbolErrors]", "LinkErrorRecoveryCounter [LinkRecovers]", "PortRcvErrors
[RcvErrors]", "LocalLinkIntegrityErrors [LinkIntegrityErrors]") often indicate a bad or
loose cable or port issues.

Verify InfiniBand switch software version is 1.3.3-2 or higher

Priority Added Machine Type OS Type Exadata Oracle

Version Version
Critical 11/01/11 X2-2(4170), X2-2, Linux, 11.2.x + 11.2.x +
X2-8,X4-2 [WIP:VW]Solaris
Bug(s): Unpublished bug 13325598 (exachk)

Benefit / Impact:

The Impact of verifying that the InfiniBand switch software is at version 1.3.3-2 or
higher is minimal. The impact of upgrading the InfiniBand switch(s) to 1.3.3-2 varies
depending upon the upgrade method
chosen and your current InfiniBand switch software level.

Risk:

InfiniBand switch software version 1.3.3-2 fixes several potential InfiniBand fabric
stability issues. Remaining on an InfiniBand switch software version below 1.3.3-2
raises the risk of experiencing a potential outage.

Action / Repair:

To verify the InfiniBand switch software version, log onto the InfiniBand switch and
execute the following command as the "root" userid:
version | head -1 | cut -d" " -f5

The output should be similar to:

1.3.3-2

If the output is not 1.3.3-2 or higher, upgrade the InfiniBand switch software to at
least version 1.3.3-2.

NOTE: Patch 12373676 provides InfiniBand software version 1.3.3-2 and instructions.

NOTE: Upgrading to 1.3.3-2 may be performed as a rolling upgrade without an outage.

The InfiniBand switch software is not dependent upon any other components in the
Oracle Exadata Database Machine
and may be upgraded at any time.

NOTE: If your InfiniBand switch is at software version 1.0.1-1, it will need to first be
upgraded to 1.1.3-1 or 1.1.3-2 before it can be upgraded to 1.3.3-2. The InfiniBand
switch software cannot be upgraded
directly from 1.0.1-1 to 1.3.3-2.

Verify the Master Subnet Manager is running on an InfiniBand switch

Priority Alert Date Owner Status

Engineered Engineered Bug(s)
Level System System
Platform
Critical FAIL 11/28/1 <Name Development Exadata - ALL 28862740 -
8 > Physical, exachk
Exadata -
Managment
Domain
DB/GI DB DB Role DB Exadata OS & Validation MAA
Version Type Mode Version Version Tool Scorecard
Version Section
N/A N/A N/A N/A ALL Linux exachk N/A
18.4.0
Benefit / Impact:

Having the Master Subnet Manager reside in the correct location improves the stability,
availability and performance of the InifiniBand fabric. The Impact of verifying the
Master Subnet Manager is running on an InfiniBand switch is minimal. The impact of
moving the Master Subnet Manager varies depending upon where it is currently
executing and to where it will be relocated.

Risk:

If the Master Subnet Manager is not running on an InfiniBand switch, the InfiniBand
fabric may crash during certain fabric management transitions.

Action / Repair:

To verify the Master Subnet Manager is located on an InfiniBand switch, execute the
following command set as the "root" userid on a database server:

SUBNET_MGR_MSTR_OUTPUT=$(sminfo)
IBSWITCHES_OUTPUT=$(ibswitches)
SUBNET_MGR_MSTR_GID=$(echo "$SUBNET_MGR_MSTR_OUTPUT" | cut -d" " -f7 | cut
-c3-16)
SUBNET_MGR_MSTR_LOC_RESULT=1
for IB_NODE_GID in $(echo "$IBSWITCHES_OUTPUT" | cut -c14-27)
do
if [ $SUBNET_MGR_MSTR_GID = $IB_NODE_GID ]
then
SUBNET_MGR_MSTR_LOC_RESULT=0
SUBNET_MGR_MSTR_LOC_SWITCH=$(echo "$IBSWITCHES_OUTPUT" | grep
$IB_NODE_GID)
fi
done
if [ $SUBNET_MGR_MSTR_LOC_RESULT -eq 0 ]
then
echo -e "SUCCESS: the Master Subnet Manager is executing on InfiniBand
switch:\n$(echo "$SUBNET_MGR_MSTR_LOC_SWITCH")"
else
echo -e "FAILURE: the Master Subnet Manager does not appear to be executing
on an InfiniBand switch:\n$(echo "$SUBNET_MGR_MSTR_OUTPUT")"
fi

The output should be similar to:

SUCCESS: the Master Subnet Manager is executing on InfiniBand switch:

Switch : 0x002128469b03a0a9 ports 36 "SUN DCS 36P QDR randomsw-iba0 <IP>"
enhanced port 0 lid 1 lmc 0

Example of a "FAILURE" result:

FAILURE: the Master Subnet Manager does not appear to be executing on an

InfiniBand switch:
sminfo: sm lid 3 sm guid 0x10e0cdce81a0a9, activity count 3362634 priority 8
state 3 SMINFO_MASTER

If the result is "FAILURE", investigate the guid provided, relocate the Master
Subnet Manager to a correct InfiniBand switch, and prevent the Subnet
Manager from starting on the component where the Master Subnet Manager
was found to be executing.

NOTES:

1. The InfiniBand network can have more than one Subnet Manager, but
only one Subnet Manager is active at a time. The active Subnet Manager
is the Master Subnet Manager. The other Subnet Managers are the
Standby Subnet Managers. If a Master Subnet Manager is shut down or
fails, then a Standby Subnet Manager automatically becomes the Master
Subnet Manager.
2. There are typically several Standby Subnet Managers waiting to take over
should the current Master Subnet Manager either fail or is manually
moved to some other component with an available Standby Subnet
Manager. Only run Subnet Managers on the InfiniBand switches specified
for use in Oracle Exadata Database Machine, Oracle Exalogic Elastic
Cloud, Oracle Big Data Appliance, and Oracle SuperCluster. Running
Subnet Manager on any other device is not supported.
3. For pure multirack Exadata deployments with less than 4 racks, the
Subnet Manager should run on all spine and leaf InfiniBand switches. For
deployments with 4 or more Exadata racks, the Subnet Manager should
run only on spine InfiniBand switches. For additional configuration
information, please see section "4.6.7 Understanding the Network Subnet
Manager Master" of the "Exadata Database Machine Maintenance Guide".
4. For InfiniBand fabric configurations that involve a mix of different Oracle
Engineered Systems, please refer to: MOS note 1682501.1
5. Moving the Master Subnet Manager is sometimes required during
maintenance and patching operations. For additional guidance on
maintaining the Master Subnet Manager, please see section "4.6
Maintaining the InfiniBand Network" of the "Exadata Database Machine
Maintenance Guide".

  Verify the Subnet Manager is properly disabled

Priority Alert Date Owner Status

Engineered Engineered Bug(s)
Level System System
Platform
Critical FAIL 11/28/1 <Name Development Exadata - ALL 28768896-
8 > Physical, exachk
 Exadata - 14534296-
Managment exachk
Domain 16270663-
exachk
16795289-
exachk
DB/GI DB DB Role DB Exadata OS & Validation MAA
Version Type Mode Version Version Tool Scorecard
Version Section
N/A N/A N/A N/A ALL Linux exachk N/A
18.4.0
Benefit / Impact:

NOTE: The Subnet Manager should only execute on InfiniBand switches. It should be
disabled on any other component attached to an InfiniBand fabric.

Having the Subnet Manager executing in the correct locations improves the
stability, availability and performance of the InifiniBand fabric. The Impact of
verifying the Subnet Manager is disabled on components where the Master
Subnet Manager should never reside is minimal. The impact of disabling the
Subnet Manager varies depending upon the component type where it is found
to be incorrectly executing, and whether or not the Master Subnet Manager is
incorrectly executing on that component.

Risk:

Unexpected behavior, such as connectivity or performance loss, can occur if the

Subnet Manager is executing on an unexpected component in the InfiniBand fabric.

Action / Repair:

To Verify the Subnet Manager is disabled on components where the Master Subnet
Manager should never reside, execute the following command set as the "root" userid
on all database and storage servers:

unset COMMAND_OUTPUT
COMMAND_OUTPUT=$(ps -ef | grep -i [o]pensm)
if [ -n "$COMMAND_OUTPUT" ]
then
echo -e "FAILURE: the Subnet Manager is
executing.\nDETAILS:\n$COMMAND_OUTPUT"
else
echo -e "SUCCESS: the Subnet Manager is not executing."
fi
The expected output is:

SUCCESS: the Subnet Manager is not executing.

Example of a "FAILURE" output:

FAILURE: the Subnet Manager is executing.

DETAILS:
root 2627 1 0 Mar24 ? 12:14:31 /usr/sbin/opensm --daemon

If the result is "FAILURE", investigate why the Subnet Manager is executing,

relocate the Master Subnet Manager if necessary, and prevent the Subnet
Manager from starting in the future.

NOTES:

1. The command set provided is for Oracle Exadata Database Machines

only. If there are non-Exadata components residing on the InifiniBand
fabric (e.g., a media server), refer to the provided documentation for that
component.
2. There are typically several Standby Subnet Managers waiting to take over
should the current Master Subnet Manager either fail or is manually
moved to some other component with an available Standby Subnet
Manager. Only run Subnet Managers on the InfiniBand switches specified
for use in Oracle Exadata Database Machine, Oracle Exalogic Elastic
Cloud, Oracle Big Data Appliance, and Oracle SuperCluster. Running
Subnet Manager on any other device is not supported.
3. For pure multirack Exadata deployments with less than 4 racks, the
Subnet Manager should run on all spine and leaf InfiniBand switches. For
deployments with 4 or more Exadata racks, the Subnet Manager should
run only on spine InfiniBand switches. For additional configuration
information, please see section "4.6.7 Understanding the Network Subnet
Manager Master" of the "Exadata Database Machine Maintenance Guide".
4. For InfiniBand fabric configurations that involve a mix of different Oracle
Engineered Systems, please refer to: MOS note 1682501.1
5. Moving the Master Subnet Manager is sometimes required during
maintenance and patching operations. For additional guidance on
maintaining the Master Subnet Manager, please see section "4.6
Maintaining the InfiniBand Network" of the "Exadata Database Machine
Maintenance Guide".
 

Verify There Are No Memory (ECC) Errors

Alert Date Owner Status

Engineered Bug(s
Priority Level System )
Critical FAIL 11/16/16     Production Exadata -  
Physical,
Exadata -
Management
Domain,
SSC
DB DB Engineered Exadata OS & Validation TBD
Version Role System Platform Version Version Tool Version
N/A N/A X2-2(4170), X2-2, 11.2.2.2.0+ Solaris - 11 EXAchk  
X2-8, X3-2, X3-8, Linux x86- 12.2.0.1.2
X3-2, X3-8, 64 exachk 2.2.4
X4-2, X4-8, X5-2,
X5-8, X6-2, X6-8
Benefit / Impact:

Memory modules that have corrected Memory Errors (ECC) can show degraded
performance, IPMI driver timeouts, and BMC error messages in /var/log/messages file.

Correcting the condition restores optimal performance.

The impact of checking for memory ECC errors is slight. Correction will likely require
node downtime for hardware diagnostics or repair.

Risk:

If not corrected, the faulty memory will lead to performance degradation and other
errors.

Action / Repair:

To verify there are no memory (ECC) errors, run the following commands as the "root"
userid on all database and storage servers:

if [ -x /usr/bin/ipmitool ]

then
  #Linux

  IPMI_COMMAND=ipmitool;

else

  #Solaris

  IPMI_COMMAND=/opt/ipmitool/bin/ipmitool

fi;

ECC_OUTPUT=$($IPMI_COMMAND sel list | grep Memory | grep ECC)

if [ -z "$ECC_OUTPUT" ]

then

  echo -e "SUCCESS: No memory ECC errors were found.\nECC list:\n\n$ECC_OUTPUT"

else

  echo -e "FAILURE: Memory ECC errors were found.\nECC list:\n\n$ECC_OUTPUT"

fi

The expected output should be:

SUCCESS: No memory ECC errors were found. ECC list:

Example of a FAILURE result:

FAILURE: Memory ECC errors were found. ECC list:  24f | 09/16/2016 | 09:32:59 |
Memory #0x53 | Correctable ECC | Asserted

If any errors are reported, take the following corrective actions in order:
1) Reseat the DIMM.
2) Open an SR for hardware replacement.

Verify celldisk configuration on disk drives

Priority Added Machine Type OS Type Exadata Oracle

 Version Version
Critical 12/06/11 X2-2(4170), X2-2, X2- Linux, 11.2.x + 11.2.x +
8,X4-2 Solaris
Benefit / Impact:

The definition and maintenance of storage server celldisks is critical for optimal
performance and outage avoidance.

The impact of verifying the basic storage server celldisk configuration is minimal.
Correcting any abnormalities is dependent upon the reason for the anomaly, so the
impact cannot be estimated here.

Risk:

If the basic storage server celldisk configuration is not verified, poor performance or
unexpected outages may occur.

Action / Repair:

To verify the basic storage server celldisk configuration on disk drives, execute the
following command as the "celladmin" user on each storage server:

cellcli -e "list celldisk where disktype=harddisk and status=normal" | wc -l

The output should be:

12

If the output is not as expected, investigate the condition and take corrective
action based upon the root cause of the unexpected result.

NOTE: On a storage server configured according to Oracle best practices, there should
be 12 celldisks on disk drives with a status of "normal".

Verify celldisk configuration on flash memory devices

Priority Alert Date Owner Status

Level
Critical FAIL 11/15/2017 <Name> Production
 DB DB Engineered System Exadata OS &
Version Role Version Version
N/A N/A X2(4170), X2-2, X2-8, X3-2, X3-8, X4-2, 11.2+ Linux x86-
X4-8, X5-2, X5-8, X6-2, X6-8, X7-2, X7-8 64
 

Benefit / Impact:

The definition and maintenance of storage server celldisks is critical for optimal
performance and outage avoidance. The number of celldisks configured on flash
memory devices varies by hardware version. Each celldisk configured on flash memory
devices should have a status of "normal".

The impact of verifying the celldisk configuration on flash memory devices is minimal.
The impact of correcting any anomalies is dependent upon the reason for the anomaly
and cannot be estimated here.

Risk:

If the celldisk configuration on flash memory devices is not verified, poor performance
or unexpected outages may occur.

Action / Repair:

To verify the celldisk configuration on flash memory devices, execute the following
command as the "root" userid on each storage server:

cellcli -e "list celldisk where disktype=flashdisk and status=normal" | wc -l

The output should be similar to the following and match one of the rows in the
"Celldisk on Flash Memory Devices Mapping Table": 

16

Celldisk on Flash Memory Devices Mapping Table

System Common Disk Number of

Description Name Type Devices
X4275  X2-2(4170)  MIXED  16 
X4270 M2  X2-2, X2-8  MIXED  16 
X4270 M3  X3-2, X3-8  MIXED  16 
X4270 M3  EIGHTH  MIXED  8 
X4-2L  X4-2  MIXED  16 
X4-2L  EIGHTH  MIXED  8 
X5-2L  X5-2, X5-8  MIXED  4 
X5-2L  X5-2, X5-8  FLASH  8 
X5-2L  EIGHTH  MIXED  2 
X5-2L  EIGHTH  FLASH  4 
X6-2L  X6-2, X6-8  MIXED  4 
X6-2L  X6-2, X6-8  FLASH  8 
X6-2L  EIGHTH  MIXED  2 
X6-2L  EIGHTH  FLASH  4 
X7-2L  X7-2, X7-8  MIXED  4 
X7-2L  X7-2, X7-8  FLASH  8 
X7-2L  EIGHTH  MIXED  2 
X7-2L  EIGHTH  FLASH  4 

If the output is not as expected, execute the following command as the "root" userid: 

cellcli -e "list celldisk where disktype=flashdisk and status!=normal"

Perform your root cause analysis and corrective actions based upon the key words
returned in the "status" field. For additional information, please reference the
following:

The "Maintaining Flash Disks" section of "Oracle® Exadata Database Machine, Owner's
Guide 11g Release 2 (11.2), E13874-24"

Troubleshooting guide for Sick or underperforming storage cell/Performance Issue

(Doc ID 1348736.1)

Troubleshooting guide for Underperforming FlashDisks (Doc ID 1348938.1)

Verify there are no griddisks configured on flash memory devices

Priority Alert Date Owner Status Engineered Bug(s)

Level System
Critical FAIL 06/10/20 <Name> Production Exadata - 31475887 -
Physical, exachk
Exadata - 22331852 -
Management exachk
Domain, 13477133 -
 BDA, Exalogic, exachk
Exalytics, SSC,
ZDLRA
DB DB Engineered Exadata OS & Validation TBD
Version Role System Version Version Tool Version
Platform
N/A N/A All Supported 11.2.2.2.0+ Linux x86- exachk 20.2  
64
Benefit / Impact:

NOTE: This does not apply to XT storage servers.

The definition and maintenance of storage server griddisks is critical for optimal
performance and outage avoidance.

The impact of verifying the storage server griddisk configuration is minimal. Correcting
any abnormalities is dependent upon the reason for the anomaly, so the impact cannot
be estimated here.

Risk:

If the storage server griddisk configuration is not verified, poor performance or

unexpected outages may occur.

Action / Repair:

To verify there are no storage server griddisks configured on flash memory devices,
execute the following command as the "celladmin" user on each storage server:

cellcli -e "list griddisk where disktype=flashdisk" | wc -l

The output should be:

If the output is not as expected, investigate the condition and take corrective action
based upon the root cause of the unexpected result.

NOTE: [Modified 12/08/15,02/29/12]

Experience has shown that the Oracle recommended Best Practice of using all available
flash device space for Smart Flash Log and Smart Flash Cache provides the highest
overall performance benefit with lowest maintenance overhead for an Oracle Exadata
Database Machine.

In some very rare cases for certain highly write-intensive applications, there may be
some performance benefit to configuring grid disks onto the flash devices for datafile
writes only. With the release of the Smart Flash Log feature in 11.2.2.4, redo
logs should never be placed on flash grid disks. Smart Flash Log leverages
both hard disks and flash devices with intelligent caching to achieve the
fastest possible redo write performance, optimizations which are lost if redo
logs are simply placed on flash grid disks.

The space available to Smart Flash Cache and Smart Flash Log is reduced by the
amount of space allocated to the grid disks deployed on flash devices. The usable
space in the flash grid disk group is either half or one-third of the space allocated for
grid disks on flash devices, depending on whether the flash grid disks are configured
with ASM normal or high redundancy.

If after thorough performance and recovery testing, a customer chooses to deploy grid
disks on flash devices, it would be a supported, but not Best Practice, configuration.

Verify griddisk count matches across all storage servers where a given prefix
name exists

Priority Added Machine Type OS Type Exadata Oracle

Version Version
Critical 12/06/11 X2-2(4170), X2-2, X2- Linux, 11.2.x + 11.2.x +
8,X4-2 Solaris
Benefit / Impact:

The definition and maintenance of storage server griddisks is critical for optimal
performance and outage avoidance.

The impact of verifying the basic storage server griddisk configuration is minimal.
Correcting any abnormalities is dependent upon the reason for the anomaly, so the
impact cannot be estimated here.

Risk:

If the storage server griddisk configuration as designed is not verified, poor

performance or unexpected outages may occur.

Action / Repair:

To verify the storage server griddisk count matches across all storage server where a
given prefix name exists, execute the following command as the "root" userid on the
database server from which the
onecommand script was executed during initial deployment:

for GD_PREFIX in `dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l

root "cellcli -e list griddisk attributes name" | cut -d" " -f2 | gawk -F
"_CD_" '{print $1}' | sort -u`;
do
GD_PREFIX_RESULT=`dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l
root "cellcli -e list griddisk where name like \'$GD_PREFIX\_.*\' | wc -l" |
cut -d" " -f2 | sort -u | wc -l`;
if [ $GD_PREFIX_RESULT = 1 ]
then
echo -e "$GD_PREFIX: SUCCESS"
else
echo -e "$GD_PREFIX: FAILURE"
dcli -g /opt/oracle.SupportTools/onecommand/cell_group -l root "cellcli -e
list griddisk where name like \'$GD_PREFIX\_.*\' | wc -l";
fi
done

The output should be similar to:

DATA_SLCC16: SUCCESS
DBFS_DG: SUCCESS
RECO_SLCC16: SUCCESS

If the output is not as expected, investigate the condition and take corrective
action based upon the root cause of the unexpected result.

NOTE: On a storage server configured according to Oracle best practices, the total
number of griddisks per storage server for a given prefix name (e.g: DATA) should
match across all storage servers
where the given prefix name exists.

NOTE: Not all storage servers are required to have all prefix names in use. This is
possible where for security reasons a customer has segregated the storage servers, is
using a data lifecycle management methodology,
or an Oracle Storage Expansion Rack is in use. For example, when an Oracle Storage
Expansion Rack is in use for data lifecycle management, those storage servers will
likely have griddisks with unique names that
differ from the griddisk names used on the storage servers that contain real time data,
yet all griddisks are visible to the same cluster.
NOTE: This command requires that SSH equivalence exists for the "root" userid from
the database server upon which it is executed to all storage servers in use by the
cluster.

Verify griddisk ASM status

Priority Added Machine Type OS Type Exadata Oracle

Version Version
Critical 12/06/11 X2-2(4170), X2-2, X2- Linux, 11.2.x + 11.2.x +
8,X4-2 Solaris
Benefit / Impact:

The definition and maintenance of storage server griddisks is critical for optimal
performance and outage avoidance.

The impact of verifying the storage server griddisk configuration is minimal. Correcting
any abnormalities is dependent upon the reason for the anomaly, so the impact cannot
be estimated here.

Risk:

If the storage server griddisk configuration as designed is not verified, poor

performance or unexpected outages may occur.

Action / Repair:

To verify the storage server griddisk ASM status, execute the following command as
the "celladmin" user on each storage server:

ASM_STAT_RESLT=`cellcli -e "list griddisk attributes name,status,

asmmodestatus,asmdeactivationoutcome" | egrep -v
".*\<active\>.*\<ONLINE\>.*\<Yes\>" | wc -l`;
if [ $ASM_STAT_RESLT = 0 ]
then
echo -e "\nSUCCESS\n"
else
echo -e "\nFAILURE:";
cellcli -e "list griddisk attributes name,status,
asmmodestatus,asmdeactivationoutcome" | egrep -v
".*\<active\>.*\<ONLINE\>.*\<Yes\>";
echo -e "\n";
fi;

 
The output should be:

SUCCESS

If the output is not as expected, investigate the condition and take corrective
action based upon the root cause of the unexpected result.

NOTE: On a storage server configured according to Oracle best practices, all griddisks
should have "status" of "active", "asmmodestatus" of "online" and
"asmdeactivationoutcome" of "yes".

Verify that griddisks are distributed as expected across celldisks

Alert Engineered Engineered

Priority Date Owner Status
Level System System Platform
Exadata - Physical,
Exadata -
Critical FAIL 10/11/17 <Name> Production ALL
Management
Domain
DB/GI DB DB Exadata Validation Tool
DB Role OS & Version
Version Type Mode Version Version
N/A N/A N/A N/A ALL Linux exachk 12.2.0.1.4
 

Benefit / Impact:

The definition and maintenance of storage server griddisks is critical for optimal
performance and outage avoidance.

The impact of verifying the storage server griddisk configuration is minimal. Correcting
any abnormalities is dependent upon the reason for the anomaly, so the impact cannot
be estimated here.

Risk:

If the storage server griddisk configuration as designed is not verified, poor

performance or unexpected outages may occur.

Action / Repair:

NOTE: The recommended best practice is to have each griddisk distributed across all
celldisks. For older versions of Exadata storage server software and hardware, the
griddisks "SYSTEM" or "DBFS_DG" had a slightly different distribution, and the code
below correctly accounts for those cases.

To verify that griddisks are distributed as expected across celldisks, execute the
following command as the "root" userid on each storage server:

RAW_CELLDISK=$(cellcli -e "list celldisk attributes name" | sed -e 's/^[

\t]*//')
RAW_GRIDDISK=$(cellcli -e "list griddisk attributes name" | sed -e 's/^[
\t]*//')
if [ `echo -e $RAW_CELLDISK | grep CD | wc -l` -ge 1 ]
then
PARSED_CELLDISK=$(echo -e "$RAW_CELLDISK" | grep CD)
else
PARSED_CELLDISK=$(echo -e "$RAW_CELLDISK")
fi
CELLDISK_COUNT=$(echo -e "$PARSED_CELLDISK" | wc -l)
if [ `echo -e $RAW_GRIDDISK | grep CD | wc -l` -ge 1 ]
then
SHORT_GD_NAME_ARRAY=$(echo -e "$RAW_GRIDDISK" | awk -F "_CD_" '{print $1}'
| sort -u)
else
SHORT_GD_NAME_ARRAY=$(echo -e "$RAW_GRIDDISK" | awk -F "_FD_" '{print $1}'
| sort -u)
fi
RETURN_RESULT=0
for GD_SHORT_NAME in $SHORT_GD_NAME_ARRAY
do
if [[ $GD_SHORT_NAME = "SYSTEM" || $GD_SHORT_NAME = "DBFS_DG" ||
$GD_SHORT_NAME = "CATALOG" ]]
then
GD_COUNT=$(expr `echo "$RAW_GRIDDISK" | grep $GD_SHORT_NAME | wc -l` + 2)
else
GD_COUNT=$(echo "$RAW_GRIDDISK" | grep $GD_SHORT_NAME | wc -l)
fi
if [ $GD_COUNT -eq $CELLDISK_COUNT ]
then
:
else
OUTPUT_ARRAY+=`echo -e "\n$GD_SHORT_NAME: FAILURE:\tGriddisk count:
$GD_COUNT\tCelldisk count: $CELLDISK_COUNT"`
RETURN_RESULT=1
fi
done
if [ $RETURN_RESULT -eq 0 ]
then
echo -e "SUCCESS: All griddisks are distributed as expected across
celldisks."
else
echo -e -n "FAILURE: One or more griddisks are not distributed as
expected across celldisks. Details:"
echo -e "${OUTPUT_ARRAY[@]}"
fi
The expected output should be:

SUCCESS: All griddisks are distributed as expected across celldisks.

Example of a "FAILURE" result:

FAILURE: One or more griddisks are not distributed as expected across

celldisks. Details:
C_DATA: FAILURE: Griddisk count: 7 Celldisk count: 8

If the output is not as expected, investigate the condition and take corrective action
based upon the root cause of the unexpected result.

Verify the percent of available celldisk space used by the griddisks

Priority Alert Date Owner Status

Engineered
Level System
Critical INFO 11/09/16 <Name> Production Exadata -
Physical,
Exadata -
Management
Domain
DB DB Engineered System Exadata OS & Validation Tool
Version Role Platform Version Version Version
N/A N/A X2-2(4170), X2-2, X2-8, 11.2+ Linux x86- exachk
X3-2, X3-8, X4-2, X4-8, X5- 64 12.2.0.1.2
2, X5-8, X6-2, X6-8
Benefit / Impact:

The impact of verifying the percent of available celldisk space used by the griddisks is
minimal.

Risk:

If the percent of available celldisk space used by the griddisks is not verified, an
unexpected configuration change may be missed.

Action / Repair:

To verify the percent of available celldisk space used by the griddisks, execute the
following command set as the "root" userid on each storage server:
ALLFLASHCELL=$(cellcli -e "list cell attributes makemodel"|egrep -ic
'ALLFLASH|EXTREME_FLASH');
RAW_GRIDDISK_SIZE=$(cellcli -e "list griddisk attributes size");
TOTAL_GRIDDISK_SIZE=$(echo "$RAW_GRIDDISK_SIZE" | sed 's/\s//g'|awk '/G$/
{ print $0 } /T$/ { size=substr($0, 0, length($0)-1); size=size*1024; print
size "" "G";}'|awk '{ SUM += $1} END { print SUM}');
if [ $ALLFLASHCELL -eq 0 ]
then
RAW_CELLDISK_SIZE=$(cellcli -e "list celldisk attributes size where
disktype=harddisk");
else
RAW_CELLDISK_SIZE=$(cellcli -e "list celldisk attributes size where
disktype=flashdisk");
fi;
TOTAL_CELLDISK_SIZE=$(echo "$RAW_CELLDISK_SIZE" | sed 's/\s//g'|awk '/G$/
{ print $0 } /T$/ { size=substr($0, 0, length($0)-1); size=size*1024; print
size "" "G";}'| awk '{ SUM += $1} END { print SUM}');
GRIDDISK_CELLDISK_PCT=$(echo $TOTAL_GRIDDISK_SIZE $TOTAL_CELLDISK_SIZE | awk
'{ printf("%d", ($1/$2)*100) }');
echo -e "INFO: The percent of available celldisk space used by the griddisks
is: $GRIDDISK_CELLDISK_PCT\nThe total griddisk size found is:
$TOTAL_GRIDDISK_SIZE\nThe total celldisk size found is:
$TOTAL_CELLDISK_SIZE";

The expected output will be similar to:

INFO: The percent of available celldisk space used by the griddisks is: 99
The total griddisk size found is: 87818.7
The total celldisk size found is: 87819.3

If the output is not as expected for a given known configuration, investigate and take
corrective action based upon the root cause of the unexpected result.

NOTE: On a storage server not in an Oracle Virtual Machine environment configured

according to Oracle best practices, the percent utilization will typically be >= 99 for
spinning disk and >= 94 <= 95 for Extreme Flash. The lower percentage of utilization
for Extreme Flash is because the griddisks, Flash Log, and Flash Cache are all built on
the same flash hardware.

NOTE: In an Oracle Virtual Machine environment, it is not unusual for the percentage
of available celldisk space used by the griddisks to be in the middle 60 range. This is
due in part to the fact the DBFS griddisk is not created by default, and user
requirements to reserve free space for future use. For example:
INFO: The percent of available celldisk space used by the griddisks is: 63
The total griddisk size found is: 4236
The total celldisk size found is: 6636.06

  
 Verify Database Server ZFS RAID Configuration

Priority Added Machine OS Exadata Oracle

Type Type Version Version
Critical 01/27/12 X2-2, X2-8, Solaris 11.2.x + 11.2.x +
X4-2
Benefit / Impact:

For a database server running Solaris deployed according to Oracle standards, there
will be two ZFS RAID-1 pools, named "rpool" and "data". Each mirror in the pool
contains two disk drives. For an X2-2,
there is one mirror for each name. For an X2-8, there is one mirror for "rpool" and 3
for "data". Verifying the database server ZFS RAID configuration helps to avoid a
possible performance impact, or an outage.

The impact of validating the ZFS RAID configuration is minimal. The impact of
corrective actions will vary depending on the specific issue uncovered, and may range
from simple reconfiguration to an outage.

Risk:

Not verifying the ZFS RAID configuration increases the chance of a performance
degradation or an outage.

Action / Repair:

To verify the database server ZFS RAID configuration, execute the following command
as the "root" userid:

/opt/oracle.SupportTools/disks_map.pl | ggrep mirror -A3

The output will be similar to:

------------------- mirror-0 ---------------------

16:5 c1t2d0s0 rpool
16:4 c1t1d0s0 rpool
--------------------------------------------------
------------------- mirror-0 ---------------------
16:6 c1t3d0 data
16:7 c1t4d0 data
--------------------------------------------------
------------------- mirror-2 ---------------------
16:0 c1t5d0 data
16:2 c1t6d0 data
--------------------------------------------------
------------------- mirror-1 ---------------------
16:3 c1t0d0 data
16:1 c1t7d0 data
--------------------------------------------------

For an X2-2, the expected output is one pool named "rpool", and one named "data",
each comprised of 1 mirror with 2 disk drives.

For an X2-8, the expected output is one pool named "rpool", comprised of 1 mirror
with 2 disk drives, and one pool named "data" comprised of 3 mirrors each with 2 disk
drives.

If the reported output differs, investigate and correct the condition.

Verify InfiniBand is the Private Network for Oracle Clusterware

Communication

Priority Added Machine Type OS Exadata Oracle

Type Version Version
Critical N/A X2-2(4170), X2-2, Linux 11.2.x + 11.2.x +
X2-8, X4-2
Benefit / Impact:

The InfiniBand network in an Oracle Exadata Database Machine provides superior

performance and throughput characteristics that allow Oracle Clusterware to operate at
optimal efficiency.

The overhead for these verification steps is minimal.

Risk:

If the InfiniBand network is not used for Oracle Clusterware communication,

performance will be sub-optimal.

Action / Repair:

The InfiniBand network is preconfigured on the storage servers. Perform the following
on the database servers:

Verify the InfiniBand network is the private network used for Oracle Clusterware
communication with the following command:

$GI_HOME/bin/oifcfg getif -type cluster_interconnect

For X2-2 the output should be similar to:

bondib0 192.xxx.8.0 global cluster_interconnect

For X2-8 the output should be similar to:

bondib0 192.xxx.8.0 global cluster_interconnect

bondib1 192.xxx.8.0 global cluster_interconnect
bondib2 192.xxx.8.0 global cluster_interconnect
bondib3 192.xxx.8.0 global cluster_interconnect

If the InfiniBand network is not the private network used for Oracle Clusterware
communication, configure it following the instructions in MOS Note 283684.1,
"How to Modify Private Network Interface in 11.2 Grid Infrastructure".

NOTE: It is important to ensure that your public interface is properly marked as public
and not private. This can be checked with the oifcfg getif command. If it is
inadvertantly marked private,
you can get errors such as "OS system dependent operation:bind failed with status"
and "OS failure message: Cannot assign requested address".
It can be corrected with a command like oifcfg setif -global eth0/<public IP
address>:public

 In each database verify that it is using the private IB interconnect withe following
query :
SQL> select name,ip_address from v$cluster_interconnects;

NAME IP_ADDRESS
--------------- ----------------
bondib0 192.xxx.40.25

Or in the database alert log you can look for the following message:

Cluster communication is configured to use the following interface(s) for this instance
192.xxx.40.26

Verify InfiniBand Address Resolution Protocol (ARP) Configuration on

Database Servers

Priority Alert Date Owner Status

Level
Critical FAIL 7/13/16   Production
Status
 DB DB Role Engineered System Platform Exadata OS Version
Version Version
N/A N/A X2-2(4170), X2-2, X2-8, X3-2, X3-8, 11.2.2.2.0+ Linux x86-64
X4-2, X4-8, X5-2, X5-8, X6-2, X6-8
Benefit / Impact: There are specific ARP configurations required for Real Application
Clusters (RAC) to work correctly that vary between an active/passive or active/active
configuration.

For an active/passive configuration, the settings for all IB interfaces should be:

  accept_local = 1
  rp_filter = 0

For an active/active configuration, the settings for all IB interfaces should be:

  accept_local = 1
  rp_filter = 0
  arp_announce = 2 (8 socket only!)

  - AND the three single attributes -

  net.ipv4.conf.all.rp_filter = 0
  net.ipv4.conf.default.rp_filter = 0
  net.ipv4.conf.all.accept_local = 1

The impact of verifying the ARP configuration is minimal. Correcting a configuration

requires editing "/etc/sysctl.conf" and restarting the interface(s).

Risk:

Incorrect ARP configurations may prevent RAC from starting, or result in dropped
packets and inconsistent RAC operation.

Action / Repair:

To verify the InfiniBand interface ARP settings for a database server, use the following
command as the "root" userid:

RAW_OUTPUT=$(sysctl -a)
RF_OUTPUT=$(echo "$RAW_OUTPUT" | egrep -i "\.ib|bondib" | egrep -i
"\.rp_filter")
AL_OUTPUT=$(echo "$RAW_OUTPUT" | egrep -i "\.ib|bondib" | egrep -i
"\.accept_local")
if [ `echo "$RAW_OUTPUT" | grep -i bondib | wc -l` -ge 1 ]
then #active/passive case
  if [[ `echo "$AL_OUTPUT" | cut -d" " -f3 | sort -u | wc -l` -eq 1 &&
`echo "$AL_OUTPUT" | cut -d" " -f3 | sort -u | head -1` -eq 1 ]]
  then
    AL_RSLT=0 #all AL same value and value is 1
  else
    AL_RSLT=1
  fi;
  if [[ `echo "$RF_OUTPUT" | cut -d" " -f3 | sort -u | wc -l` -eq 1 &&
`echo "$RF_OUTPUT" | cut -d" " -f3 | sort -u | head -1` -eq 0 ]]
  then
    RF_RSLT=0 #all RF same value and value is 0
  else
    RF_RSLT=1
  fi;
  if [[ $AL_RSLT -eq 0 && $RF_RSLT -eq 0 ]]
  then
    echo -e "Success:  The active/passive ARP configuration is as
recommended:\n"
  else
    echo -e "Failure:  The active/passive ARP configuration is not as
recommended:\n"
  fi;
  echo -e "$AL_OUTPUT\n\n$RF_OUTPUT"
else #active/active case
  NICARF_OUTPUT=$(echo "$RAW_OUTPUT" | egrep -i
"net.ipv4.conf.all.rp_filter")
  NICDRF_OUTPUT=$(echo "$RAW_OUTPUT" | egrep -i
"net.ipv4.conf.default.rp_filter")
  NICAAL_OUTPUT=$(echo "$RAW_OUTPUT" | egrep -i
"net.ipv4.conf.all.accept_local")
  NICARF_RSLT=$(echo "$NICARF_OUTPUT" | cut -d" " -f3)
  NICDRF_RSLT=$(echo "$NICDRF_OUTPUT" | cut -d" " -f3)
  NICAAL_RSLT=$(echo "$NICAAL_OUTPUT" | cut -d" " -f3)
  IB_INTRFCE_CNT=$(echo "$RAW_OUTPUT" | egrep "\.ib.\." | cut -d"." -f4 |
sort -u | wc -l)
  if [[ `echo "$AL_OUTPUT" | cut -d" " -f3 | sort -u | wc -l` -eq 1 &&
`echo "$AL_OUTPUT" | cut -d" " -f3 | sort -u | head -1` -eq 1 ]]
  then
    AL_RSLT=0 #all AL same value and value is 1
  else
    AL_RSLT=1
  fi;
  if [[ `echo "$RF_OUTPUT" | cut -d" " -f3 | sort -u | wc -l` -eq 1 &&
`echo "$RF_OUTPUT" | cut -d" " -f3 | sort -u | head -1` -eq 0 ]]
  then
    RF_RSLT=0 #all RF same value and value is 0
  else
    RF_RSLT=1
  fi;
  if [ $IB_INTRFCE_CNT -eq 2 ] # 2 socket case
  then
    if [[ $AL_RSLT -eq 0 && $RF_RSLT -eq 0 && $NICARF_RSLT -eq 0 &&
$NICDRF_RSLT -eq 0 && $NICAAL_RSLT -eq 1 ]]
    then
      echo -e "Success:  The active/active ARP configuration is as
 recommended:\n"
    else
      echo -e "Failure:  The active/active ARP configuration is not as
recommended:\n"
    fi;
    echo -e
"$AL_OUTPUT\n\n$RF_OUTPUT\n\n$NICARF_OUTPUT\n$NICDRF_OUTPUT\n$NICAAL_OUTP
UT"
  else # 8 socket case
  NICIAA_OUTPUT=$(echo "$RAW_OUTPUT" | egrep "\.ib.\." | egrep
arp_announce)
  if [[ `echo "$NICIAA_OUTPUT" | cut -d" " -f3 | sort -u | wc -l` -eq 1
&& `echo "$NICIAA_OUTPUT" | cut -d" " -f3 | sort -u | head -1` -eq 2 ]]
  then
    NICIAA_RSLT=0 #all arp_announce same value and value is 2
  else
    NICIAA_RSLT=1
  fi;
    if [[ $AL_RSLT -eq 0 && $RF_RSLT -eq 0 && $NICIAA_RSLT -eq 0 &&
$NICARF_RSLT -eq 0 && $NICDRF_RSLT -eq 0 && $NICAAL_RSLT -eq 1 ]]
    then
      echo -e "Success:  The active/active ARP configuration is as
recommended:\n"
    else
      echo -e "Failure:  The active/active ARP configuration is not as
recommended:\n"
    fi;
  echo -e
"$AL_OUTPUT\n\n$RF_OUTPUT\n\n$NICIAA_OUTPUT\n\n$NICARF_OUTPUT\n$NICDRF_OU
TPUT\n$NICAAL_OUTPUT"
  fi;
fi;

The expected output should be similar to:

Success: The active/passive ARP configuration is as recommended:

net.ipv4.conf.ib0.accept_local = 1
net.ipv4.conf.ib1.accept_local = 1
net.ipv4.conf.bondib0.accept_local = 1

net.ipv4.conf.ib0.rp_filter = 0
net.ipv4.conf.ib1.rp_filter = 0
net.ipv4.conf.bondib0.rp_filter = 0
- OR -

Success: The active/active ARP configuration is as recommended:

net.ipv4.conf.ib0.accept_local = 1
net.ipv4.conf.ib1.accept_local = 1

net.ipv4.conf.ib0.rp_filter = 0
net.ipv4.conf.ib1.rp_filter = 0

net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.accept_local = 1

- OR -

Success: The active/active ARP configuration is as recommended:

net.ipv4.conf.ib0.accept_local = 1
<outpout truncated>
net.ipv4.conf.ib7.accept_local = 1

net.ipv4.conf.ib0.rp_filter = 0
<output truncated>
net.ipv4.conf.ib7.rp_filter = 0

net.ipv4.conf.ib0.arp_announce = 2
<output turncated>
net.ipv4.conf.ib7.arp_announce = 2

net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.accept_local = 1

If a "FAILURE: ..." message appears, investigate for root cause, make the necessary
edits to "/etc/sysctl.conf", and restart the interface(s).

NOTE: These recommendations are for the InfiniBand interfaces on database servers
only! They do not apply to the Ethernet interfaces on the database servers. No
changes are permitted on the storage servers.

 
Verify Oracle RAC Databases use RDS Protocol over InfiniBand Network.

Priority Alert Date Owner

Level
Critical FAIL 03/01/2017   Production

DB DB Role Engineered System Platform Exadata

Version Version
11.2.0.2+ Primary, X2-2(4170), X2-2, X2-8, X3-2, X3-8, X4-2, All Linux x86-
Standby, X4-8, X5-2, X5-8, X6-2, X6-8, SL6
ASM Sparc Linux
Benefit / Impact:

The RDS protocol over InfiniBand provides superior performance because it avoids
additional memory buffering operations when moving data from process memory to
the network interface for IO operations.
This includes both IO operations between the Oracle instance and the storage servers,
as well as instance to instance block transfers via Cache Fusion.

There is minimal impact to verify that the RDS protocol is in use. Implementing the
RDS protocol over InfiniBand requires an outage to relink the Oracle software.

Risk:

If the Oracle RAC databases do not use RDS protocol over the InfiniBand network, IO
operations will be sub-optimal.

Action / Repair:

To verify the RDS protocol is in use by a given Oracle instance, set the ORACLE_HOME
and LD_LIBRARY_PATH variables properly for the instance and execute the following
command as the oracle userid
on each database server where the instance is running:

$ORACLE_HOME/bin/skgxpinfo

The output should be:

rds
Note: For Oracle software versions below 11.2.0.2, the skgxpinfo command is not
present. For 11.2.0.1, you can copy over skgxpinfo to the proper path in your 11.2.0.1
environment from an
available 11.2.0.2 environment and execute it against the 11.2.0.1 database home(s)
using the provided command.

Note: An alternative check (regardless of Oracle software version) is to scan each

instance's alert log (must contain a startup sequence!) for the following line:

Cluster communication is configured to use the following interface(s)for this

instance 192.xxx.20.21 cluster interconnect IPC version:Oracle RDS/IP
(generic)
 

If the instance is not using the RDS protocol over InfiniBand, relink the Oracle binary
using the following commands (with variables properly defined for each home being
linked):

 (as oracle) Shutdown any processes using the Oracle binary

 If and only if relinking the grid infrastructure home, then (as root)
GRID_HOME/crs/install/rootcrs.pl -unlock
 (as oracle) cd $ORACLE_HOME/rdbms/lib
 (as oracle) make -f ins_rdbms.mk ipc_rds ioracle
 If and only if relinking the Grid Infrastructure home, then (as root)
GRID_HOME/crs/install/rootcrs.pl -patch

Note: Avoid using the relink all command due to various issues. Use the make
commands provided.

Verify Database and ASM instances use same SPFILE

Priority Added Machine Type OS Type Exadata Version Oracle Version

Critical March 2013 All All All All
Benefit / Impact:

All instances for a particular database or ASM cluster should be using the same spfile.
Making changes to databases and ASM instances needs to be done in a reliable and
consistent way across all instances.
Risk:

Multiple 'sources of truth' can cause confusion and possibly unintended values being
set.

Action / Repair:

Verify what spfile is used across all instances of one particular ASM or database cluster.
If multiple spfiles for one database are found, provide a recommendation to
consolidate them into one.

Scope includes all machine types, os types and db versions

SQL> select name, value from gv$parameter where name = 'spfile';

NAME VALUE
------------------------------
------------------------------------------------------------
spfile +DATA/racone/spfileracone.ora

The value for pfile should be empty:

SQL> select name, value from gv$parameter where name = 'pfile';

no rows selected

Verify Berkeley Database location for Cloned GI homes

Priority Added Machine Type OS Type Exadata Oracle

Version Version
Critical
March X2-2(4170), X2-2, X2- Linux, 11.2.x + 11.2.x +
2013 8, X4-2 Solaris
Benefit / Impact

After cloning a Grid Home the Berkeley Database configuration file

($GI_HOME/crf/admin/crf<node>.ora) in the new home should not be pointing to the
previous GI home where it is cloned from. During previous patch set updates Berkeley
Database configuration files were found still pointing to the
'before (previously cloned from) home'. It was due an invalid cloning procedure the
Berkeley Database location of the 'new home' was not updated during
the out of place bundle patching procedure
NOTE:401749.1 - Oracle Linux: Shell Script to Calculate Values Recommended Linux
HugePages / HugeTLB Configuration

Oracle Exadata Initialization Parameters and Diskgroup Attributes Best Practices

(Doc ID 2062068.1)
Oracle Database Cloud Service - Version N/A and later
Exadata X3-2 Hardware - Version All Versions and later
Gen 1 Exadata Cloud at Customer (Oracle Exadata Database Cloud Machine) -
Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Schema Service - Version N/A and later
Linux x86-64

PURPOSE

 Initialization Parameters and Diskgroup Attributes Best Practices for Oracle Database
Machine

SCOPE

People working with  Oracle Exadata Database Machine 

DETAILS

ASM rebalance and power limit considerations for GI 12.1.0.2

Verify ASM Instance Initialization Parameters for 12.1.0.2
Verify ASM Diskgroup Attributes for 12.1.0.x
Verify ASM Instance Initialization Parameters for 12.2.0.1
Verify ASM Diskgroup Attributes for 12.2.0.x
Verify Common Instance Database Initialization Parameters for 12.1.0.x
Verify Common Instance Database Initialization Parameters for 12.2.0.1
Verify OLTP Instance Database Initialization Parameters for 12.1.0
Verify Multitenant Instance Database Initialization Parameters for 12.1.0.x
Verify DW/BI Instance Database Initialization Parameters for 12.1.0.x
Verify DBFS Instance Database Initialization Parameters for 12.1.0.x

ASM rebalance and power limit considerations for GI 12.1.0.2

Alert Date Owner Status

Engineered Bug(s
Priority Level System )
Critical WARN 8/12/2015 Michael Production Exadata-User  
Nowak Domain, Exadata-
Physical, SSC
DB DB Engineered Exadata OS & Validation Tool TBD
Version Role System Platform Version Version Version
12.1.0.2+ N/A X2-2(4170), X2-2, 12.1+ Solaris - 11 TBD  
X2-8, X3-2, X3-8, Linux x86-
X4-2, X4-8, X5-2 64 el5uek
Linux x86-
64 el6uek
Benefit / Impact:

For physical deployments:

The default ASM power limit, ie the asm_power_limit parameter in the ASM instance,
should be set to ensure the application performance is not impacted while rebalance is
running. The default 12.1.0.2 deployment sets asm_power_limit=4 for all Exadata
racks. Here are some other considerations for ASM rebalance and the setting of
asm_power_limit

* The default power of 4 was chosen after an internal MAA assessment showed it did
not impact the application. Also the following bug fixes should exist to ensure efficient
performance with minimal application impact

 All known ASM related fixes from MOS Exadata critical issues note 1270094.1

 Fixes for bugs 16087033 & 21218243 (GI side) and 13967870 (cell side)

For OVM deployments :

The default 12.1.0.2 deployment sets asm_power_limit to as below for OVM

deployments :

deployments with 1 OVM RAC cluster - asm_power_limit=4

deployments with 2 OVM RAC clusters - asm_power_limit=2

deployments with more than 2 OVM RAC clusters - asm_power_limit=1

* The maximum power that should be used is 64 as anything higher adds no

performance benefit, only risk.

* Setting asm_power_limit=0 disables ASM rebalance operations. In normal operation,

asm_power_limit should not be set to 0.

* All MAA Exadata best practice attributes should be used to ensure an efficient
rebalance. Amongst other things, these attributes ensure the compaction phase is
disabled and appliance mode is set properly.

* Fixed extents will be allocated using Exadata and Grid Infrastructure 12.1.0.2 if the
compatible.rdbms diskgroup attribute is set to 11.2.0.4 or higher at allocation time.
Note this means no databases using versions < 11.2.0.4 can mount the diskgroup.

* During a rebalance that occurs after a failure where extent copies are lost, the most
critical files are rebalanced first, and evidence of this can be found in the ASM alert log.
For more information on priority rebalance, see MOS note 1968607.1.

* To determine the amount of work that a rebalance needs to do ahead of time, run
the 'EXPLAIN WORK...' command and then query v$asm_estimate. Explain work is
documented here.

* To monitor a running rebalance, query v$asm_operation. Along with the classic

columns to assess work done so far and time remaining, the new 'PASS' column can be
used to see what phase of rebalance is being executed.

Risk:

Not following MAA Exadata best practices with respect to asm_power_limit can cause
application impact during rebalance and/or cause excessive rebalance runtimes.

Action / Repair:

Customers should evaluate the performance impact of different ASM power limit
settings to their application while conducting an ASM rebalance operation under full
application workload in a non-production environment and choose their
ASM_POWER_LIMIT setting based on the results of the testing and the business
requirements for the application performance and availability.

Verify ASM Instance Initialization Parameters for 12.1.0.2

Benefit / Impact: Experience and testing has shown that certain ASM initialization
parameters should be set at specific values. These are the best practice values set at
deployment time. By setting these ASM initialization parameters as recommended,
known problems may be avoided and performance maximized. The parameters are
specific to the ASM instances. Unless otherwise specified, the value is for both 2 socket
and 8 socket Database Machines. The impact of setting these parameters is minimal.

Risk: If the ASM initialization parameters are not set as recommended, a variety of
issues may be encountered, depending upon which initialization parameter is not set as
recommended, and the actual set value.

Action / Repair: To verify the ASM initialization parameters, compare the values in
your environment against the table below (* = default value):

Recommended Value
Parameter
cluster_interconnect Colon delimited IB IP addresses
s

asm_power_limit For Exadata physical: 4

For Exadata OVM:

1 OVM RAC cluster: 4

2 OVM RAC clusters: 2
3 or more OVM RAC clusters: 1
sga_target 2048M
pga_aggregate_targ
400M
et

memory_target 0

memory_max_target 0

processes For < 10 instances per node, set processes=1024

For >= 10 instances per node,set processes using the following formula

MAX((50*MIN(db_instances_per_node+1,11))
+(10*MAX(db_instances_per_node-10,0)),1024)
audit_sys_operations TRUE

audit_syslog_level local0.info

Correct any Priority 1 parameter that is not set as recommended. Evaluate and correct
any Priority 2 parameter that is not set as recommended.
The proper way to implement the memory related parameters is as follows. This is
important as it works around an issue where memory_target remains set despite
setting it to 0.

 alter system set sga_target=2048M sid='*' scope=spfile;

 alter system set pga_aggregate_target=400M sid='*' scope=spfile;
 alter system set memory_target=0 sid='*' scope=spfile;
 alter system set memory_max_target=0 sid='*' scope=spfile;
 alter system reset memory_max_target sid='*' scope=spfile;

Verify ASM Diskgroup Attributes for 12.1.0.x

Benefit / Impact: Experience and MAA testing has shown that certain diskgroup
attributes should be set at specific values. These are the best practice values set at
deployment time. By setting these diskgroup attributes as recommended, known
problems may be avoided and performance maximized. The attributes are specific to
ASM diskgroups. The impact of setting these parameters is minimal.

Risk: If the diskgroup attributes are not set as recommended, performance and or
availability issues can be encountered.

Action / Repair: To verify the diskgroup attributes, compare the values in your ASM
diskgroups against the table below.

Recommended Value Priority Notes

Parameter
Enables fixed partnering which en
true for diskgroups with
rebalance timings. If appliance.m
appliance.mode compatible.asm=12.1.0.2 or 1
required for it to take effect. Note
higher
with version 12.1.0.1.
Grid Infrastructure software
compatible.asm version in use, for example 1 Enables many important ASM fea
12.1.0.2
Grid Infrastructure software Required to reduce resync timing
compatible.advm version in use, for example 1
12.1.0.2 NOTE: This attribute should be se
compatible.rdbms The lowest RDBMS software 1
version in use by databases that NOTE: Reducing "compatible.rdbm
will access the diskgroup higher version requires the diskgr
the value of "compatible.rdbms" s
and set according to customer re
plans.
 NOTE: "compatible.rdbms" is set
Exadata deployments to provide m
and to assure all space allocation

NOTE: important features for Exa

compatible.rdbms are listed below
the ASM documentation.

 Disk resync checkpointing

compatible.rdbms=12.1.0.
 Sparse disk group with wit
 Extended and Flex disk gro
compatible.rdbms=12.2.0.

cell.smart_scan_capable true 1 Ensures smart scans run

au_size 4M 1 Default and well tested AU size fo
Content type attributes should be
diskgroups, DATA, RECO, and DB
type attributes provides better re
'data' for DATA diskgroup
and recovery point objective (RPO
half and full rack configurations, b
content.type 'recovery' for RECO diskgroup 1
quarter rack configurations that m
criteria in case a future expansion
'system' for DBFS_DG diskgroup
Note a rebalance is required after
attribute.
disk_repair_time indicates how lo
ASM before being dropped. The p
disk_repair_time >=3.6h 1 on Exadata is for when a disk has
of its slot. 3.6h is the default sett
it hasn't been changed.
failgroup_repair_time indicates ho
offline in ASM before being dropp
attribute on Exadata is for when
failgroup_repair_time 24.0h 1
unplanned outages or planned m
default setting so we are just ma
changed
Correct any Priority 1 attribute that is not set as recommended. Evaluate and correct
any Priority 2 parameter that is not set as recommended. 

 
Verify ASM Instance Initialization Parameters for 12.2.0.1

Benefit / Impact: Experience and testing has shown that certain ASM initialization
parameters should be set at specific values. These are the best practice values set at
deployment time. By setting these ASM initialization parameters as recommended,
known problems may be avoided and performance maximized. The parameters are
specific to the ASM instances. Unless otherwise specified, the value is for both 2 socket
and 8 socket Database Machines. The impact of setting these parameters is minimal.

Risk: If the ASM initialization parameters are not set as recommended, a variety of
issues may be encountered, depending upon which initialization parameter is not set as
recommended, and the actual set value.

Action / Repair: To verify the ASM initialization parameters, compare the values in
your environment against the table below (* = default value):

Recommended Value
Parameter
cluster_interconnect Colon delimited IB IP addresses
s

asm_power_limit For Exadata physical:

 4

For Exadata OVM:

 1 OVM RAC cluster: 4

 2 OVM RAC clusters: 2
 3 or more OVM RAC clusters: 1
 

Understanding ASM Redundancy State and Restoration Status

ASM storage partnership exists at both the disk and fail group level, and is designed to
reduce the possibility of data loss if multiple failures occur using a large storage
configuration with many disks. Partnering also ensures that a given ASM extent set is
sufficiently spread across storage to avoid single points of failure. For example, using
ASM high redundancy on Exadata, every extent (primary, secondary and tertiary) in a
given extent set is guaranteed to be on different Exadata cells, since every cell is a
failure group. The content.type diskgroup attribute is available to minimize or eliminate
partnership intersections across different diskgroups. One example of this with Exadata
is having different content.type for the DATAC1 and RECOC1 diskgroups created at
deployment time. With enough disks and cells to support it, this configuration ensures a
single catastrophic full partner failure does not forcibly dismount both diskgroups thus
losing both data and recovery files.

When storage goes offline, either temporarily or permanently, redundancy restoration is

required. The ASM ARB0 process performs this rebalance work, asynchronously issuing
the number of outstanding extent relocations defined by the specified or default ASM
power limit. Specific phases of rebalance can be observed via gv$asm_operation.pass
column when using Oracle Grid Infrastructure 12.1 or higher. If the pass is RESYNC or
RESILVER, redundancy restoration is in process. If the pass is REBALANCE, redundancy
restoration may be in process, depending on the sequence of events and the reason the
storage went offline. Since there isn’t an airtight way to determine redundancy
restoration status via the v$asm_operation view, it is recommended that the
v$asm_disk.mode_status column be used instead as it works in all cases. If
v$asm_disk.mode_status is not ‘ONLINE’ for all disks currently in a given diskgroup,
then the redundancy of that diskgroup is compromised. When redundancy is
compromised, care should be taken to ensure partner storage does not go offline as it
would result in a forcible diskgroup dismount. Oracle tools such as patchmgr check
v$asm_disk.mode_status before taking storage offline for planned software updates.

Here is an example query run in the ASM instance showing a case where not all
disks are online:

 SQL> select distinct mode_status from v$asm_disk where

 group_number=3;

MODE_ST

-------------

OFFLINE

ONLINE

Here is an example query run in the ASM instance showing a case where all disks
are online:

SQL> select distinct mode_status from v$asm_disk where

group_number=3;

MODE_ST

-------------

ONLINE

Follow MAA Exadata ASM best practices using Grid Infrastructure 12.1.0.2

The following represent the consolidated list of MAA Exadata best practices for ASM.
They supersede all previous best practices for any prior Grid Infrastructure versions.

Use High Redundancy for all Production ASM Diskgroups on Exadata

ASM High Redundancy Exadata MAA Best Practice - Introduction

Configuring ASM diskgroups to use high redundancy, or triple mirroring of extents in an

extent set, is an Exadata MAA (Maximum Availability Architecture) best practice because
it provides the ultimate protection against downtime and data loss caused by partner
storage failures and corruptions. High redundancy not only protects against the less
likely case of a double partner disk failure, but more importantly protects from all of the
other combinations of issues that can occur with partner storage. The following table
details this benefit.

One discernible example from this table is having an Exadata storage cell offline for
planned maintenance while encountering an IO error or corruption reading a secondary
extent on a partner disk [1].

High redundancy is an MAA best practice for all Exadata configurations. The only MAA
compliant alternative to high redundancy is normal redundancy with a DataGuard
standby database that is ready to take over the production workload. This includes
application support and fully tested operational procedures that have proven failover
will work seamlessly. More information can be found on this MAA page.

ASM Rebalance Resources and Redundancy Restoration

An ASM rebalance is run when storage is dropped or added to an ASM disk group, most
commonly when a disk fails or when adding storage servers to the configuration. The
ASM power limit [2] defines the number of outstanding asynchronous IOs the ASM
Rebalance Process (ARB0) process will issue during the rebalance. The Exadata MAA
best practice default for ASM power limit is 4, and set at deployment time. The Exadata
MAA best practice maximum for ASM power limit is 64. A higher power limit such as 64
enables rebalance to run faster, but the extra IOs can impact the application service
level. The aforementioned best practice default of 4 has been tested in MAA labs and
shown to have minimal impact on several different application service levels [3].
Internal tests have also shown that there is no IO throughput or rebalance runtime
benefit beyond a power limit of 64.

When a rebalance is triggered by a planned maintenance operation using ASM

redundancy (normal or high), for example the adding of a new Exadata storage server,
the diskgroup’s redundancy is never compromised and thus running at the Exadata
default ASM power limit of 4 is advised if maintaining the application service level is a
priority. When a rebalance is triggered by a storage failure, for example a disk failure,
the diskgroup’s redundancy is compromised until the rebalance is complete [4]. It is in
this case where redundancy restoration via a timely rebalance completion is important
since the permanent failure of remaining partner storage will cause the diskgroup to
forcibly dismount and data will be lost [5]. This is particularly relevant when using a
normal redundancy diskgroup since there is only one remaining copy of the extents
from the extent sets on the failed disk, so one might want to use a higher ASM power
limit to restore redundancy faster, potentially at the expense of the application service
level. With high redundancy, two extent copies remain, diminishing the risk of losing all
remaining partner storage, thus a lower ASM power limit, one that maintains application
service levels, may be preferred.

ASM Rebalance Performance Factors

At this point, it should be clear that redundancy restoration is the most critical task of
rebalance, and it is only necessary when storage is lost. It should also be clear why
high redundancy is an Exadata MAA best practice that protects your application RTO
(recovery time objective) and RPO (recovery point objective). Now let’s look at what
factors into rebalance runtime.

Two primary factors influencing rebalance runtime are the amount of allocated space
on the storage that caused the reconfiguration (ex: allocated space on disk for disk
failure), and the ASM power limit. For example, using the same ASM power limit, the
rebalance resulting from the failure of a fully allocated 8TB disk will take twice as long
as if that disk was only half allocated. Increasing the ASM power limit (up to the
Exadata MAA best practice maximum of 64) yields higher IO throughput and lower
rebalance runtimes, and scales, though less so at the higher power limits.

The diskgroup’s redundancy type can also affect rebalance runtime. With the same
amount of allocated storage, the rebalance of a high redundancy diskgroup moves up
to twice the number of extents compared to a normal redundancy diskgroup. The extra
extent movement is required to maintain disk partnerships and diskgroup balance for
non primary extents. While the pure cost of rebalancing a high redundancy diskgroup is
more expensive, the aforementioned benefits far outweigh this cost.

Lastly, IO is a shared resource, thus an IO intensive application workload can also

impact ASM rebalance timing.

Rebalance Work and Time Estimations

To understand how many extents need to move for a given rebalance, the EXPLAIN
WORK FOR ALTER DISKGROUP statement [6], which populates the V$ASM_ESTIMATE
view, may be used. It is not possible to use this technique to estimate the amount of
time a rebalance will take, because it does not have IO timings to take into
consideration.

To understand how many extents have been processed and how much time remains for
a running rebalance, the V$ASM_OPERATION view may be used. This view tracks the
effective rate of rebalance and can fluctuate with the IOs coming from application
workloads.

Normal and High Redundancy Comparison – an Example

The following table compares rebalance times using normal and high redundancy after
the loss of a single hard disk in an Exadata HC quarter rack with fully populated 8TB
disks [7]

For 4TB disks, the database space allocated and rebalance timings are simply half of
the values provided in this table, and all other columns are the same.

ASM High Redundancy Exadata MAA Best Practice - Conclusion

Exadata MAA best practices exist to enable our most demanding customers to meet
their most stringent application service level RTOs and RPOs. High Redundancy is an
Exadata MAA best practice because it protects from the crashes and data loss
associated with every type of partner storage loss, and it enables the repair from disk
failure to run at a power limit that minimizes application service level impact.

[1] Exadata scrubbing runs on an intelligent, adaptive schedule to mitigate the risk of
encountering latent bad sectors, but it cannot prevent it in all cases.

[2] The default ASM power limit is defined by the ASM instance’s asm_power_limit
parameter. Alternate ASM power settings can be specified using the alter diskgroup
command.

[3] It is impossible to simulate every possible customer workload but workloads

generating sufficient IO within data sheet limits were used

[4] 15% of allocated diskgroup space should be reserved to ensure ASM rebalance
completes without an out of space error such as ORA-15041. This number is factored
into the allocated space section of the Exadata data sheets.

[5] When the storage failure is small and isolated, such as a single bad sector or set of
bad disk sectors, ASM ensures that only the data affected by the failure is lost, not any
of the other valid data on the disk. ASM also helps the Administrator to identify the
affected storage by writing a well known string to a new extent location, and this well
known string can be used to facilitate other methods of repair such as RMAN BMR
(block media recovery).

[6] Available as of Oracle Database release 12.1

[7] For high redundancy, an assumption is made that no pathological condition exists
that affects all three disk partners

[8] Rebalance times were generated in MAA labs with no competing application IOs to
document expectations without a workload profile variable (which changes from
customer to customer), and enable a pure comparison between normal and high
redundancy

[9] This may be done in certain cases where the rebalance completion is a priority, for
example if there is no application workload or one where service level impact doesn’t
matter

Verify important ASM instance parameters

Priority Alert Date Owner Status Engineered System

Level
Critical WARN 06/24/15 Michael Production
Exadata-User
Nowak Domain, Exadata-
Physical, SSC,ZDLRA
DB DB Engineered System Exadata OS & Validation Tool
Version Role Platform Version Version Version
12.1.0.2+ N/A X2-2(4170), X2-2, X2- 12.1+ Solaris - 11 TBD
8, X3-2, X3-8, X4-2, Linux x86-
X4-8, X5-2 64 el5uek
Linux x86-
64 el6uek
 

Benefit / Impact:
Aligning with MAA best practice parameters for the ASM instance improves stability,
security, and performance of ASM operations. Check the ASM documentation for
instructions and impact of changing specfic parameters.

Risk:

The risk of not using MAA best practice parameters for the ASM instances is security
vulnerability, performance issues, and possible data loss.

Action / Repair:

To verify important ASM instance parameters, run the following script with the
environment configured for the ASM instance. Note this script cannot check for all best
practice parameters because some are customer specific. Please refer to the 12.1.0.2
ASM instance best practice parameter and attribute reference for a complete list as well
as more details on why these parameters are best practices.

for record in asm_power_limit,4 audit_syslog_level,local0.info

audit_sys_operations,true
do
parameter_name=`echo $record | awk -F, '{print $1}'`
pass_value=`echo $record | awk -F, '{print $2}'`
current_value=`sqlplus -S "/ as sysasm" << eof
set head off
select lower(value) from v\\$parameter where name='$parameter_name';
exit
eof`
current_value=`echo $current_value | tr -d '\r'`
if [ $current_value == $pass_value ]
then
ret_val=0
echo -e "SUCCESS: $parameter_name current value $current_value matches best
practice value $pass_value"
else
echo -e "FAILURE: $parameter_name current value $current_value does not match
best practice value $pass_value"
ret_val=1
fi
done

The output should look like this:

SUCCESS: asm_power_limit current value 4 matches best practice value 4

SUCCESS: audit_syslog_level current value local0.info matches best practice
value local0.info
SUCCESS: audit_sys_operations current value true matches best practice value
true
If the output does not match please adjust any parameter listed with a FAILURE prefix

ASM rebalance and power limit considerations for GI 12.1.0.2

Priority Alert Date Owner Status Engineered

Level System
Critical WARN 8/12/2015 Michael Production Exadata-User
Nowak Domain, Exadata-
Physical, SSC
DB DB Engineered System Exadata OS & Validation Tool
Version Role Platform Version Version Version
12.1.0.2+ N/A X2-2(4170), X2-2, X2- 12.1+ Solaris - 11 TBD
8, X3-2, X3-8, X4-2, Linux x86-
X4-8, X5-2 64 el5uek
Linux x86-
64 el6uek
 

Benefit / Impact:

For physical deployments:

The default ASM power limit, ie the asm_power_limit parameter in the ASM instance,
should be set to ensure the application performance is not impacted while rebalance is
running. The default 12.1.0.2 deployment sets asm_power_limit=4 for all Exadata
racks. Here are some other considerations for ASM rebalance and the setting of
asm_power_limit

The default power of 4 was chosen after an internal MAA assessment showed it did not
impact the application. Also the following bug fixes should exist to ensure efficient
performance with minimal application impact

 All known ASM related fixes from MOS Exadata critical issues note
1270094.1
 Fixes for bugs 16087033 & 21218243 (GI side) and 13967870 (cell
side)
For OVM deployments :

The default 12.1.0.2 deployment sets asm_power_limit to as below for OVM

deployments :

deployments with 1 OVM RAC cluster - asm_power_limit=4

deployments with 2 OVM RAC clusters - asm_power_limit=2

deployments with more than 2 OVM RAC clusters - asm_power_limit=1

 The maximum power that should be used is 64 as anything higher

adds no performance benefit, only risk.
 Setting asm_power_limit=0 disables ASM rebalance operations. In
normal operation, asm_power_limit should not be set to 0.
 All MAA Exadata best practice attributes should be used to ensure
an efficient rebalance. Amongst other things, these attributes
ensure the compaction phase is disabled and appliance mode is set
properly.
 Fixed extents will be allocated using Exadata and Grid
Infrastructure 12.1.0.2 if the compatible.rdbms diskgroup attribute
is set to 11.2.0.4 or higher at allocation time. Note this means no
databases using versions < 11.2.0.4 can mount the diskgroup.
 During a rebalance that occurs after a failure where extent copies
are lost, the most critical files are rebalanced first, and evidence of
this can be found in the ASM alert log. For more information on
priority rebalance, see MOS note 1968607.1.
 To determine the amount of work that a rebalance needs to do
ahead of time, run the 'EXPLAIN WORK...' command and then
query v$asm_estimate. Explain work is documented here.
 To monitor a running rebalance, query v$asm_operation. Along
with the classic columns to assess work done so far and time
remaining, the new 'PASS' column can be used to see what phase
of rebalance is being executed.

Risk:

Not following MAA Exadata best practices with respect to asm_power_limit can cause
application impact during rebalance and/or cause excessive rebalance runtimes.

Action / Repair:

Customers should evaluate the performance impact of different ASM power limit
settings to their application while conducting an ASM rebalance operation under full
application workload in a non-production environment and choose their
ASM_POWER_LIMIT setting based on the results of the testing and the business
requirements for the application performance and availability.

Follow MAA Exadata ASM best practices using Grid Infrastructure 11.2.0.x

The following represent the consolidated list of MAA Exadata best practices for ASM
using Grid Infrastructure 11.2.0.x. They supersede all previous best practices for any
prior Grid Infrastructure versions.

Verify there is enough diskgroup free space for a rebalance operation

Priority Alert Date Owner Status

Engineered Engineered Bug(s)
Level System System
Platform
Critical FAIL 04/03/1 <Name Production Exadata - ALL 29248427 -
9 > Physical,  exachk
Exadata - 27733004 -
User Domain exachk
27656375 -
exachk
27092380 -
exachk
25559650 -
exachk
DB DB DB Role DB Exadata OS & Validation MAA
Version Type Mode Version Version Tool Scorecard
Version Section
11.2+ ASM N/A N/A ALL Linux exachk 19.2.0 N/A
Benefit / Impact:

Having enough diskgroup freespace available to accomodate a rebalance operation

helps to prevent an outage or loss of data.

The Impact of verifying that there is enough diskgroup freespace available for a
rebalance operation is minimal. The impact of correcting a lack of freespace will vary
depending upon the reason for the lack of freespace, and cannot be estimated here.
Risk:

If there is not enough diskgroup freespace available for a rebalance operation, there is
an increased risk of an outage or loss of data with an additional disk failure.

Action / Repair:

The following table displays the required free space for a complete rebalance operation
after one disk failure:

Grid Infrastructure Number of Required %

Version Failgroups free
12.1.0 any 15
greater than or equal to 12.2 less than 5 15
greater than or equal to 12.2 5 or more 9
To determine if there is sufficient free space to permit rebalancing after the loss of a
single disk, as the Grid Infrastructure owner, run the following query in SQLPlus on any
ASM instance in the cluster :

set serveroutput on
set linesize 160
declare
FGnum number;
gnum number;
gname varchar2(20);
freespace number;
totmb number;
freemb number;
usedmb number;
reqfmb number;
pctfree number;
reqpctfree number;
version varchar2(40);
cursor c1 is select x.group_number,count(x.numfg)
from ( select group_number,count(failgroup) numfg from v$asm_disk where
failgroup_type='REGULAR' and group_number > 0 group by group_number,failgroup
) x
group by x.group_number;
begin
dbms_output.enable(500000);
open c1;
loop
fetch c1 into gnum,FGnum;
exit when c1%notfound;
select name into gname from v$asm_diskgroup where group_number=gnum;
select version into version from v$instance;
select to_number(substr(regexp_replace(version,'\.'),1,3)) into version
from v$instance;
SELECT dg.name, dg.total_mb, dg.free_mb, dg.total_mb-dg.free_mb
INTO gname,totmb,freemb,usedmb
 FROM v$asm_diskgroup dg WHERE dg.group_number= gnum;
if (version >= 122) then
if (FGnum < 5) then
reqpctfree := 0.15;
else
reqpctfree := 0.09;
end if;
else
reqpctfree := 0.15;
end if;
pctfree := round(freemb/totmb,3);
if (pctfree < reqpctfree) then
dbms_output.put_line('FAILURE: Diskgroup '||gname||' has '||
pctfree*100||'% free space which is lower than the '||reqpctfree*100||'% free
space threshold. A rebalance may fail with ORA-15041'); dbms_output.new_line;
else
dbms_output.put_line('SUCCESS: Diskgroup '||gname||' has '||
pctfree*100||'% free space which is enough to complete the rebalance');
dbms_output.new_line;
end if;
end loop;
close c1;
end;
/

Examples of the output returned by the script:

FAILURE: Diskgroup DATAC1 has 13% free space which is lower than the 15% free
space threshold. A rebalance may fail with ORA-15041
SUCCESS: Diskgroup DBFS_DG has 100% free space which is enough to complete the
rebalance
SUCCESS: Diskgroup RECOC1 has 99% free space which is enough to complete the
rebalance
SUCCESS: Diskgroup TEST has 50% free space which is enough to complete the
rebalance

SUCCESS: Diskgroup DATAC1 has 99% free space which is enough to complete the
rebalance
FAILURE: Diskgroup DBFS_DG has 6% free space which is lower than the 9% free
space threshold. A rebalance may fail with ORA-15041
SUCCESS: Diskgroup RECOC1 has 100% free space which is enough to complete the
rebalance

If a diskgroup does not have sufficient free space, you should do one of the following:

 evaluate the space usage within the diskgroup and free up space by removing
tablespaces that are no longer needed
 move datafiles to another diskgroup that has enough space (datafile moves can
be done on-line as of Oracle 12.1 )
 increase the size of the diskgroup, assuming that some other diskgroup can be
 shrunk or, there is unallocated freespace on celldisks

NOTE: Although it is not a recommended best practice, some customers may want to
reserve enough space to tolerate a double disk failure in a high redundancy diskgroup.
For those cases, we have the following free space requirements:

Grid Infrastructure Number of Required % free

Version Failgroups
greater than or equal to 12.2 less than 5 29 (see Exception
1)
greater than or equal to 12.2 5 or more 15 (see Exception
2)
Exception 1: A diskgroup having failgroups with less than 8 disks (ex: eighth rack
configuration) requires 37% free space

Exception 2: A diskgroup having failgroups with less than 8 disks (ex: eighth rack
configuration) requires 18% free space

ASM power limit considerations and rebalance estimates for GI 11.2.0.x

Priority Added Machine Type OS Type Exadata Oracle

Version Version
Important 11/29/2011 X2-2(4170), X2-2, X2-8, Linux, 11.2.x + 11.2.x +
X3-2, X3-8, X4-2 Solaris
Benefit / Impact:

The default ASM power limit, ie the asm_power_limit parameter in the ASM instance,
should be set to ensure the application performance is not impacted while rebalance is
running. The default deployment sets asm_power_limit=1 for a eighth/quarter rack and
asm_power_limit=2 for all other rack types. Here are some other considerations for
ASM rebalance and the setting of asm_power_limit

 Redundancy is restored, and diskgroup data is protected, when the

compaction phase of rebalance begins. The compaction phase of
rebalance is active when the message "SUCCESS: refreshed
membership for <diskgroup number>/<hex string> (<diskgroup
name>) " is printed to the alert log, and v$asm_operation still
shows a rebalance running, usually with est_minutes at 1 or 0. The
compaction phase is removed in version 11.2.0.4.
 The highest power limit that should be used on a Database
Machine running is 32. This assumes 11.2.0.2 GI_HOME and a
diskgroup with the compatible.asm attribute set to 11.2.0.2.
 The anonymous PL/SQL block below* can be run against an ASM
instance before executing a rebalance. It will provide a rough
 estimate runtime for rebalance at the default power and the
maximum recommended power of 32. Note this estimate does not
include the compaction phase of rebalance and assumes HP (high
performance SAS) disks. Also, there are a number of variables
associated with this kind of calculation which is why it is a rough
estimate.
 If a rebalance is running at a lower power level than 32, and not
proceeding fast enough according to v$asm_operation, the
currently running rebalance may be replaced by executing a new
rebalance command (ex: alter diskgroup <dg name> rebalance
power 32;). Increasing the power should only be done when the
application SLAs can tolerate the increased IO level. There is a
some work that will repeated when a running rebalance is replaced
with a new one, but an increased power level can make up for it,
especially when the rebalance replacement is done toward the
beginnning of the original rebalance.
 Setting asm_power_limit=0 disables ASM rebalance operations. In
normal operation, asm_power_limit should not be set to 0.
 The semantics of the asm_power_limit have changed with 11202
software and a diskgroup with both rdbms and asm compatible
attributes set to 11202. The power limit with this new
software/config represents the number of outstanding relocations
and is implemented with async IO (therefore you will only see a
single process - ARB0 - executing the rebalance).

Risk:

Longer ASM rebalance time

Action / Repair:

Replacing a running rebalance to change the power limit.

* The following PL/SQL can be run against an ASM instance ahead of time to give a
very rough estimate for rebalance time both with the current default power and the
recommended DBM max power of 32. Note it assumes the diskgroup is already
balanced. There are MOS notes available with scripts to check diskgroup balance, but
imbalance should be an uncommon case.

set serveroutput on
declare
diskspace number;
cellspace number;
disk_rebal_time number;
cell_rebal_time number;
default_power_limit number;
max_rate_diff_cell number;
max_rate_diff_disk number;
failgroup_name varchar2(30);
diskgroup_name varchar2(30);
begin
dbms_output.put_line('--------------------------------------------------------
-------');
dbms_output.put_line('Estimates are provided for both the current default
power ');
dbms_output.put_line('(asm_power_limit) and the recommended DBM max of 32');
dbms_output.put_line('The current recommended default power limit for a DBM is
4');
dbms_output.put_line(chr(10));
dbms_output.put_line('Note these estimates are for a DBM with High Performance
drives');
dbms_output.put_line('and do not include the compaction phase of rebalance.
The ');
dbms_output.put_line('compaction phase is not required for redundancy
restoration');
dbms_output.put_line('--------------------------------------------------------
-------');
select value into default_power_limit from v$parameter where
name='asm_power_limit';
max_rate_diff_cell:=32/default_power_limit;
max_rate_diff_disk:=16/default_power_limit;
for dg in (select group_number from v$asm_Diskgroup where group_number>0 and
state='MOUNTED') loop
select failgroup into failgroup_name from v$asm_disk where
group_number=dg.group_number and rownum=1;
select name into diskgroup_name from v$asm_diskgroup where
group_number=dg.group_number;
select round(sum(total_mb-free_mb)/1024,0) into diskspace from v$asm_disk
where disk_number=1 and group_number=dg.group_number;
select round(sum(total_mb-free_mb)/1024,0) into cellspace from v$asm_disk
where failgroup=failgroup_name and group_number=dg.group_number;
disk_rebal_time:=round((diskspace/280)*60,0);
cell_rebal_time:=round((cellspace/1024)*60,0);
dbms_output.put_line(chr(10));
dbms_output.put_line('********************************************************
********');
dbms_output.put_line('Rough time estimates for rebalance of diskgroup '||
diskgroup_name||':');
dbms_output.put_line('DISK based rebalance at the default power of '||
default_power_limit||': '||disk_rebal_time*max_rate_diff_disk||' minutes');
dbms_output.put_line('CELL based rebalance at the default power of '||
default_power_limit||': '||cell_rebal_time*max_rate_diff_cell||' minutes');
dbms_output.put_line('DISK based rebalance at the maximum recommended power of
32: '||disk_rebal_time||' minutes');
dbms_output.put_line('CELL based rebalance at the maximum recommended power of
32: '||cell_rebal_time||' minutes');
dbms_output.put_line('********************************************************
********');
end loop;
end;
/
 

Ensure the ASM Disk Repair Timer is Set Correctly

Priority Added Machine Type OS Type Exadata Oracle

Version Version
Critical N/A X2-2(4170), X2-2, X2-8, X3- Linux, 11.2.x + 11.2.x +
2, X3-8, X4-2 Solaris
The ASM disk repair timer represent the amount of time a disk (or failure group..ie cell)
will remain offline before ASM drops it and run the subsequent rebalance. While the
disk or cell is offline, ASM is tracking the changed extents so the disk can be resynced if
the problem was a temporary. The default disk repair timer is 3.6 hours and can be
changed. If the default is inadequate, set the disk repair timer to the maximum amount
of time it takes to detect and correct a temporary disk failure. Here is an example of
changing the disk repair timer to 8.5 hours for the DATA diskgroup.

alter diskgroup data set attribute 'disk_repair_time'='8.5h';

It is important to understand that changing the disk_repair_timer parameter does NOT

change the repair timer in use for disks currently offline. The repair timer for those
offline disks is either the repair timer specified on the command line (if disks manually
offlined) or the default repair timer in effect when the offline occurred. To change the
repair timer for disks currently offlined, you can re-issue the offline command with the
desired repair timer specified.

It is also important to understand that, typically, disks and cells have different failure
and repair characteristics. Typically, a disk failure is permanent (ex: disk dies), and a
cell failure is transient (ex: cell reboot). Starting with release 11.2.1.3, a feature was
added to proactively drop disks that are predictive failure or failed. This allows the
setting of disk_repair_time to be geared almost exclusively toward cell failure, thus
easing the current aforementioned trade-off. In other words, you can set
disk_repair_time primarily for cell failure.

Check ASM rebalance forward progress if you suspect a problem

Priority Added Machine OS Exadata Oracle Version

Type Type Version
n/a Apr X2-2(4170), Linux 11.2.x + 11.2.0.2 or above with diskgroup
2012 X2-2, X2-8, X3- attribute compatible.asm=11.2.0.2
2, X3-8, X4-2 or higher
Benefit / Impact:

This check can be used when one has a long running rebalance and want to ensure it is
making forward progress. We have estimates in v$asm_operation but they can be
misleading, especially when an IO intensive application is running alongside the
rebalance.

Risk:

Apprehension that long rebalance runtime is stuck or won't finish.

Action / Repair:

Set the environment for ASM, and run the script below on the node running the
rebalance. The considerations and requirements should be read before execution.

Script:

#!/bin/bash
# Goal
# 1. Provide assurance that rebalance is making progress by reporting the
# current file (Phase 1) or disk (Phase 2) being processed.
# Considerations
# 1. Works file by file so bigger files will be reported longer
# 2. Only works for diskgroups with compatible.asm=11.2.0.2 or higher
# 3. Only works with a running rebalance
# 4. Does not account for files added while the script is running
# Requirements to run:
# 1. Set environment for ASM instance before running
# 2. Execute with "./rebalance_file_progress.sh <optional check interval
time where default is 10 minutes>"

function check_env {
if [[ $ORACLE_SID != +ASM* ]]; then
echo "Environment not set for ASM."
exit 1
fi
}

function check_running {
arbpid=`ps -ef | grep asm_arb0 | grep -v grep | awk '{print $2}'`
if [ -z "$arbpid" ]; then
echo "Rebalance is not running (no ARB0 ospid)"
exit 1
fi
}

function goto_trace_directory {
td=`sqlplus -s "/ as sysasm" << eof
set head off
set pagesize 0
select value from v\\$parameter
where name='background_dump_dest';
exit;
eof`
read -rd '' td <<< "$td"
cd $td
}

function check_trace_file {
tfile=${ORACLE_SID}_arb0_${arbpid}.trc
if [ ! -f $tfile ]
then
echo "The ARB0 trace file $tfile doesn't exist."
exit 1
fi
}

function collect_static_data {
dg=`grep 'relocating file' $tfile | tail -1 | awk -F "+" '{print $2}' | awk -F
"." '{print $1}'`;
maxfile=`sqlplus -s "/ as sysasm" << eof
set head off
set pagesize 0
select max(file_number) from v\\$asm_file
where GROUP_NUMBER = (select group_number from v\\$asm_Diskgroup where
name='$dg');
exit;
eof`
read -rd '' maxfile <<< "$maxfile"
disklistall=`sqlplus -s "/ as sysasm" << eof
set head off
set pagesize 0
set feedback off
select DSKNUM_KFDDD from x\\$kfddd
where GRPNUM_KFDDD = (select group_number from v\\$asm_Diskgroup where
name='$dg');
exit;
eof`
disklist=`echo $disklistall | tr ' ' '\012' | uniq`
i=1
for disk in ${disklist[@]}
do
dla[${disk}]=$i
let "i++"
done
}

function get_current_disk_compact {
currentdisk=`sqlplus -s "/ as sysasm" << eof
set head off
set pagesize 0
select disk_kfgmg from x\\$kfgmg
where NUMBER_KFGMG = (select group_number from v\\$asm_Diskgroup where
name='$dg');
exit;
eof`
read -rd '' currentdisk <<< "$currentdisk"
if [ $currentdisk -eq "65535" ]; then
echo "Incorrect Phase 2 calculation..please rerun the script"
exit 1
fi
}

# Main
declare -a dla
declare -i oldfile
declare -i currentfile
declare -i maxfile
declare -i currentdisk
check_env
check_running
goto_trace_directory
check_trace_file
collect_static_data
currentfile=`grep 'relocating file' $tfile | tail -1 | awk -F "+" '{print $2}'
| awk -F "." '{print $2}'`;
oldfile=$currentfile
if [ -z "$1" ]; then
check_interval=600
else
check_interval=$1
fi
echo "######################################################################"
echo "This script will monitor Phase 1 (rebalance) file by file and Phase 2"
echo "(compaction) disk by disk. Both phases should increment, showing
progress."
echo "This script will *not* estimate how long the rebalance will take."
echo "######################################################################"
echo " "
echo "Diskgroup being rebalanced is $dg"
echo "ASM file numbers for databases start at 256. "
echo "Default check interval is 600 seconds. This run using $check_interval
seconds..."
echo " "
while [ $currentfile -ge $oldfile ]
do
check_running
echo "`date`: PHASE 1 (of 2): Processing file $currentfile out of $maxfile"
sleep $check_interval
oldfile=$currentfile
currentfile=`grep 'relocating file' $tfile | tail -1 | awk -F "+" '{print $2}'
| awk -F "." '{print $2}'`;
done
sleep 30
echo "*******************************************************"
echo "`date`: PHASE 1 (of 2) complete."
echo "*******************************************************"
while true
do
check_running
get_current_disk_compact
echo "`date`: PHASE 2 (of 2): ${dla[$currentdisk]} disks processed out of $
{#dla[@]}"
sleep $check_interval
done
Sample output:

$ ./rebalance_progress.sh 300
######################################################################
This script will monitor Phase 1 rebalance file by file and Phase 2
rebalance disk by disk. Both phases should increment, showing progress.
This script will *not* estimate how long the rebalance will take.
######################################################################

Diskgroup being rebalanced is DATA

ASM file numbers for databases start at 256.
Default check interval is 600 seconds. This run using 300 seconds...

Wed May 2 07:47:44 PDT 2012: PHASE 1 (of 2): Processing file 258 out of 365
Wed May 2 07:52:44 PDT 2012: PHASE 1 (of 2): Processing file 294 out of 365
Wed May 2 07:57:44 PDT 2012: PHASE 1 (of 2): Processing file 306 out of 365
Wed May 2 08:02:44 PDT 2012: PHASE 1 (of 2): Processing file 329 out of 365
Wed May 2 08:07:44 PDT 2012: PHASE 1 (of 2): Processing file 346 out of 365
Wed May 2 08:12:44 PDT 2012: PHASE 1 (of 2): Processing file 346 out of 365
Wed May 2 08:17:44 PDT 2012: PHASE 1 (of 2): Processing file 346 out of 365
Wed May 2 08:22:44 PDT 2012: PHASE 1 (of 2): Processing file 346 out of 365
Wed May 2 08:27:44 PDT 2012: PHASE 1 (of 2): Processing file 346 out of 365
Wed May 2 08:32:44 PDT 2012: PHASE 1 (of 2): Processing file 347 out of 365
Wed May 2 08:37:44 PDT 2012: PHASE 1 (of 2): Processing file 347 out of 365
Wed May 2 08:42:44 PDT 2012: PHASE 1 (of 2): Processing file 348 out of 365
Wed May 2 08:47:44 PDT 2012: PHASE 1 (of 2): Processing file 348 out of 365
Wed May 2 08:52:44 PDT 2012: PHASE 1 (of 2): Processing file 348 out of 365
Wed May 2 08:57:44 PDT 2012: PHASE 1 (of 2): Processing file 348 out of 365
*******************************************************
Wed May 2 09:02:49 PDT 2012: PHASE 1 (of 2) complete.
*******************************************************
Wed May 2 09:02:51 PDT 2012: PHASE 2 (of 2): 44 disks processed out of 168
Wed May 2 09:07:52 PDT 2012: PHASE 2 (of 2): 44 disks processed out of 168
Wed May 2 09:12:53 PDT 2012: PHASE 2 (of 2): 76 disks processed out of 168
Wed May 2 09:17:54 PDT 2012: PHASE 2 (of 2): 76 disks processed out of 168
Wed May 2 09:22:55 PDT 2012: PHASE 2 (of 2): 108 disks processed out of 168
Wed May 2 09:27:57 PDT 2012: PHASE 2 (of 2): 140 disks processed out of 168
Wed May 2 09:32:58 PDT 2012: PHASE 2 (of 2): 168 disks processed out of 168
Rebalance is not running (no ARB0 ospid)

Check forward progress on an long running ASM resync if a problem is

suspected

Priority Added Machine Type OS Type Exadata Oracle

Version Version
Important 5/16/2012 X2-2(4170), X2-2, X2- Linux, 11.2.x + 11.2.x +
8,X3-2, x3-8, X4-2 Solaris
Benefit / Impact:

Ensures a disk resync is making forward progress.

Risk:

Long running resyncs may be killed by customers if they are perceived to be

stuck/hung/spinning

Action / Repair:

The following script can be run to check forward progress on running disk resyncs.

#!/bin/bash
# Goal
# 1. Provide assurance that resync is making progress by reporting the
# current file and extent being processed by each active resync slave.
# Considerations
# 1. Works file by file so bigger files will be reported longer
# 2. Only works with resyncs that are active; cannot be run ahead of time
# 3. Does not account for files added while the script is running
# 4. Interval < 16 seconds is not useful due to the tracing rate, therefore
# a minimum of 20 is enforced

# Requirements to run:
# 1. Set environment for ASM instance before running
# 2. Execute with "./resync_progress.sh <optional check interval
# greater than or equal to 20 seconds (default is 10
minutes)>"

function check_env {
if [[ $ORACLE_SID != +ASM* ]]; then
echo "Environment not set for ASM."
exit 1
fi
}

function check_running {
resyncrunning=`sqlplus -s "/ as sysasm" << eof
set head off
set pagesize 0
select count(*) from v\\$asm_operation where operation='ONLIN';
exit;
eof`
read -rd '' resyncrunning <<< "$resyncrunning"
if [ $resyncrunning -eq 0 ]; then
echo "Resync is not running on this instance (no ONLINE operation
detected)"
exit 1
fi
}

function goto_trace_directory {
td=`sqlplus -s "/ as sysasm" << eof
set head off
set pagesize 0
select value from v\\$parameter
where name='background_dump_dest';
exit;
eof`
read -rd '' td <<< "$td"
cd $td
}

function process_updates {
while true
do
check_running
if [ $resyncrunning -ne $prior_running ]; then
echo " "
echo "*** $resyncrunning disk resyncs running on the local ASM instance ***"
echo " "
fi

# Things can change drastically in between intervals so we must collect

everytime
slavepids=`ps -ef | grep 'asm_s[0-9][0-9][0-9]_' | sort -k 2 | grep -v grep |
awk '{print $2}'`
if [ -z "$slavepids" ]; then
echo "Resync is not running on this instance (no slaves ospids detected)"
exit 1
fi
if [ $resyncrunning -gt 1 ]; then
echo "***"
fi
for slave in ${slavepids[@]}
do
slavename=`ps -ef | grep $slave | grep -v grep | awk '{print substr($8,5,4)}'`
read -rd '' $slavename <<< "$slavename"
sa[${slave}]=$slavename
tfile=${ORACLE_SID}_${sa[$slave]}_${slave}.trc
if [ ! -f $tfile ]
then
echo "The trace file $tfile doesn't exist."
exit 1
fi
slavegrp=`grep "NOTE: PST update grp =" $tfile | tail -1 | awk '{print $6}'`
sga[${slave}]=$slavegrp
maxfile=`sqlplus -s "/ as sysasm" << eof
set head off
set pagesize 0
select max(file_number) from v\\$asm_file
where GROUP_NUMBER = $slavegrp;
exit;
eof`
read -rd '' maxfile <<< "$maxfile"
smfa[${slave}]=$maxfile
secs_since_mod=`perl -e '@q=stat($ARGV[0]); print time-$q[9]' $tfile`
currentfile=`grep '^f[0-9][0-9][0-9]' $tfile | tail -1 | awk '{print $1}' |
awk -F "f" '{print $2}'`
currentextent=`grep '^f[0-9][0-9][0-9]' $tfile | tail -1 | awk '{print $2}' |
awk -F "x" '{print $2}' | awk -F"," '{print $1}'`
if [ $currentfile -eq 0 ] || [ $secs_since_mod -gt 20 ]; then
echo "`date`: RESYNC slave ${sa[$slave]}, ospid $slave : Not actively
resyncing"
else
echo "`date`: RESYNC slave ${sa[$slave]}, ospid $slave : Processing diskgroup
${sga[$slave]}, file $currentfile (out of ${smfa[$slave]}), extent
$currentextent"
fi
done
prior_running=$resyncrunning
sleep $check_interval
done
}

# Main
declare -a sa
declare -a sga
declare -a smfa
declare -i currentfile
declare -i maxfile
declare -i slavegrp
declare -i prior_running
declare -i secs_since_mod
check_env
check_running
goto_trace_directory
if [ -z "$1" ]; then
check_interval=600
else
if [ $1 -ge 20 ]
then
check_interval=$1
else
check_interval=20
fi
fi
echo
"#########################################################################"
echo "This script will monitor disk resync, showing progress for each active"
echo "slave. This script will *not* estimate how long the resync will take."
echo
"#########################################################################"
echo " "
echo "ASM file numbers for databases start at 256. "
echo "Default check interval is 600 seconds. This run using $check_interval
seconds..."
prior_running=resync_running
process_updates

Sample output

$./resync_progress.sh 60
#########################################################################
This script will monitor disk resync, showing progress for each active
slave. This script will *not* estimate how long the resync will take.
#########################################################################

ASM file numbers for databases start at 256.

Default check interval is 600 seconds. This run using 60 seconds...

*** 1 disk resyncs running on the local ASM instance ***

Tue May 15 10:18:43 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 15624
Tue May 15 10:19:44 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 21755
Tue May 15 10:20:46 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 22973
Tue May 15 10:21:48 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 23918
Tue May 15 10:22:49 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 24922
Tue May 15 10:23:51 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 36232
Tue May 15 10:24:53 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 43283
Tue May 15 10:25:55 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 43955
Tue May 15 10:26:56 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 48147
Tue May 15 10:27:58 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 48483
Tue May 15 10:29:00 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 48884
Tue May 15 10:30:01 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 49206
Tue May 15 10:31:03 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 60938
Tue May 15 10:32:05 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 61251
Tue May 15 10:33:07 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 61662
Tue May 15 10:34:08 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 61998
Tue May 15 10:35:10 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 62372
Tue May 15 10:36:11 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 62670
Tue May 15 10:37:13 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 63065
Tue May 15 10:38:14 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 63406
Tue May 15 10:39:16 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 63786
Tue May 15 10:40:18 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 325 (out of 387), extent 64122
Tue May 15 10:41:19 CDT 2012: RESYNC slave s000, ospid 106162 : Processing
diskgroup 1, file 326 (out of 387), extent 824
Resync is not running on this instance (no ONLINE operation detected)