Article Number: 000115696 Print

PowerEdge: How to Recover from a Corrupt BIOS or

Interrupted BIOS Update
Summary:
This article details the cause and solution of a "No POST" issue caused by a corrupted BIOS or
interrupted BIOS update.

Article Content

Instructions

This article details how to recover from a "No POST" issue caused by a corrupted BIOS or interrupted BIOS update.

This article is also applicable in case of 14G servers and newer hanging in POST at "Configuring memory...... Please wait! "

Table of Contents
Description
Issue Identification
Solution
Troubleshooting

Description

This article details how to recover from a "No POST" issue caused by a corrupted BIOS or interrupted BIOS update.

It is not recommended to power cycle or otherwise interrupt a system BIOS update while it is in progress. This procedure is
likely to cause a "No power, No POST" issue, where the system does not power on when the power button is pressed.

Dell EMC PowerEdge Servers, beginning with systems which support Intel Xeon Scalable Processors (formerly Skylake) and second
Generation Intel Xeon Scalable Processors (formerly Cascade Lake), have a built-in BIOS recovery process.

This BIOS recovery process may be able recover systems where a BIOS update was interrupted or the BIOS was found corrupt and
the system was hanging as described above. Follow the steps in this article to begin the BIOS recovery process.

Issue Identification

To verify that a BIOS update interruption occurred, check the server's LifeCycle Controller (LCC) logs for a similar sequence of events.
You can find out how to access the LCC logs here.

 
SYS1003 System CPU Resetting.

SYS1000 System is turning on.

RAC0182 The iDRAC firmware was rebooted with the following reason: ac.

SUP0516 Updating firmware for System BIOS (BIOS.Setup.1-1) to version 2.2.9.

SYS1003 System CPU Resetting.

RED030 Reboot is complete.

 SYS1000 System is turning on.

SYS1001 System is turning off.

SYS1003 System CPU Resetting.

RAC0702 Requested system power cycle.

USR107 The operation SetupJobQueue of the DCIM_JobService was performed by root.

RED029 A reboot is pending.

JCP027 The (installation or configuration) job RID_633911391716 is successfully created on iDRAC.

USR107 The operation CreateRebootJob of the DCIM_SoftwareInstallationService was performed by root.

RED002 Package successfully downloaded.

RED111 Successfully downloaded the update package details 28.7065 MB in 16.33 secs at 1.7579 MBps (14.0632 Mbps)
[FOLDER05611398M_1_BIOS_5XWV7_WN64_2.2.9.EXE].

RED110 Downloading the FOLDER05611398M_1_BIOS_5XWV7_WN64_2.2.9.EXE update package.

Figure 1: Log excerpt showing interrupted BIOS update.

In the above log excerpt, starting from the bottom and going up, the BIOS update is initiated (Event ID JCP027). The system then
experienced a power AC loss and subsequent power restoration (Event ID RAC0182) during the update.

If the update had completed without interruption the logs would show the following:
Event ID SUP0518: "Successfully updated the System BIOS (BIOS.Setup.1-1) firmware to version X.Y.Z" (with X.Y.Z being the BIOS
version)
Event ID SYS1003: "System CPU Resetting".

Solution

Once the issue has been confirmed as an interrupted BIOS update issue, use the following steps to perform the BIOS recovery.

Steps to perform BIOS recovery:

1. Check the LCC job queue, and delete all jobs to stop any interfering the BIOS recovery.

 
This can be performed from the iDRAC web GUI from Maintenance -> Job Queue. Select all jobs and click the "Delete" button.

Figure 2: iDRAC9 Job Queue with job IDs displayed

Alternatively, the job queue can be deleted with local or remote racadm command: racadm jobqueue delete -i
JID_CLEARALL_FORCE

Figure 3: SSH session showing "jobqueue delete -i JID_CLEARALL_FORCE" command

 
 The Job Queue page should return no jobs.

Figure 4: iDRAC9 Job Queue cleared

 
2. SSH to iDRAC and run the command: racadm recover BIOS.Setup.1-1

Figure 5: SSH session showing racadm recover BIOS.Setup.1-1 command

Figure 5 A: SSH session showing racadm recover BIOS.Setup.1-1 command when console shows racadm>>
3. In the LCC log, a message ID of UEFI0298 with description of "The BIOS Recovery process is started" should be present.

Figure 6: iDRAC9 LCC log with Message IDs displayed

 
4. After a few minutes, the system will power on from a recovery ROM image. The boot screen displays the "[Recovery BIOS]" text.

 Figure 7: Boot screen showing commencement of BIOS recovery

 
5. The system then performs a BIOS update to flash the primary ROM. This is completed with the full BIOS image that is stored in
iDRAC from the last successful BIOS update performed from iDRAC/LC.

Figure 8: Boot screen showing progress bar of BIOS recovery

  
6. Once the BIOS flash completes, the system should restart and boot from the primary ROM image. The LC log should show
"UEFI0299: The BIOS Recovery process is completed".

Figure 9: iDRAC9 LCC logs showing confirmation of BIOS recover completion

7. Once BIOS is recovered, it will come up as a new system board detected as seen below in figure 10 and the standard easy recovery
process applies.

Figure 10: POST picture showing standard easy recovery options

8. Press Y to start restore process.
 9. The server will restart and then it will show the "The system content was successfully restored." (figure 11)

Figure 11: Picture showing system configuration restore

10. Press Y again and the server will restart multiple times as in Figure 12

Figure 12: Example of server restarting during configuration restore

** Cautionary note**

After doing the BIOS recovery if the Lifecycle controller goes into recovery mode then run the following commands to re-enable it: 
 1. racadm set LifecycleController.LCAttributes.LifecycleControllerState 0
This command will disable the lifecycle controller
2. racadm set LifecycleController.LCAttributes.LifecycleControllerState 1
This command will enable the lifecycle controller

 
 

Troubleshooting

In the event either the BIOS recovery does not work or an auto recovery occurs, but does not resolve, a BIOS image can be specified to
perform recovery. It is recommended to provide a BIOS image of the same version as that which was on the system before the latest
BIOS update attempt.

The process is similar to the steps above, with the addition of uploading the BIOS Dell Update Package (DUP) to the iDRAC web GUI.
1. Clear the LCC job queue through the methods shown in the solutions section.

 
2. Upload BIOS DUP to iDRAC web GUI. From initial iDRAC web page, select Maintenance -> System Update. Click "Choose File"
button, and browse to BIOS DUP file.

Figure 10: iDRAC9 System Update screen showing Manual Update option

 
3. Select "Upload". Activate the check box for the BIOS update and select "Install Next Reboot"

Figure 11: iDRAC9 Manual Update screen showing Update Detail check box and install Next Reboot command

 
4. Perform steps 2 through 6 as in the first section above (SSH to iDRAC, run "racadm recover BIOS.Setup.1-1").
 

Article Properties

Affected Product
PowerEdge

Last Published Date

09 May 2022

Version

Article Type
How To