Question 1 :  

     Can you describe about your VMware Infrastructure that you are supporting?
What are the daily operations?
Hint:                Interviewer wants to know weather you have live experience or not. Based on your
answer he will conclude couple of items.
 
Answer:
I am part of VMware team (5 to 10 members) and working as L1/L2/L3/SME level supporting
Global Customer Account. Tell him/her customer details are confidential and can’t be disclosed.
 
 
 
Technical details:
We have 3 vCenter servers configured for two Data Center locations of the Customer. Two of
them are production and other one is for Dev/Test purpose. Production vCenter servers having
multiple (say 5) clusters and each of them has 10+ ESXi servers.
We are supporting around 150+ ESXi servers which are running 1500+ Virtual Machines
We have 5.x versions of ESXi in the Infrastructure but for one cluster they are running at same
version. VCenter servers are configured at 5.5 U2 version.
 
Daily tasks:
 
1)   VMware Health Check Report – Running the script and sending the reports to Management
Check my other post
2)    Checking vCenter server console for alarms and alerts
3)    Scheduling changes required for VMware tasks
4)    Attend meetings with Architects
5)    Working on VMware related incidents like backup failure, VM not pinging, ESXi host down,
vCenter service failure …. Etc.

Question 2:        There is Virtual Machine which is not pingable/rdp and in vCenter console it was
hung. All options are grayed out at VM options. How to recover this Virtual machine?
Hint:                Interviewer wants to know your ESXi command line skills to troubleshoot the
scenario
 
Answer:          
From the symptoms it is clear that there is no option available from vCenter server except you
can see Events & Tasks to understand if any action performed before it went to hung state.
For eg:  Backup jobs taking snapshot, L1/L2 Admin tried to hit multiple tasks to
Shutdown/Restart/Power-off VM
 
 
With all these symptoms let us identify the ESXi host on which VM is running and get the root
password either from your Team Lead or Tool where you can get the shared ID password.
 
Step 1:    Determining the virtual machine’s location
Determine the host on which the virtual machine is running. This information is available in the
virtual machine’s Summary tab in VI Client. Subsequent commands will be performed on, or
remotely reference, the ESXi host where the virtual machine is running.
 
Step 2:    Open a console session either in the ESXi Shell or Putty session to ESXi host via
Name/IP
 
Step 3:    Get a list of running virtual machines, identified by World ID, UUID, Display Name, and
path to the .vmx configuration file
esxcli vm process list

 
Step 4: Power off one of the virtual machines from the list using this command:
esxcli vm process kill –type=[soft,hard,force] –world-id=WorldNumber
Three power-off methods are available – Soft is the most graceful, hard performs an immediate
shutdown, and force should be used as a last resort.
 
Step 5:    Check the Virtual Machine process again to make sure it is no more exist
esxcli vm process list

Question 3:     What are the steps that you will take when vCenter service failed to start in your
Infrastructure? It is running with 5.1 or 5.5 version.
Hint:   Interviewer wants to know your troubleshooting skills and confirm weather you worked on
this issue at-least once

 
 
Answer:
Validate if each troubleshooting step below is true for your environment. Each step provides
instructions or a link to a document that helps eliminate possible causes and take corrective
action as necessary
.
Note: If you perform a corrective action in any of the following steps, attempt to restart the
VMware Virtual Center Server service.
1. Verify that the VMware Virtual Center Server service cannot be restarted. Try to restart the
service once again and check for logs for error messages
 
2. Verify that the configuration of the ODBC Data Source (DSN) used for connection to the
database for vCenter Server is correct.
Based on your Infrastructure – SQL/DB Server either on vCenter or on other Production SQL
Cluster
 
3. Verify there is enough free disk space on the vCenter Server. Also disk space on SQL DB is
running, DB configured with dynamic size, DB logs are grown .. etc
Sometimes you need to contact SQL Team who can perform advanced troubleshooting steps
 
4. Verify that ports 902, 80, 8080, 8433 and 443 are not being used by any other application.
If another application, such as Microsoft Internet Information Server (IIS) (also known as Web
Server (IIS) on Windows 2008 Enterprise), Routing and Remote
Access Service (RAS), World Wide Web Publishing Services (W3SVC), Windows Remote
Management service (WS-Management) or the Citrix Licensing Support service are
utilizing any of the ports, vCenter Server cannot start.
If you see an error similar to one of the following when reviewing the logs, another application may be
using the ports:
Failed to create http proxy: Resource is already in use: Listen socket: :<port>
Failed to create http proxy: An attempt was made to access a socket in a way forbidden by its access
permissions.
proxy failed on port <port>: Only one usage of each socket address (protocol/network address/port) is
normally permitted
 
5. Verify the health of the database server that is being used for vCenter Server. If the hard
drives are out of space, the database transaction logs are full, or
if the database is heavily fragmented, vCenter Server may not start.
Sometimes you need to contact SQL Team who can perform advanced troubleshooting steps
 
6. Verify the VMware VirtualCenter Service is running with the proper credentials.
vpxd.exe utility helps you to update DB credential
KB 1006482
 
7. Verify that critical folders exist on the vCenter Server host
 
8. Verify that no hardware or software changes have been made to the vCenter server that may
have caused the failure. If you have recently made any changes to the vCenter server, undo
these changes temporarily for testing purposes
 
9. Before launching vCenter Server, ensure that the VMwareVCMSDS service is running
 
10. Verify that the vpxd.exe is present in C:\Program Files\VMware\Infrastructure\VirtualCenter
Server\vpxd.exe location. If this file is not present, reinstall vCenter Server
 
Your troubleshooting skills will be useful to identify error messages and use Google to find
nearest solution. Logical thinking is always required.

Question 4:     How do you perform ESXi patching in your Infrastructure?

Hint:   Interviewer wants to understand your ITIL Skills like Change management along with
Technical answer
 
Answer:
Based on your Infrastructure size and Internet Connectivity status your answer will vary. Let us
cover generic information followed by each case.
 
ITIL Process:
Take outage approval from Customer
Submit the Change record based on outage window
Discuss with management and Customer via meetings to get approval
Once approved – apply the patches at approved outage window
Submit the artifacts about successful closure of patching and Change record
 
Patching will be performed once in 3months [OR] once in 6 months and before we start patching
the hosts we need to configure the Update Manager.
Open VMware vCenter server, and on the Home page click the Update Manager icon select
Network Connectivity
Network connectivity – On this section you can change the ports on which clients and ESX/ESXi
servers communicate with the Update Manager server.
Download Settings
Direct connection to the Internet – If the Update Manager server has an internet connection you
should choose this option to download patches from the VMware repository.
Use a share repository – This is for those that don’t have an internet connection on the Update
Manager server, and they are using an internal web server to publish VMware patches.
Use proxy – Use this only if your Update Manager server needs to pass trough a proxy server to
connect to the internet.
When you are done with your configuration hit the Apply button to save the changes. To start
downloading the patches press the Download Now button. By pressing the download button it
will not start to download the patches only an index of them.
As soon as you click the download button, the patches index is downloaded. When the process
is done you can see all the available updates on the Patch Repository tab.
The next step is to create a Baseline, where we tell Update Manager what updates to download,
and what type of updates to use for patching. Usually the default baselines are sufficient, but we
can customize it based on requirement. Go to the Baseline and Groups tab and click the Create
link.
Give the baseline a name and leave the default baseline type which is Host Patch
If you go with the first option, future updates will not be included in this baseline and you will

need to create a new baseline, or edit this one to include those updates.
Choose the patch type you want to include in this baseline based on you ESX/ESXi hosts and

select Finish button for successful creation of baseline  

Click the data center object in the Inventory pane. If you want to patch one server only, click the
server object. It is not recommended to patch all your hosts in the datacenter at once, especially
if you are in a production environment because your VMs will stop, and customer will be
unhappy.
Click the Attach link in the upper right corner.
In the Attach Baseline window select the baseline which is created earlier then click the Attach
button.
Remember, all your VMs will stop because the ESX/ESXi hosts need to be in maintenance mode
before the actual patching begins. Move the VMs to another host if you are in a production
environment.
At the Ready to Complete screen click the Finish button to start the patching process.
This is going to take a while, because those patches need to be downloaded from the VMware
repository.
Your ESXi hosts will reboot a couple of times maybe, depends on the updates.

Question 5: How do you troubleshoot Virtual Machine Backup failures in your Infrastructure?

Hint:   Interviewer wants to understand your skills with various backup solutions at vCenter
 
Answer:
We have many vendors to provide backup solutions like Symantec NetBackup, Veeam, IBM
Tivoli, VMware Data Protector … etc
Vendors started releasing Backup solutions exclusively for Hyper-visors like VMware vSphere,
Microsoft Hyper-V & Citrix Xen Server
  Based on your customer Infrastructure Architecture you will fall in below cases:
1)    Run backup clients from within a virtual machine performing file‐level or image‐level
backups.
2)    Run backup clients from the ESX service console, backing up virtual machines in their
entirety as files residing in the VMFS file system.
3)    Back up virtual machine data by running a backup server within a virtual machine that is
connected to a tape drive or other SCSI‐based backup media attached to the physical system.
4)    When virtual machine files reside on shared storage, use storage‐based imaging on storage
such as SAN, NAS, or iSCSI, or an independent backup server (a proxy backup server or
NDMP) to back up virtual machine files
 
Sample Error messages:
The virtual machine identified by <unset> is unknown. Exit Code: 16
status: 156: snapshot error encountered
status: 13: file read failed
the backup failed to back up the requested files(6)
 
Troubleshooting backup failures purely depends on error message received during backup
failure like “Can not create a quiesced snapshot”
You need to logically separate the failure scenarios like if all backups are failing then it should be
your vCenter server service.
If only one VM failing with specific error then check for status of below services:
VSS  :  Volume Shadow Copy
SWPRV : Microsoft Software Shadow Copy Provider
 
Check for output of command “VSSADMIN LIST WRITERS”  –  Make sure no errors are reported
 
  Take manual snapshot from vCenter and check for errors (if any)
 
VMware vSphere backup best practices:
 
1) Don’t back up virtual machines at the guest OS layer
2) Leverage the vStorage APIs
3) Know how quiescing and VSS works
4) Don’t skip on backup resources
5) Snapshots are not backups
6) Schedule backups carefully
7) Know your Fault Tolerance backup alternatives
8) Don’t forget to back up host and vCenter Server configs

Question 6:     How do you troubleshoot ESXi/ESX host that is disconnected or not responding in

your Infrastructure?
Hint:   Interviewer wants to understand your skills for common issues in VMware with ESX/ESXi
 
Answer:
It is common problem faced by every VMware Administrator in their Infrastructure. Most of them
will stuck at Network troubleshooting as they feel disconnected happened due to connectivity
problems and their approach is correct for few scenarios.

Let us discuss various reasons for the host disconnect which is expected by Interviewer 
1)  Verify that the ESXi host is in a powered on state – sounds silly but most of the people forget
to check server status via ILO/DRAC/RIB … etc
 
2)  Verify that network connectivity exists from vCenter Server to the ESXi host with the IP and
FQDN –  this is VMware Administrator common suspicious point.
3)  Verify that the ESXi host can be reconnected, or if reconnecting the ESXi host resolves the
issue – Simple and sometimes it will resolve the problem
 
4)  Verify that the ESXi host is able to respond back to vCenter Server at the correct IP address.
If vCenter Server does not receive heartbeats from the ESXi host, it goes into a not responding
state. To verify if the correct Managed IP Address is set, see Verifying the vCenter Server
Managed IP Address and ESXi 5.0 hosts are marked as Not Responding 60 seconds after being
added to vCenter Server. (Known issue)
5) ESXi/ESX host disconnects from vCenter Server after adding or connecting it to the inventory
(VMware KB2040630)
 
6)  Verify that you can connect from vCenter Server to the ESXi host on TCP/UDP port 902. If
the host was upgraded from version 2.x and you cannot connect on port 902, then verify that you
can connect on port 905.
Use simple Telnet command for checking the ports status
 
7) Verify if restarting the ESXi Management Agents resolves the issue – You ran these

commands many times right 

Run these commands:
/etc/init.d/hostd restart
/etc/init.d/vpxa restart
To restart all management agents on the host, run the command:
services.sh restart
 
8) ESXi hosts can disconnect from vCenter Server due to underlying storage issues – Complex
one to explain but you should know the pain points from HBA card of ESXi server to LUN/Disk of
the Storage Box for fixing these issues
These points are sufficient to keep the Interviewer happy and let us discuss more Real time
Scenarios in next post …
======================================================================
Question 7:     How do you troubleshoot unable/failed to power-on Virtual Machine Scenario?
Hint:   Interviewer wants to understand your skills for common issues in VMware with Virtual
Machines
 
Answer:
Let me ask you this question – what happens if production SQL DB Virtual Machine failed to

Power - on which is hosting critical data bases for HR-Payroll department 

No salary slips are generated and your Company can’t pay your salary (Just kidding)
Now you can easily understand the impact for Customer of this real time scenario when their
Virtual Machines are in Power-off state. Let us focus on Troubleshooting methodology for this
Scenario.
 
Symptoms:
 Cannot power on the virtual machine
 Powering on the virtual machine fails
 You see the errors similar to: Failed to power on VM
Could not power on VM: Admission check failed for memory resource See the VMware ESX
Resource Management Guide for information on resource management settings.
Group vm.3582: Invalid memory allocation parameters for VM vmm0:New_Virtual_Machine.
(min: 524288, max: -1, minLimit: -1, shares: -1, units: pages)
Group vm.13327: Cannot admit VM: Memory admission check failed. Requested reservation:
311199 pages
 You see the error:  Unsupported and/or invalid disk type
 In the Events tab on the ESXi hosts or vCenter Server, you see the error:Module
DevicePowerOn power on failed.
Unable to create virtual SCSI device for scsi0:0,
‘/vmfs/volumes/datastorename/VirtualMachineHome/VirtualMachineDisk.vmdk’
Failed to open disk scsi0:0: Unsupported and/or invalid disk type 7. Did you forget to import
the disk first?
 
Troubleshooting Steps:
Couple of straight answers for above symptoms will be
 Issue occurs if a virtual machine that is meant for VMware Hosted products such as VMware
Workstation, VMware Player or VMware Fusion is powered-on on an ESX/ESXi host. The
underlying format used to store virtual machines on VMware Hosted products differs from the
format used to store virtual machines on ESX/ESXi hosts.
 User or Distributed Resource Scheduling (DRS) have assigned limited resource to a resource
pool
 Virtual machine’s host does not have enough memory for the reservation required
 Virtual machine’s resource usage does not match its resource settings
 
Detailed steps are:
1. The virtual machine monitor may be asking a question to be answered during startup. A
virtual machine may pause the power-on task at 95% and requires manual intervention to
bring the VM online. Check the Summary tab of VM and select “I copied it “
2. Creating a new power-on task may fail if another task for the virtual machine or other
component is already in progress, and multiple concurrent tasks on the object are not
permitted
3. A virtual machine may fail to power on if licensing requirements are not met and you can see
Error Message “Cannot Power on Virtual Machines, Not enough licenses installed to perform
the operation”. It is simple to fix by adding proper licenses for ESXi & vCenter server
4. The virtual machine may be configured to reserve physical memory on the host, but the host
memory is over-committed and the required memory is unavailable
5. The virtual machine may be starting in a VMware High Availability cluster with strict admission
control enabled, and there are insufficient resources to guarantee failover for all virtual
machines
6. A file required for starting the virtual machine, such as a virtual disk or swap file, may be
unavailable or missing
7. The virtual machine may have been previously suspended and making use of CPU features
which are unavailable or incompatible with the CPU features available on this host. The virtual
machine cannot be started without the required features
8. The virtual machine may require both a VT-capable CPU and the VT feature to be enabled in
the host system’s BIOS. This is true for all 64-bit virtual machines. If the VT feature is
unavailable, the virtual machine may produce the message msg.cpuid.noLongmode
9. The virtual machine may require another CPU feature which is unavailable on this host. The
virtual machine may produce a message similar to msg.cpuid.<FeatureName>, identifying the
specific feature it has been configured to require. Move the virtual machine back to the host
which has the required CPU features, or edit the virtual machine’s configuration to remove the
requirement.
10. The virtual machine may start, but quickly fail with an error during startup. Review the
contents of the log file in the virtual machine’s directory for any errors or warnings, and
search the Knowledge Base for the error or warning. Base your troubleshooting on the
specific messages seen in the logs – this is the best method to fix this problem
11. If the virtual machine does successfully power on, but the guest OS doesn’t start correctly,
there may be an incompatibility between the virtual hardware and drivers within the guest OS.
For example, a missing SCSI driver may be required for booting.
12. If the guest OS, or a driver or application within the virtual machine experiences a problem
during startup, the guest OS may become unresponsive.
 

Question 8:     How do you troubleshoot V-Motion failures for Virtual Machine Scenario?

Hint:   Interviewer wants to understand your skills for common issues in VMware with Virtual
Machines
 
Answer:
How many times you see below messages at your vCenter Server? V-Motion failing at 14% –
10%  – 82% – 90 to 95%
 
You are attempting a vMotion migration between two ESX/ESXi hosts, and the vMotion task
reaches 14%, then times out with this error message:
The vMotion migrations failed because the ESX hosts were not able to connect over the vMotion
network. Check the vMotion network settings and physical network configuration.
VMotion fails at 82% Cannot migrate a virtual machine with vMotion
In the /var/log/vmware/hostd.log file on source ESX host, you see the error:
ResolveCb: Failed with fault: (vmodl.fault.SystemError) {
reason = “Source detected that destination failed to resume.”
msg = “”
}
vMotion fails at 90-95% Cannot perform a vMotion times out In vCenter Server, you see the
error:
Operation timed out vMotion stops at 90% then fails with the error
a general system error occurred: failed to resume on destination
VMware vMotion fails at 10% vMotion times out The VirtualCenter/vCenter Server reports these
errors:
A general system error occurred: Failed waiting for data. Error 16. Invalid argument
A general system error occurred: failed to look up VMotion destination resource pool object
 
Enough of error messages and let us see how to answer this question in the Interview:
Thumb rule for V-Motion failure issues is like if the operation fails below 15% then you can
assume it as Network/configuration issue.
You can tell him below are the settings to be verified for any V-Motion failures:
 Ensure that vMotion is enabled on all ESX/ESXi hosts.
 Determine if resetting the Migrate. Enabled setting on both the source and destination ESX or
ESXi hosts addresses the vMotion failure
 Verify that VMkernel network connectivity exists using vmkping
 Verify that VMkernel networking configuration is valid
 Verify that the virtual machine is not configured to use a device that is not valid on the target
host
 If Jumbo Frames are enabled (MTU of 9000) (9000 -8 bytes (ICMP header) -20 bytes (IP
header) for a total of 8972), ensure that vmkping is run like vmkping -d -s 8972
<destinationIPaddress>. You may experience problems with the trunk between two physical
switches that have been misconfigured to an MTU of 1500
 Verify that Name Resolution is valid on the host
 Verify that Console OS network connectivity exists
 Verify if the ESXi/ESX host can be reconnected or if reconnecting the ESX/ESXi host
resolves the issue
 Verify that the required disk space is available
 Verify that time is synchronized across environment
 Verify that valid limits are set for the virtual machine being vMotioned
 Verify that hostd is not spiking the console
 This issue may be caused by SAN configuration. Specifically, this issue may occur if zoning is
set up differently on different servers in the same cluster
 Verify and ensure that the log.rotateSize parameter in the virtual machine’s configuration file
is not set to a very low value
 If the virtual machine being vMotioned is a 64-bit virtual machine, verify that the VT option is
enabled on both of your ESX hosts.
 Restart the host management agents
 Verify that time is synchronized across your environment
 Verify that valid limits are set for the virtual machine being vMotioned
 Verify that host management agents are not spiking the Service Console (ESX only)
 Verify that there are no issues with the shared storage.
 
To check the health of the vMotion network:
1. Check for IP address conflicts on the vMotion network. Each host in the cluster should have a
vMotion vmknic, assigned a unique IP address
2. Check for packet loss over the vMotion network. Try having the source host ping (vmkping)
the destination host’s vMotion vmknic IP address for the duration of the vMotion.
3. Check that each vMotion VMkernel port group have the same security settings. A security
mismatch causes a vMotion operation to fail. For example, a failure occurs if a source
VMkernel port group is set to allow promiscuous mode and the destination VMkernel port
group is set to disallow promiscuous mode.
4. Check for connectivity between the two hosts. Try having the source host ping (vmkping) the
destination host’s vMotion vmknic IP address
 
Hope this information helps you to crack your Interview and share with others who need this real
time scenarios.
Question 9:     How do you troubleshoot ESXi host PSOD problems?
Most of the Windows Administrators are familiar with Blue Screen of Death and it is time to know
new term PSOD – Purple Screen of Death (VMware seems try to be unique by not using Blue
colour)
 
Interviewer explain this as scenario based question:
I have a ESX 5.5 installed on my HP Proliant DL 380 G9 with a configuration of 8X Intel(R)
Xeon(R) CPU E5540 @ 2.53GHz, 24 GB RAM
Recently the server has crashed four times, showing the Purple Screen of Death. Once this
happens all of the virtual machines on the server stops and crashes until I restart this server.
 

 
 
Answer 9:    You can start with definition of what is PSOD to impress him/her and followed by
Troubleshooting steps
 
“A Purple Screen of Death (PSOD) is a diagnostic screen with white type on a purple background
that is displayed when the VMkernel of an ESX/ESXi host experiences a critical error, becomes
inoperative and terminates any virtual machines that are running”
 You need to highlight important step to capture log file information after the PSOD occurred.
To resolve this issue, extract the log file from a vmkernel-zdump file using a command line utility
on the ESX or ESXi host. This utility differs for different versions of ESX or ESXi.
 For ESXi 3.5, ESXi/ESX 4.x and ESXi 5.x, use the esxcfg-dumppart utility:# esxcfg-dumppart
-L vmkernel-zdump-filename
To extract the log file from a vmkernel-zdump file:
1. Find the vmkernel-zdump file in the /root/ or /var/core/ directory:# ls /root/vmkernel*
/var/core/vmkernel*
/var/core/vmkernel-zdump-073108.09.16.1
2. Use the vmkdump or esxcfg-dumppart utility to extract the log. For example:# vmkdump -l
/var/core/vmkernel-zdump-073108.09.16.1
created file vmkernel-log.1# esxcfg-dumppart -L /var/core/vmkernel-zdump-073108.09.16.1
created file vmkernel-log.1
3. The vmkernel-log.1 file is plain text, though may start with null characters. Focus on the end
of the log, which is similar to:
VMware ESX Server [Releasebuild-98103]
PCPU 1 locked up. Failed to ack TLB invalidate.
frame=0x3a37d98 ip=0x625e94 cr2=0x0 cr3=0x40c66000 cr4=0x16c
es=0xffffffff ds=0xffffffff fs=0xffffffff gs=0xffffffff
eax=0xffffffff ebx=0xffffffff ecx=0xffffffff edx=0xffffffff
Note: The file name created for the log in this example is vmkernel-log.1. If another file with the
same name already exists, the new file is created with the number suffix incremented.
 
Most of the times it will be hardware issue and you need to open a case with Hardware vendors,
in this case it is HP. Based on findings you need to replace the Hardware devices or upgrade the
firmware as suggested by Hardware vendors via ITIL Change Management process.
In some cases it may be problem with software installed on ESXi server like additional agents for
monitoring both software & hardware, additional VIBs added for Storage … etc
 
Finally if you want to be expert to analyze the logs on your own, then here is the good KB Article
from VMware. It’s rare that Interviewer asking about debugging this issue but he wants to check
your understanding about procedure followed in case of PSOD.

Question 10:   How do you troubleshoot P2V Failure Issues in your Infrastructure? (P2V =
Physical to Virtual)
Answer:      There is lot of discussion about which Physical server is good candidate for VMware
Infrastructure like Exchange, SQL or Cluster … etc
Interviewer also show interest to hear from you that, how you judge which Physical server is
good candidate for Virtualization
Answer for this point is first we need to analyze 3 months of data from any Performance reporting
tools, If you notice server utilization is 80% of CPU & Memory then most likely that Physical
server not much suitable for VMware Infrastructure.
 
If the Server utilization is less than 70% then you can recommend it for VMware Infrastructure.
Once the server is selected for P2V and you started the process (hope you have Pre & Post P2V
checklists) and ran into some issue. Here you can find good check list to fix P2V problems.
 
 To eliminate permission issues, always use the local administrator account instead of a
domain account.
Note: Disable UAC for Windows Vista, Windows 7, or Windows 8 prior to converting.
 To eliminate DNS problems, use IP addresses instead of host names.
 Ensure that you do not choose partitions that contain any vendor specific Diagnostic
Partitions before proceeding with a conversion.
 To reduce network obstructions, convert directly to an ESX host instead of vCenter Server as
the destination.
Notes:     This is only an option in VMware vCenter Converter Standalone
If you are unable to convert directly to an ESX host in vCenter Server 5.0, see   vCenter
Converter Standalone 5.0 errors when an ESXi 5.0 host is selected as a              destination.
Check KB2012310
 VMware vCenter Converter Standalone has many more options available to customize your
conversion. If you are having issues using the Converter Plug-in inside vCenter Server,
consider trying the Standalone version.
 If a conversion fails using the exact size of hard disks, decrease the size of the disks by at
least 1MB. This forces VMware Converter to do a file level copy instead of a block level copy,
which can be more successful if there are errors with the volume or if there are file-locking
issues.
 Make sure there is at least 500MB of free space on the machine being converted. VMware
Converter requires this space to copy data.
 Shut down any unnecessary services, such as SQL, antivirus programs, and firewalls. These
services can cause issues during conversion.
 Run a check disk on the volume before running a conversion as errors on disk volumes can
cause VMware Converter to fail.
 Do not install VMware Tools during the conversion. Install VMware Tools after you confirm
that the conversion was successful.
 Do not customize the new virtual machine before conversion.
 Ensure that these services are enabled:
Workstation Service
Server Service
TCP/IP NetBIOS Helper Service
Volume Shadow Copy Service
 Check that the appropriate firewall ports are opened
 Check that boot.ini is not looking for a Diagnostic/Utility Partition that no longer exists.
 If you are unable to see some or all of the data disks on the source system, ensure that you
are not using GPT on the disk partitions.
 In Windows XP, disable Windows Simple File Sharing. This service has been known to cause
issues during conversion.
 Unplug any USB, serial/parallel port devices from the source system. VMware Converter may
interpret these as additional devices, such as external hard drives which may cause the
conversion to fail.
 If the source machine contains multiple drives or partitions and you are having issues failing
on certain drives, consider converting one drive or partition at a time.
 Verify that there are no host NICs or network devices in the environment that have been
statically configured to be at a different speed or duplex. This includes settings on the source
operating system, switches and networking devices between the source and destination
server. If this is the case, Converter sees the C: drive but not the D: drive.
 If you are using a security firewall or Stateful Packet Inspecting (SPI) firewall, check firewall
alerts and logs to make sure the connection is not being blocked as malicious traffic.
 If you have static IP addresses assigned, assign the interfaces DHCP addresses prior to
conversion.
 If the source server contains a hard drive or partition larger than 256GB, ensure that the
destination datastores block size is 2MB, 4MB, or 8MB, and not the default 1MB size. The
1MB default block size cannot accommodate a file larger than 256GB.
 Clear any third-party software from the physical machine that could be using the Volume
Shadow Copy Service (VSS). VMware Converter relies on VSS, and other programs can
cause contention.
 Disable mirrored or striped volumes. Mirrored or striped volumes cannot be converted
 Verify that the VMware Converter agent is installed on the source machine. It may not be if
the conversion fails right away.
 Verify that DNS and reverse DNS lookups are working. It may be necessary to make entries
into the local hosts file on source machine. Use IP addresses, if possible.
 Run msconfig on the source server to reduce the number of services and applications running
at startup. Only Microsoft services and the VMware Converter Service should be running.
 Inject VMware SCSI drivers into the machine before conversion. Windows tries to Plug-n-Play
the new SCSI Controller, and Windows may fail if the proper drivers are not installed.
 If you customized permissions in your environment, ensure that local administrator has rights
to all files, directories, or registry permissions before conversion.
 Uninstall any UPS software. This has been known to cause issues after Conversion.
 Ensure that you do not have any virtual mounted media through an ILO- or DRAC-type
connection. Converter can misinterpret these as convertible drives, and fails upon detecting
them. As a precaution, disconnect your ILO or DRAC to prevent this issue.
 
Your answer should also cover Logs information which will prove your real time experience
VMware Converter logs
There are also several ways to diagnose issues by viewing the VMware Converter logs. The logs
can contain information that is not apparent from error messages. In newer versions of VMware
Converter, you can use the Export Log Data button. Otherwise, logs are typically stored in these
directories:
Windows NT, 2000, XP, and 2003:
C:\Documents and Settings\All Users\Application Data\VMware\VMware Converter Enterprise\
Logs
C:\WINDOWS\Temp\vmware-converter
C:\WINDOWS\Temp\vmware-temp
Windows Vista, 7, and 2008:
C:\Users\All Users\Application Data\VMware\VMware Converter Enterprise\Logs
Windows 8 and Windows 2012:
C:\ProgramData\VMware\VMware vCenter Converter Standalone\logs
Note: In order to access this location in Windows Vista, 7, or 2008, you may need to go into the
folder options and ensure that Show Hidden Files is enabled and that Hide Protected Operating
System Files is disabled.
C:\WINDOWS\Temp\vmware-converter
C:\WINDOWS\Temp\vmware-temp
Windows NT and 2000:
C:\WINNT\Temp\vmware-converter
C:\WINNT\Temp\vmware-temp
Linux:
$HOME/.vmware/VMware vCenter Converter Standalone/Logs
/var/log/vmware-vcenter-converter-standalone
Question 11:   VMware VSAN is popular now a days and do you have any experience in that
one? Can you brief pre-requisites of VMware VSAN? Have you ever faced any issues with VSAN
setup? If yes how you recovered it?
Most of us not blessed to Install & Configure VMware VSAN in Customer Infrastructure but soon
you will be assigned to that role as the Storage slowly moving to VSAN world. Virtual SAN 6.0
comes with a Health Services plugin. This feature checks a range of different health aspects of
Virtual SAN, and provides insight into the root cause of many potential Virtual SAN issues. The
recommendation when triaging Virtual SAN is to begin with the Virtual SAN Health Services.
Once an issue is detected, the Health Services highlights the problem and directs administrators
to the appropriate VMware knowledge base article to begin problem solving.
 
Please refer to the Virtual SAN Health Services Guide for further details on how to get the
Health Services components, how to install them and how to use the feature for troubleshooting
common Virtual SAN issues. It is better to know the pre-requisites before entering into
Troubleshooting mode. Let’s review couple of pre-requisites for VMware Virtual SAN (List is
really big and try to remember only important ones)
Once you know the pre-requisites and next step is collecting Virtual SAN (VSAN) support
logs. Virtual SAN support logs are contained in a normal ESXi support bundle in the form of
VSAN traces. The VSAN support logs are gathered automatically by gathering the ESXi support
bundle for the hosts. As VSAN is distributed across multiple ESXi hosts, VMware recommend
you gather the ESXi support logs for all hosts configured for VSAN. As Virtual SAN is a software-
based storage product, it is entirely dependent on the proper functioning of its underlying
hardware components such as the network, the storage I/O controller, and the storage devices
themselves. As Virtual SAN is an enterprise storage product, it can put an unusually demanding
load on supporting components and subsystems, exposing flaws and gaps that might not be
seen with simplistic testing or other, less-demanding use cases. Indeed, most Virtual SAN
troubleshooting exercises involve determining whether or not the network is functioning properly,
or whether the Virtual SAN VMware Compatibility Guide (VCG) has been rigorously followed.
 
As Virtual SAN uses the network to communicate between nodes, a properly configured and fully
functioning network is essential to operations. Many Virtual SAN errors can be traced back to
things like improperly configured multicast, mismatched MTU sizes and the like. More than
simple TCP/IP connectivity is required for Virtual SAN. Virtual SAN uses server-based storage
components to recreate the functions normally found in enterprise storage arrays. This
architectural approach demands a rigorous discipline in sourcing and maintaining the correct
storage I/O controllers, disks, flash devices, device drivers and firmware, as documented in the
VMware Virtual SAN VCG.
 

Tools to troubleshoot Virtual SAN

1)            vSphere Web Client
2)            ESXCLI (esxcli vsan)
3)            Ruby vSphere Console – RVC
4)            VSAN Observer
 

Recovering a VMware Virtual SAN cluster after a

vCenter Server rebuild

Finally When you attempt to configure or deploy VMware Virtual SAN (VSAN), you experience
these symptoms:
 VSAN configuration fails
 In the vSphere Web Client, you see an error similar to:
Found another host participating in the VSAN service which is not a
member of this host’s vCenter cluster
 
Resolution
To resolve this issue, check all ESXi host nodes in the VSAN cluster and ensure that their
configuration is correct and all VSAN hosts are in the same network subnet.

Question 12:   Have you ever fixed metadata issues with ESXi data stores? Can you list
commands or Tools used for such activity? Is there any pre-requisite checklist before you use
such tools? Help me to understand the troubleshooting procedure for this scenario.
Answer:     Once Interviewer got confidence at your basics of VMware, he/she will start asking
questions in advanced troubleshooting or topics. You need to answer such questions wisely by
adding Customer scenario along with answer. Let me help you to answer it properly with this
post. You can start the answer like recently one of Customer had SAN outage which resulted
couple of data stores not coming online which resulted Major Incident/Priority One ticket. I
worked with VMware support team and used VOMA (vSphere On-disk Metadata Analyzer) tool to
check VMFS metadata consistency.This utility scans the VMFS volume metadata and highlights
any inconsistencies to which you need to work with SAN team to fix it faster in real time
scenarios.
 
Since ESXi 5.x it is possible to check VMFS for metadata inconsistency with a tool called VOMA
(VMware Ondisk Metadata Analyser). With VOMA you can check VMFS3 and VMFS5
datastores. Please note, that the tool can only identify problems, as it runs in a read-only mode.
So it does not help you to fix detected errors.
 
Reasons to use VOMA:
 occurrence of metadata errors in the vmkernel log
 If you experience SAN outage
 After Rebuilt RAID
 Disk replacement
 Partition table update
 Reports of metadata errors in the vmkernel.log file
 Unable to access files on the VMFS volume that are not in use by any other host
 if you cannot modify, erase or access files on a VMFS datastore, that is not in use by another
host
 
Before you start VOMA from the CLI of your ESXi host, take care of the following guidelines:
 Shut down all virtual machines running on the VMFS datastore (or migrate them)
 make sure that the VMFS volume is not in use by other hosts (best practice: unmount the
datastore on the other hosts)
 make sure that the datastore is not in use by vSphere HA for heartbeating
 make sure that the datastore is not in use by other features like Storage I/O control,…
 make sure that the volume is not a multi-extent volume
 
Now log on to your ESXi host and let’s take a look at the available parameters of VOMA (voma -
h)
 
Procedure:
To perform a VOMA check on a VMFS datastore and send the results to a specific log file, the
command syntax is:
voma -m vmfs -d /vmfs/devices/disks/naa.00000000000000000000000000:1 -s /tmp/analysis.txt
where naa.00000000000000000000000000:1 is replaced with the LUN NAA ID and partition to
be checked. Note the “:1” at the end.
This is the partition number containing the datastore and must be specified. See note below. As
an advisory, if you run voma more than once, add the NAA ID and a time stamp to the output log
file name. EG: -s /tmp/naa.00000000000000000000000000:1_analysis_<<hhmm>>.txt
Note: VOMA must be run against the partition and not the device. If VOMA is run against a
device, it produces an error similar to:
Error: Missing LVM Magic. Disk doesn’t have a valid LVM Device
Error: Failed to Initialize LVM Metadata

Question 13:     As part of Data Center Network devices upgrade/change – someone changed
vCenter IP Address. How do you tackle this Scenario as a VMware Administrator? What is the
Technical plan that you will follow for this Change Record? (ITIL Process)
 
Answer:
Hint:   Interviewer looking at your Technical direction/plan along with ITIL Chanage management
procedures
 
We may think that changing IP Address is easy job like going to vCenter VM Console (most of
the cases) [OR]  Remote console for Physical servers and modify the Network Adapter Settings.
But what happens to your ESXi servers, NSX VM’s and Update Manager?? will they
communicate to your vCenter server with new IP directly without any modification? Here is the
detailed Technical Plan to answer this question.
 
 Create backups of the vCenter Server & underlying SQL database for Backup Plan
 Set DRS to manual mode to avoid anything moving around
 Identify the ESXi host running the vCenter VM and connected directly to the host with the
vSphere Client – Do not forget your vCenter going to disconnect and you can’t manage it
anymore via vSphere client
 Close any sessions you have open to the vCenter Server (Web Client, vSphere Client
sessions
 Open a console window to the vCenter Server by way of the ESXi host.
 Stop all VMware related services
 Change the IPv4 address and IPv4 gateway as per new Networking configuration
 Put DRS back to fully automated (optional based on your setup)
 Uninstall Update Manager software from the VM (Some times it’s installed other than
vCenter)
 Install Update Manager and point it new vCenter Server IP Address
          NOTE:  There is easy method to update vCenter IP Address at Update Manager via
command line (we will discuss it in future posts)
 Update the vCenter Managed IP Address with below procedure

 NSX requires your attention as vCenter re-registration is complex procedure – leave this for
Network Specialists to provide technical plan
 Disconnect host from vCenter to flush out the database entry
 Reconnect to the ESXi host to use new vCenter IP Address for communication and agents
Installation
 
Finally I tried to bring most of the related items for vCenter IP Address change from my
Experience and knowledge but do not treat this as final Technical Plan. You need to refer your
Infrastructure for better planning Change Records (CR’s) as per ITIL Procedure.

Question 14:   Have you ever fixed vSphere ESX Agent Manager (EAM) failure issues with NSX
Solution? What do you think about below error message?
In the /var/log/netcpa.log file on the ESXi host, you see entries similar to:
Netcpa error: error netcpa[FFCFFB70] [Originator@6876 sub=Default]
Failed to get host id, returned , len 0
Answer:    NSX became core Network solution for most of Cloud Companies. IBM Soft Layer –
VMware became Strategic Partners to deliver VMware solutions for Soft Layer customers. You
can find more details here. Interviewer wants to check your Host preparation skills for NSX usage
in this question. Host preparation involve couple of steps including installing VIB modules for esx-
dvfilter-switch-security, esx-vsip & esx-vxlan.
 
Host preparation is the process in which the NSX Manager:
1) Installs NSX kernel modules on ESXi hosts that are members of vCenter clusters
2) Builds the NSX control-plane and management-plane fabric
NSX kernel modules packaged in VIB files run within the hypervisor kernel and provide services
such as distributed routing, distributed firewall, and VXLAN bridging capabilities. To prepare your
environment for network virtualization, you must install network infrastructure components on a
per-cluster level for each vCenter server where needed. This deploys the required software on all
hosts in the cluster. When a new host is added to this cluster, the required software is
automatically installed on the newly added host. vSphere ESX Agent Manager automates the
process of deploying and managing vSphere ESX agents. Once you are clear with basics then
you can explain the possible reasons to troubleshoot VIB’s deployment failure using VMware
ESX Agent Manager (EAM), Installation Status showing Not Ready on the Clusters and Hosts.
 
 
If you can speak or explain 4 or more steps to the Interviewer, then he/she will be happy to

consider you for the applied position of NSX Engineer 

 
1. Ensure that the DNS is configured correctly on the ESXi hosts and the environment. The
VMware ESX Agent Manager (EAM) may fail to deploy all required VIBs during installation of
network virtualization components on vSphere hosts during Host Preparation in NSX for
vSphere 6.x
2. Verify that all required ports are open between ESXi hosts, vCenter Server and NSX
Manager, and is not being blocked by a Firewall.
3.  If you are using VMware Update Manager, ensure that VUM is available and running.
4. Verify that the URL http://VC_IP/eam/mob/ is accessible
5. Ensure that the EAM is able to access the VIBs from NSX Manager. Review the eam.log file
and look for VIB inaccessible messages. Also check the vCenter Server registration in NSX
Manager.
6. Ensure that the ESXi hosts are able to access VIBs from the vCenter Server. You see this
error: esxupdate: ERROR: MetadataDownloadError:IOError: <urlopen error [Err no -2] Name=
or service not known in the esxupdate.log file. This can occur if the ESXi host is unable to
access the vCenter Server’s Fully Qualified Domain Name (FQDN).
7. Verify that the cluster contains the correct agencies. If an EAM agency is missing on an
already prepared cluster, new hosts can fail to prepare. To work around this issue, create a
dummy cluster and prepare it. This forces NSX/EAM to update the configuration of all existing
clusters, creating a new EAM Agency for the problematic cluster.
8. Ensure that the ESXi hosts are rebooted after an upgrade. Failure to manually reboot the
ESXi host marks them as red in the Installation Status. Note: In a DRS enabled cluster, an
ESXi host reboot can be initiated by clicking the Resolve option on the cluster.
9. If you experience stale agencies with the error: Netcpa error: error netcpa[FFCFFB70]:
1. Create a new cluster.
2. Add the new cluster to the transport zone.
3. Move the affected ESXi host to the new cluster.
4. Re-attempt to install the VIBs on the affected ESXi host.
10. If you see errorCode =99 on the eam.log file, this means that EAM incorrectly mark Host
Preparation as successful. To resolve this issue, manually reboot the affected ESXi host.
11. If you see Failed to complete VIB task on the eam.log file, this means that the vSphere
Update Manager (VUM) are having issues installing NSX VIBs. To resolve this issue, restart
the vSphere Update Manager service.
12. If you see the error esxupdate: ERROR: MetadataDownloadError:IOError: <urlopen error
[Errno -2] Name= or service not known) in the esxupdate.log file, this may happen if the ESXi
host cannot reach the vCenter Server Fully Qualified Domain Name (FQDN). To resolve this
issue, set the correct the vCenter Server Managed IP address.
13. Sometimes after restarting NSX Manager, EAM or vCenter Server, you notice the cluster
status reported Not Ready in Host Preparation tab, in Installation section of NSX Manager.
This is a false-positive status that is a result of restarting one of the components. To get the
state updated, click the Resolve All button.

Question 15:    Can you explain me any major issue that you fixed in VMware Infrastructure?
How you handle that situation? Can you list the steps that you followed to fix that problem?
Answer:   It’s long back that I wrote the post for real time scenarios and today going to help you
guys with one of common Interview question for this “Thanks Giving Day”. It’s always good to
pick Storage related issues which creates high impact for both Physical & VMware
Infrastructures. I’m going to take the scenario of couple of VMs’ are not responding including
vCenter Server which is also running as Virtual Machine.

 
You can start the narration like – you got a call from Helpdesk about some VM’s are inaccessible
in the vCenter server. When you connected to the vCenter server, you can’t access the console
of VM’s as they stuck at black screen. From the screenshot it’s clear that, CPU & Memory usage
is high for couple of hosts in the Cluster. You want to understand the reason for CPU & Memory
utilization on the ESXi hosts and ran the “esxtop” command to know which VM/process creating
more CPU/Memory utilization. If you are new to esxtop, then follow this article to know the usage
of this command. As the issue is related to couple of virtual machine’s only then you downloaded
vmware.log file to understand more about this issue from VM’s point of view. You notice that
there is some latency to download the log files. While this issue is running, vCenter VM went to
unresponsive state which makes the situation more worse. From the attempt of esxtop &
vwware.log there is not much information to extract and management wants you to fix the
vCenter VM issue first. Also SQL DB server also running as Virtual Machine which is holding
vCenter & Update Manager DB’s. Also you installed PernixData (acquired by Nutanix) to improve
storage I/O Performance for VM’s. These factors make the situation complex and below diagram
helps you to understand the scenario in better way.
Now let’s talk about resolution step by step and how you found the issue with Storage. Help desk
reported that another set of VM’s are not reachable in the network and they are hung at black
screen when they opened the VM console from making direct connection to ESXi server. It
means issue is really growing and impacting more business functions like SharePoint, Exchange,
Citrix, Finance Applications … etc. Management declared this as MI (Major Incident) and
requested you to join the bridge call along with other Technical teams like Storage, Network &
Incident escalation managers. This is going to be collaboration effort rather than you fix this issue
alone from VMware team. As there was latency to download the file and black screen for the VM
console brings your attention to Storage related problems. esxtop has specific switches to
understand storage related problems and check this article to know more details. For this
escalation you need to know KAVG & DAVG values and thresholds as listed below.
 
 
So from your troubleshooting it’s clear that there is ongoing Storage related problem in the
Infrastructure which shows in esxtop and VM consoles are with black screen. You requested
Storage team to validate the VNX configuration and performance to make sure they are also
seeing any alerts or issues from their side. They came back quickly with an update that ALL stats
are looking good for read & write operations. Management asking what is the problem and you
are confident that there is some Storage issue but they can’t see in VNX side. So you requested
the Storage team to give some recommendations and they comeback with a plan that, they are
going to try to switch the Storage Processor from A to B. They executed this plan with
management approval but there is no change in the situation. Later they want to try to reboot
storage processors one by one but it has major impact if they can’t come online after reboot.
Management agreed as ALL the production VM’s are down at that time. Storage team finished
rebooting both processors successfully in 1 hour but that didn’t change anything in the VM’s
status. Which means Storage team performed all the Troubleshooting including rebooting the
SPA & SPB. Now management wants to know next action plan from VMware Infrastructure.
You are confident that there is Storage problem but VNX looks good then there should be a
problem with another Storage layer which is Pernixdata. Upon investigating more from
Pernixdata process – you found that there are lot of pending I/O jobs waiting at one of ESXi host
server. You killed the pernix process on that specific host which brought the services online to
normal state. One the VM’s became normal, you recommended the Management to perform
clean ESXi reboot to avoid any immediate potential outage to the Production. They agreed for it
and clean reboot performed for each ESXi host and conference call got closed after validating
the Applications from the end users. (UAT – User Acceptance Test)
 

 
Resolution:   Pernixdata helps to improve the Storage I/O performance by offloading that
service to ESXi host local cache, however in this situation it’s holding lot of jobs which created
above situation. By killing pernix process from ESXi host console and clean ESXi reboot brought
the situation to normal state. Hope this helps you to answer Interview questions in better way and
be social to share the knowledge.

Question 16: Can you explain the options that you see when you format new Data store in
vSphere 6.5 via Web client?
Answer: vSphere 6.5 released some time ago and Interview panel members now started asking
for questions related to 6.5 like 16differences and what’s new in 6.5. Today let me help you to
understand the Storage part. In real-time, there are couple of scenarios like new VM’s build
project, migrating to new Storage Array, Increasing current VM disk file size requires additional
storage in VMware Cluster. VMware administrator is responsible to make sure that he/she
understand the over-commit challenges with Data store. For example if we have 2 TB of Data
store and you already allocated 1.8 TB of space, which means you consider the thin disk vs thick
disk precaution before assigning the leftover 200 GB storage space. To keep it simple – thumb
rule for VMware administrator is not to allocate more space than available in the Data store for
Production Infrastructure.
 
Also you need to understand the performance challenges with THIN on THIN vs THICK on
THICK options selected by Storage and VMware Administrators. Do not forget Storage
administrator also can create THIN disks to allocate the storage for VMware Cluster. Let’s talk
more options available while you format the Data store at VMware cluster, after Storage
administrator confirmed that LUN allocation to all ESXi servers. First we will see the standard
steps performed by VMware Administrator for VMFS type LUN which has options to format with
version 5 or 6 followed by NFS & VVOL based storage. You can find the differences between
these version in my previous post “VMFS 5 vs VMFS 6“. Previous versions won’t show VVol
option and Interviewer expectation is that you will highlight that point and describe it’s
functionality. VMFS 6 enables advanced format (512e) and automatic space reclamation support.
You can mention the options – Space Reclamation Priority and Granularity which are not
available in previous versions. You can also mention NFS versions 3 & 4.1 which makes your
answer in real time format. Interviewer is keen about new options available or enabled with
vSphere 6.5 and make sure you mention all these in your answer to make better impression.
 

VMFS LUN
 
 NFS LUN

 
 VVOL Based Storage
If you are new to VVOL term, here is the quick explanation about it.  Virtual Volumes (VVols) is a
new integration and management framework that virtualizes SAN/NAS arrays, enabling a more
efficient operational model that is optimized for virtualized environments and centered on the
application instead of the infrastructure. Virtual Volumes simplifies operations through policy-
driven automation that enables more agile storage consumption for virtual machines and
dynamic adjustments in real time, when they are needed. It simplifies the delivery of storage
service levels to individual applications by providing finer control of hardware resources and
native array-based data services that can be instantiated with virtual machine granularity. With
Virtual Volumes (VVols), VMware offers a new paradigm in which an individual virtual machine
and its disks, rather than a LUN, becomes a unit of storage management for a storage
system.Virtual volumes encapsulate virtual disks and other virtual machine files, and natively
store the files on the storage system.
 

Question 17:  Can you draw on the whiteboard/paper for high level architecture design that
details a cluster of 4 ESXi hosts, connected to a vCenter Server that has 150+ VM’s balanced
across the hosts.There are 2 network switches vSS-1 & vDS-2 and ESXi servers are
connected to for VM & management traffic via VLAN’s .The hosts are connected to four 500
GB shared storage LUN’s that are presented via fiber channel.
Answer:  If you answer theory questions correctly then Interviewer is going to check your
understanding of vSphere Architecture skills. This is tough question as it involves lot of
factors to explain specific answer. However I will try to cover my answer with some
assumptions to keep it simpler. When I cleared my Data center design exam, there are lot of
these examples, we prepared in the Google+ community and shared with other members.
There are two phases for any design like Logical & Physical diagrams, where logical talks
about high level design but won’t cover any specific product details. Physical diagram will
cover most of product specific details like Cisco UCS, storage and software versions.
 
What Is a Technical Design?
A technical design is a way of communicating an end product or solution. By creating a
technical design, a group of people can work together to create a final solution. A design
methodology is a multi phase process used to direct a technical design process which
involves below steps.
 Iterative
 Involves other people
 Helpful and necessary to the success of a project
 

The basic components of a good design are:

Vision: The vision component represents the actual idea. The vision must, therefore, be kept
at the forefront throughout the project.
Scope: The scope is a quantitative statement of what is included in a project, or, more
pertinently, what is not included in a project.
Requirements:  A requirement is an attribute that must be achieved; Requirements affect
design choice substantially.
Constraints: A constraint is an attribute that may limit a design choice, and it could be
technical or business driven
Assumptions:  An assumption is something that has been decided to be true for the project
design but that has not been tested nor verified.
Risks: A risk is an attribute that could prevent the completion of a project or change the
project design considerably.
 
Enough of theory classes and it’s time to represent given requirements into the whiteboard for
Interviewer:
 
 
 
NOTE:  Each interviewer has their own thoughts for design and he/she may ask you some
questions about why you selected this component vs other component [or] product. You need
to justify design decisions from the Infrastructure qualities.
 ===============================================================
Question 18:   We are looking for VMware Administrator with VSAN experience. Have you ever
configured VSAN cluster? What kind of challenges you ran into while VSAN deployment? What
tools you used to troubleshoot those issues? Help me to understand your VSAN understanding
and implementation experience.
Answer:   Most of the VMware Administrators answer this question in theory part of VSAN
fundamentals rather than explaining real time challenges with deployment. Interviewer looking for
some real time issues as part of your answer which helps me to trust your skills in VSAN
implementation. Today let’s see how to answer such tough questions by adding production issue
of disk failed to detect when you tried to configure VSAN at cluster level. There are many blogs
cover the steps required to configure/enable VSAN and here is the VMware feature walk through
for VSAN beginners.
 
We are going to use the tool “Ruby vSphere Console” to troubleshoot the local hard disks which
are not visible during the VSAN configuration but they are available at ESXi for regular disk
operations. The Ruby vSphere Console is a console user interface for VMware ESXi and Virtual
Center.The Ruby vSphere Console comes bundled with both the vCenter Server Appliance
(VCSA) and the Windows version of vCenter Server.  Most importantly, RVC is one of the
primary tools for managing and troubleshooting a Virtual SAN environment. You can find the
white paper published by VMware for Ruby vSphere console command reference for Virtual
SAN.
 
Production issue:   Customer requested you to enable vSAN at Cluster where Flash/HDD disks
are populated. When you start the VSAN wizard, you can’t see local disks connected to couple of
ESXi servers
 
Connect to RVC with below procedures based on your vCenter version:
 

 
Run the command:  vsan.disks_info ~/computers/Clustername/hostname
/10.30.40.61/CICD-SDDC/computers> vsan.disks_info ~/computers/vDC-NSX/hosts/10.30.40.51
2017-05-16 14:52:55 -0700: Gathering disk information for host 10.30.40.51
2017-05-16 14:53:08 -0700: Done gathering disk information
Disks on host 10.30.40.51:
+——————————————————–+——-+———+
—————————————————————————————-+
| DisplayName                                            | isSSD | Size    |
State                                                                                  |
+——————————————————–+——-+———+
—————————————————————————————-+
| Local ATA Disk (naa.55cd2e404c09019c)                  | SSD   | 223 GB  |
eligible                                                                               |
| ATA INTEL SSDSC2BB24                                   |       |         |          
|
+——————————————————–+——-+———+
—————————————————————————————-+
| Local ATA Disk (naa.55cd2e404c090c3b)                  | SSD   | 223 GB  |
ineligible (Existing partitions found on disk ‘naa.55cd2e404c090c3b’.)                 |
| ATA INTEL SSDSC2BB24                                   |       |         |                        
|
|                                                        |       |         | Partition
table:                                                                       |
|                                                        |       |         | 1: 223.57 GB, type = vmfs
(’51_SSD-Datastore’)                                         |
+——————————————————–+——-+———+
—————————————————————————————-+
 From the output you can clearly see that there is ineligible disk as  there is existing partition
found on that disk. Which means this disk has previous file system and failed to detect by VSAN
as it will look for disks without any file system/partition. Now it’s time to delete old partition on the
disk and make it usable for VSAN operations. When you try to de-commission the VSAN cluster,
you may get same challenge as file system/ disk partitions are not deleted during the VSAN
disable process from vCenter server. Now you have proper real time/ production scenario to
explain it to the Interviewer. Start your answer with the number of ESXi servers in the cluster
going to be part of VSAN operations, number of disks used and FTT values.
 

 
Summary:    When you answer this question, you will explain about VSAN Infrastructure
planning/sizing decisions taken by Architect. You are responsible to implement the solution and
come across the ineligible disks issue during the deployment. RVC tool is used to find the
problem and later partedUtil is used to delete the existing partition from the disk. Finally local
disks are visible for VSAN enablement and VSAN Datastore is created successfully.
Question 19:   Have you installed vCenter 6.5 in HA mode? ( new feature) How do you
troubleshoot if there is any issue with vCenter availability? What happens if the failover process
is not executed in HA? Can you access the vCenter server with regular IP address?
Answer:   It’s long time that I posted scenario based question. You may not get this much
detailed question from the Interviewer but I tried to cover some aspects of question can be
asked. Having said that, VMware announced this vCenter HA feature from vSphere 6.5 with

there vCenter server standard license. So no additional cost involved for this feature   &
thanks for not making licensing model more complex topic. This feature is exclusively available
for the vCenter Server Appliance (VCSA). When vCenter HA is enabled, a three-node vCenter
Server cluster (Active, Passive, and Witness nodes) is deployed. vCenter HA provides an RTO of
about 5 minutes for vCenter Server greatly reducing the impact of host, hardware, and
application failures with automatic failover between the Active and Passive nodes.
 
vCenter HA can also be enabled, disabled, or destroyed at any time allowing customers to easily
take advantage of this new capability. There is also a maintenance mode that prevents planned
maintenance from causing an unwanted failover. It’s lot of theory and here is the famous vCenter
HA diagram. Do not get confused with cluster HA feature which is there from long time. It’s
always tricky with HA feature as it creates additional machines. Let’s focus about our explanation
towards vCenter HA crashed.
 

 
You can begin the story with the production scenario – our vCenter server is not responding even
it’s configured in HA mode. Upon investigation we notice that vCenter is not in network and can’t
manage it remotely. When the connection made to ESXi server directly to see what’s running in
vCenter console it’s up and running but not reachable in the network. When we ran ifconfig
command at the console – there is only private IP but management IP is not visible. It sounds
like vCenter is still able to make heartbeat connections with it’s witness node and there is no
vCenter failover happened. So you have production outage as vCenter is not available and any
software depends on vCenter functionality is going to be affected like backups,disaster recovery
… etc. It’s good explanation about the problem and let’s cover the recovery steps required for
this problem.
 
If all nodes in a vCenter HA cluster cannot communicate with each other, the Active node stops
serving client requests.
Problem
Node isolation is a network connectivity problem.
Procedure
1. Attempt to resolve the connectivity problem. If you can restore connectivity, isolated nodes
rejoin the cluster automatically and the Active node starts serving client requests.
2. If you cannot resolve the connectivity problem, you have to log in to Active node’s console
directly.
             Power off and delete the Passive node and the Witness node virtual machines.
             Log in to the Active node by using SSH or through the Virtual Machine Console.
             To enable the Bash shell, enter shell at the appliancesh prompt.
             Run the following command to remove the vCenter HA configuration.
                       destroy-vcha -f
              Reboot the Active node. The Active node is now a standalone vCenter Server
Appliance.
              Perform vCenter HA cluster configuration again.
 
These steps helps you to bring your vCenter back online into the management network and tip

here is you need to use VMRC to correct network card status to connected   this is not listed
in VMware official document right. I found that vCenter VM has two network adapters but only
heartbeat network card seems connected but management is not. I tried to run some linux based
command to bring it online but trick is it’s in edit settings. I need to look for same setting via
HTML client for ESXi but for now you can use VMRC console as your answer. That’s it for this
post and will publish more troubleshooting scenarios to share my knowledge.

Question 20:   Do you know the purpose & use case of vRA? Have you ever troubleshooted VM
provision failure issues in vRA? Let’s assume there are two flavors of Desktop Win 7 & Win 10
with couple of software’s like Chrome, Mozilla, IE & Java. Can you help me to explain the
troubleshooting steps which you follow for the VM provision failure problem with given
software’s?
Answer:  As the focus shifted from VMware operations to automation, most of the Interviewers
asking questions related to vRealize Automation with the primary use case of automated
provision failures when requested from the user portal. Let’s recap how to define this vRealize in
VMware words – vRealize Automation enables IT Automation through the creation and
management of personalized infrastructure, application and custom IT services (XaaS). This IT
Automation lets you deploy IT services rapidly across a multi-vendor, multi-cloud infrastructure.
 
Let me introduce some known errors from the vRA console:
Let’s start the explanation with the scenario like – You got the call from DevOps team as they
failed to provision the VM’s from the self service portal. Assumption for this case is – vRA
deployed in minimal setup with 3 servers, Tenants, Blueprints, catalogs and necessary
configurations are pre-configured. You use blueprints to define machine deployment seĴingsǯ
Published blueprints become catalog items, and are the means by which entitled users provision
machine deployments. vRealize Automation provides a secure portal where authorized
administrators, developers, or business users can request new IT services and manage specific
cloud and IT resources, while ensuring compliance with business policies. Requests for IT
services, including infrastructure, applications, desktops, and many others, are processed
through a common service catalog to provide a consistent user experience.
 

 
What next – you want to understand more about the problem, so first check with catalog item has
a problem and capture the details of vCenter server & Template name associated with it. You
can connect to the vCenter server and validate for any error messages for VM clone and guest
customization errors (if you are using them from vCenter server). From the vRA portal, you can
request the VM and monitor the provision process to troubleshoot further. Make sure there are
no communication problems within vRA components – SQL Server Database, DEM (Distributed
Execution Manager), IaaS & Web server components. Once it’s confirmed that vRA components
are functional, then you need to review the logs on Guest VM deployed. You need to have local
administrator or darwin user account to perform further troubleshooting steps for this scenario.

We need to know the logs location right 

 

 
You need to start with GuestAgent.log file to see if there any connection failures. We speak a lot
about the overall process but please note vRA product has many areas to look at before you
conclude the problem area. So let me conclude the answer for the interview as – first error
message occurred as the new ESXi host joined recently missing particular VLAN 150, which is
required for VM to communicate with IaaS server. To fix the issue we need to work with Network
team to trunk/enable VLAN 150 for this ESXi server Physical NIC’s. Second error message
which I mentioned earlier caused by darwin account has an expired password as the Company
password policy requires all the user account passwords require to change the password in 90
days or get an exception fpr special services accounts. In this case darwin is not treated as
service account as this one is local to the specific VM provisioned. If you can explain 50% of the
content which I explained in this blog post, your Interviewer can be convinced that you have
some knowledge in vRA product. If you want to talk more about key features of this product, here
is the required information.
 
Deliver a Self-Service Experience
• Unified IT service catalog – deliver infrastructure, container, application and custom services
(XaaS).
• Policy-based governance ensures the right service level to meet specific business needs.
• Automation accelerates IT service delivery.
Unified Blueprint Model via Design Canvas, Command Line or API
• Streamline IT service design process by assembling applications from prebuilt components
using a visual canvas with a drag and      drop interface.
• Blueprints as Code – Export, import and edit automation blueprints as text.
• API – enable the entire design and management process via API calls.
• Leverage VMware and partner-provided blueprints in VMware Cloud Management Marketplace.
Deploy Across Multi-Vendor, Hybrid Cloud Infrastructure
• Flexibility to choose the right cloud platform and location that meets the business needs.
• Consistent governance and control across hybrid cloud deployments.
Accelerate Time to Value via an Extensible Automation Platform
• Extensible platform that enables customization and extensibility at multiple levels across IT
ecosystem.
• Design and automate the delivery of any IT services (XaaS) through service orchestration.
• Leverage VMware and partner-provided integration solutions in VMware Cloud Management
Marketplace.
 

Question 20:   Do you know the recent security patches released for Intel Spectre and
Meltdown? Have you patched ESXi servers (Cluster/standalone/DMZ/Lab)? Can you share
couple of issues you ran into while patching those hosts?
Answer:    This is hot topic for many Administrators not limited to VMware to start the new year
2018. Some background for “Meltdown and Spectre” which are critical vulnerabilities existing in
several modern CPU: these hardware bugs allow programs to steal data which is currently
processed on the computer. Meltdown and Spectre can affect personal computers, mobile
devices, server and several cloud services.
Actually, the only way to minimize those security risks is to patch your operating systems or the
hypervisor level (if you are using virtual machines) and here is the latest VMware KB
Article about these patches.
 
These kind of questions from the Interviewer to check your awareness to active issues in the IT
Infrastructure space and followed by troubleshoot depth based on the scenario you picked to
explain the answer. Let me share couple of good scenarios
 

Scenario 1:   Standalone ESXi host in remote branch office can’t be

patched via Update Manager
 

 
 
 
ESXUPDATE Log file
2018-01-11T16:24:39Z esxupdate: 76791: downloader: DEBUG: Downloading
http://vumserver:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip to
/tmp/tmp4m_ng4…
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: An esxupdate error exception
was caught:
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: Traceback (most recent call
last):
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: File “/usr/sbin/esxupdate”, line
238, in main
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: cmd.Run()
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: File “/build/mts/release/bora-
5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/
esx5update/Cmdline.py”, line 105, in Run
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: File “/build/mts/release/bora-
5224934/bora/build/esx/release/vmvisor/sys-boot/lib/python2.7/site-packages/vmware/
esximage/Transaction.py”, line 73, in DownloadMetadatas
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: ERROR: MetadataDownloadError:
(‘http://vumserver:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip’, None,
“(‘http://vumserver:9084/vum/repository/hostupdate/vmw/vmw-ESXi-6.0.0-metadata.zip’,
‘/tmp/tmp4m_ng4’, ‘[Errno 4] IOError: [Errno 104] Connection reset by peer’)”)
2018-01-11T16:24:39Z esxupdate: 76791: esxupdate: DEBUG: <<<
 
You can start the explanation with there is a remote branch office which needs to be patched by
shutting down VM’s (as your design has single ESXi host) but failed to apply the required
patches with the above error message. Log file and symptoms are giving the indication about
communication failure between VMware update manager server and ESXi host. Then you tried
to copy the patches manually to the data store and tried to apply them but failed. As this is
problem seems to be complex, then you opened a case with VMware support to resolve it.
VMware support requested for log bundle from the ESXi server and Update manager server to
analyze this issue. They found issues with patch repository which is resolved by renaming the file
–  “D:\ProgramData\VMware\VMware Update Manager\Data\hostupdate\vmw\vmw-ESXi-6.0.0-
metadata.zip” and followed by downloading metadata from the patch source.
 

Scenario 2:   Failed to migrate VM’s from ESXi host while trying to keep it
in Maintenance Mode
 

 
Migrate virtual machine:
The vMotion failed because the destination host did not receive data from the source host on the
vMotion network. Please check your vMotion network settings and physical network
configuration and ensure they are correct
 
This scenario is kind of V-Motion issue stopping your patching activity as this is only way to
perform live migration of running virtual machines and keep the ESXi maintenance mode. You
started checking the V-Motion related settings like VMKernel, IP address and port group settings
but all of them looks good. You tried to perform vmkping command from source to destination
host but it failed. It seems there is communication issue for the V-Motion network and VLAN
associated with it. When you engaged Network team to validate the connectivity, they confirm no
firewall rule changes and other IP’s are able to communicate in the same subnet. This scenario
forced you to check other settings. Upon more investigation you found that these are HP Blade
servers and V-Motion network is configured as Internal network which doesn’t have an uplink
associated. HP confirmed it’s known issue with the current Openview/c7000/Flex backend
network settings. After contacting HP support they suggested to move the V-Motion network to
another port group where communication is not broken. This helped you to migrate all the VM’s
to other ESXi servers in the cluster to complete the patching activity.

Question 22:   What is HCI (Hyper-Converged Infrastructure) and how to manage VMware
Infrastructure when there is HCI solution deployed like Nutanix (for this scenario). What are the
commands to run against the cluster to check the health status? Have you ever run the upgrades
for Control VM (AOS), Prism Central (PC – like vCenter), BMC/BIOS & Hypervisor? Can you
explain the responsibilities to configure/manage the Nutanix based clusters?
Answer:  HCI aka Hyper-Converged Infrastructure became a turn-key solution and migrations
from Traditional Infrastructure to HCI based solutions. Legacy infrastructure—with separate
storage, storage networks, and servers—is not well suited to meet the growing demands of
enterprise applications or the fast pace of modern business. The silos created by traditional
infrastructure have become a barrier to change and progress, adding complexity to every step,
from ordering to deployment to management. New business initiatives require buy-in from
multiple teams, and IT needs must be predicted 3-to-5 years in advance. As most IT teams
know, this involves a substantial amount of guesswork and is almost impossible to get right.
 
Hyper-converged infrastructure combines common datacenter hardware using locally attached
storage resources with intelligent software to create flexible building blocks that replace legacy
infrastructure consisting of separate servers, storage networks, and storage arrays. Nutanix
converges the entire data center stack, including compute, storage, storage networking, and
virtualization. Complex and expensive legacy infrastructure is replaced by Nutanix Enterprise
Cloud OS running on state-of-the-art, industry-standard servers that enable enterprises to start
small and scale one node at a time.

Each server, also known as a node, includes Intel-powered x86 or IBM Power hardware with
flash SSDs and HDDs. Nutanix software running on each server node distributes all operating
functions across the cluster for superior performance and resilience. The Acropolis Distributed
Storage Fabric simplifies storage and data management for virtual environments. By pooling
flash and hard disk drive storage across a Nutanix cluster and exporting it as a data store to the
virtualization layer as iSCSI, NFS, and SMB shares, DSF eliminates the need for SAN and NAS
solutions. A key concept for HCI is using SSD disks for hot data (active) and HDD disks for cold
data (non-active).

Acropolis Distributed Storage Fabric joins HDD and SSD resources from across a
cluster into a storage pool.
 

 
Enough of theory and let me help you with Nutanix Administrator useful commands for day to day
operations. The key aspect of HCI is to make sure the storage online during the maintenance
window or for any node/disk/chassis failures. AOS (Operating System) on Control VM requires
frequent upgrades to get the newer feature sets and 1-Click upgrade is a famous selling point for
the Nutanix Sales campaign. LCM (Life Cycle Manager) is a 1-Click tool to upgrade the
BMC/BIOS and other devices firmware like LSA Controller, Disks & SATADOM. Prism Central is
another Virtual Machine to be deployed in the Infrastructure like vCenter and always keep the
Prism Central at the higher version than your Prism Element Clusters. (This point helps to make
you real-time Nutanix Administrator). You can explain about some theory from above HCI notes
and follow by below commands, will give the confidence for the Interview panel to select you for
Nutanix Administrator jobs. Make sure you stress the point of AOS upgrade failures with /home
full, Failed SATADOM replacement, CVM rescue, Nutanix Files, Async DR, NearSync, Metro
Clusters & shutdown token failed to release from CVM terms to keep the discussion more real-
time and promote discussion from theory to practical knowledge. 
1. svmips && hostips && ipmiips - Display all the IP's in the cluster
2. cluster status | grep -v UP - Display the status of the services in
Control VM
3. nodetool -h 0 ring - Display the storage stack is online or not
4. ncli host list - Display the hypervisors list
5. ncli cluster get-redundancy-state - Display the Redundancy Factor value
6. allssh date - Display the date on ALL the Control VM's
7. hostssh date- Display the date on ALL the Hypervisors
8. ncli cluster info - Display the Cluster details (number of nodes -
cluster ID - VIP)
9. ncli ms list - Display the Hypervisor version
10. allssh ntpq -pn - Display the NTP stats
11. ncc --version - Display the Nutanix Cluster Check (NCC) version
12. ncc healthchecks run_all - Run the Nutanix Health Checks
13. ncc log_collector run_all - Run the Nutanix logs collection (Logbay
is the future tool)
 
SAMPLE OUTPUT FROM NUTANIX CLUSTER - THIS EXAMPLE HELPS YOU TO UNDERSTAND
THE COMMANDS ONLY

nutanix@NTNX--A-CVM:10.xx.xx.176:~$ svmips && hostips && ipmiips

10.xx.30.176
10.xx.29.176
10.xx.28.176
nutanix@NTNX--A-CVM:10.xx.xx.176:~$ cluster status | grep -v UP
2019-05-04 16:25:26 INFO zookeeper_session.py:135 cluster is attempting to
connect to Zookeeper
2019-05-04 16:25:26 INFO cluster:2642 Executing action status on SVMs
10.xx.30.176
n2019-05-04 16:25:26 INFO cluster:2755 Success!
The state of the cluster: start
Lockdown mode: Disabled

CVM: 10.xx.30.176 Up, ZeusLeader

nutanix@NTNX--A-CVM:10.xx.xx.176:~$ nodetool -h 0 ring
Address Status State Load Owns Token
10.xx.30.176 Up Normal 78.83 MB 100.00%
00000000VIxeuKCNP8v1koBev1UVdCAQ9TmNo7GlyXQtpS6Xaxjm3kIYf3BD

nutanix@NTNX--A-CVM:10.xx.xx.176:~$ ncli host list

Id : 00058751-8a31-a11b-0000-00000000e3da::3
Uuid : 19878e31-0ff7-4f72-be10-d480fc8706ff
Name : NTNX-SERIAL-A
IPMI Address : 10.xx.28.176
Controller VM Address : 10.xx.30.176
Controller VM NAT Address :
Controller VM NAT PORT :
Hypervisor Address : 10.xx.29.176
Hypervisor Version : Nutanix 20170830.184
Host Status : NORMAL
Oplog Disk Size : 200 GiB (214,748,364,800 bytes) (0.5%)
Under Maintenance Mode : null (-)
Metadata store status : Metadata store enabled on the node
Node Position : Node physical position can't be displayed for this model.
Please refer to Prism UI for this information.
Node Serial (UUID) : OM162S010412
Block Serial (Model) : 16SM52340075 (NX-6155-G5)

nutanix@NTNX--A-CVM:10.xx.xx.176:~$ ncli cluster get-redundancy-state

Current Redundancy Factor : 1

Desired Redundancy Factor : 1
Redundancy Factor Status :
kCassandraPrepareDone=true;kZookeeperPrepareDone=true

nutanix@NTNX--A-CVM:10.xx.xx.176:~$ allssh date

================== 10.xx.30.176 =================
Sat May 4 16:25:56 UTC 2019

nutanix@NTNX--A-CVM:10.xx.xx.176:~$ hostssh date

============= 10.xx.29.176 ============
Sat May 4 16:25:59 UTC 2019

nutanix@NTNX--A-CVM:10.xx.xx.176:~$ ncli cluster info

Cluster Id : 00058751-8a31-a11b-0000-00000000e3da::58330
Cluster Uuid : 00058751-8a31-a11b-0000-00000000e3da
Cluster Name : Diehard
Cluster Version : 5.10.3.1
Cluster Full Version : el7.3-release-euphrates-5.10.3.1-stable-
655d4def34bf18785782f2adb8cdd5f8457d1fe3
External IP address :
Node Count : 1
Block Count : 1
Shadow Clones Status : Enabled
Has Self Encrypting Disk : no
Cluster Masquerading I... :
Cluster Masquerading PORT :
Is LTS : true
External Data Services... :
Support Verbosity Level : BASIC_COREDUMP
Lock Down Status : Disabled
Password Remote Login ... : Enabled
Timezone : UTC
NCC Version : ncc-3.7.0.2
Common Criteria Mode : Disabled
Degraded Node Monitoring : Enabled

nutanix@NTNX--A-CVM:10.xx.xx.176:~$ ncli ms list

Name : 10.xx.29.176
Uuid : dce5b69a-5905-41b9-97bf-a32c5c4d0b45
Hypervisor Type : AHV
Access URL : qemu+ssh://10.xx.29.176/system
User Name : root
Password : ****
Hypervisor Version : el6.nutanix.20170830.184

nutanix@NTNX--A-CVM:10.xx.xx.176:~$ allssh ntpq -pn

================== 10.xx.30.176 =================
remote refid st t when poll reach delay offset jitter
===========================================================================
===
*10.xx.1.20 69.164.213.136 3 u 735 1024 377 0.542 0.584 0.186
127.127.1.0 .LOCL. 10 l 9d 64 0 0.000 0.000 0.000

nutanix@NTNX--A-CVM:10.xx.xx.176:~$ ncc --version

3.7.0.2-3a444087

nutanix@NTNX--A-CVM:10.xx.xx.176:~$ ncc health_checks run_all

+--------------------------------------------------------------------------
----------------------------+
| Type | Name | Impact | Short help |
+--------------------------------------------------------------------------
----------------------------+
| M | cassandra_tools | N/A | Plugins to help with Cassandra ring analysis
|
| M | config_based | N/A | All config based plugin |
| M | fix_failures | N/A | Fix failures |
| M | hardware_info | N/A | Plugin to get or update node hardware
information |
| M | health_checks | N/A | All health checks |
| M | help_opts | N/A | Show various options for ncc. |
| M | log_collector | N/A | Collect logs on all CVMs. |
| M | performance_checks | N/A | This module performs various performance
checks on the cluster. |
| M | pulsehd_collectors | N/A | Plugin to start the insights collectors |
| M | upload_to_ftp | N/A | Uploads a file to SFTP/FTP server |
+--------------------------------------------------------------------------
------------------------+
 
 
 1. An administrator wants to connect ESXi host directly from vSphere Web Client.
Which ports are required for this purpose?
Normally vSphere Web Client is used to connect vCenter Server and VClient is used to connect
ESXi hosts. But vSphere Web Client can also be used to connect vCenter Server, but for this
purpose you will need 443 TCP, 902 TCP and UDP, and 903 TCP ports to be opened from
Security Profile.

2. Clock time of an ESXi 6.x host is not correct. What should an administrator do to
correct this issue?
To correct the time on ESXi host, modify the time for the host using the vSphere client and,
correct the NTP settings in the /etc/ntp.conf file.

3. An administrator wants to shutdown the host using ESXi host. Which option would
be used in Direct Console User Interface to perform this task?
To shutdown the host for Direct Console User Interface (DCUI), administrator will press F12 Key.

4. An administrator can access ESXi host via vCenter Server using vSphere Web Client
but unable to access directly via VClient. What should he do to access ESXi host
directly?
If ESXi host connected to vSphere Web Client is being accessed and can’t be accessed directly,
we should check that Lockdown is not enabled. If it enabled, we should be disabled. Because if
Lockdown is enabled, ESXi hosts can only be accessed via vCenter Server, you cannot directly
access any host.

5. An administrator wants to use the VMware Certificate Authority (VMCA) as an

Intermediate Certificate Authority (CA). He already replaces the Root Certificate and
Machine Certificates (Intermediate CA)
What should he do next?
After replacing root certificate and machine certificate (intermediate CA), following two steps are
needed to perform.

1. Replace Solution User certificates (Intermediate CA)

2. Replace the VMware Directory Service certificate.
6. If Strict Lockdown Mode is enabled on an ESXi host, which action should an
administrator perform to allow ESXi Shell or SSH access for users with administrator
privileges?
Administrator will add the users to Exception Users and enable the service to allow ESXi Shell or
SSH access.

7. SSO is an important component of vCenter Server. Which SSO component issues

Security Assertion Markup Language tokens?
VMware Security Token Service component of SSO grants SAML tokens.

8. What valid Identity Source is used to configure vCenter Single Sign-On?

OpenLDAP is a valid Identity Source for configuring vCenter SSO.

9. What happens to the files contained on shared storage When a Content Library is
deleted?
When Content Library is deleted, all stored files in content library will be deleted.

10. What is the maximum number of vCPUs are required for a VM in vSphere 6.0?
Maximum 128 vCPUs can be allocated to a VM vSphere 6.0.

11. A windows domain user can be logged into vSphere using vSphere Web Client. What
are the requirements to be met for this feature to be available and functional?
An administrator can allow users to login to vSphere Web Client using Windows session
authentication. For this purpose, Install the vSphere Web Client Integration browser plug-in on
each computer from where a user will sign in. The users must be signed into Windows using
Active Directory user accounts. And, administrator must create a valid Identity Source in Single
Sign-On for the users’ domain.

12. An administrator wants to clone a virtual machine using the vSphere Client. Which
explains why the Clone option is missing?
To clone a VM can be perform from vCenter Server either you connected  via vSphere Web
Client or VClient. If you are directly connected to an ESXi host, you cannot perform cloning of a
VM.

13. What will happen if the .nvram file is deleted accidently from a VM?
.nvram file is used to store the BIOS state of a VM. If it deleted for some reason, then, .nvram file
will be created again when the virtual machine is powered on.

14. An administrator wants to connect the vSphere 5.5 Client to ESXi 6.x host. What will
occur?
If administrator tries to connect the vSphere 5.5 Client to ESXi 6.x host, the operation will prompt
the administrator to run a script to upgrade the vSphere Client.

15. Which one of secondary Private VLANs (PVLANs) type can send packets to Isolated
PVLAN?
Promiscuous type of PVLAN can communicate and send packets to an Isolated PVLAN.

16. What sample roles are provided by default when vCenter is installed?
When vCenter is installed, Virtual machine user and Network Administrator roles are provided.

17. What will happen when all paths down (APD) event occurs for the software FCoE
storage?
If all paths down event occurs, Spanning Tree Protocol is enabled on the network ports.

18. What methods are available for upgrade a host from ESXi 5.x to ESXi 6.x?
vSphere Update Manager (VUM), esxcli command line tool, and vSphere Auto Deploy can be
used to upgrade.

19. What administrator should do before upgrading virtual machine hardware?

Before upgrading a VM hardware, we should create a backup or snapshot of the VM, should
upgrade VMware Tools to the latest version, and verify that the VM is stored VMFS or NFS
datastore.

20. vCenter Server upgradation fails at the vCenter Single Sign-On installation. What
should do to complete the upgrade process?
Before upgrading vCenter Server, verify that the VMware Directory service can stop by manually
restarting it. If it stopped manually, then you can start upgradation process of vCenter Server.

21. What prerequisites should be considered before upgrading the vCenter Server
Appliance?
In case of upgradation of vCenter Server Appliance (vCSA) or after fresh installation, Client
Integration Plugin (CIP) will be installed in both cases.

22. After deploying a PSC, vCenter Server is not being installed and shows the following
error:
Could not contact Lookup Service. Please check VM_ssoreg.log.
If this error appears then verify that the clocks on the host machines running the PSC, vCenter
Server, and the vSphere Web Client are synchronized with each other. And also ensure that
there is no firewall blocking port 7444 between the PSC and vCenter Server.

23. An administrator installed Windows Server 2008 and want to install vCenter Server
on it but failed when installing on a Windows virtual machine?
vCenter Server installation requires 64bit Windows OS to install. If you try to install it on Windows
Server 2008, it would not be installed and installation will be failed. vCenter Server will be
installed in Windows Server 2008 R2 or higher Windows OS.

24. What is the minimum Virtual Hardware version required for vFlash Read Cache?
vFlash Read Cache was first in vSphere 5.5, and the minimum Virtual Hardware version for
vSphere 5.5 is version 10.

25. ESXi host is added in vCenter Server but not responding in vSphere Web Client. If
this issue occurs due to firewall, which port should be opened?
If administrator sees no response of added ESXi 6.x host in vCenter Server, and the issue is
caused by network firewall blocking traffic. Then he should check that port 902 (UDP) is not
blocked by firewall. If it happens, enable the port from Security Profile by using vSphere Web
Client by selecting said ESXi host in vCenter Server.

26. If a VM is unexpectedly powered off. Which VM logs files should be considered to

troubleshoot the issue?
If it happens, administrator should check vmware.log and hostd.log log files to troubleshoot the
issue.

27. Why a VM appeared as orphaned VM?

If a VM appears in orphaned state, this could cause a VMware High Availability host failure has
occurred. And the virtual machine was unregistered directly on the ESXi host.

28. While upgrading an ESXi 5.5 host to ESXi 6.x, following error appears:
MEMORY_SIZE
What does this require to do?
It indicates insufficient memory on the ESXi host to complete the upgrade process of an ESXi
host from ESXi 5.5 to ESXi 6.x.

29. To remove a host from a vSphere Distributed Switch (vDS), following error message
is observed:
The resource ’10’ is in use
Before removing vDS, it is ensured that VMkernel network adapters on the vDS are not in use. If
any of resource of vDS is being used, then above mentioned error message with resource ID will
appear.

30. An administrator wants to monitor network traffic and tries to capture network traffic
for a VM, but cannot see the expected traffic in the packet capture tool. What should
he do to resolve the problem?
If administrator needs to capture network traffic for a VM, he should Enable Promiscuous Mode
on the relevant port group. Then he can capture the network traffic by using any networking
traffic capturing tool.

31. A vSAN Cluster is created with six nodes along with the fault domain, and three of
them moved into fault domain. One-member node of fault domain fails. What will
happen with the remaining two nodes exist in fault domain?
When a member node of fault domain fails, the remaining two fault domain members will be
treated as failed.
 32. At which level a vSAN Fault Domain configured?
Fault domain is configured at vSAN Cluster level and nodes will be added in this domain. If any
member node fails due to any reason, remaining members of fault domain will also be
considered as fail.

33. It is observed that a VM storage activity on an ESXi 6.x host is negatively affecting a
VM storage activity on another host that is accessing the same VMFS Datastore.
Which action would mitigate the issue?
To control the storage activity of a VM to affecting another VM’s storage activity, Storage IO
Control (SIOC) should be enabled. Storage I/O Control provides much needed control of storage
I/O and should be used to ensure that the performance of your critical VMs are not affected by
VMs from other hosts when there is contention for I/O resources.

34. While upgrading an ESXi host from 5.5to 6.0, administrator runs the following
command:
esxcli software vib list –rebooting-image
What will be shown by this command?
This command will show all active VIBs (vSphere Installation Bundle). VIB is is a collection of
files like tarball or zip that packaged into a single archive to facilitate distribution.

35. To troubleshoot a CPU performance issues of a VM, which counters will be used to
demonstrate CPU contention?
To test the performance of an ESXi host in the form of memory, CPU, and network utilization,
ESXTOP tool is used. It is a very good tool available for VMware administrators to troubleshoot
the performance issues. For configuring ESXTOP, you’ll need vSphere Client and putty and SSH
session should be enabled. For CPU performance testing, %RDY, %MLMTD, and %CSTP
counters are used.

36. An administrator tries to run esxtop by enabling SSH and using putty to
troubleshoot CPU performance issues, but no output displayed. How to resolve this
issue?
To display output in ESXTOP, press f and place an asterisk next to each field that should be
displayed.

37. An administrator wants to monitor VMs on a host using vCenter Server and send
notifications when memory usage crosses 80%. What should an administrator do in
vCenter Server to accomplish this?
To monitor VM’s memory usage that reaches 80%, vCenter Server alarm will be created that will
monitor VM’s memory usage and set an action to email the notification.

38. An administrator created a DRS cluster and it became unbalanced. What are likely
causes to become unbalanced?
DRS cluster can become unbalance when Affinity rules are preventing VMs from being moved.
And a device is mounted to a VM is preventing vMotion from one host on another.

39. An IT administrator configured two vCenter Servers within a PSC, and needs to grant
a user privileges that can access all environments. What access level is required to
access all the environment?
To access multiple vCenter Servers within a PSC, it requires Global Permission to access all
environments.

40. An administrator created 10 ESXi 6.x hosts via Auto Deploy for a new Test/Dev
cluster and all hosts are configured to obtain their IP address via DHCP. Which DCUI
option should the administrator use to renew the DHCP lease for the hosts?
To renew the DHCP lease for the hosts, “Reset Management Network” of Direct Console User
Interface (DCUI) option is used.