Methods of Viewing Log Files.

Using the vSphere client

The direct console user interface (DCUI)
A web browser
A syslog or vMA appliance
An SSH connection to the host
PowerCLI using the Get-Log command
ESXi Configuration Files & System Logs:
http://<insert ESXi host name or IP here>/host

Checking log files - First Way

Select the host, cluster, or datacenter in the inventory that you would like to generate the bundle
for.
Select File->Export->Export System Logs
If you selected a cluster or datacenter, you can check or uncheck which hosts you would like to
include here.
Select which components you would like to include in the bundle and whether or not to gather
performance data.
Done.

Second Way

1. Click Administration->Export System Logs

2. Select the hosts you wish to export and/or vCenter.
3. Select whether to gather performance data.
4. DONE.

Preserving the ESX/ESXi Host Configuration

During the upgrade, most of the ESX/ESXi host configuration is retained

The following key configuration settings not carried forward during an upgrade:

1. The service console port group

2. Local users and groups on the ESX/ESXi host
3. NIS settings
4. Rule sets and custom firewall rules
5. Any data in custom disk partitions
6. Any custom or third-party scripts/agents running in the ESX service console
7. SSH configurations for ESX hosts (SSH settings are kept for ESXi hosts)

HA New Features in 5.0 - FDM

New HA Agent – FDM - Fault Domain Manager is the name of the agent
FDM replaces AAM agent in 4.1

No dependency on DNS – HA use IP not DNS

Primary node concept – The primary/secondary node mechanism removed.

Master , Slave concept came now.

Supports management network partitions – Can have multiple “master nodes” when multiple
network partitions exist.
Enhanced isolation validation - Avoids false positives when the complete management
network has failed.

Datastore Heartbeating – This additional level of heart beating reduces chances of false
positives by using the storage layer to validate the state of the host and to avoid unnecessary
downtime when there’s a management network interruption.

Enhanced Admission Control Policies.

 The Host Failures based admission control - allows for more than 4 hosts to be
specified (31 is the maximum)
 The Percentage based admission control policy - allows to specify percentages for both
CPU and memory separately
 The Failover Host based admission control policy - allows to specify multiple
designated failover hosts

Three major components of HA as of vSphere 5.0:

FDM , hostd , vCenter

The FDM Agent is responsible for

- Communicating host resource information,
- virtual machine states and
- HA properties to other hosts in the cluster
- heartbeat mechanisms, virtual machine placement, virtual machine restarts, logging
and much more

log file is stored  /var/log/ fdm.log

One host takes the role of Master. The other Agents on other hosts are Slaves, and can
become Masters if master fails.

FDM Master
---- The FDM Master monitors all hosts and VMs and slave hosts.
---- if host failure happens, the master restarts all VMs on another host.
---- It keeps a list of VMs protected, which is updated after every power off or on done by user.
---- keeps track of any adding/removing of hosts.
---- sends periodically status reports to vCenter

Slave
--- A slave monitors the vms it is running and informs the master about any changes.
--- The slave also monitors the health of the master by monitoring heartbeats.
--- If the master fails, the slaves initiate and participate in the election process.

A Master Election occurs

--- when HA is enabled on a cluster
--- when the host on which the master is running fails
--- becomes network partitioned or isolated
--- is disconnected from VC
--- is put into maintenance or standby mode

Master-election algorithm
Takes 15 to 25s (depends on reason for election)
Elects a host which has access to greatest number of datastores.
if there is a tie, host with highest Managed Object Id will be chosen.

Re-Election : If the master fails there is a re-ellection

When the cluster is split in two sites due to a link failure each “partition” will get its own
master. This allows for workloads to be restarted even in a geographically dispersed
cluster when the network has failed….

Network Heartbeat mechanism

--- Sends heartbeat between slaves and master in every second
--- Election using UDP and master-slave communication using TCP
--- When a slave isn’t receiving heartbeats from the master, it will try to determine whether it is
isolated or the master is isolated or has failed
Before vSphere 5.0, virtual machine restarts were always initiated, even if only the management
network of the host was isolated and the virtual machines were still running

Datastore Heartbeat
--- If the primary network goes down this secondary network is used to find out
Whether the host is failed or isolated / partitioned host.
--- It Prevents Unnecessary Restarts due to isolation

Two different files used : PowerOn file , Host hb file

These files are located on vmfs volumes in the hidden directory “.vSphere-HA/<FDM cluster
ID>
PowerOn- This file is used to determine Isolation
PowerOn file created by Slave which lists the powered on VMs. (host-<number>-poweron)
Host hb file is created by host in the datastore (Host-<number>-hb)
Datastore heartbeat mechanism is only used if the master has lost network connectivity with the
slaves

2 datastores are automatically selected by vCenter for this mechanism. Max is 5.

For VMFS datastores, the Master reads the VMFS heartbeat region(Uses locking mechanism).
For NFS datastores, the Master monitors a heartbeat file that is periodically touched by the
Slaves

By default it will select two datastores. We can configure an advanced setting

(das.heartbeatDsPerHost) to allow for more datastores for datastore heartbeating.

Locking mechanism

HA use existing VMFS file system for Datastore heartbeat mechanism.

All hosts in the cluster update host-xx-hb file used for datastore heartbeat and HA will check
whether the file is updated.
To update a datastore region, a host needs to have at least one open file which is created by HA
for heartbeating.
When the master is isolated or fails it locks the file on the datastore. the lock will expire and the
new master will relock the file

To see which datastores have been selected for heartbeating. go to your summary tab on your
cluster and click “Cluster Status”, the 3rd tab “Heartbeat Datastores” will reveal it.
Isolated vs Partitioned
there are two different states:

 Isolated
o Is not receiving heartbeats from the master
o Is not receiving any election traffic
o Cannot ping the isolation address
 Partitioned
o Is not receiving heartbeats from the master
o Is receiving election traffic
o (at some point a new master will be elected at which the state will be
reported to vCenter)
Network Partition
When multiple hosts are isolated it can communicate within each other over the management
networks, it is called a network partition. When a network partition exists, a master election
process will be issued so that a host failure or network isolation within this partition will result in
appropriate action on the impacted virtual machine(s).

Restarting VMs
Restart priority changes
The order in which virtual machines will be restarted:
 Agent virtual machines
 FT secondary virtual machines
 Virtual Machines configured with a restart priority of high,
 Virtual Machines configured with a medium restart priority
 Virtual Machines configured with a low restart priority

Agent virtual machine is a virtual machine which performs a specific function like
vShield Endpoint appliance which is an anti-virus solution. By "tagging" this virtual
machine as an agent VM we ensure that this virtual machine gets powered on first in the
case of a host failure before all other VMs start.

All hosts down

But what happens when all hosts in a cluster fail simultaneously?
 Power Outage, all hosts down
 Power for hosts returns
 As soon as the first host has booted an election process will start and a master will be
elected
 Master reads protected list which contains all VMs which are protected by HA
 Master initiates restarts for those VMs which are listed as protected but not running

if the VM was cleanly powered off, initiated by the admin, or powered-off due to a
failure/isolation. If a vm is cleanly powered off they will not be restarted, If they are not cleanly
powered off such VMs will be powered on.

VM Port Group - Virtual Machine Network

VMkernel Port - Management Network
Appreciation
Creating iSCSI network in vSphere ESXi 5.0

Storage Adapter:

by default it’s not installed, you must manual add the adapter by clicking Configuration >
Storage Adapter > Add Storage Adapter
 - Select: Add Software iSCSI Adapter

A new software iSCSI adapter will be added to the Storage Adapters list. After it has been
added, select the software iSCSI adapter in the list and click on Properties to complete the
configuration

Event: Change Software Internet SCSI Status = Completed

s): -
Network label: iSCSI01 - Configure IP Address and Subnet Mask

..

- Finish the configuration, repeat this step to configure the 2nd iSCSI02 port
vSwitch2 result:

VMkernel – Override switch failover order:

Now we need to change the failover order for each VMkernel:

iSCSI01: Active Adapter VMNIC4 – Unused Adapter: VMNIC5

iSCSI02: Active Adapter VMNIC5 – Unused Adapter: VMNIC4
Configure Software iSCSI Initiator:
Go to the properties of the iSCSI initiator: Configuration > Storage Adapters > iSCSI Storage
Adapter: vmhba** > Properties
And setup Dynamic Discovery – Sent Target location

Next we need to bind a pNIC to the iSCSI VMkernel

When you go to tab: Network Configuration > Add..
- Click: Add..
- Now you can select a Physical NIC to bind the VMkernel for your iSCSI targets

Vcenter Crashed
What happens if the vCentre server looses connection to the database server,
HP blade centre, both the database server and the HP blade server?

ESX hosts and vms will run, HA will continue to work

DRS and vMotion will stop working .

After a connection failure , ping the vCenter box

check vCenter service is running or not
If the service is stopped, restart it

what if it fails to start?

verify everything on the DB is fine.
Check the logs.
vCenter logs is vpxd logs will help.
Windows 2008, the vpxd.log file is located at C:\ProgramData\VMware\VMware VirtualCenter\
Logs

--- Stop IIS service in the vcenter server

the vmware-vxpa service was crashing and needing to be restarted frequently.

# cat /var/log/vmware/vpx/vpxd.log
# service vmware-vpxa stop
# service vmware-vpxa start

Steps to Recover Vcenter Server

If we don’t have backup of the vCenter server,

we have to reinstall vCenter and build everything
re-register vm’s, reconfigure everything, re-create and re-configure Resource pools and DRS.
So it is important to have a good backup of your vcenter server.

Backup vCenter:
 The vCenter Database

 The SSL Certificates located in:

Windows 2003:%ALLUSERSPROFILE%\Application Data\VMware\VMware
VirtualCenter\SSL“
Windows 2008: “%ALLUSERSPROFILE%\VMware\VMware VirtualCenter\SSL“.

 The Sys prep files- used for customizations during template deployments.
C:\ProgramData\VMware\VMware VirtualCenter\Sysprep

If SSL certificates are not backed up, we have to disconnect and reconnect esxi hosts.
once you would restore your vCenter server.
Hence it’s better to backup those SSL Certificates.

Restoring vCenter :
--- Reinstall your Windows Server
--- Reinstall sql
--- Restore database
--- Create the ODBC connections and test it.
--- Install vCenter server, Note : when installing you receive a message that there already is a
vCenter database, do not replace the database and use it.
--- After Installation completes Restore SSL and Sysprep files.
To configure the TSM timeout value from the vSphere Client:

1. Select the host and click the Configuration tab.

2. Click Software > Advanced Settings.

ESXi 5.x – Change the UserVars.ESXiShellTimeOut field to the desired value.

ESXi 4.1 – Change the UserVars.TSMTimeOut field to the desired value.

3. Click OK.

To access the local TSM:

1. At the main DCUI screen, press ALT+F1 simultaneously. This opens a virtual console
window to the host.
2. Provide credentials when prompted.

Note: When typing the password, characters are not displayed on the console.

To access the remote TSM:

1. Open an SSH client.

2. Specify the IP address or domain name of the ESX host.

Notes:
o Directions may vary depending on what SSH client you are using. For more
information, consult vendor documentation and support.
o By default, SSH works on TCP port 22.
3. Provide credentials when prompted.

ESX or ESXi host fails with a Purple Screen Error

Extract the log file from a vmkernel-zdump file using a command line utility on the ESX or
ESXi host. This utility differs for different versions of ESX or ESXi.

For ESX 3.0 and 3.5, use the vmkdump utility:

# vmkdump -l <vmkernel-zdump-filename>

For ESXi 3.5, ESX/ESXi 4.x and 5.0, use the esxcfg-dumppart utility:
# esxcfg-dumppart -L <vmkernel-zdump-filename>

To extract the log file from a vmkernel-zdump file:

1. Find the vmkernel-zdump file in the /root/ or /var/core/ directory:

# ls /root/vmkernel* /var/core/vmkernel*
/var/core/vmkernel-zdump-073108.09.16.1

2. Use the vmkdump or esxcfg-dumppart utility to extract the log. For example:

# vmkdump -l /var/core/vmkernel-zdump-073108.09.16.1
created file vmkernel-log.1

# esxcfg-dumppart -L /var/core/vmkernel-zdump-073108.09.16.1
created file vmkernel-log.1
 3. The vmkernel-log.1 file is plain text, though may start with null characters. Focus on the
end of the log, which looks similar to:

VMware ESX Server [Releasebuild-98103]

PCPU 1 locked up. Failed to ack TLB invalidate.
frame=0x3a37d98 ip=0x625e94 cr2=0x0 cr3=0x40c66000 cr4=0x16c
es=0xffffffff ds=0xffffffff fs=0xffffffff gs=0xffffffff

Note: The file name created for the log in this example is vmkernel-log.1. If another file with
the same name already exists, the new file is created with the number suffix incremented.

Esx Host Not Responding Error

It is caused by the host agent service (mgmt-vmware) failing due to a dead process.

First restart the mgmt-vmware service:

# service mgmt-vmware restart
If this is hanging when trying to restart the host agent, then kill the process causing the issue.

Open another console session and do the following:

# ps -ef | grep hostd

This will output a list of processes .

root 23955 1 0 10:42 pts/1 00:00:00 /bin/sh /usr/bin/vmware-watchdog -s hostd -u 60 -

q 5 -c /usr/sbin/vmware-hostd-support /usr/sbin/vmware-hostd -u
root 23961 23955 4 10:42 ? 00:00:15 /usr/lib/vmware/hostd/vmware-hostd
/etc/vmware/hostd/config.xml -u
root 24211 23422 0 10:48 pts/1 00:00:00 grep hostd

The first process is using the vmware-watchdog,

The second line is using hostd (config.xml -u). kill the process.
virtual machines will continue to run so don’t worry about that.

# kill -9 23961
Now the hostd service will start and after a few seconds
Host and virtual machines will be up in vCenter.

If esxi hosts Not Responding or Disconnected .

check – the DNS addresses on the ESXi hosts, the hosts files on the ESXi hosts & vCenter
Server, the Managed IP Address setting in vCenter .

/var/log/vmware/vpx/vpxa.log Shows Agent can’t send heartbeat, No route to host. errors.

/var/log/vmware/aam/vmware_hostname.log Shows date and timestamp and “Node hostname
has started receiving heartbeats from node hostname” informational events for intra-esxi host
communications

What are the basic commands to troubleshoot connectivity between vSphere

Client /vCenter to ESX server?

service mgmt-vmware restart (restarts host agent (hostd) on vmware esx server)
service vmware-vpxa restart (restarts Vcenter agent service)
service network restart (restarts management networks on ESX)

VMs not responding Issue

First list all your vms on the host:: vim-cmd vmsvc/getallvms

It will list:
 Vmid : virutal machine id
 Name : of the server in vsphere
 File :: which is the location and name of the VMX,
 Guest OS :: os type
 Version :: of hardware

Example:
For 250-DC01
 Vmid : is 7
 Name : is 250-DC01.
 File :: located in datastore2 under 250-DC01
 Guest OS :: win 2008 64 is listed as windows7server64guest
 Version :: of hardware is 8

The VMID is the key to the following steps.

To get the power state of 250-DC01 we would enter

vim-cmd vmsvc/power.getstate VMID

vim-cmd vmsvc/power.getstate 7

To gracefully to a shutdown, enter

vim-cmd vmsvc/power.shutdown VMID

vim-cmd vmsvc/power.shutdown 7

If you are unable to shutdown or you don’t have vmware tools installed, you will need
to use another command.
vim-cmd vmsvc/power.off VMID

Powering off a virtual machine fails with the error: Cannot power Off: Another task is
already in progress
To resolve this issue:

1. Open the .vmx file of the virtual machine using a text editor.
2. Comment this line:

#log.fileName =
"/vmfs/volumes/4b8bd18f-c1f89b5a-1914-002219c8e7a3/vmware.log"

3. Restart the virtual machine for the changes to take effect.

Note: Instead of restarting the virtual machine, you can reload the .vmx file using
these commands:

# vmware-vim-cmd vmsvc/getallvms
This command returns the VMID.

# vmware-vim-cmd vmsvc/reload VMID

In ESXi 4.x/5.x, run this command:

# vim-cmd vmsvc/getallvms
This command returns the VMID.

# vim-cmd vmsvc/reload VMID

In ESXi 3.5-5.0, use kill command to terminate, a running virtual machine process.

1. On the ESXi console, enter Tech Support mode and log in as root

To check the virtual machine process is running on the ESXi host, run this command:

ps | grep vmx

The output appears similar to:

7662 7662 vmx /bin/vmx

7667 7662 vmx /bin/vmx
7668 7662 mks:VirtualMachineName /bin/vmx
7669 7662 vcpu-0:VirtualMachineName /bin/vmx

The first column contains the PID, and the Second contains the parent's PID.
Kill only the Parent process.

2. If the vmx process is listed, Kill the process using this command:
kill ProcessID

3. Wait 30 seconds and repeat step 2 to check for the process again.
4. If it is not terminated, run this command:
kill -9 ProcessID

5. Wait 30 seconds and check for the process again.

vmkfstools – VMware ESX file system management tool

all information on a computer is written in binary form: ones (1) and zeros (0). So when you zero
out data, the computer actually writes zeros over the locations where the data resided, thus
covering the tracks of the previous information with nothing but empty space (empty space is not
actually empty – it’s a bunch of zeros).

write zeros in the unused areas of your hard drive, overwriting any scraps and fragments of old
files there so that they can’t be easily recovered by snoops

When files are normally deleted they are just removed from the filesystem. The file itself is still
on the hard drive though, which is how some programs are able to "recover" deleted files. For
security reasons you may want to zero out your free space, which will get rid of all chances of
recovering deleted files.

c:\>sdelete -z c:
-c Zero free space (good for virtual disk optimization
-z Clean free space

How to Shrink a Thin VMDK on ESXi 5.0

To check virtual disk size : ls –lh *.vmdk. In my example, I have a 40 GB virtual size.

To see the ‘real’ size of the vmdk, run du –h *.vmdk.

This gives us a 36 GB size (while the VM itself actually uses about 10 GB).

Issue the command vmkfstools –punchzero VM.vmdk. you VM must be powered off

On a 40 GB VMDK it took 10 minutes, but this will depend mostly on your SAN speed.

After it’s done, let’s re-issue du –h *.vmdk.

Nice, now it’s actually 10.8 GB. Makes it a lot easier to transfer around

To Extend an existing Virtual Disk

vmkfstools -X 12G ./testing.vmdk

To extend an existing Virtual Disk to 12GB.
How to extend the OS drive of a guest OS (windows VM)

vmkfstools -X 50M /vmfs/volumes/Storage2/testvm/testvm.vmdk

vmkfstools -X 50M /vmfs/volumes/Storage2/testvm/testvm.vmdk

Extending a VM Virtual Disk Size

VM Click to Edit Properties Hardware tab

Then go into Computer Management, under Start -> Administrative Tools. Here, under
Storage, click on Disk Management.

Right-click on the allocated piece of the disk and click Extend Volume
Recover Orphaned (left Alone) Virtual Machines
Virtual machines in host with (orphaned) appended to their name.

Cause
Virtual machines can become orphaned if a host failover is unsuccessful, or when the virtual
machine is unregistered directly on the host.
If this situation, Move the orphaned virtual machine to another host .

Solution
1 Right-click the virtual machine and select Migrate.
2 Click Change Host and click Next.
3 Select the host on which to place the virtual machine.

If no hosts are available, add a host that can access the datastore on which the virtual machine's
files are stored.

4 Click Finish to save your changes.

The virtual machine is connected to the new host and appears in the inventory list.

Virtual Machine Does Not Power On After Cloning or Deploying from Template

Virtual machines do not power on after you complete the clone or deploy from template

Cause
The swap file size is not reserved when the virtual machine disks are created.

Solution
Reduce the size of the swap file of the vm. To do this increase the vm memory reservation.
  right-click the virtual machine and select Edit Settings.
 Select the Resources tab and click Memory.
 Use the Reservation slider to increase the amount of memory allocated to the virtual
machine.
 Click OK.

Step :2 Increase space for the swap file. by moving other vm disks off of the datastore that is
being used for the swap file.

 select the datastore and click the Virtual Machines tab.

 right-click the virtual machine and select Migrate.
 Select Change datastore.
 Proceed through the Migrate Virtual Machine wizard.

Step :3 Increase space for the swap file by changing the swap file location to a datastore with
enough space.

 Select the host and click the Configuration tab.

 Under Software, select Virtual Machine Swap file Location.
 Click Edit.

Note
If the host is in cluster use the Cluster Settings to change the swap file location policy for the
cluster.

Select a datastore from the list and click OK

vCenter Server Cannot Connect to Managed Hosts

After you replace default vCenter Server certificates and restart the system, vCenter Server might
not be able to connect to managed hosts.

Problem
vCenter Server cannot connect to managed hosts after server certificates are replaced and the
system is restarted.

Solution
Log into the host as the root user and reconnect the host to vCenter Server.

vCenter map
A vCenter map is a visual representation of your vCenter Server topology. Maps show the
relationships between the virtual and physical resources available to vCenter Server.

Maps are available only when the vSphere Client is connected to a vCenter Server system.

The maps helps to check which clusters or hosts are most densely populated,
which networks are most critical, and which storage devices are being utilized.

vCenter Server provides the following map views.

Virtual Machine Resources- Displays virtual machine-centric relationships.

Host Resources - Displays host-centric relationships.
Datastore Resources - Displays datastore-centric relationships.
vMotion Resources - Displays hosts available for vMotion migration

Checking Telnet Port

To do a Telnet test to a port from Windows:
In the command prompt window, type: telnet server port

Where server is the hostname or IP address of the server, and port is the port that you want to
connect to.
Connecting to port 902 on an ESXi/ESX host:

C:\>telnet server 902

Connecting...

220 VMware Authentication Daemon Version 1.10: SSL Required,

ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC

VM disk Resize - Failed to extend the disk

Issue
- Edit a VM's settings and then select one of its virtual disks, the option to resize the disk is
greyed out.
-When trying to extend the disk using vmkftools for resizing, you will receive the below error:
Failed to extend the disk. Failed to lock the file.
-When you extend the disk using vsphere client, the change is not getting reflected.

Root cause
The VM will be having a snapshot. Because when a snapshot is taken the actual VMDK file
will be locked and the changes will be written to a new delta file. So, the VMDK file could not
be resized.

Resolution
Remove the snapshot file and then try to resize.

Esxtop
esxtop command used to check whether the ESX/ESXi server is being overloaded

1.check the load average on the first line of the command output.

load average of 1.00 means the ESXi Server’s physical CPUs are fully utilized,
load average of 0.5 means they are half utilized.
load average of 2.00 means the system is overloaded.

2. Check %READY for the percentage of time that the virtual machine was ready but could not
be scheduled to run on a physical CPU.

it should be under 5%.

If the ready time values are high on the virtual machines that experience bad performance,
then check for CPU limiting:
If the load average is too high & the ready time is not caused by CPU limiting, adjust the
CPU load on the host. To adjust the CPU load on the host, either:
Increase the number of physical CPUs on the host

OR
Decrease the number of virtual CPUs allocated to the host.

To decrease the number of virtual CPUs allocated to the host, either:

Reduce the total number of CPUs allocated to all vms running on the ESX host
OR
Reduce the number of virtual machines running on the host.

Memory overcommitment
Use the esxtop command to check whether the ESX/ESXi server's memory is overcommitted

check the MEM overcommit avg on the first line of the command output. This value says the
ratio of the requested memory to the available memory, minus 1.

Examples:

- If the vms require 4 GB of RAM, and the host has 4 GB of RAM, then there is a 1:1 ratio.
After subtracting 1 (from 1/1), the MEM overcommit avg field reads 0.
There is no overcommitment and no extra RAM is required.
- If the vms require 6 GB of RAM, and the host has 4 GB of RAM, then there is a 1.5:1
ratio. After subtracting 1 (from 1.5/1), the MEM overcommit avg field reads .5. The
RAM is overcommited by 50%, meaning that 50% more than the available RAM is
required.

If the memory is being overcommited, adjust the memory load on the host.
To adjust the memory load, either:
Increase the amount of physical RAM on the host
OR
Decrease the amount of RAM allocated to the virtual machines.

To decrease the amount of allocated RAM, either:

Decrease the total amount of RAM allocated to all of the virtual machines on the host

OR
Reduce the total number of virtual machines on the host.
Determine whether the virtual machines are ballooning and/or swapping.

To detect any ballooning or swapping:

a. Run esxtop.
b. Type m for memory
c. Type f for fields
d. Select the letter J for Memory Ballooning Statistics (MCTL)
e. Look at the MCTLSZ value.

MCTLSZ (MB) displays the amount of guest physical memory reclaimed by the
balloon driver.
f. Type f for Field
g. Select the letter for Memory Swap Statistics (SWAP STATS).
 h. Look at the SWCUR value.

SWCUR (MB) displays the current Swap Usage.

To check if multiple vCPUs assigned to vm is causing poor performance:

Type esxtop and press Enter.
In the CPU screen, check the %CSTP value. If this number is higher than 3, the performance
issues may be caused by the vCPU count. Reduce the vCPU count of the virtual machine by 1.
To lower the vCPU count:
The virtual machine must be powered off to perform these steps.

1. Right-click on the virtual machine and click Edit Settings.

2. Click CPUs.
3. Use the Number of virtual processor drop-down to lower the vCPU count by 1.
4. Click OK.
5. If your virtual machine still experiences performance issues, and if its kernel or HAL
can handle switching to a single vCPU, lower the vCPU count to 1.

To monitor storage performance on a per-LUN basis:

1. Start esxtop by typing esxtop from the command line.

2. Press u to switch to disk view (LUN mode).
3. Press f to modify the fields that are displayed.
4. Press b, c, f, and h to toggle the fields and press Enter.
5. Press s, then 2 to alter the update time to every 2 seconds and press Enter.
6. See Analyzing esxtop columns for a description of relevant columns.

 esxtop – Provides real time CPU, memory, disk and network statistics for virtual
machines and their host. This utility is accessible from a direct connection to an ESXi
host. It does not report statistics for NFS datastores.

esxtop has 8 different "displays" that show CPU, interrupt, memory, network, disk adapter,
disk interface, disk VM, and power management

 resxtop – A remote version of esxtop. It is included as part of the Linux vCLI (vSphere
Command Line) and the vMA (vSphere Management Assistant). Discussed below,
resxtop has three modes of operation including Interactive, Batch, and Replay. Both
esxtop and resxtop may only connect to a single ESXi host and requires root level access.

What's the difference between esxtop on ESX and ESXi?

The only difference will be that esxtop on ESX will show information about the service console.
On ESX, you'll also find a man page for esxtop by typing man esxtop.

What's the difference between esxtop and resxtop?

The only difference is that esxtop is run locally on an ESX or ESXi server.
Resxtop is "remote esxtop" and it is only used to run esxtop on another server. For example, you
would use resxtop to monitor performance of an ESXi server if you were currently logged into a
vMA (vSphere Management Assistant) virtual appliance.

Esxtop run directly on the vSphere host by connecting via SSH.

Resxtop can be run remotely from the vMA perhaps.
Inaccessible virtual machines are named as Unknown VM

If the vm is renamed as Unknown VM, check the /etc/vmware/hostd/vmInventory.xml file on

the host to check which vm is registered. Make a note of the virtual machines and the datastores
on which they are located and then re-register the virtual machines.

less /etc/vmware/hostd/vmInventory.xml | grep -i vmx

This gives link to each VM registered to each ESX hosts in the format:
/vmfs/volumes/datastore/VirtualMachine folder/Virtual machine name.vmx

To re-register the virtual machine:

1. Right-click the Unknown VM entry and click Remove From Inventory.

2. Browse to the appropriate datastore for the virtual machine and open its folder
3. Right-click the *.vmx file and click Add to Inventory.
4. Power on the virtual machine.
5. Repeat Steps 1 to 4 for all unknown virtual machines

Find the owner of the locked file of a virtual machine

In ESXi 5.x and ESXi 4.1, to find the owner of locked file of a vm
# vmkvsitools lsof | grep Virtual_Machine_Name

You see output similar to:

11773 vmx 12 46 /vmfs/volumes/Datastore_Name/VirtualMachineName/VirtualMachineName-
flat.vmdk

then run this command to get the PID of the process for the vm
ps | grep Virtual_Machine_name

You can kill the process with this command:

kill -9 PID

To generate a core dump after killing the running virtual machine (but hung and
nonresponsive), use the command kill -6 PID or kill -11 PID.

In ESXi 4.1 and ESXi 5.x, Use k command in esxtop to kill a running virtual machine
process. On the ESXi console, enter Tech Support mode and log in as root.

a. Run the esxtop utility using the esxtop command.

b. Press c to switch to the CPU resource utilization screen.
c. Press Shift+f to display the list of fields.
d. Press c to add the column for the Leader World ID.
e. Identify the target virtual machine by its Name and Leader World ID (LWID).
f. Press k.
g. At the World to kill prompt, type in the Leader World ID from step 5 and press
Enter.
h. Wait 30 seconds and validate that the process is no longer listed.
ESXi 5 Unresponsive VM – How-to Power Off
Step 1 - connect via SSH by using puty for example and enter esxtop.
Enter ”esxtop”, then press “c” for the CPU resource screen and shift + V to display VMs only.

Step 2 – changing the display and locating the LWID number

Press “f” to change the display fields and press “c” to show the LWID (Leader World Id) and
press ENTER.

Step 3 – Use the k (kill) command with the number

Check the LWID column you can see the VM’s LWID number.
press “k” and enter the LWID number of the VM to stop.

Next time when the VM boots we can see this screen (depending on your guest OS of course).
 To Patch an ESXi 5.0 host from the command line:

“ Install “ command overwrites the existing packages. It may downgrade the existing packages,
Overwrite existing drivers.
“ Update” command is the recommended. It will update from lower version to new.

Patching Steps

1. Patches can be downloaded from the VMware patch portal. Select ESXi (Embedded and
Installable) in the product dropdown and click Search.

2. Click the Download link below the patch Release Name to download the patch.

3. Upload the patch to a datastore using the Datastore Browser from vCenter or a direct
connection to the ESXi 5.0 host using the vSphere client.

Note: VMware recommends creating a new directory on the datastore and uploading the patch
file to this directory.

4. Log into the local Tech Support Mode console of the ESXi 5.0 host.

5. Migrate or power off the virtual machines and put the host into maintenance mode.
# vim-cmd hostsvc/maintenance_mode_enter

6. Navigate to the directory on the datastore where the patch file was uploaded
# cd /vmfs/volumes/Datastore/DirectoryName
# ls

Where Datastore is the datastore name where the patch file was uploaded to, and
DirectoryName is the directory you created on the datastore.

o Install or update a patch on the host using these esxcli commands:

o To Install: Using local setup:

# esxcli software vib install -d "/vmfs/volumes/Datastore/DirectoryName/PatchName.zip"

o esxcli software vib install -d /vmfs/volumes/[DATASTORE]/[PATCH_FILE].zip

Where PatchName.zip is the name of the patch file you uploaded to the datastore.
# esxcli software vib install -d "/vmfs/volumes/datastore1/patch-directory/ESXi500-
201111001.zip"

o Using http setup:

# esxcli software vib install -v viburl

Where viburl is the URL to the http depot where VIB packages reside.

# esxcli software vib install -v

https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-
light/VMware_locker_tools-light_5.0.0-0.7.515841.vib
 To Update: Using local setup:
# esxcli software vib update -d "/vmfs/volumes/Datastore/DirectoryName/PatchName.zip"
Where PatchName.zip is the name of the patch file you uploaded to the datastore.

# esxcli software vib update -d "/vmfs/volumes/datastore1/patch-directory/ESXi500-

201111001.zip"

o Using http setup:

# esxcli software vib update -v viburl
Where viburl is the URL to the http depot where VIB packages reside.

o # esxcli software vib update -v

https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/tools-
light/VMware_locker_tools-light_5.0.0-0.7.515841.vib

7. Verify that the VIBs are installed on your ESXi host:

# esxcli software vib list

Name Version Vendor Acceptance Level Install Date

----------------- --------------------------- ------ ----------------------------------------------------
ata-pata-amd 0.3.10-3vmw.500.0.0.469512 VMware VMwareCertified 2012-05-04
ata-pata-atiixp 0.4.6-3vmw.500.0.0.469512 VMware VMwareCertified 2012-05-

8. After the patch has been installed, reboot the ESX host:
# reboot

9. After the host has finished booting, exit maintenance mode and power on the vms .
# vim-cmd hostsvc/maintenance_mode_exit

vcenter inventary service linked mode port is 10111

Editing the vm mac address in .vmx file through the GUI

To use the GUI to change the MAC address of a virtual machine:

1. Shut down the virtual machine.

2. Right-click the virtual machine and click Remove from Inventory.
3. Download the .vmx configuration file for the virtual machine to your desktop using the
Datastore Browser.
4. Open the .vmx configuration file with an plain text editor and add or change the
following entries:

ethernetN.checkMACAddress = “false”
ethernetN.addressType = “static”
ethernetN.Address = “XX:XX:XX:XX:XX:XX″

Where XX:XX:XX:XX:XX:XX is the new desired MAC address for the virtual machine.

5. Upload the new (modified) .vmx file back to the same location (datastore) using the
Datastore Browser.
 6. Copy the original .vmx file (using a name similar to vmname.vmx.old) for backup
purposes.
7. Register the virtual machine back to the inventory.
8. Start the virtual machine.

ethernet0.address = 00:50:56:XX:YY:ZZ

Where XX is a valid hex number between 00 and 3F and YY and ZZ are valid hex numbers
between 00 and FF. The value for XX must not be greater than 3F in order to avoid
conflict with MAC addresses that are generated by the VMware

What happens if a datastore fills up?

When a datastore runs out of space, thin-provisioned virtual disks can no longer dynamically
grow to accommodate additional storage demand. When VMware ESX detects this condition,
virtual machines in need of additional storage are instantly paused to prevent guest
operating systems from failing
Free up some space by deleting or moving files — ISO images.
powered-off non critical VMs or idle vms.
Resume one of the paused VMs
Use Storage VMotion to move the disks for that VM to another datastore
Resume the remaining VMs

What if it’s not possible to free enough space?

1. Powering off the virtual machine
This operation reduces the swap file. A virtual machine’s swap file is usually the same
size as the allocated amount of RAM. This operation does not have to be done to the
machine which is running off of the snapshot. If there are non-critical machines residing
on the same partition, they can be powered off to free up storage for the commit
operation.
2. Add an extent to the existing partition
If there is a lack of disk space, the Add Extent wizard can be used to increase the amount
of space available. The Add Extent operation is irreversible, and creates a dependence of
multiple LUNs for a single one.
3. Clone the virtual machine to a partition or storage that has more space

Improving Storage Performances

Storage vMotion is separated into two components: Data movers and Data mirroring.
Data movers read blocks from the original location and copy to a new destination.
Data mirroring writes data to both the original VMDK location and the new location.
The guest does not get a write confirm until the data has been written in both locations.

Data Movers
The ESX hypervisor uses one of three mechanisms, which affects the speed of the Storage
vMotion. The two different kind of data movers are: (s/w & h/w based)
fsdm - Software Datamovers
fs3dm - Software Datamovers
fs3dm-hw - Hardware offloading datamover

fsdm is the slowest.

fs3dm is a faster mechanism.
fs3dm-hw is a VAAI hardware offload, it offloads the copy to the SAN or NAS layer.
The hypervisor chooses the data mover to use based on a specific set of conditions. If the source
and destination datastore are in the SAN, uses the same VMFS block size, and the SAN/NAS is
VAAI capable, then it uses fs3dm-hw.
If all the above conditions are met but the SAN is not VAAI capable, or it uses fs3dm.
If the block sizes are different, then it uses the oldest (and slowest) fsdm mechanism.

How do I know if VAAI is enabled?

To check if VAAI is enabled using the vSphere Client:
select hostConfiguration Software  Advanced Settings

DataMover.HardwareAcceleratedMove - options are set to 1 (enabled):

DataMover.HardwareAcceleratedInit - options are set to 1 (enabled):
VMFS3.HardwareAcceleratedLocking - options are set to 1 (enabled):
Note: These options are enabled by default

Storage IO Control

It dynamically allocates portions of hosts’ I/O queues to VMs running with shares assigned
to the VMs. Administrators can set priority for vm with critical workloads during peak load
periods (by means of disk shares)
Setting I/O priorities for VMs results in better performance during periods of congestion.
If you try to customize threshold, you can click Advanced button.
Esxi 5.0

Network I/O Control (Net IOC)

New in vSphere 4.1, Network I/O Control (NetIOC) allows the allocation of network bandwidth
to six network resource groups: vMotion, NFS, iSCSI, Fault Tolerance, virtual machine, and
management. Network bandwidth can be allocated using either shares or limits:

􀂄 Using shares, each network traffic type is allowed the network bandwidth equal to the share
value. With shares, unused bandwidth is available for use by other traffic types.
􀂄 Using limits, a maximum bandwidth utilization (in Mbps) is set for each traffic type.
Unused bandwidth is not available for other traffic types.

This feature can guarantee bandwidth for specific needs and can prevent any one traffic type
from impacting the others

Setting Priority for VMs

Giving Priority for VMs to restart when a host is down or isolated , .
right-click the cluster click on Edit Settings.
  Leave Powered On: If the host is isolated, the VM remains powered on.
 Power Off: If the host is isolated the VM is powered off. This is a hard power off.
 Shut down: the VM is shut down gracefully using VMware Tools. If not shutdown
completed within 5 minutes, a power off is immediately executed. If VMware Tools is
not installed, a power off is executed instead.
 Use Cluster Setting: VMs are restarted based on the setting given at the cluster level.

Disk performance
2-5 ms latencies are healthy storage 5-12 ms latencies reflecting a healthy storage architecture
were data is being randomly read across the disk,
15 ms latencies or greater possibly representing an over-utilized or misbehaving array.

 esxtop – Provides real time CPU, memory, disk and network statistics for virtual
machines and their host. This utility is accessible from a direct connection to an ESXi
host. It does not report statistics for NFS datastores.
 resxtop – A remote version of esxtop. It is included as part of the Linux vCLI (vSphere
Command Line) and the vMA (vSphere Management Assistant). Discussed below,
resxtop has three modes of operation including Interactive, Batch, and Replay. Both
esxtop and resxtop may only connect to a single ESXi host and requires root level access.

Use esxtop/resxtop when troubleshooting virtual machine CPU performance issues.

VMs with multiple vCPUs (SMP), statistics can be displayed in an aggregate form or per vCPU

To access a virtual machines CPU performance statistics using esxtop/resxtop, perform the
following steps. Note: For more information on establishing an esxtop/resxtop session

1. Establish an SSH (Putty) connection to either an ESXi host or the vMA.

2. Enter resxtop –server <server>. For example: resxtop –server tfl-esxi02.thefoglite.net
3. Enter c for the CPU view.
4. Enter f to manipulate which fields to display.
5. Enter V to switch to display virtual machine specific metrics.
6. Enter e and the GID of a specific VM to display worlds for that VM.
%RDY - Amount of time a VM’s vCPU was ready to perform an operation but could not get
scheduled on the ESXi host’s physical CPU. This indicate not enough physical CPUs in the Esxi
host.

5% - 1000ms 10% - 2000ms 100% - 20,000ms

If ready time <= 5 percent, this is normal.

If ready time is between 5 and 10 percent, ready time is start watching now.
If ready time is > 10 percent action is required to address performance issues.

What is CPU Ready?

CPU ready is the time it takes a VM to be scheduled on physical core after it is placed in the
CPU scheduling queue.

What is High CPU Ready?

In my opinion, during peak load, anything above 2% (or 400ms) is a concern and should be
monitored. Above 5% will be impacting performance (resulting in lower CPU utilization) and
10% or more, should be considered a serious problem and remediated immediately

CPU Scheduler
The scheduler is a VMKernal component .It schedules VM’s virtual CPUs requests to the
physical CPUs of the host server. When a VM uses its virtual CPU, the VMkernel has to find a
free physical CPU (or core).
The scheduler’s job is to find CPU time for all the VMs that are requesting it and to do it in a
balanced way, so performance for any one VM does not suffer.

Single vCPU vs Multiple vCPUs

To check multiple vCPUs assigned to your VM is causing poor performance:
 Open a console prompt on the ESX host or initiate an SSH connection to it. .
 Type esxtop and press Enter.
 On the CPU screen, check the %CSTP value. If this number is higher than 100, the
performance issues may be caused by the vCPU count. Try lowering the vCPU count of
the virtual machine by 1.
Note: The %CSTP value represents the amount of time a virtual machine with multiple virtual
CPUs is waiting to be scheduled on multiple cores on the physical host. The higher the value, the
longer it waits and the worse its performance. Lowering the number of vCPUs reduces
the scheduling wait time.
To lower the vCPU count:
Note: The virtual machine must be powered off to perform these steps.
Right-click on the virtual machine and click Edit Settings.
Click CPUs.
Use the Number of virtual processor drop-down to lower the vCPU count by 1.
Click OK.
If your virtual machine still experiences performance issues, and if its kernel or HAL can handle
switching to a single vCPU, lower the vCPU count to 1.
Warning: If your virtual machine's kernel or HAL cannot handle switching to a single vCPU,
unexpected behaviour may occur.
 A high value indicates that more CPU resources are being requested by VMs than are
available on the host and queueing is likely. If %RDY exceeds 10% consistently, a
contention issue with other VMs is apparent and should be investigated.

1. %USED - Percentage of the host’s physical CPU cycles are being used by a VM. A high
value does not necessarily indicate an issue unless coupled with queueing. A high
%USED value in conjunction with a high %RDY value could indicate that the host is
overcommitted.
2. %WAIT - Amount of time the VM spent in a blocked or busy wait state. The VM is
likely waiting for a vmkernel operation to complete before it can be scheduled. Note that
this includes idle time.
3. %MLMTD – Idle time due to a configured CPU limit.
4. %CTSP - Amount of time an SMP VM was ready to run, but experienced a delay due to
vCPU scheduling contention.

Is an ESXi host actively swapping VM memory?

From the vSphere Client, connect to vCenter Server.
Go to the Hosts and Clusters view (Ctrl+Shift+H).
Select host  click the Performance tab.
Select the Advanced view, click Chart Options.
Under Chart Options, select Memory > Real-time.
The default value for Chart Type, Line graph is fine.
Under counters, select Swap in rate and Swap out rate.
Click Apply, OK.
If the value for either of these counters is greater than zero, then the host is most likely actively
swapping.
If a host is actively swapping, use esxtop to check which VMs performance is being degraded
by host swapping. The %SWPWT gives which VM is being affected.
A value greater than zero says a VM is waiting for memory pages to be swapped back in from
the host.
Adding a vm to the Inventory – (How to add vm under host )
Right click the vmx file and add

It will open a wizard… choose the host …finish.

Click next next and finish

Enable ssh

Through console
Setting the number of cores per CPU in a virtual machine
Power off the virtual machine--Right-click on the virtual machine -- click Edit Settings.
Click Hardware and select CPUs. -- Choose the number of virtual processors.
Click the Options tab -- Click General, in the Advanced options section.
Click Configuration Parameters  Include cpuid.coresPerSocket in the Name column.
Enter a value (try 2, 4, or 8) in the Value column.

Notes: Ensure that the number of vCPUs is divisible by the number of cpuid.coresPerSocket in
the virtual machine. That is, when you divide the number of vCPUs by the number of
cpuid.coresPerSocket , it must return an integer value. For example, if your virtual machine is
created with 8 vCPUs, coresPerSocket can only be 1, 2, 4, or 8.

The virtual machine now appears to the operating system as having multi-core CPUs with the
number of cores per CPU given by the value that you provided in step 9.

1. Click OK.
For example:

Using 4 vCPU
Configuration you want Settings needed for this configuration
Number of Set
Number of cores per Total Set vCPU cpuid.coresPerSocket/soc
sockets socket cores to: kets to:
1 4 4 4 4
2 2 4 4 2

Using 8 vCPU
 Configuration you want Settings needed for this configuration
Number of Set
Number of cores per Total Set vCPU cpuid.coresPerSocket/soc
sockets socket cores to: kets to:
1 8 8 8 8
2 4 8 8 4
4 2 8 8 2
Notes:

 To assign more than 4 vCPUs or if the processor supports more than 6 cores per
processor, you should have Enterprise Plus license, which supports up to 8 vCPUs and 12
cores per processor.
 Only values of 1, 2, 4, 8 for the cpuid.coresPerSocket are supported for the multi-core
vCPU feature in ESX 4.x.
 In ESX 4.0, if multi-core vCPU is used, hot-plug vCPU is not permitted, even if it is
available in the UI.
 Only HV 7 virtual machines support the multi-core vCPU feature.

http://www.petri.co.il/vsphere-hot-add-memory-and-cpu.htm

Increasing RAM , CPU

If it is grayed, we can’t modify the amount of RAM

Enabling Hot-Add and Hot-Plug

To enable go to your vSphere Client and power off the VM you want to activate
Hot-Plug/Hot-Add on.
In the Virtual Machine Properties Options  Memory/CPU Hotplug.
Memory Hot Add  “Enable memory...” option.
CPU Hot Plug Enable CPU...” option. Then click OK.

How to Hot-Add RAM and Hot-Plug CPU to a Running VM

go to the Summary  Edit Settings  Hardware tab select Memory.

To add more CPUs. click CPUs from the list of Hardware.
Add CPUs by selecting a number from the drop-down list at the right (e.g. we changed ours from
1 CPU to 3 CPUs)

Click OK to proceed.

Now CPU & Memory increased in Summary tab.

Adding the LUN to ESXI

login ESXI host  Congifuration  Storage Adapters

first item will be the iScsi initiator. Select it, and click on “Properties” in the below section

first enable iScsi initiator. click “Configure” in the properties dialog.

Check the “Enable” tickbox, and in the iSCSI Name field, you will see the name of the iSCSI
initiator (Which you will use on your san for LUN masking) and click “Ok”
On the next Tab (Dynamic Discovery), click on “Add” and enter the IP address of one of your
iSCSI ports of your san, mine is 10.1.1.27

Click “Ok” and then “Close”

vSphere will prompt you to rescan the iSCSI host for any new luns available, click “Yes”
Once the adapter is rescanned (Takes a couple of seconds), your LUN will be shown in the list
view below.

Now SAN LUN is Added to ESXI.

Hardware  Storage Add Storage”

In the Add Storage wizard first page, select “Disk/LUN” and click Next

Your Added LUN will be in the list, select it, and click “Next”
Next will be a summary of your disk layout, click next

On the next page you will see the file formatting config, I left the defaults, and clicked next

Once finished the wizard, you will see your LUN storage ready in the storage list.

Deselect Maximum Capacity check box and enter the LUN size if you want to decrease this
VMFS datastore

How to Add LUN to VMware ESX

Step 1
Select the ESX host and then click the "Configuration" tab.
Step 2
Select "Storage Adapters" from under Hardware. Click "Rescan."
Step 3
Confirm "Scan for New Storage Devices" is selected click "OK" to assign LUN to the host.
Step 4
Configuration tab Storage to open the Add Storage Wizard.
Step 5
Select "Disk/LUN" click "Next." Select the LUN from the list of devices and then click "Next."
Click "Next" again.

Step 6
give a name for new volume. Click "Next." Choose a block size -- 1, 2, 4 or 8MB -- from the
drop-down menu or use the default setting.
Step 7
Deselect "Maximum Capacity" and then enter the desired storage size of the volume, if
preferred. Click "Next."
Step 8
Click "Finish" to add the new volume to VMware ESX.

Un-presenting LUN
ESX/ESXi 4.1

The best practices for removing a LUN from an ESX 4.1 host

1. Unregister all objects from the datastore including VMs and Templates
2. Ensure that no 3rd party tools are accessing the datastore
3. Ensure that no vSphere features, such as Storage I/O Control, are using the device
4. Mask the LUN from the ESX host by creating new rules in the PSA (Pluggable Storage
Architecture)
5. Physically unpresent the LUN from the ESX host using the appropriate array tools
6. Rescan the SAN
7. Clean up the rules created earlier to mask the LUN
8. Unclaim any paths left over after the LUN has been removed

Esxi 5.0

1. Unregister all objects from the datastore including VMs and Templates
2. Ensure that no 3rd party tools are accessing the datastore
3. Ensure that no vSphere features, such as Storage I/O Control or Storage DRS, are using
the device
4. Detach the device from the ESX host; this will also initiate an unmount operation
5. Physically unpresent the LUN from the ESX host using the appropriate array tools
6. Rescan the SAN

To change the 1mb block size

Migrate the virtual machines to another datastore(s), delete the existing datastore, and re-create
it using VMFS-5.

The main change with VMFS 5 is the unified block size, which is 1 MB.
VMFS 3 was able to format at 1, 2, 4, or 8 MB
With VMFS 5 supporting 1 MB block sizes, the maximum sizes for Virtual Machine Disk
Formats (VMDKs) are not limited to 256 GB like previous block sizes

VMFS 3 datastores can be upgraded, but it retain their block size. So reformat the volume to
VMFS 5 at 1 MB (the default size).
Another important change is related to the sub-block algorithm allocation. VMFS implements a
unique sub-block algorithm that works well for the polar distribution of file types: large VMDKs
and small VMX and others. The previous sub-block was 64 KB
Upgrading VMFS 3 to VMFS 5

Migration Between Clusters

You can migrate VM’s between vSphere clusters (even between different versions) as long as
below conditions are met:

vSphere clusters must be managed by single vCenter server

vSphere clusters must be within single DataCenter object in vCenter server
Pre vSphere 5.1 – clusters must have an access to the same datastore.
vMotion network is stretched between clusters.
Processors must be from the same vendor (Inter or AMD) and family (model) on both clusters or
both clusters have common EVC baseline applied.
Virtual Machine hardware version is supported by hypervisor – very useful during migration to
new hardware or new version of vSphere platform.
If you have vDS implemented, make sure dvportgroup is span across both clusters

LUN paths
vmhba0:C0:T0:L23
Adapter: vmhba0
Channel: 0
Target: 0
LUN: 23

esxcli storage core path list gives a list of all LUN paths currently connected to the ESXi host.
esxcli storage core device list gives a list of LUNs currently connected to the ESXi host.

For example, vmhba0:2:3:1 refers to the first partition on LUN 3, target 2, HBA 0.

The block size defines the maximum size of a file on the datatore (e.g. a virtual disk file - .vmdk
if you select a 1MB block size on your data store the maximum file size is limited to 256GB. So
when you create a VM you cannot assign it a single virtual disk greater then 256GB.
1MB block size – 256GB maximum file size
2MB block size – 512GB maximum file size
4MB block size – 1024GB maximum file size
8MB block size – 2048GB maximum file size

To change the block size of a vmfs filesystem, use vmkfstools to reformat the partition.
The command is : vmkfstools --createfs vmfs3 --blocksize 8M vmhba0:0:0:3
It will destroy any data on the partition, so make sure you move your data before you do this

With VMFS-5 this is history and the unified block size for all file sizes is 1MB.