SER1289BE

vSphere Troubleshooting
t i o n
Tips and Tricks i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w

Javier Menendez & Francis Daly

#VMworld #SER1289BE
Agenda

Use of the VMware vSphere Command-line Interface (vCLI),

1 t i o n
ibu
the ESXi Shell, and Log Files to Diagnose and Correct Problems
i s tr
or d
2 t ion
Troubleshooting Networking Issues
bli c a
r p u
fo
3
n Not
Troubleshooting Storage Issues
t:
ont e
4 1 7 CvSphere HA Cluster and Performance Issues
d0
Troubleshooting
2
or l
VMw

#SER1289BE CONFIDENTIAL 3
 t i o n
i s tribu
or d
Troubleshooting Overview t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w
Troubleshooting Process
The troubleshooting process consists of the following tasks:

t i o n
i s tribu
or d
t ion
Defining Identifying the cause bli c a Solving
p u
the problem of the problem
t f or the problem
: N o
t en t
• Identifying symptoms 17 Co• nIdentifying possible • Identifying possible
d 2 0
l causes solutions
• Gathering
informationVM
wor • Determining the root • Implementing the
cause best solution

#SER1289BE CONFIDENTIAL 6
Solving the problem …
Can I do it with the Web Client or should I use the esxi shell?

t i o n
tr ibu
What are my choices:
or di s
t ion
bli c a
r p u
• Can I fixNthe
o fo
t problem with the Web, Host or
en t:
ont
vSphere
C
Client?
1 7
o rl d 20 • Do I want or need to use the command
VM w line utilities?

#SER1289BE CONFIDENTIAL 7
 t i o n
i s tribu
or d
a t i on
Troubleshooting Tools and pChoices u blic
t f or
: N o
t en t
C on
2 017
o r ld
VMw
Command-line Troubleshooting Choices
Choices to obtain command-line access on a VMware® ESXi™ host:

t i o n
i s tr ibu
VMware vSphere® ESXi™ Shell vSphere or d
t i n
oAssistant
Management
ic
bla (VMA)
p u
t or
f
: o
N • Appliance deployed via ovf
• Can be access via:
ten t
C o n
2 017
• r ld via ALT + F1)
1. The DCUI (if enabled
o
• Contains vCLI package
VMw
• 2. Via putty after enabling ssh. • Does NOT use port 22.

#SER1289BE CONFIDENTIAL 9
The esxcli Command

The esxcli command

offers the following t i o n
namespaces, i s tribu
or d
as well as many new
t ion
esxcli functions added in bli c a
r p u
the vSphere 6.5.
o t fo
n t: N
ont e
1 7 C
If working with vSAN,
o d 20
rl
check out the new
VM w
esxcli
vsan debug namespace

#SER1289BE CONFIDENTIAL 10
Example: The esxcli vsan Namespace

t i o n
i s tribu
or d
The vSAN related
t ion
commands are found in bli c a
r p u
esxcli vsan
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w

#SER1289BE CONFIDENTIAL 11
New commands added with every new vSphere release.

Ex: The esxcli vsan debug Namespace

t i o n
i s tribu
or d
The new vSAN debug
t ion
related commands are bli c a
r p u
amazing to discover the
o t fo
state of your objects, vms, n t: N
etc. ont e
1 7 C
o rl d 20
VM w

#SER1289BE CONFIDENTIAL 12
Tip: Remember the esxcli esxcli command list command

esxcli esxcli command list t i o n

displays all the command
i s tribu
options. or d
t ion
bli c a
r p u
o t fo
To quickly find the command
n t: N
needed, combine it with the
ont e
grep command
1 7 C
o r l d 20
VM w

#SER1289BE CONFIDENTIAL 13
The VMA’s vicfg-* Commands

vicfg- [tab] [tab]

• Commands with the vicfg- prefix enable you to manage t i o n
i s tribu
your storage, network, and host configuration.
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o r l d 20
VM w
• Type vicfg- followed by the TAB key twice so see all the available commands.
• For example, vicfg-cfgbackup allows you to back up the esxi settings in a few seconds.

#SER1289BE CONFIDENTIAL 14
The esxi shell’s esxcfg- Commands are still there

esxcfg- [tab] [tab]

• Commands with the esxcfg- t i o n
i s tribu
prefix enable you to manage
or d
your storage, network, and
t ion
host configuration. bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w
• Type esxcfg- followed by the TAB key twice so see all the available commands.
• Remember the –h (help) option to master these commands.

#SER1289BE CONFIDENTIAL 15
Tip: Remember the “-h” option for esxcfg-* commands

Use the –h option (for help)

t i o n
to see options for this
i s tribu
particular command and in or d
t ion
some cases, even
bli c a
examples on how to use the
r p u
command. o t fo
n t: N
How many times did you ont e
17 C
struggle to create a
2 0
ld
orcli?
vmkernel port with the
VMw

#SER1289BE CONFIDENTIAL 16
Manipulating virtual machines with the cli from the V.M.A.?

vmware-cmd
t i o n
• The vmware-cmd command is used to interact with virtual machines.
• vmware-cmd
i s tribu
– The vmware-cmd command is exclusively used for virtual machines. o r d
t io n
bli c a
p u
t f or
: N o
t en t
C on
2 017
o r ld
VMw

#SER1289BE CONFIDENTIAL 17
Example: Listing your Virtual Machines

vmware-cmd -l
t i o n
• Lists the virtual machines that are located on the target host.
i s tribu
Lists virtual machines by path to the .vmx file.
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o r l d 20
VM w

#SER1289BE CONFIDENTIAL 18
Tip: Learn More On Your Own

Type vmware-cmd --help to vmware-cmd allows you to The esxi hostnhas a similar
u t i omanipulate vms
see what the command allows start a vm, take snapshots, command
i s t r ib to
you to do. register a vm, etc.
o d vim-cmd vmsvc.
r called
a t i on
u blic
or p
o t f
n t: N
o nte
17 C
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 19
Manipulating virtual machines from the esxi shell

t i o n
i s tribu
or d
vim-cmd vmsvc can t ion
perform similar bli c a
r p u
tasks from the o t fo
esxi shell. n t: N
ont e
Wondering what 017 C
the destroywor l d 2
VM
option does?
Better try it on
a test vm.

#SER1289BE CONFIDENTIAL 20
Tip: Location of VMware vCenter Server 6.5 Log Files

The VMware vCenter Server The VMware vCenter Server

t i o n
6.5 logs are located in the Appliance 6.5 logs are
t r i bu
%ALLUSERSPROFILE%\VM located in the r dis
o
Ware\vCenterServer\logs ion
/var/log/vmware tfolder.
c a
bl i
folder.
r pu
o t fo
n t: N
ont e
1 7 C
o rl d 20Recommendation
VM w
See http://kb.vmware.com/kb/2110014 and 1021806
for additional details.

#SER1289BE CONFIDENTIAL 21
Useful ESXi Host Logs for Troubleshooting

ESXi hosts write to multiple log files,

t i o n
ibu
depending on which action is being performed.
di s tr
on or
Log file a t i
Purpose
p ublic
hostd.log
r
Host Management service logs
fo
N o t
Management service initialization, watchdogs, scheduled
syslog.log
en t:
ont
tasks, and DCUI use

1 7 C Core VMkernel logs, including device discovery, storage

2 0
vmkernel.log
d
and networking device and driver events, and virtual
l
VM wor machine startups
A summary of warning and alert log messages excerpted
vmkwarning.log
from the VMKernel logs
A summary of ESXi host startup and shutdown, and an
vmksummary.log hourly heartbeat with uptime, number of virtual machines
running, and service resource consumption

#SER1289BE CONFIDENTIAL 22
Tip: Other Critical Logs to Be Aware Of and how to view them

/var/log/fdm.log displays HA /var/log/shell.log displays

o n
/var/log/clomd.log displays vSAN
ti information for
cluster related events and commands executed inside b u
health irelated
tr among the output
notifications. of the esxi shell. r di s
devices
on o
c a t i
p u bli
t for
: N o
ten t
C o n
2 017
w o rld
V M

#SER1289BE CONFIDENTIAL 23
Tip: Remember /var/log

t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
Lots of logs available in
n t: N
this directory
ont e
1 7 C
o r l d 20
VM w

#SER1289BE CONFIDENTIAL 24
Collecting Diagnostic Data for
VMware Technical Support
t i o n
Methods for collecting diagnostic information
i s tribu
to send to VMware Technical Support include
or d
the following: t ion
bli c a
• Use the GUI to export files to a log bundle.
r p u
− vSphere Client or vSphere Web Client o t fo
n t: N
• Use the vm-support command
o nt e to collect
1 7 C ESXi host.
information from an individual
d 2 0
w o rl
VM

#SER1289BE CONFIDENTIAL 25
Tip: Running vm-support when engaging support

t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w

#SER1289BE CONFIDENTIAL 26
 t i o n
Network Troubleshooting i s tribu
or d
Commands t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w
Tip: Useful Networking Related Commands

Use esxcfg-vmknic –l or Use esxcfg-vswitch –l or n

u t i o
vicfg-vmknic -l to view your vicfg-vswitch –l to view virtual
t r i b
vmkernel port related switch related information. rIsdis
o n o
information. Are they still there the switch connected
c a t i to at
and enabled? ub l
least one uplink? i
o r p
N otf
en t:
t
7
Use esxcfg-nics –l or1vicfg-Con esxcli also has options to
2 0
o ld related
nics –l to view ruplink perform these tasks.
V MwAre the nics up or
information.
down? For example:
esxcli network nic list is similar
to esxcfg-nics -l

#SER1289BE CONFIDENTIAL 28
Tip: Did You Know That…

Somebody could accidentally If so, type esxcli network nic up t i o n

t r i bu
down a nic? –n vmnic2. is
r d
on o
c a t i
r p ubli
tf o
: N o
t en t
17 Con
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 29
Tip: Did You Know That…

t i o n
Somebody could disable a vmkernel port
t r i bu
If so, type esxcfg-vmknic -e to re-
with commands like esxcfg-vmknic -D? is
enable the vmkernel port.
r d
on o
c a t i
r p ubli
tf o
: N o
t en t
17 Con
2 0
w o rld
V M

#SER1289BE CONFIDENTIAL 30
Tip: Rebuilding the Management Network from scratch

Somebody could accidentally t i o n

disconnect an uplink from a i s tribu
or d
virtual switch?
t ion
bli c a
r p u
o t fo
n t: N
ffaaa
ont e
1 7 C
o r l d 20
VM w

#SER1289BE CONFIDENTIAL 31
Need a packet sniffer? Remember pktcap-uw

t i o n
i s tribu
or d
t ion
bli c a
r p u
There is a built-in packet o t fo
n t: N
sniffer called pktcap-uw
ont e
and an older one called
1 7 C
tcpdump-uw.
o r l d 20
VM w

#SER1289BE CONFIDENTIAL 32
 t i o n
i s tribu
or d
Storage Troubleshooting t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w
Tip: Did You Know That…

You can see how many dastastores exist and how full
t i o n
the datastores are? This command displays vmfs, nfs,
i s tribu
vsan and vvol datastores. There is also a vdf –h or d
t ion
command.
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w

#SER1289BE CONFIDENTIAL 34
Tip: Did You Know That…

t i o n
i s tribu
or d
t ion
You can easily view all
bli c a
sorts of information
r p u
with esxcli?
o t fo
n t: N
ont e
1 7 C
o r l d 20
VM w

#SER1289BE CONFIDENTIAL 35
Tip: Use esxcli Storage Core Device Smart for SSD information

vSphere 6.5
Includes t i o n
i s tribu
Use this command to
or d
t ion
retrieve data about a
bli c a
specific SSD device. r p u
o t fo
Critical for vSAN,Flash
n t: N
Read Cache or any ont e
1 7 C
d 20
datastore using SSDs.
o rl
VM w

36 36
Tip: Be familiar with partedUtil

t i o n
i s tribu
or d
partedUtil is the command
t ion
that partitions luns. It can also
bli c a
r p u
create, resize and delete
o t fo
partitions.
n t: N
ont e
1 7 C
o r l d 20
VM w

37
Tip: Suspect metadata corruption?

t i o n
i s tribu
voma is the on-disk metadata or d
t ion
analyzer for vmfs datastores.
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w

#SER1289BE CONFIDENTIAL 38
 t i o n
Troubleshooting Clusters and i s tr ibu
o r d
Performance t io n
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o rl d 20
VM w
vSphere HA
A reliable network connection between the hosts and VMware® vCenter Server™
is essential for enabling vSphere HA.
t i o n
i s tr ibu
or d
a t i on Heartbeat

u blic Datastores
r p
o t fo
FDM
n t:
FDM N FDM

C onte
vpxa hostd
1 7 vpxa hostd vpxa hostd

orl d 20
ESXi host (slave) ESXi host (slave) ESXi host (master)

VMw Management
Network
vCenter vpxd
Server

40
Tip: Remember F.D.M

t i o n
There is a log for HA that
i s tribu
you can’t see with the Web or d
t ion
or vSphere Client?
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o r l d 20
VM w

#SER1289BE CONFIDENTIAL 41
Tip: Did You Know That…

You can stop, start or restart the HA service from the cli if needed?

t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
n t: N
ont e
1 7 C
o r l d 20
VM w

#SER1289BE CONFIDENTIAL 42
Tip: You can restart all services at once if needed …

t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
Remember services.sh restart
n t: N
ont e
1 7 C
o r l d 20
VM w

#SER1289BE CONFIDENTIAL 43
Tip: Did You Know That…

esxtop is an amazing tool for esxtop has options for CPU esxtop has annoption for vSAN
u t i o
performance troubleshooting? (c), RAM (m), Disk (d,u,v)
i s ib
(-x option)?
tr
and Network (n) related
or d
statistics? tion
l i c a
pu b
for
N o t
en t:
C ont
1 7
o rl d 20
VM w

#SER1289BE CONFIDENTIAL 44
Tip: Did You Know That…

t i o n
i s tribu
or d
t ion
bli c a
r p u
o t fo
esxtop has built-in help?
n t: N
ont e
1 7 C
o r l d 20
VM w

#SER1289BE CONFIDENTIAL 45
Tip: Did You Know That…

t i o n
i s tribu
or d
You can add more
t ion
output to esxtop by bli c a
r p u
pressing “f”?
o t fo
n t: N
For example, you can ont e
1 7 C
add NUMA related
information (G Option)orld
20
VMw

#SER1289BE CONFIDENTIAL 46
Tip: Learn the vsish command and its capabilities

t i o n
vsish has hundreds of i s tribu
or d
subcommands.
t ion
bli c a
Just type the command and r p u
take a look … o t fo
n t: N
ont e
You can type commands like 7
1 C
2 0
cd to change directories
o r ld and
Mw
get to view information.
V

#SER1289BE CONFIDENTIAL 47
Tip: Learn the vsish command and its capabilities

t i o n
vsish has hundreds of i s tribu
or d
subcommands.
t ion
bli c a
r p u
o t fo
n t: N
Just type the command and
ont e
take a look …
1 7 C
o r l d 20
VM w

#SER1289BE CONFIDENTIAL 48