Scribd
Upload
604 views4 pages

VMware KB - Poor Network Performance or High Network Latency On Windows Virtual Machines

This document discusses symptoms of poor network performance and high latency on Windows virtual machines. It identifies three potential causes: the power plan setting, high CPU ready time indicating overcommitment, and improper configuration of Receive Side Scaling (RSS). The resolution section provides steps to check and address each potential cause, including changing the power plan, using esxtop to check CPU ready time, and ensuring RSS is properly enabled if supported by the hardware and operating system.

Uploaded by

Raj232
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
604 views4 pages

VMware KB - Poor Network Performance or High Network Latency On Windows Virtual Machines

This document discusses symptoms of poor network performance and high latency on Windows virtual machines. It identifies three potential causes: the power plan setting, high CPU ready time indicating overcommitment, and improper configuration of Receive Side Scaling (RSS). The resolution section provides steps to check and address each potential cause, including changing the power plan, using esxtop to check CPU ready time, and ensuring RSS is properly enabled if supported by the hardware and operating system.

Uploaded by

Raj232
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

KnowledgeBase

PoornetworkperformanceorhighnetworklatencyonWindowsvirtualmachines(2008925)
Symptoms
AvirtualmachinewithonevirtualCPUandahighCPUload,oravirtualmachinewithtwoormorevirtualCPUsingeneralandaWindows2003/2008/7guestoperatingsystem,mayexperiencethesesymptoms:
Poornetworkperformanceand/orhighpingresponsetimes:
Whenreceivingnetworktraffic(regardlessoftheamountofdataandtype)
WhileunderhighCPUload,orsharingCPUresourceswithhighlyutilizedvirtualmachines
Observedthroughputmaydecreaseto512kB/sonGigabitEthernet.Timeoutsandconnectivitydisruptionsmayalsobeobserved.
Pingrepliesmaytakeupto20seconds.
Sensitiveserviceslikedatabaseserversmayperformpoorlyortimeout.
Thenumberofvirtualandphysicalnetworkcardshasnoeffectonthisissue.
Thisissueoccurswithdifferentvirtualnetworkadaptertypes(E1000,VMXNET2andVMXNET3).
Measuredperformanceresults(generatedwithtoolslikeiperf)mayworsenwhenaddingmorevirtualCPUstothevirtualmachine.

Cause
Therearethreepossiblecausesforthisissue:
Powerplan
OnWindows2008and2008R2,thepowerplanissettoBalancedbydefault.MicrosofthasobservedandconfirmedthatchangingthepowerplanfromBalancedtoHighPerformancemayincreaseoverall
performance.Formoreinformation,seeGeneralGuidelinesforImprovingOperatingSystemPerformance(http://msdn.microsoft.com/enus/library/ee377075.aspx).Aggressivepowersavingplanscanadverselyaffect
performance,especiallywithlatencysensitiveapplicationslikewebanddatabaseservers.
HighCPUreadytime(alsoreferredas"%RDY"or"%RDYtime")
Note:Thisinformationissimplifiedanditsonlypurposeistoillustratethecauseofthedescribedissue.Itshouldnotbereferencedoutsideofthecontextofthisdocument.Althoughtheexampleweusehere
issufficienttodescribethecauseofthestatedissue,itdoesnotclaimtobetechnicallycorrectineverydetailduetothecomplexityoftheCPUschedulingprocessforvirtualmachines.
Thisexampleisbasedontheseassumptions:
AnESXi5.0hostwithahyperthreadingenabledquadcoreCPU,resultingin4physicaland8logicalCPUs
AWindows2008R2virtualmachinewith4virtualCPUs
ThreeWindows2003virtualmachineswith2virtualCPUseach
Inthisconfiguration,theESXihostexposes8logicalCPUsas10virtualCPUstothevirtualmachines.Inotherwords,theESXihostisovercommitted.Dependingontheutilizationofthevirtualmachines,
theESXihostwillnotbeabletoprovideallvirtualmachineswiththerequestedCPUtime,thustheperformanceofthevirtualmachineswillbeasexpected.However,iftheloadonmultiplevirtualmachines
increases,theESXihosthastodecidewhichvirtualmachinewillbeservedfirstwiththecurrentlyavailableCPUtime.ItisimportanttonotethattheESXihostwillservemulticorevirtualmachinesonly
whenitisabletoserveallthevirtualCPUsfromtheparticularvirtualmachineatonce.OtherwiseavirtualmachinewithalowernumberofvirtualCPUswillbeservedfirst.AlthoughthisishowtheCPU
schedulerissupposedtowork,itcanalsoleadtosituationswherecertainvirtualmachineshavetowaitforanunreasonableamountoftimefortherequestedCPUtime.Insuchcasesyoucanobservea
degradedoverallperformanceandincreasedresponsetimes.FormoreinformationaboutCPUschedulingandthemeaningoftheCPU%RDYtime,seetheVMwareTechnicalPaper,Performance
TroubleshootingforvSphere4.1(http://www.vmware.com/resources/techresources/10179).
ReceiveSideScaling(RSS)
RSSisamechanismwhichallowsthenetworkdrivertospreadincomingTCPtrafficacrossmultipleCPUs,resultinginincreasedmulticoreefficiencyandprocessorcacheutilization.Ifthedriverorthe
operatingsystemisnotcapableofusingRSS,orifRSSisdisabled,allincomingnetworktrafficishandledbyonlyoneCPU.Inthissituation,asingleCPUcanbethebottleneckforthenetworkwhileother
CPUsmightremainidle.
Note:TomakeuseoftheRSSmechanism,thehardwareversionofthevirtualmachinemustbe7orhigher,thevirtualnetworkcardmustbesettoVMXNET3,andtheguestoperatingsystemmustbecapable
andconfiguredproperly.Onsomesystemsithastobeenabledmanually.TheseoperatingsystemsarecapableofusingRSS:
Windows2003SP2(enabledbydefault)
Windows2008(enabledbydefault)
Windows2008R2(enabledbydefault)
WindowsServer2012(enabledbydefault)
Linux2.6.37andnewer(enabledbydefault)
ForfurtherinformationonMicrosoftRSS,seeReceiveSideScaling(RSS)(http://technet.microsoft.com/enus/library/hh997036.aspx).FormoreinformationonLinuxReceiveSideScaling,seeRSSandmultiqueue
supportinLinuxdriverforVMXNET3(2020567)(/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2020567).

Resolution
Beforeproceedingwiththesesteps,ensurethat:

therearenoproblemsinyouexternalinfrastructurelikefaultyhardwareorpossiblemisconfigurations(commonconfigurationproblemsareIPconflicts,unintendedtrafficshaping,misconfiguredtrunkand
EtherChannelports)
thenetworkisnotcongested
thenetworktheESXihostisonisstableandperformsasexpected
thevirtualmachinesareconfiguredwiththeVMXNET3networkadapter
thehardwaredriversandfirmwareversionsarerecent
theBIOSisrecentandconfiguredappropriately
thevirtualmachineisrunningthelatestversionofVMwareTools(theycontainthedriversforthevirtualhardware)
anysecuritysoftwarelikeintrusiondetection/preventionsystemsorpacketinspectorshaveenoughresourcesavailableandareconfiguredcorrectly(checkthelogsforincorrectlyfilteredtrafficordropped
packets)
Afteryouhaveconfirmedthatyourinfrastructureishealthyandallcomponentsareconfiguredcorrectly,checkthepowersavingconfiguration.ForvirtualmachineswithmorethanonevirtualCPU,alsocheckifhigh
CPU%RDYtimeshaveanegativeimpactonthesevirtualmachines.
ThefinalstepistochecktheRSSsettings.ChangingtheRSSsettingsshouldonlybedonebytrainednetworkadministrators.VMwarealsorecommendsconfirmingthatallrelevantapplications(includingthe
operatingsystem)supportchangestotheRSSconfiguration.

Powerplan
Toensurethatthesystemtakesadvantageoftheavailableresources,itisimportanttodisableallpowersavingfeatureswhileinvestigatingperformanceissues.Ifthepowersavingconfigurationappearstoberelatedto
theperformanceproblems,acustomizedpowerplanbasedontheperformanceandpowersavingrequirementsshouldbecreated.Ifyouareunsureaboutwhichpowersavingconfigurationisrecommendedforyour
system,engageyourhardwarevendor.
ToadjustthepowerplansettingsonaWindows2008Server:
1. ClickStart,typepowercfg.cpl,andpressEnter.
2. EnsurethattheHighperformanceoptionisselected.
Note:Steps3through6areoptional.
3. ClickChangeplansettings.
4. ClickChangeadvancedpowersettings.
5. Toenableaccesstoallsettings,clickChangesettingsthatarecurrentlyunavailable.
6. Browsetheavailablesettingsandadjustasnecessary.
7. ClickOKtoconfirmandcloseallwindows.
Note:Somechangesmightrequirearebootoftheguestsystem.
CheckingCPU%RDYtimes
TodetermineifavirtualmachineisimpactedbyhighCPU%RDYtimes,useoneofthesemethods:
CountallvirtualCPUsonaparticularhostorcluster,anddividebythenumberoflogicalCPUs.Aresultofoneorhighermeansthatthehostorclusterisovercommittedandshouldbeinvestigated.Valuesof
fourorhigherareconsideredoverloadedandmustbeinvestigatedimmediately.
Notes:
Theintentofthismethodistoquicklydetermineifahostisovercommitted,ratherthandeterminingifitisnot.VMwarerecommendsusingesxtoptoobservedetailedhostperformance.
Althoughhyperthreadingdoublesthenumberoflogicalprocessors,itcannotprovidethesameperformanceastwophysicalprocessorcores.Ifitislikelythatthehostisovercommitted,calculate
usingthenumberofphysicalCPUs,ratherthanlogicalCPUs.
TheesxtopcommanddisplaysthevaluesfortheCPU%RDYtimewhenrunonthehostwiththeaffectedvirtualmachines.Formoreinformationonhowtouseandinterprettheoutputofesxtop,seethe
WorldStatisticssectionintheInterpretingesxtopStatistics(http://communities.vmware.com/docs/DOC9279)Communitiesdocument.Youcanalsorunresxtop,whichisprovidedwiththevSphereManagement
Assistant(vMA).
Thevmsupportcommandprovidesthecapabilitytocreateperformancesnapshots.Formoreinformation,seeCollectingperformancesnapshotsusingvmsupport(1967)(/selfservice/search.do?
cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1967).
Torelieveanovercommittedhost,useoneofthesemethods:
Movetheaffectedvirtualmachinetoahostwithmoreavailableresources
Moveothervirtualmachinesoffthehost
DecreasethenumberofvirtualCPUsontheaffectedvirtualmachine
Note:ChangingtheCPUcountmightnotbesupportedbytheguestoperatingsystem.Formoreinformation,contacttheoperatingsystemvendor.

EnablingandconfiguringReceiveSideScaling(RSS)
BeforeenablingRSS:
EnsurethatthehardwareversionofthevirtualmachineissettoVersion7orhigher.Formoreinformation,seeVirtualmachinehardwareversions(1003746)(/selfservice/search.do?
cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1003746).
EnsurethatthevirtualnetworkadapterissettoVMXNET3andthattheoperatingsystemissupportedbythisadapter.Formoreinformation,seeChoosinganetworkadapterforyourvirtualmachine
(1001805)(/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1001805).
EnsurethatRSSisenabledintheguestoperatingsystem.ToverifythisinaWindowsguestoperatingsystem,openacommandpromptandrunthecommand:
netshinttcpshowglobal
TheoutputindicateswhetherReceiveSideScalingStateisenabledornot.
EnsurethatthenetworkadapterinthevirtualmachineisconfiguredtouseRSS.ToverifythisinaWindowsguestoperatingsystem:
1. OpentheDeviceManager,navigatetoNetworkadapters,andrightclicktheadapteryouwishtoenableRSSon.
2. InthePropertieswindow,clicktheAdvancedtab,thenclickRSSinthelistontheleftside.
3. ChangetheValuetoEnabledandclickOKtoclosethewindow.Arebootmightbenecessaryforthechangestotakeeffect.

Note:Enabling/disablingtheRSSfeatureinterruptsthenetworkconnectionontheadapterforseveralseconds.Ifyouareaccessingthesystemviaaremotedesktopsession,ensurethatyoucanaccess
thesysteminanotherwayincaseanissueoccursthatcausesthenetworkconnectiontonotreturn.
Bydefault,WindowsusesuptofourCPUsforRSS.
FormoreinformationonhowtoconfigureCPUaffinitiesorthenumberofCPUsusedforRSS,seetheMSDNarticleSettingtheRSSCPUConfiguration(http://msdn.microsoft.com/enus/library/ff570796%28v=VS.85%29.aspx)
.
FormoreinformationonenablingordisablingRSS,seetheMicrosoftSupportArticleInformationabouttheTCPChimneyOffload,ReceiveSideScaling,andNetworkDirectMemoryAccessfeaturesin
WindowsServer2008(http://support.microsoft.com/kb/951037).
Note:TheprecedinglinkswerecorrectasofDecember10,2013.Ifyoufindalinkisbroken,providefeedbackandaVMwareemployeewillupdatethelink.
Note:TSO(TCPSegmentationOffload)isafeatureofsomeNICsthatoffloadsthepacketizationofdatafromtheCPUtotheNIC.TSOissupportedbytheE1000,EnhancedVMXNET,andVMXNET3virtual
networkadapters(butnotbythenormalVMXNETadapter).InESXi,TSOisenabledbydefaultintheVMkernel,butissupportedinvirtualmachinesonlywhentheyareusingtheVMXNET3device,theEnhanced
VMXNETdevice,ortheE1000device.TSOcanimproveperformanceeveniftheunderlyinghardwaredoesnotsupportTSO.

AdditionalInformation
Forrelatedinformation,see:
PerformanceBestPracticesforVMwarevSphere5.0(http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf)
ReceiveSideScaling(http://technet.microsoft.com/enus/network/dd277646)(MicrosoftTechNetarticle)
ReceiveSideScaling(http://msdn.microsoft.com/enus/library/ff567236%28v=VS.85%29.aspx)(MSDNLibraryarticle)
DegradedoverallperformanceonWindowsServer2008R2(http://support.microsoft.com/kb/2207548)(Microsoftsupportarticle)
Windows2008/2008R2defaultPowerPlanof"Balanced"canincreaselatencyandreducethroughput(http://web.archive.org/web/20130411021814/http://windowsazurecat.com/2011/06/windows20082008r2defaultpowerplanofbalancedcan
increaselatencyandreducethroughput/)(archivedcopyofMicrosoftCustomerAdvisoryTeamarticle)
Note:TheprecedinglinkswerecorrectasofFebruary5,2014.Ifyoufindalinkisbroken,providefeedbackandaVMwareemployeewillupdatethelink.
Ifyouobservedroppedpacketsinesxtop,seeTheoutputofesxtopshowsdroppedreceivepacketsatthevirtualswitch(1010071)(/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1010071).

SeeAlso
Choosinganetworkadapterforyourvirtualmachine(1001805)(http://kb.vmware.com/kb/1001805)
Virtualmachinehardwareversions(1003746)(http://kb.vmware.com/kb/1003746)
TroubleshootingnetworkperformanceissuesinavSphereenvironment(1004087)(http://kb.vmware.com/kb/1004087)
Theoutputofesxtopshowsdroppedreceivepacketsatthevirtualswitch(1010071)(http://kb.vmware.com/kb/1010071)
CollectingperformancesnapshotsusingvmsupportinESXandESXi(1967)(http://kb.vmware.com/kb/1967)
RSSandmultiqueuesupportinLinuxdriverforVMXNET3(2020567)(http://kb.vmware.com/kb/2020567)
Windows(2093679)(http://kb.vmware.com/kb/2093679)
Windows(2097316)(http://kb.vmware.com/kb/2097316)

UpdateHistory
11/14/2013AddedESXi5.5toProducts
12/10/2013AddedMicrosoftlinktoenableordisableRSS

RequestaProductFeature
TorequestanewproductfeatureortoprovidefeedbackonaVMwareproduct,pleasevisittheRequestaProductFeature(http://www.vmware.com/contact/contactus.html?department=prod_request)page.
Permalinkto:PoornetworkperformanceorhighnetworklatencyonWindowsvirtualmachines(http://kb.vmware.com/kb/2008925)
Readourblog(http://blogs.vmware.com/kb/)

WatchKBTV(http://blogs.vmware.com/kbtv/)

Followus(http://www.twitter.com/vmwarekb)

request.html)

1(javascript:void(rate(1,this)))
2(javascript:void(rate(2,this)))
3(javascript:void(rate(3,this)))
4(javascript:void(rate(4,this)))
5(javascript:void(rate(5,this)))
39Ratings

Actions
BookmarkDocument(javascript:addBookmark(actionBoxUtil.docURL,actionBoxUtil.docTitle))
PrintDocument(javascript:window.print())
SubscribetoDocument(/selfservice/microsites/microsite.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2008925&format=rss)
(http://www.addthis.com/bookmark.php?v=250&username=xa4b5f42f36e60a29e)

KB:2008925
Updated:Apr25,2014
Categories:
Howto
Troubleshooting
Languages:
English
Product(s):
VMwareESX

RequestNewContent(http://www.vmware.com/landing_pages/knowledgebasecontent

VMwareESXi
ProductVersion(s):
VMwareESX3.5.x
VMwareESX4.0.x
VMwareESX4.1.x
VMwareESXi3.5.xEmbedded
VMwareESXi3.5.xInstallable
VMwareESXi4.0.xEmbedded
VMwareESXi4.0.xInstallable
VMwareESXi4.1.xEmbedded
VMwareESXi4.1.xInstallable
VMwareESXi5.0.x
VMwareESXi5.1.x
VMwareESXi5.5.x

You might also like