Debugging ServiceNow Performance Issues

Transaction Logs
The main tool we will use is ServiceNow itself. The transaction log will contain most
of the evidence we need to investigate performance issues.

1. Under System Logs navigate to one of the Transaction Logs.

2. On the List view add all the columns referencing Time there are about a dozen totals.
3. On the List view add Session, System ID and User Agent
4. On the form view add every single field.
I like to create a separate section for the time fields.

Scenarios
I will break these scenarios into one of three categories:

Client is the users browser or machine

Server is the ServiceNow server
Network is everything in between

For each scenario I will list symptoms and potential root causes.

Scenario 1: Poor Network Performance

A fairly common scenario, users may experience poor network performance especially
when there are multiple locations, in an on-premise installation, where the users must
VPN in or where additional devices such as firewalls or proxies come into play.

Symptoms

One or more users will report intermittent degraded performance

If multiple users report issue it may only occur at specific times or locations
If a single user reports issue it may only happen at specific locations

Verifying The Issue Users should try the following.

Logout, completely clear cache, delete all cookies and try logging in again
Log in from a different browser
Log in from a different machine on the same network

If the issue still persists it may be an issue with the network. To verify:
1. Navigate to Transaction Logs (All User)
2. Find all transactions for a particular users session
3. Find all transaction where the Client response time or Network timeis excessive
(>5000ms)
4. If there are excessively high Network times then those should be flagged and you
should look for additional logs around the same time
5. If additional log entries are found this trend will likely point to a network issue
6. If there are excessively high Client response times you should compare this time to
the other time fields (excluding Browser Time)
7. If the sum of the other time fields does not closely match the Client response time then
it is likely a network issue.

Once you have determined you have a network issue it is necessary to look for trends
to isolate the problem.

Do affected users have the same location or network?

Is the issue isolated to a specific time of day?
Do any users access ServiceNow through a proxy?
Is the issue intermittent or consistent?

Potential Root Causes

Overloaded or malfunctioning networking equipment This would almost certainly affect

other applications as well.
Overloaded proxy If users access ServiceNow through a proxy, is the machine
correctly configured and will it support the current load?
Insufficient bandwidth.
Poor wireless signal.

Fixes

Unfortunately for networking issues there is not much that can be done from
ServiceNow. Infrastructure may need to be upgraded or reconfigured to support the
load.

Scenario 2: Mis-configured ACL

Usually occurring for non-admin users or those with specific roles, mis-configured ACLs
can impact forms but especially lists where ACLs run multiple times.

Symptoms

One or more users will report excessively long load times for lists or forms.

Verifying The Issue

1. Navigate to Transaction Logs (All User).
2. Find all transactions for a particular users session.
3. Find all transaction where the ACL time is excessive (>100ms).
4. For ACLs even a fraction of a second is a very long time.
5. Search for any other transactions for the same URL.
6. If the same URL has a trend of long ACL times there may be a mis-configured ACL.

Validate the ACL is mis-configured

1. Based on the URL you should be able to find the affected table. Usually it will be in the
name however if it is not a standard List or Form page (for instance a UI Page) you may
need to look through the contents of the page to see the source table(s).
2. View the ACLs for the table.
3. Look for any ACLs with Scripted Conditions.
4. Verify the Script does not have nested for loops or any other violations of best practice.

Potential Root Causes

Poorly written Scripted ACLs

Excessive number of records being accessed

Fixes

Scripts should be re-written to be as efficient as possible.

Lists and queries should be properly limited where necessary.
Additional filters can be added to lists to cut down on the number of records returned.
Potentially re-structuring tables or columns to simplify the script.

Scenario 3: Mis-configured Business Rule

A very common scenario is when Business Rules are not written correctly or several
rules are written poorly and stack up to create a poor user experience. Most commonly
when a user creates or updates but this can occur any time a business rule runs with
the exception of asynchronous types.

Symptoms

One or more users will report excessively long wait times saving or loading a record.

Verifying The Issue

1. Navigate to Transaction Logs (All User).

2. Find all transactions for a particular users session.
3. Find all transaction where the Business rule time is excessive (>2000ms).
4. Find all transaction where the Business rule count is excessive (>10).
5. Search for any other transactions for the same URL.
6. If the same URL has a trend of long Business rule times or counts there may be an
issue with the BRs.

Validate incorrect Business Rules

1. Open the Business Rules for the particular table.

2. Look for Business Rules which match the conditions reported by user (when saving,
when creating).
3. Look for all After or Before rules (Async rules will not apply since they do not stop the
user session).
4. Verify there are not an excessive number or rules.
5. While having many Business Rules is not bad or uncommon the efficiency of those
rules needs to increase with the number. Otherwise many rules that are somewhat slow
can add up to a long wait time for end users.
6. Verify the content of the Business Rules.
7. Like ACLs you should avoid making an excessive amount of queries or nested queries
in a BR.

Potential Root Causes

Multiple poorly written Business rules

Example: 10 or more rules which each take 250ms will end up 2.5 seconds for the
user, this is only a part of the process though so the over all time may end up being
excessive.
Single poorly written Business rule
A single rule taking more than 1000ms will directly impact the users experience

Fixes

Scripts should be re-written to be as efficient as possible.

Minimizing the number of GlideRecord queries and number of results returned.
Potentially moving some scripts to asynchronous where possible

Scenario 4: Under-performing Node

A very uncommon scenario but also one that is not fixable except by ServiceNow. This
will usually involve ServiceNow restarting and possibly decommissioning the node.

Symptoms

Multiple users will report excessive wait times performing any action in the system
Issue will usually be intermittent
Issue will occur more frequently at peak times
 Verifying The Issue

1. Navigate to Transaction Logs (All User).

2. Find all transactions where the Response Time is excessive (>5000ms).
3. Add the System ID column
4. Verify that no other cause (Network, BR, ACL) is affecting the performance.
5. This may manifest as a longer Session wait time, Semaphore Wait time or SQL time
6. You may also try to eliminate transactions that are expected to take longer, such as
those with larger outputs (Output length)
7. You may also eliminate those requests which affect multiple System IDs
8. If there is a node (System ID) which is constantly under-performing and where the
same requests execute quickly on other nodes it may be isolated to that node.
9. You may open a HI server ticket to confirm the node is not functioning correctly

Potential Root Causes

If the issue consistently happens even after restarting the node hardware may be
defective
Many other causes are unknown especially since we do not have access to the back-
end

Fixes

Ask ServiceNow to restart the node via HI ticket or Support

Scenario 5: Browser Performance

A very common scenario especially for environments with older browser (specifically
Internet Explorer). Poor performance is almost exclusively limited to IE but there are
scenarios where Chrome and Firefox will not work correctly.

Symptoms

One or more subset of users will report a performance issue especially loading forms or
custom pages
Issue will be consistent and easily reproduced
Issue will only occur on a certain browser or machine

Verifying The Issue

Before verifying check that the user is actually using a Browser on their desktop or
laptop and it is not related to a mobile device.

1. Navigate to Transaction Logs (All User).

2. Add the User Agent column.
3. Find all the Transactions for a particular users session who is reporting the issue.
4. OR Find all transactions where the Response Time is excessive (>5000ms).
5. Eliminate any transactions which can be explained by other issues.
6. Try to correlate any poor performance to a particular User Agent.
7. Look up the Users User Agent string to determine which browser they are
using. UserAgentString.com

Validate the browser is not running in Compatibility Mode

1. For Internet Explorer ask the User to verify their version by going to About Internet
Explorer
2. If the user reports a version which does NOT match the User Agent reported in the
transaction log they may have compatibility mode on which should be disabled.

Validate the Browser is supported

1. Older versions of Internet Explorer (<10) generally do not perform well enough to have a
good User Experience in ServiceNow
2. Versions of Internet Explorer (<9) may not work at all

Validate extensions are not interfering

1. Ask the user to disable extensions OR run an instance of the browser in private
browsing mode (this usually disables all extensions)
2. If the issue does not persist with extensions disabled review the extensions to see if any
may be causing an issue
3. Disable extensions one a time to verify.

Validate no errors are being reported

1. Ask the user to verify there are no errors in the JavaScript console. Most browsers have
a developer console which will output all JavaScript errors.
2. Believe it or not some errors are expected but an excessive amount especially those
which report missing resources could be affecting the user experience.

Potential Root Causes

Unsupported Browser.
Browser running in compatibility mode.
Browser extension impacting performance.
GPO pushing browser setting which impacts performance.
GPO pushing system setting which prevents caching.
Cache disabled through browser setting.
 Browser unable to reach resources this may also manifest as the UI appearing
strange.

Fixes

Upgrading to the latest browser version.

Disabling Compatibility Mode.
Removing or resolving extension or browser configuration.
Verifying network configuration (firewall or proxy) allows users machine to retrieve
correct resources.

Scenario 6: Excessive or Incorrect Client Side Scripts

Another common scenario is for the user to see poor performance due to too much form
automation on the client side. Offloading some work to Business rules or re-architecting
it for performance will usually resolve these.

Symptoms

One or more users will report an issue with forms taking a long time to load.
Users may reports jittery or unresponsive form when changing values.

Verifying The Issue

1. Navigate to Transaction Logs (All User).

2. Find all transactions for the table or form where the issue is reported which have an
excessive (>2000ms) Client script or UI Policy time
3. Note that these values will only include those scripts run when loading.
4. If script time is consistently long there may be an issue with the number or efficiency of
the script.

Validate the script is impacting performance

1. For either/both UI Policies and Client Scripts disable all scripts.

2. Enable each script testing the form.
3. Take note of the load time. If a single script increases the load time dramatically there
may be an issue with the script.
4. If each script or policy increases the load slightly resulting in a long load time, there may
be an issue with the quantity of scripts.

Validate the form layout

1. Validate there is not an excessive amount of fields,tabs, related lists or embedded lists
on the form.
2. Validate there are not custom Formatters present.
3. If there are an excessive amount of controls on a form it will directly impact the load
time for the user.

Potential Root Causes

Too many Scripts or Policies.

Poorly written Scripts.
Excessive amount of AJAX transactions in scripts.
Some AJAX is OK, but while those transactions are occuring synchronously the user
is unable to do anything.
Synchronous AJAX should only be used when the result is NECESSARY for the
user to proceed.
Too many fields present on form.
Too many lists or embedded lists.

Fixes

Re-writing Scripts to be more efficient.

Refactoring or combining UI Policies.
Changing synchronous AJAX to asynchronous where possible.
Changing some form automation to run as Business rules where possible.
Removing fields or creating additional views to only show necessary fields.
Removing unnecessary fields or lists.

Scenario 7: Incorrect User Expectations

A perfectly valid expectation is that every page will respond quickly but not every page
in ServiceNow is equal. Sometimes 5, 10 or even 30 seconds is reasonable when the
system is crunching 1000s or rows to display in a report.

Symptoms

User reports performance issue for particular pages.

Verifying The Issue

1. Navigate to Transaction Logs (All User).

2. Find all the transactions for the users session.
3. Find any excessively long transactions (>5000).
4. Take note of the URL.
5. If the user is loading home.do or a report it may be valid to expect wait times greater
than 5 seconds.

Potential Root Causes

 User is loading a Report or Dashboard.
User is loading too many records on a list.

Fixes

Explain to the user the wait times are valid.

Reconfigure the report to have additional conditions.
Reset the user preference to display less rows.
Change the users Home page if possible.