Analyzing HTTP

A Practical Guide to Profiling and

Troubleshooting Web Performance

Jean Tunis
Copyright © 2015 by Jean Tunis

Editing by Byrnes Publishing, LLC

All rights reserved. No part of this publication may be

reproduced, distributed, or transmitted in any form or by any
means, including photocopying, recording, or other electronic
or mechanical methods, without the prior written permission
of the publisher, except in the case of brief quotations
embodied in critical reviews and certain other non-commercial
uses permitted by copyright law.

ISBN: 978-1-6873-4191-4
Table of Contents
About This Book........................................................................v

Brief History of HTTP................................................................1

Simple Beginnings: HTTP 0.9.......................................................1

The Popular Kid: HTTP/1.0..........................................................2

The Standard: HTTP/1.1................................................................3

The HTTP Protocol....................................................................4

Types of Requests..........................................................................4

Types of Responses........................................................................5

Header Fields.................................................................................6

Message Body................................................................................6

HTTP Performance Analysis......................................................8

The Network is Slow......................................................................9

Components of Network Delay....................................................10

Performance Testing and Analysis..............................................12

Nine Performance Metrics to Check.........................................26

Five Common HTTP Analysis Recommendations.....................39

The Here Future: HTTP/2........................................................45

HTTP/1.1 Limitations..................................................................45

Workarounds...............................................................................47

Next Version................................................................................48

Conclusion (and Thank You)...................................................50

About This Book
My goal for this book is to help provide you with guidelines,
tips and recommendations to analyze your organization’s web
applications in your current and future role. That role may
have many different titles. You have probably, at some point,
been a Network Engineer, Network Analyst, Performance
Analyst or Performance Engineer. You may have even been
called “developer” or “consultant”. Whatever your title or role,
this book will give you information about what it takes to
analyze HTTP applications.

Whether you are a beginner to this or have done it for a while,

you should continue reading. As a professional with more than
a decade and a half in this field, I think there are a number of
things in this book you can walk away with to implement the
next time you need to profile or troubleshoot a web
application.

I wrote this book to help save you some time before, during
and after you have tested your web application. Who doesn’t
want to save time?

I start with giving a brief history of the HTTP protocol. It’s

fascinating that such a simple protocol has come such a long
way. I think we all must know and understand where the
technologies we use today come from.

Next, I discuss some more details about the protocol itself and
how it works.

Later, I dig into a number of things to look at and consider for

HTTP analysis. I don’t go into the details of which tool to use,
but I discuss use of packet analysis software for analyzing
HTTP traffic.

Finally, I finish with the future of HTTP.

 ANALYZING HTTP

Brief History of HTTP

As you may know, the World Wide Web began as a project

launched by Timothy Berners-Lee in the United Kingdom
(UK). He wanted a method to share his research from CERN,
the European Organization for Nuclear Research, with other
researchers. He was able to combine some of the earlier work
he had done with hypertext and elements of the Internet then
– namely, the Transmission Control Protocol (TCP) and
Domain Name System (DNS) – to create what he called the
World Wide Web. Yes, it’s true, there were no spaces between
the words then.

To do this, Berners-Lee had to develop a transfer protocol and

language to allow a client to make requests of a server, and the
server to reply with a response. The protocol was the Hypertext
Transfer Protocol (HTTP) and the language was the HyperText
Markup Language (HTML), which form the foundation of all
websites today. Berners-Lee also created the first web browser
in order to facilitate the communication.

Simple Beginnings: HTTP 0.9

HTTP is one of many application layer protocols within the
TCP/IP protocol stack and as you know, is used for the
communication of web traffic.

Users utilize HTTP with their client web browser to access

services provided by web servers. A client is required to make a
connection to the web server, so HTTP runs on top of TCP to

1
 JEAN TUNIS

facilitate this connection. This is one of the links between

hypertext, TCP and DNS that Berners-Lee saw that he could
take advantage of, namely running HTTP on top of TCP for the
connection.

The first documented version of HTTP is HTTP 0.9, which was

released in 1991. It originally contained one HTTP request
type, known as a method. This method was the HTTP GET you
likely know about today. The communication was a relatively
simple process:

- Connection: the client browser opens a connection

with the server
- Request: the client browser makes a GET request to
the server
- Response: the server responds with an HTML page
that the client browser will read and display to the user
- Disconnection: the server will close the connection
with the client.

The Popular Kid: HTTP/1.0

As HTTP gained popularity because of its usefulness, it became
clear that this relatively simple communication protocol was
not enough. There grew more interest in expanding the
protocol to make it even better and more efficient. Work on
this started in 1995, and in 1996, HTTP/1.0 became
standardized with RFC 1945.

Before HTTP/1.0 could be standardized, work had already

started on the next version to get around some of the
limitations of HTTP/1.0. As RFC 2068 was being written,
which ultimately became the standard for the next HTTP
version, the major browsers at the time, such as Mosaic and
Netscape, were already implementing the changes to get
around the limitations. In 1997, HTTP/1.1 was officially
standardized as RFC 2068.

2
 ANALYZING HTTP

The Standard: HTTP/1.1

There were a number of changes in HTTP/1.1 to improve on
HTTP/1.0, but two key changes were Persistent Connections
and Pipelining.

In HTTP/1.0, similar to HTTP/0.9, once a client made a

request, the server closed the connection. However, as more
people got online and the distance between client and server
began to increase, this was developing into an issue because
websites were becoming slow.

One solution was to allow the client to specify to the server that
it wanted the connection to remain open. Another solution to
aid with this was to allow the client to send a request without
waiting for its response before sending the next request. This
allowed the client to pipeline multiple HTTP requests.

Today, HTTP/1.1 is widely deployed across the Web.

However, a number of limitations exist with this version as

well. This has led to development and standardization of the
next version, HTTP/2. HTTP/2 is outside the scope of this
book, but we will explore it a bit more later on.

3
 JEAN TUNIS

The HTTP Protocol

The HTTP protocol consists of three parts: request/response,

header, and body. I will briefly go over each, and provide a bit
more detail about the HTTP requests and responses.

Types of Requests
There are a number of types of HTTP requests, but the two
most utilized are HTTP GETs and HTTP POSTs. Some of the
lesser utilized requests include the HEAD, PUT and DELETE.

A GET request is used to do exactly what the name implies…to

get data from a server. The POST request typically does the
reverse of a GET – it posts or sends data to a server. However,
the POST can also be used to get data from a server as well.
This usually occurs for dynamic data that the web application
does not want to be cached by the browser.

A third common request type you will find in some cases is an

HTTP CONNECT request. This type of request is a bit more
common because it is used when a client is talking to a web
server via a proxy. In this instance, the client makes requests
to the proxy server to connect to the web server. The proxy, in
turn, makes the appropriate request (GET or POST) on behalf
of the client.

4
 ANALYZING HTTP

Types of Responses
There are a number of HTTP response types from a web server
to a client. However, some of the most commonly known and
seen are 100, 200, 301, 404 and 500. Here is a quick
discussion on these response types.

HTTP 100
The HTTP 100 response from a server tells a client that it can
continue on and send the request it wanted to send. This often
happens when a browser does not want to make a request for a
file that may take some processing time. Instead of requesting
the whole file, the browser requests for the existence of the file,
and waits for the server to tell it that it can continue with its
requests, if the file exists. A client that makes this request
usually specifies the following in its HTTP header: “Expect:
100-continue”.

HTTP 200
The 200 response is the most common and most expected
response. It signifies that the server has received the client's
request and that it can fulfill it. In your packet analysis tool,
you should see “200 OK”.

HTTP 301
The 301 response is used by the server to redirect client
requests to another server. This code tells the client browser
that the redirect is permanent, and the resource it is looking
for has permanently moved. A similar 302 code specifies that
the move is temporary. In either case, redirects tend to be
frowned upon as they can adversely affect web performance.

HTTP 404
This is commonly seen when a particular requested resource
(such as a file) on a server is not found. This can happen when

5
 JEAN TUNIS

changes were made to the application, but the code was not
updated to reflect those changes, and the browser continues to
request files no longer being used by the application. This is
the dreaded 404 error. Some companies today plan for this
error page and provide custom pages so that the user still finds
something useful, instead of nothing at all.

HTTP 500
This generic code is sent by the server when an internal server
error has been detected, but no additional information is
known about the reason for the error. This error usually means
that something has gone terribly wrong with your web or
application server.

Header Fields
The HTTP header contains information used by both client
and server to determine how to handle requests and responses.
There are many fields in the header, specifying what both
client and server should do or are doing with data that is
requested or received.

All of the above requests and responses are provided in the

header. In addition, the header also provides the policy for
caching, data compression, redirection, and more.

Message Body
The HTTP message-body is the actual data being sent or
received. This part of the HTTP protocol follows the header
information above. This field is actually optional since there
doesn’t have to be any data sent.

The most common type of message occurs during a GET

request when the server returns the requested data to the
client browser. One received, the client browser must render
that data to the screen for the user to see. Another type of

6
 ANALYZING HTTP

message occurs during GET or POST requests when a user is

submitting form information or uploading a file, for example,
to the server.

The HTTP header contains all the information about the body
of the message, such as what type of message it is, so that the
receiver (either client or server) can understand how to, or
even when it can, render or receive the message.

7
 JEAN TUNIS

HTTP Performance Analysis

Over the years, the Web has become more integrated into our
daily lives. We use the Internet for everything these days, from
buying goods and services to providing them ourselves.

Many companies choose to allow their employees to work from

home a certain number of days out of the week, while others
are entirely remote, with all or most of of their employees
working from home all the time. For these employees, there is
no office to go to.

I was both types of employees for many years. I was either

working from home or traveling to customer locations, so I was
always remote from the office. You may very well be in a
similar situation.

As a result of this, the performance of websites and web

applications has become more important. The HTTP protocol
has undergone a number of changes over the years to help
make things faster. Also, a number of tweaks and workarounds
have been identified to make things faster, or at least give the
user the impression that everything is faster.

But sometimes, things just don’t work. And when that

happens, the usual inclination is to blame the network being
used to access the website.

8
 ANALYZING HTTP

The Network is Slow

“The network is slow” is a common response when I ask
questions about why a website or web application is slow. If
you’ve done this for any number of years, I’m sure you have
gotten the same or similar responses. Granted, years ago, the
network was slow. In some corporations, that’s still the case
due to the lack of a need (or want) of the use of more
bandwidth while having remote users.

We’ve had dial-up at rates up to 56Kbps bandwidth; we’ve had

128Kbps ISDN lines; we’ve had 256Kbps Frame Relay PVCs;
we’ve had T1/E1 private line connections.

As web applications became more dynamic, these connections

could not keep up. However, these days, that’s not really the
case. According to the State of the Internet report for the first
quarter of 2015 from Akamai Technologies1, the average
connection speed for the United States of America (USA) is
11.9Mbps. That’s over 130x the speed of dial-up in less than
about 20 years! Globally, connection speed is 5Mbps!

In more recent years, I consistently came across companies

with 100Mbps to user desktops (some with 1Gbps), 1Gbps
between servers, and in a growing number of cases, 10Gbps in
the network’s core.

So the typical response about the network being slow is fast

becoming a thing of the past. Also, network connections have
become much more robust than in years’ past.

Network devices are being configured much more effectively.

For example, issues, such as duplex problems, don’t occur as
much as they used to.

CRC errors on WAN links don’t occur as often as I’ve seen in

the past. Therefore, it is clear that things are much better on
the network.

9
 JEAN TUNIS

This isn’t to say that the network is never at fault any more.
Things happen.

Circuits get accidentally cut, causing new network routes to be

taken. Someone misconfigures something on a router or
switch, causing interfaces to go down.

All these things still happen. However, when they do, from my
experience, it is not because of the network not being robust
enough.

Components of Network Delay

Latency on the network is something that you should pay
attention to, if you’re not doing it already. With the increased
interactivity of web applications, latency has become the key
bottleneck. Latency is the time that it takes for the data a client
sends to get to the server.

There are four main delay components of latency.

We have processing delay. This delay is the time it takes for

any device to process the data that it receives to determine
where this data is going and how to get it to its next stop to
that destination.

We have queuing delay. This delay is the time that the data
sent is waiting in a device’s queue along the path between the
client and the server before it can be transmitted.

We have serialization delay. This delay, also referred to as

transmission delay, is the time that it takes to put the bits of
data onto the network links along the path.

We have propagation delay. This is the time that it takes the

bits of data sent to travel from one side (where the client is) to
the other side (where the server is).

10
 ANALYZING HTTP

All of the above delays, except one, are functions of things that
we can reduce with better technology.

For example, processing delay can decrease if we develop and

add better processing hardware to our network devices.
Queuing delay can improve with similarly better components
or faster, more efficient queuing algorithms. Serialization delay
can be improved by reducing the amount of data we send or
increasing the bandwidth capacity of our network links.

Propagation delay, however, is a function of the distance

between the client and server. Better technology is not going to
bring two cities physically closer together - at least not now.

This delay is one component on the network that only Mother

Nature can improve for us. This is something we can do
nothing about.

Yet, as the Internet becomes more global, with users and

customers located practically everywhere, the distance
between user and server has steadily increased. So when a user
says that the network is slow these days, that may very well be
the case. However, that’s not the network’s fault. Mother
Nature’s to blame!

With the speed of light in a vacuum traveling at about 186,000

miles per second (300,000 km per second), it will take roughly
10 milliseconds for light to travel 1,000 miles and back. That’s
the theoretical value for propagation delay alone!

Companies are more global than ever. Employees are more

geographically dispersed than ever. For example, if a company
has employees located in Frankfurt, Germany using its human
resources application server located in Washington, DC, which
is about 4,055 miles (6,526 km), it is expected that the
propagation delay will be about 41 milliseconds to get from one
side to the other and back. And that’s the best case!

11
 JEAN TUNIS

So distance is a big factor for web performance. But as with

everything else, we have some workarounds for this as well
that I will be going into more detail about later, even though
we currently cannot reduce distance

Performance Testing and Analysis

As websites and web applications become more dynamic and
complex, and the distance between user and server increase,
the field of performance engineering and analysis has gotten
more and more attention. Some organizations develop a whole
process behind ensuring that their sites and applications
perform as efficiently as possible for its users

Many integrate performance testing and analysis processes

with the development process. Others have whole teams ready
to troubleshoot performance issues occurring on these sites.

If you are on one of those teams, below are some steps you can
follow to help you with profiling and/or troubleshooting a
website or web application.

Tools
The process of analyzing a website or web application involves,
first and foremost, having the appropriate knowledge about
how the HTTP protocol and the TCP/IP stack function. While I
provided an overview of the HTTP protocol, this book is not
intended to delve into any sort of detail of how TCP/IP works.
It is firstly assumed you have some of the basic knowledge.

Secondly, you must have the appropriate tools to help you

capture and analyze the data going from the user to the server
and back. There are plenty of such tools. You probably have
your favorite already.

Some of the most common are Wireshark, tcpdump, Riverbed

OPNET Transaction Analyzer, NetScout Sniffer, and many
others. Each product has its benefits and drawbacks, so I won’t

12
 ANALYZING HTTP

be focusing on the use of any one tool. Instead, I will be

focusing on the process by which you can use your preferred
tool to get the job done - and that is to profile or troubleshoot
whatever website or web applications you’ve been asked to
analyze.

Pre-Capture Preparation
When preparing to analyze a website or web application, you
cannot just dive right in and start capturing data. That is a
recipe for disaster and potentially a waste of yours and
everybody else’s time.

You must have a plan and then create a plan of attack. As they
say, if you fail to plan, you plan to fail.

In order to put a plan together, you must get some preliminary

information about the application, its user base, and most
importantly, what is the issue being experienced by the user.

Here are some questions to ask and get answers to before you
go off capturing data. These are questions you ask the
application team in many cases, but may also include the
involvement of existing users.

Key Must-Ask General Questions

Whether you are analyzing an application before deployment
or troubleshooting it after it has already begun to perform
badly, there are some general questions you should always try
to get some answers to before capturing any data.

1. How does the application work?

Clearly, it is always a good idea to understand how an
application is supposed to be working so you can better
analyze its performance. I recommend that you get a visual
presentation of how an application works. Whether it gets
done via a face-to-face meeting or web conference, seeing the

13
 JEAN TUNIS

application in action can be invaluable to your performance

analysis efforts.

2. How many users are using the application?

Knowing this information helps to better understand how the
application can scale. You may not be involved in any stress
testing of the application, but this will give you some insight
into whether the application is able to support the number of
users currently using it or the number that is planned to use it.

3. Where are the users located?

As I mentioned above, the distance between user and server is
increasing as organizations become more global. Knowing this
information helps to start putting a thought process together
of how this application is expected to perform. HTTP can be a
chatty protocol, particularly if chattier protocols that contain
XML, like SOAP (simple object access protocol), are used. If
users and servers are far apart with high latency, you may
already be able to correctly form a hunch that the cause of any
slowness is chattiness.

4. What is the architecture between the clients and

the server?
Similar to the above question, you need to know what all the
network infrastructure is between the user and the server.
What’s the bandwidth connectivity? How many network hops
does it take to get from one side to the other? What’s the
round-trip delay? What are the network devices that a request
or response traverse? Knowing these things, will help you
formulate a plan for analysis. Getting this information in the
form a network diagram is most helpful. So get a diagram!

14
 ANALYZING HTTP

5. What types of transactions or tasks are executed by

a user?
Because of its request/response nature, the HTTP protocol can
be very chatty depending on the types of transactions or tasks
that are allowed to be executed by a user. If you know what
some of these transactions are, you will have a sense of what
type of performance you can expect. For example, if a user
needs to log into a site or web application, you expect that to
perform fairly well. A user expects an almost immediate
response. If a lot of other data is being sent during the login
task, it’s very likely to be slow. Knowing information like this
helps to get a sense of how the application will work before you
even start doing any testing.

6. What is the expected performance for this

application?
Knowing what users expect performance to be can save a lot of
time. Every organization and group of users is different.
According to Akamai, Google and others, the common
recommendation for website performance is about 1 second.
Any request from a user in the browser should return a
response within about 1 second. Otherwise, the user will start
to think the site is slow. Knowing this information will help
keep you from chasing a performance metric you believe to be
bad when it’s within the user’s expectation or a customer’s
service level agreements (SLA).

7. What is the frequency of use of the application, i.e.

transactions per hour?
You need to know how often the application or some of the
transactions in the application are being used. This is a key
question to know the answer to especially if you are profiling
an application that is yet to be deployed on the entire network
or even an area on the network. You can then size the network

15
 JEAN TUNIS

infrastructure appropriately, if needed, before users are calling

the Helpdesk complaining of slow performance.

8. What TCP ports are used for this application?

The default HTTP port is 80. If the site is secure, the port is
443. Either of these can be changed, and often is for many
organizations or non-browser based web applications. You
don’t want to be looking at captured data, expecting to see the
default ports and not finding them. Find this out early.

9. What are the hardware specifications for the server

and client devices?
This can be very critical to know, especially for the server. You
want to know that the client and server machines are powerful
enough to handle the application and all that is required to run
it.

10. Any encryption or compression taking place?

This one is big! Depending on the nature of the packet analysis
you are doing, being able to “see” all of the traffic can make
your job a lot easier. So you want to know in advance how
easier or difficult it will be during analysis of the data you
capture. If the data is encrypted or compressed, you are limited
in what you can see. Both compression and encryption are
typical recommendations for increased performance and
security, respectively, so either one or both are likely to be
enabled for the application or website. You should know that
going in.

Troubleshooting Questions
The general questions above should be asked for any analysis
process. However, when you’re being asked to troubleshoot an
existing web application issue, there are some additional
questions you will need some answers to that will help you

16
 ANALYZING HTTP

more effectively identify the root cause of the problem being

experienced.

Here are some of the questions you need answers to before

capturing data:

1. What exactly is the problem?

This is an obvious question to ask. You need to know the
problem right? The thing is that you need to be very specific
about this answer. You need to find out exactly at what point
users start experiencing the problem. Does it happen as soon
as they log in? Is it after filling out some forms? Find out
exactly what the problem is and when it happens. This is where
being able to see the problem for yourself, either in-person, or
via web conference is key.

2. When does the problem typically occur?

You have to know when the problem occurs. It can save you
time if you know this before running your test to capture data.
Depending on the application and the time the issue occurs,
you may be able to identify the cause of the issue without doing
any tests. What if the user is experiencing slow performance
when the site is being used at night time? “Oh yeah, that’s
when data is being backed up”, you might say. In such a case, a
test is probably not needed, and you will have saved yourself
some time

3. Are there any virtual technologies in use?

This is a sign of the times. To save money, many organizations
are implementing websites and web applications on or using
virtual technologies, such as virtual machines from VMware,
Microsoft and Citrix, among others. This can impact how the
web application works. But it can also impact how you capture
data. So knowing this can help save you some time for not only
capturing application data, but also troubleshooting it.

17
 JEAN TUNIS

4. Does the problem occur for a particular transaction

or is everything affected?
This seems similar to question #5 about in the General
section. However, it’s slightly different. When troubleshooting,
you want to get to the root cause as quickly as possible. You
can do this by capturing data for one or two transactions
instead of everything transaction in the application.

5. Has it ever worked at any point? If so, what has

changed since then?
This question can easily be the very first question you ask
during a troubleshooting exercise. This can save you a lot of
troubleshooting time. Knowing when a problem started
occurring can be invaluable. Was this website always
performing slowly? Or is this something that started
happening after a recent upgrade? You know where to focus
your efforts after getting this answer

Capturing Data Tips

You now know how to prepare for capturing data. You’ve asked
the questions you believe will help capture and analyze the
data as effective as possible. In this section, I discuss how you
go about capturing the data and some things to be aware of
that could affect your capture.

Tiers Involved
In order to properly analyze an HTTP application, or any
application for that matter, you have to be able to capture the
appropriate data. To do this, you must identify the correct
tiers. As mentioned in the above Questions section, you should
have gotten either a diagram or some information about the
application architecture that would have helped you identify all
the tiers involved.

18
 ANALYZING HTTP

Since HTTP is a client/server protocol, your tiers will include a

client and one or more servers. At many organizations, the
standard implementation of a web application includes three
tiers – client, web server, and database server. In many other
cases, there are four tiers, which includes an application server
between the web tier and the database tier.

The names or IP addresses of these tiers should be identified

first.

It is possible that many of the above questions in the previous

section may not be answered before you test. You can make a
call to delay this testing or go forward anyway. Each
organization has different needs and pressures. Many
organizations have an application development process that
involves cataloging every application and the information
about them. Many others, unfortunately, have nothing in
place, and know very little about the applications they have
deployed.

However, at the very least, you will need the IP address

information of the tiers involved so you can capture the data
with a capture agent. Unfortunately, there are times when even
the IP address or hostname is not fully available. In those
instances, you must know who and where to go to within your
organization to get some information. This can make for a
more difficult profiling and troubleshooting process, but that’s
why you’re the expert professional.

Agent-based and Agentless Captures

There are two ways to capture application packet data: 1) with
an agent installed on the client and server machines, or 2) with
a hardware appliance outside of the client and server
machines, sitting on the network in or near the path between
client and server. Both methods have their advantages and
disadvantages.

19
 JEAN TUNIS

Agent-based Packet Capture

Capture agents have been around for years. The first modern-
day packet capture agent is tcpdump, which was created in
1987 by Craig Leres, Steve McCanne, and Van Jacobson while
working at the Lawrence Berkeley National Laboratory. The
tcpdump agent runs on Unix-based systems. A Windows-based
version, called windump, was also created.

Tcpdump has been the building block of many of the popular

packet analyzers, including arguably the most popular one,
Wireshark. The intent of this book is not to go into the details
of Wireshark, but it is commonly the preferred open-source
tool of choice for analysts profiling and troubleshooting
applications. A number of application performance
management (APM) products utilize either Wireshark or its
command line version, Tshark, as part of their packet analysis
component.

Using a software agent tends to be very inexpensive financially.

However, if done incorrectly or haphazardly, it can be
expensive resource-wise. I discuss a couple of tips on how to
do this effectively.

Once you have the information you need from all your
questions, you can submit any requests that are needed to get
the agents installed on the appropriate tiers. Don’t forget about
Change Management! An organization's Change Management
process can derail your analysis efforts, if you’re not aware of
them.

Agentless Packet Capture

There are a number of vendors that offer what’s considered
agentless packet capturing. This is does not mean that there is
no agent involved when the application data is being captured.
Instead, this term usually refers to the fact that the agent is
external to the machine (client or server) running the
application.

20
 ANALYZING HTTP

Agentless packet capturing is done by a hardware appliance

that you install on your network with all the components of a
software-based capture tool, like Wireshark, already installed.
The sniffer is the most well-known type of appliance.

A sniffer can be deployed in one of two ways. The first method

is to connect it to the network link you want to capture data on
via a network tap. While this is usually the best way to ensure
you are able to capture all of the application’s packets, it’s also
not always as feasible. The second method is via a SPAN port
or port mirroring. It is usually much easier to configure port
mirroring on your switch and send the packets to the sniffer.
However, port mirroring has had a history of losing packets as
the packet rate increases.

Using such an appliance can tend to get expensive both

financially and resource-wise. Not only do you have to concern
yourself with Change Management, you also need to concern
yourself with network link issues.

Agent Installation
When installing your agent, there are some things to be
cognizant of to help make installation and capture go as
smoothly as possible. Each organization has its own sets of
rules, processes and procedures for getting software installed
on client and server machines. Be sure to follow whatever
process is required in a timely fashion to get the agents on the
machines well in advance of the testing.

Installation Tips
Below are some tips to help with the installation and testing of
your web application:

21
 JEAN TUNIS

1. Ensure that your client and server machines will

have enough storage space to capture the data you
need.
In most organizations, client and server machines have ample
storage to capture application data for any testing needed for
profiling or troubleshooting. However, you do not want your
test to cause an application to run out of space and stop
working. Make sure that you understand how much space is
available on your test machines. I often recommend having at
least 5GB of storage space. This is usually enough for some
quick on-demand testing. This may not be enough for long-
term testing, so tread carefully.

2. Verify that there can be root access to your test

machines
All packet analyzers I’ve come across require administrative or
root access in order their for installation. The last thing you
want is to have testing delayed because it was later discovered
that this level of access was needed to install and run the
capture agent. Ensure that this is known and verified.

3. Ensure network devices do not block remote

capturing
Being able to remotely capture data from agent installed on a
machine at another location is very handy. Wireshark, via
remote packet capture (rpcap), and other tools allow for this.
However, most networks have firewalls and other such devices
that block certain ports. Make sure that the tool you are using
to remotely reach your capture agents has been given access
across the network.

22
 ANALYZING HTTP

4. Understand limitations due to server operating

system
There are many operating systems available that web
applications run on. You have Linux, Solaris, and Windows to
name just a couple. These can be implemented in a number of
different ways, such as Solaris Zones, Linux containers, and
Windows NLB, and that can impact whether you are able to
capture data. Make sure that you understand the operating
system and any capabilities that might limit or prevent you
from capturing data.

Data Importing Tips

Once the data has been captured, you will download it to a
specified location on your machine. Importing the captured
data into your preferred tool is analyzer-specific, so I will not
go into any detail here. However, I will say that unless you did
some pre-capture filtering, there will likely be much more
post-capture filtering to be done. This is particularly the case
with HTTP applications, since there can be many other
elements of a website or web application that generate
communication to many other servers, like DNS and file
servers, in addition to the typical three tier model involving a
database server. Filtering can make the file you ultimately
work with much smaller and more manageable.

A couple tips to keep in mind for importing captured data

include:

1. Filter your captured data as much as possible. If

possible, don’t try to import everything.
2. When there is a firewall or proxy between the client
and server, the ports used between the client and
firewall/proxy are often different between the
firewall/proxy and the server. It really depends on the
vendor and technology being used. In cases where they
are different, utilizing the IP identification header field
may help identify the appropriate ports. If the

23
 JEAN TUNIS

firewall/proxy changes this, it makes for a more difficult

analysis task.

Data Analysis Tips

After importing data for a website or web application, you
want to go back to what the reason was for the test. This will
help determine where to go first in your analyzer for your
analysis. Are you profiling or troubleshooting the application?
The answer to this helps you narrow what you want to focus on
first.

After identifying the reason for the analysis, you want to better
understand the application architecture. You want to confirm
that what you were told when you were asking the above
questions is what is actually happening. Any erroneous
information is often not the fault of the users or application
team. Sometimes, this information is either not known or
simply out of date. Organizations that have an application
testing process will more than likely (although not always)
have provided accurate information.

One way to verify that you were provided correct information

is to look at a chart or view to see all of the tiers that are
communicating. Is there anything in that view or chart, outside
of any DNS servers, that was not mentioned? If so, you will
need to find out from the application team what function these
elements serve.

Second, you also want to confirm the protocols being used, the
amount of data exchanged, any retransmissions, and more.
You want to verify some typical metrics:

- Protocols being used

- Number of legitimate retransmissions
- Data volume exchanged
- TCP Receive Window Size
- TCP Maximum Segment Size
- Observed latency during TCP connection setup

24
 ANALYZING HTTP

Next, you want to take a look at individual HTTP requests and

their responses, if the data is not encrypted. Most packet
analyzers are usually able to decode each request and display
them, largely due to Wireshark. This helps to identify issues
down to the specific request that was sent and what response
was received. Take advantage of this!

If the HTTP data is encrypted, you will obviously will not be

able to see any of the HTTP requests or responses. You will
only be able to see the Transport Layer Security (TLS)
sessions.

On the other hand, many packet analyzers have the ability to

decrypt HTTPS traffic if the TLS private key is provided. Check
your analyzer’s instructions for how this can be done.

It is not very likely that you will be able to obtain the private
key to decrypt TLS, especially in a production environment. If
you are unable to obtain the private key, you will need to focus
on profiling or troubleshooting the application based on TCP
metrics only, such as retransmissions and window sizes.

There are some other ways to get around this, like getting
server logs and correlating data in those logs to when requests
are being made. So it’s clearly possible...just more time-
consuming and sometimes more interesting!

25
 JEAN TUNIS

Nine Performance Metrics to Check

I have had to profile and troubleshoot hundreds of

applications over the years. I have seen many of the same or
similar issues that causes web applications to perform slowly.
Below are some common items (in no particular order) to
check when profiling or troubleshooting an HTTP application.

1. Number of TCP connections

The number of TCP connections is very important for an HTTP
application. This is one of the factors that can cause an
application to be chatty. Efficient use of those connections is
important. You want an application to use as few connections
as possible to transfer the data it needs to send and receive.

Most of today’s browsers allow for six concurrent TCP

connections. Each of those connections should be reused as
more data transfer is required. If you notice more than six
different connections, this could cause an increase in
applications turns, thereby likely making the application
chatty.

Ideally, you want connections to be reused, as it is specified in

the RFC. New connections should not constantly be getting
opened, as can be seen in many HTTP applications.

Also, verify that number of appropriate connections are being

opened, as per the RFC and user-agent. I have seen cases when
a corporate standard web server software was not used. I

26
 ANALYZING HTTP

noticed that only two connections were being opened

concurrently during any given time, even though the Internet
Explorer browser being used supports up to six concurrent
connections.

In this situation, I checked the configuration on the server

machine to verify the web server software. The web server
turned out to be instead using its own custom-configured
Apache web server. Further investigation showed this server
was closing TCP connections prematurely.

This performance issue went unnoticed until user distance was

increased due to a data center migration. With relatively low
latency, something like that does not cause much of an issue
for users. However, with the increase in latency, users begin to
notice the effects of having too many TCP connections because
application performance severely decreased.

2. HTTP Version
HTTP/2 is the recently ratified version of the HTTP protocol.
However, few websites have it implemented. Most
organizations are running HTTP/1.1 as the version of the
protocol that has been standardized. This was officially defined
in RFC 2068 in 1997. The original standard version, HTTP/1.0,
was defined in RFC 1945 in 1996, even though HTTP had been
in use since 1990 in some form or another. It quickly became
evident that HTTP/1.0 had its shortcomings and work and
support for HTTP/1.1 was available in 1996.

HTTP/1.1 provides some of the following common benefits

over HTTP/1.0:

→ Persistent connections – In HTTP/1.0, each client

request would generate a new connection. With this
feature, requests could reuse existing connections.
→ Chunking – HTTP/1.1 provides support for HTTP
chunking, when a web server can break-up a large file
into smaller chunks, and provides the client browser with

27
 JEAN TUNIS

a list of chunks it can expect to receive. This can help the

browser process the file faster and begin showing some of
the file to the user before the whole thing is downloaded.
→ Pipelining - HTTP/1.1 adds support for the ability of
the client to send multiple requests without a response
from the server before sending additional requests.

There are other benefits that HTTP/1.1 provides over

HTTP/1.0. However, the above have been more commonly
known. Therefore, when analyzing an HTTP application, it is
important look at the version being used, especially if you are
troubleshooting. This may be the cause of the issue.

The way to do this is to look at the protocol decodes in your

packet analyzer. Typically, the first line of the HTTP header
specifies the version being used. You would want to confirm
this in both the client and server. Both must be using the same
version. If one is using HTTP/1.1 and the other is using HTTP/
1.0, HTTP will utilize the capabilities of HTTP/1.0.

I have analyzed HTTP applications in which client and server

for the application did not use the same HTTP version. As a
result, the application reverted to using HTTP/1.0. The effect
of this was a lack of connection reuse and requests being sent
in a serial fashion, instead of parallel.

This is something I am encountering less these days, as HTTP/

1.1 has become more entrenched as the standard web protocol.
But with HTTP/2 newly standardized, I expect to run into
applications that have HTTP/2 clients with HTTP/1.1 servers.
Watch for this!

3. User-Agent
There are many browsers available for use to access websites.
The browser makes requests on behalf of the user, and is
known as the HTTP user agent. At many organizations, the
current browser standard is Internet Explorer (IE), especially
those with a longer history.

28
 ANALYZING HTTP

The latest version of Internet Explorer is IE11, as of 2015.

However, there are still some organization that may still be
using older versions, such as IE6, IE7, IE8 and IE9. I have seen
that these older versions are still in use because of legacy
applications requiring the appropriate browser version to work
correctly.

Furthermore, there are some other users using atypical

corporate standard browsers like Chrome and Firefox. These
browsers tend to be more efficient than Internet Explorer.
Applications generally tend to perform better using these
browsers, but since they are not generally supported, some
applications may not work at all with them.

The user agent is the same as the browser or HTTP client being
used to access web services. Sometimes, if you see an issue and
want to figure out what browser a client may be using, the way
to do so in the captured data is to look at the HTTP header in
the protocol decodes in your analyzer. You may even find out
that the client isn’t using a browser at all, but an HTTP client
that is using a user agent not commonly seen at your
organization.

You will find a header field called “user-agent” when you look
at the header. Somewhere in this field will be information that
clues you in on what browser version is being used.

You are usually able to identify which browser is being used by

looking at the first few words in the user-agent header field.
For Internet Explorer 7, for example, the user-agent header
field will read MSIE 7.0, while Internet Explorer 8 will see the
user-agent header field set to MSIE 8.0.

I have analyzed applications showing the user-agent as Axis

1.1. This turned out to be the user-agent being used as part of a
SOAP implementation in this application. It would override
the user-agent of the browser itself and was used to
communicate with a web application server.

29
 JEAN TUNIS

In this particular case, this Axis version turned out to be pretty

old, and only supported HTTP/1.0, making the application
inefficient. A much newer version was available, but was not
being used in this application.

Because of the above, when analyzing an HTTP application,

you want to find out what user-agent and versions are being
used. The main reason for this is that different versions
provide support for different HTTP capabilities. For example,
IE8 and above can support as many as six concurrent TCP
connections. However, IE6 and IE7 only support two
concurrent connections.

Firefox and Chrome support six concurrent connections across

their versions. For an HTTP client application that sends a lot
of requests, say for text files, images and scripts, usually the
more concurrent connections it has, the better. Because of this,
an application that uses IE8 is likely (but not always due to
other potential factors) to be faster than one running IE6. So
pay attention to the user-agent.

4. Gzip Compression
Something else to check for when analyzing an HTTP
application is whether compression is enabled. When this is
enabled, the web server compresses files before sending them
to the client. However, in order for this to occur, both sides
have to be able to support compression. It is quite common
these days to find that the client supports compression, but the
server either does not support it or it is not enabled.

Gzip compression is the default type of compression method

used in HTTP applications. This is basically similar to using
WinZip to compress a file. Most browsers today support Gzip
compression. However, not every web server does. IIS and
Apache, the two most commonly deployed web servers,
support Gzip compression, and the option is usually enabled
by default.

30
 ANALYZING HTTP

As with other capabilities in HTTP, both client and server must

support compression in order for it to occur. The client
browser does this usually by adding “accept-encoding” field to
its HTTP header. If the server supports it, you will usually see
“content-encoding” in the HTTP response header from the
server.

If you are troubleshooting an HTTP application and clients are

complaining of a slow download, for example, check that
compression is occurring on both sides. This may help reduce
the response time if there is, in fact, no compression enabled.

One thing to keep in mind that may be affecting performance

is if your organization has WAN optimization devices and how
optimization works. While WAN optimization is outside the
scope of this book, I wanted to bring up this point.

Each WAN optimization vendor is a little different in how they

optimize traffic. Many are able to optimization application
data, particularly HTTP data, by way of compression. Some
use LZ compression, which in part makes up Gzip
compression.

Compression is most efficient when the data is text-based.

When compressed, a file is no longer text, and additional
compression will not help much, if at all. Therefore, do not
expect WAN optimization to further compress data that is
already compressed.

For reasons such as this, it is still recommended that

compression occur at the server, if possible, rather than the
optimization device to help reduce overall network capacity
use. This also may help the optimization device, as it would not
have to use up processing time to compress such data.

31
 JEAN TUNIS

5. Caching
Caching is commonly known as beneficial to application
performance, particularly with HTTP applications. By default,
most browsers support client-side caching. However,
depending on the application, not everything is cacheable and
some servers do not allow it.

When analyzing HTTP applications, you want to look out for

files that are typically cacheable and whether they are being
cached or allowed to be cached. For example, files that tend
not to change very often, like images and scripts are usually
cacheable. Also, GET requests are generally much more
cacheable than POST requests. These POST requests are often
used in applications that contain dynamically updated
information, and therefore do not generally allow for caching.

The way to check whether caching is allowed again is to check

the HTTP header. A client browser will specify whether it can
cache data using the “cache-control” or “pragma” fields. They
will contain either cache, no-cache, or some other parameters
to specify to the server its caching capability. Also, if no such
header field exists, caching is not enabled. A server that
supports caching will respond with the same header fields.

In addition to the above header fields, there are other HTTP

header fields that specify not only whether caching is
supported, but also how long data can be cached and when a
cached file was changed or expires. For example, the “last-
modified” field specifies when a particular file was last
changed. This helps the browser determine whether a cached
file it already has needs to be requested or not.

6. Connection Keep-Alives
HTTP Keep-alive (or HTTP persistent connection) is a
capability that has been part of HTTP since HTTP/1.1. With
persistent connections in HTTP/1.1, it is this capability that

32
 ANALYZING HTTP

ensures a TCP connection stays open so that it can be reused.

This will help keep the number of connections down. This will
also reduce the time of having to go through the TCP
connection setup. This can take anywhere from tens of
milliseconds to hundreds of milliseconds. Depending on the
nature of your web application, this can be seen as an eternity
by your users.

When analyzing a slow HTTP application, one of the things

you want to look at is the keep-alives. In the today’s browser,
for example, there normally are six concurrent connections. If
the application has keep-alives enabled, this should be the only
six connections throughout an application’s use.

One way to identify whether connections are being kept alive is

by looking at your packet analyzer’s port counts. If there are
many different TCP ports being opened, something is closing
the connections. You next have to confirm whether the cause
of this is the client, server, or some intermediate device, which
can be done looking at the TCP and HTTP headers in your
packet analyzer.

In order for a connection to be kept alive, both client and

server must support this. If either does not support HTTP
keep-alive, the connections will be closed once the requested
data has been sent.

The way to check that either client or server is allowing keep-

alives is to look at the HTTP header. If keep-alive is allowed,
the header will specify the following parameters: connection:
keep-alive. If not, the following parameters will be found in the
header: connection: close.

In many HTTP applications, keep-alives are allowed. However,

there are some applications that do not allow this, and it must
be investigated with the application team to determine
whether this is the desired configuration. Unfortunately, it is
all too common that keep-alives do not occur even though they

33
 JEAN TUNIS

are allowed in the headers. This is often found with HTTPS-

enabled applications.

The TLS protocol used as the underlying encryption protocol

for HTTPS contains a similar capability as the HTTP keep-alive
called TLS Session Persistence. Opening up a TLS session adds
to the overhead because the encryption keys have to be
exchanged. This takes time and slows down an HTTP
application, especially when taking distance into account. TLS
Session Persistence helps reduce the number of TLS sessions
that get opened up, thus reducing round trips, and reuses
existing sessions without exchanging the keys again.

Unfortunately, what you will find with many HTTPS

applications is that TLS persistence does not occur. For
example, in Internet Explorer, this is due to how IE (and other
browsers) handles TLS, which forces the TLS encryption keys
to be exchanged every so often during application use. More
recent versions of IE are more efficient than their
predecessors, but applications that contain pop-ups or open
new IE tabs are likely to force the TLS keys to be exchanged in
IE, and thereby reduce the effect of TLS Session Persistence
and HTTP keep-alive.

When troubleshooting or profiling an HTTP application, check

for keep-alives if you see more than six TCP connections. TCP
connections more than this suggest that either keep-alives are
not enabled or TLS is being used. As mentioned above, with
HTTPS, it is quite common that you will find more
connections. With HTTP, the number of connections should be
low as a result of keep-alives. If not, it is very likely that either
the client or server is not allowing it.

7. Application turns and chattiness

An application turn is the amount of round-trips it takes for a
particular transaction between a client and server to be
completed. HTTP applications are known to be high-
application-turn applications because of the many requests

34
 ANALYZING HTTP

that can make up a transaction. One client request for a web

page can include many requests for the download of the HTML
file, a number of images, a number of JavaScript (JS) files, and
other cascading style sheets (CSS) files. All of this can make
for a high number of application turns.

High application turns have been commonly discussed as an

effect within an application that causes it to perform worse as
latency is increased between two tiers, typically, user client and
server. High application turns can make an application very
chatty, thereby decreasing its performance over a WAN.

An application is chatty when there are not only a lot of

application turns, but also when the size of the data being sent
back and forth is relatively small. This is typical of HTTP
applications that are running over SOAP. Such applications
should try to send fewer and larger packets instead of these
many small packets.

Your packet analyzer can be used to identify both high

applications turns and chattiness. It’s easier to do in some
cases than others, but it can be done.

In order to be efficient, a web application should be sending

data at close to the maximum segment size (MSS). The MSS
can vary depending on the application and technologies in use
between the client and server, such as virtual private
networking (VPN), that may reduce the MSS.

However, by default, the MSS is about 1460 bytes. At this byte

count, I would recommend that your application be sending at
least 1000 bytes to keep the pipe close to being full. Going
below this number, particularly far below, will make the web
application susceptible to degraded performance as latency
increases.

So, I’ve talked about when to know when an application is

becoming too chatty. I want to talk about when an application
is considered to have too many application turns. There is no

35
 JEAN TUNIS

specific threshold that I know of for this metric. However, I

have learned over the years that there is a way to determine
this.

Using your packet analyzer, you find the number of application

turns. This is something that is much easier to do when the
application is using HTTP. You can then look for HTTP GET
requests and their associated responses, for example, and get a
count. Each request/response pair is one turn. These can be
grouped together as one client message as well.

What you want is for the application turns to be much less

than the message groups. If both values (turns and groups) are
close to each other, the number of application turns is
relatively high and the application can be considered to be
communicating inefficiently. For example, if you have a client
and server exchanging 90 message groups and it takes 89
application turns to accomplish it, this is too high. As latency
increases between the client and server, application
performance will begin to degrade.

On the other hand, if client and server are exchanging 800

message groups with 70 application turns, this is low and this
application is likely communicating much more efficiently.

8. HTTP Chunking
HTTP chunking is a capability in HTTP that allows a server to
break up the data it needs to send to a client into chunks. As
mentioned above, this was added in HTTP/1.1 and helps client
browsers process received data that can be displayed to the
user faster. By default, many web and application servers
break up data into 8192 bytes. What this means is that if a user
requests a 64KB file, the server will send it in about eight 8KB
chunks.

The reason this is an important thing to check is because

sometimes servers specify a small amount for the chunk size. I
have seen chunk sizes as low as a tens of bytes. This greatly

36
 ANALYZING HTTP

affects how much data TCP can send over the network,
regardless of how big its congestion window.

You can identify whether data is being chunked by looking at

the HTTP transfer-encoding field in the server’s HTTP header.
Chunking is not something that a client browser needs to
support. It is strictly a server capability to help get data to the
client faster.

If you are analyzing a slow performing web application, a small

HTTP chunk size may be one of the culprits. If large files are
being transferred, most of the chunks should be closer to 8KB,
if default, or whatever the current chunk size is on the server.

If a low chunk size is being used as the default, you should

consider having the application team increase this to 32KB or
64KB and testing it to allow the server to send more data in
one shot, thereby potentially making the application much
faster. You should also test to ensure that this change does not
have an adverse impact on server memory.

9. Number of Client Requests

The number of application turns becomes high when a client is
sending many requests to a server. The HTTP protocol, by its
very nature, consists of many client requests. As mentioned
above, there can be requests for multiple HTML pages, many
image files, and many script files, all around the same time.
Even with persistent connections in HTTP/1.1, too many client
requests can lead to high application turns, thereby causing a
degradation in performance with increased latency.

In order to see the client requests in your packet analyzer,

either the application must not be using encryption, i.e. no
HTTPS or any other form of encryption, or you have received
the server’s private key to decrypt the traffic. If either is the
case, you should be able to see all the traffic and this should
make your analysis a bit easier.

37
 JEAN TUNIS

A typical HTTP communication these days can contain many

requests between the web and application servers for a four-
tier application. There can be many different types of requests
for HTML files, images (.JPG and .GIF), scripts, CSS, and
other files (.VBS, .DO, etc). Requests such as these are simply
far too many and will make an application perform poorly once
latency is increased between user and server. Developers
testing with local server resources will not be able to tell if an
application with this many requests will have any issues unless
some testing is done with a WAN simulation or emulation
product tool.

In such a situation, the application team needs to look at

compression, caching, file consolidation, and any other means
to increase performance. File consolidation, for example, is
one that is often recommended for applications. Application
teams should look at combining a server’s image files, for
example.

There isn’t necessarily a rule of thumb for the number of client

requests, but you should always look to reduce the number of
requests as much as possible. If an application is suffering
from slowness, a common thing to look for is too many client
requests for images and scripts. Requests for many images,
especially if they are rendered near each other on a web page,
should be considered for consolidation to one image in the
form of an image map. Likewise, requests for many JavaScript
and CSS files should be considered for consolidation into one
file. Changes such as these are unlikely to occur for third-party
vendor software, but should certainly be considered for
custom-developed in-house applications for your organization.

38
 ANALYZING HTTP

Five Common HTTP Analysis

Recommendations

Having been involved in many profiling and troubleshooting

analyses of web applications over the years, I have noticed that
many of the same issues keep coming up, and therefore,
require the same or very similar recommendations. The
following are some common recommendations I have provided
for web applications, in addition to what was discussed above:

1. Investigate WAN optimization deployment

The area of WAN optimization has been around for a number
of years. This service usually involves adding two devices
between the client and the server that work to optimize traffic
before going over the WAN and then de-optimize that traffic
when it arrives at a local network. One of the first vendors to
offer this service was Riverbed in 2002. Now, on the other
hand, there are more than ten vendors in this space, with
Riverbed and Cisco being the two largest.

This recommendation is usually given to the application team,

suggesting that they look into how WAN optimization can
positively or negatively impact their application. Many
organizations have long since utilized optimization techniques
for improved performance, but not everyone is always aware.
Such devices are not always deployed everywhere on a
network. They are most often between remote locations, where
the users are, and the datacenters, where servers tend to be. An
application team needs to understand where optimization

39
 JEAN TUNIS

devices are deployed and how or whether their application

could be impacted by this.

Such a recommendation is particularly needed if an

application resides in an area of the network where there is no
optimization in place, but there are expansion plans for this
application to more locations. For example, an application in
which its servers and users only reside in a datacenter location
or campus is not going through any optimization devices.
Therefore, there would be no concern about the impact of
optimization. However, if this application has global usage
within your organization, user transactions will go through
these devices, and the application team should understand the
potential impact.

In general, WAN optimization positively impacts applications,

especially HTTP. However, I have seen instances in the past
when this was not always the case, albeit more so for non-
HTTP data. For example, some optimization techniques can
adversely affect some database traffic. In a three-tier
application architecture, it is always important to understand
the deployment of the database server with respect to the
application or web server with which it communicates.
Therefore, it is important to recommend this to the application
team so that they are not surprised later by any slowness after
application deployment. Furthermore, this is another reason
why you, as the analyst, should ask many of the questions I
previously discussed.

2. Increase TCP Receive Window Size

Since HTTP runs on top of the TCP protocol, one of the
common reasons for a web application to perform slowly,
particularly during the download of a file, is a TCP receive
window that is too small. When a user requests the download
of a PDF file, for example, the user’s PC and the server specify
their receive window sizes.

40
 ANALYZING HTTP

The default maximum TCP window size is 65,535 bytes.

Current Windows and Linux operating systems support this.
However, sometimes network settings are implemented
incorrectly on a server or changes are made on a PC causing a
deviation from the default. Furthermore, applications have the
ability to set the TCP congestion window for both receive and
send with a socket call that can limit TCP’s performance.
Therefore, when looking at a web application that users are
saying is slow, one thing to check for is the TCP window sizes,
and make the appropriate recommendation to update them.

However, be careful with this recommendation on PCs and

servers with limited RAM. Data stored for the TCP congestion
window is kept in RAM, and a machine with low memory will
be adversely affected from increasing the window size too
much.

3. Optimize Database Queries with Multiple Row

Requests
Most web applications today are multi-tiered. Much of the data
is stored in a database that is talking to a web or application
server. At many organizations, most HTTP applications
contain four tiers – client, web server, application server, and
database server.

When a user makes a request to the web server for a particular

file, some or all of that data may be stored in a database. The
application server relays the request from the web server to the
database server. However, because the database is often near
an application or web server during the testing and quality
assurance (QA) phases, application developers may not always
focus on how the two communicate.

Furthermore, much of the communication is inherent in how

databases work, and the developers may not have a whole lot
of control over this for various reasons. Therefore, the requests
are not very efficient, leading to a lot of chattiness.

41
 JEAN TUNIS

There are also cases when the client makes direct requests to
the database with this same type of inefficiency. One of the
biggest reasons for this is that the SQL queries used request
data one database row at a time. This is typical of database
communication without explicit configuration changes.

You can tell whether this is the case through a couple of

methods. One method is to look at the decode summary in
your packet analyzer and see the data transfer between the two
tiers. If your analyzer does not have the ability to decode
database traffic, you should consider using Wireshark. At any
rate, for an Oracle database, for example, the client requests
will state “fetch a row” in the decodes. This is telling you that
the database client is asking for data one row at a time.

A second method is to look at your packet analyzer time

sequence diagram. Most come with one, but may have a
different name for it. Whatever the name your analyzer uses,
look at the sequence diagram and pay attention to the amount
of interaction between the tiers. The busier the interaction, the
chattier the communication. Chatty communication very likely
means the database requests are for one row at a time.

If the above two methods used show single-row requests, you

should recommend this be changed, if possible.

4. Investigate remote desktop deployment

Remote desktops have been in use for many years. Windows
RDP and Citrix ICA have been two protocols and related
solutions that organizations have used for many years to allow
remote users to access a web application that would otherwise
be terribly slow if run over the WAN. There are many other
players today, but this option is something that can help.

This recommendation is usually a last-resort option. Many

organizations already have server farms dedicated for remote
desktop services. However, it may not always be easy or cost-
advantageous for an application to do this.

42
 ANALYZING HTTP

There are times when the application team cannot or will not
accept any of the other recommendations to remedy a slow
application. This is most often the case because the application
is vendor-based, and changes are either not likely occur or may
not occur for a long time. If that is the case, testing the viability
of remote desktop solutions should be considered.

5. Enable TCP window scaling

TCP window scaling is a capability of the TCP protocol to has a
receive window that is larger than 65,535 bytes. The reason for
this is the number of bits preserved for the window, which is 16
bits. Since the maximum this provides is 65,535, a scaling
factor is used so TCP can specify the true size of its window.
Once scaled, this allows the TCP protocol to send more data
over the network with less acknowledgments, allowing for
more network throughput and faster download times.

Window scaling is supported by practically every operating

system. However, it is only implemented by default in certain
Windows versions such as Windows 8 and Windows Server
2012. If your organization is still running older versions of
Windows, double-check your TCP settings. Linux and other
Unix-based operating systems have had this implemented by
default since 2004.

When scaling is enabled, the TCP protocol can theoretically

increase the receive window size to as much as 1,073,725,440
bytes, or about 1GB. This is done using a scaling factor from 0
to 14. To get the scaled window size, you multiply the current
window size by 2^(scaling factor). For example, if you want to
double the TCP receive window size from 65,535 to 131,070, a
scaling factor of 1 is used (65,535*2^1=131,070).

It is always a good thing to send more data if the network can

support it, which many, but certainly not enough,
organizational WANs can do today with Gigabit speeds. TCP
scaling allows for this, and therefore, should be recommended

43
 JEAN TUNIS

as often as possible, especially for HTTP applications that are

file-download intensive.

However, an adverse effect is that this increases memory use

on a PC or server. Since the increased window size in bytes is
stored in RAM, enabling TCP window scaling can degrade PC
or server performance on a machine with low memory,
especially if the scaling factor is too high. This is not likely to
be an issue on many of today’s client machines, since many
contain multiple gigabytes of RAM built-in, but there are still
companies running older server hardware for continuity
purposes. If your company is one, this is something you must
be cognizant of.

For scaling to work, like other options in TCP and HTTP, both
sides must support it. To determine whether TCP scaling is
enabled, you can check the TCP options in the protocol
decodes in your packet analyzer. When looking at the decodes
for the TCP protocol, you should see a “Window scale:” field
with the value of the scaling factor.

44
 ANALYZING HTTP

The Here Future: HTTP/2

Some of the recommendations mentioned above are due to the

limitations of the HTTP/1.1 protocol. These limitations have
been known for years, and plans for an updated version of
HTTP have been in the works. As I mentioned above,
HTTP/1.1 provided a number of benefits over HTTP/1.0, but
many other limitations remained.

HTTP/1.1 Limitations
Although HTTP/1.1 has a number of limitations, this has not
stopped the protocol from being probably the most popular
application layer protocol, and arguably the most popular
behind TCP. However, as websites and web applications and
now, mobile websites and applications, make more use of the
protocol, its limitations are becoming more of a pain. In this
section, I want to discuss some of these limitations.

Many Small Requests

A typical web application makes many requests, as mentioned
above. These requests contain data as small as a favicon image,
for example. So many small requests can be made, and such
requests make the protocol latency sensitive due to its chatty
nature. HTTP/1.1 did not do anything to address this.

45
 JEAN TUNIS

Pipelining Not Multiplexed

As I mentioned above, HTTP Pipelining was supposed to be a
great benefit over HTTP/1.0 that would allow clients to send
multiple requests before receiving a response from the server.
However, pipelined requests were not capable of being
multiplexed.

This means that when the server did receive a client request, it
still had to reply in the order the requests were received.
Unfortunately, if one of those multiple requests got lost en
route to the server, it could not respond to the other requests
until the lost request was re-sent. This is due to the nature of
how the TCP works, thereby causing what’s known as head-of-
line blocking.

This would lead to decreased performance, and because of this,

many of the browsers disabled this feature.

Short-Lived Requests
As mentioned previously, HTTP requests can be for many
images or files. Because of this, many requests don’t last very
long. As a result, the short-lived nature of HTTP always ends
up being impacted by TCP slow-start algorithm.

The TCP slow-start algorithm is a feature in the TCP protocol

that forces requests sent over TCP to slowly ramp up the
number of segments that can be sent. This is known as the
initial congestion window, and it is set as the number of TCP
segments. As TCP ACKs get exchanged between the client and
the server, the window doubles until the maximum receive
window size is reached, including any TCP scaling. This is done
to ensure that the network not overwhelmed and can support
the data needed to be sent.

By default, in most operating systems, the initial congestion

windows is set to three. The Linux 2.6 kernel update sets this
to 10 after some testing done at Google2. However, Windows
has not yet made a similar update.

46
 ANALYZING HTTP

Remember that the maximum segment size (MSS) is typically

set to 1460 bytes. So for three segments, the initial TCP
window is set to about 4380 bytes, or about 4KB. This is rather
small when you consider that an image like a favicon, for
example, can be only a few hundred bytes. Other images can
easily be less than 64KB, so its possible for an HTTP request to
never have the need for up to 64KB of data, thereby always
getting adversely impacted by TCP slow-start.

Even with the change in Linux, this is only about 14KB. As a

result of the low setting, most HTTP requests are still in TCP
slow-start when they are completed.

Workarounds
As the popularity of the HTTP protocol grew, workarounds
have had to be used to get around the HTTP/1.1 limitations.
The following are some workarounds that have come up over
the years.

Multiple Connections
As mentioned above, today’s browsers allow six concurrent
TCP connections. It wasn’t always this way. Previous versions
of Internet Explorer, for example, only allowed two concurrent
connections. As these limitations began to impact web
performance, browsers went up to six. This was done so that
multiple HTTP requests could be sent and to help avoid the
impact of TCP slow-start.

Domain Sharding
As mentioned above, there can be too many connections, and
that can adversely impact performance. The number of
concurrent connections in today’s browser is what it is in part
because you do not want too many connections. However, this
limits a website that has numerous small images or files that
need to be sent to the browser.

47
 JEAN TUNIS

Domain sharding is a way to give web applications more than

six concurrent connections. This is done by utilizing sub-
domains to render all of a website’s resources. For example,
there can be one.domain.com, two.domain.com and
three.domain.com. Each domain has its own set of resources,
and can therefore have six connections each, for up to 18
concurrent connections!

Resource Inlining
As I mentioned previously, there can be too many connections.
Too many connections introduce the user to any delay that
may be on the network. The best way to reduce this delay is to
avoid it. Resource inlining involves putting your scripting code
into the HTML so that the browser can execute it without
making a request to the server for the script file.

Next Version
To help solve, not just work around, many of the HTTP/1.1
limitations, Google released the first draft in 2009 of what they
called the SPDY (pronounced speedy) protocol. The main goal
was to reduce the time it takes for web pages to load.

SPDY’s success, although not without its criticisms, led to it

being used as the starting point for creating the next version of
HTTP. And in May 2015, the HTTP/2 protocol was published
as the new proposed standard as RFC 7540.

The new protocol is not that dissimilar from HTTP/1.1. HTTP/

2 is expected to be a more efficient version HTTP/1.1, with the
following changes:

→ It uses only one TCP connection. There will be no need

to have multiple connections.
→ Browser requests are multiplexed, instead of pipelined.
→ It is binary, and not text-based, to allow for the
multiplexing. This will limit us to some degree as analysts

48
 ANALYZING HTTP

because we will not be able to see any of the clear-text

data.
→ It contains different types of frame types. There are
header frames and data frames that get sent and received.
→ It can prioritize requests over others.
→ Headers are compressed. This helps solve a problem
that has arisen over the years with caching. The size of
the HTTP header was growing very large because of the
cookies used to manage browser caching.
→ An HTTP data stream can be reset without resetting
the TCP connection. This helps avoid having to open a
new TCP connection for future requests. This reduces
connection setup time and impact of TCP slow-start.

As HTTP/2 is only in its infancy, it has not yet been deployed

by many organizations. However, with most of today’s
browsers supporting it, we can expect more websites to
transition to it. We will need to start analyzing some of this
traffic to get accustomed to the various issues associated with
this new version of HTTP.

Some exciting times should be ahead!

49
 JEAN TUNIS

Conclusion (and Thank You)

The above was an overview of the HTTP protocol along with

numerous tips and recommendations on how to profile and
troubleshoot a website or web application using this protocol. Much
of this information is from my experience as a network analyst for
over a decade. However, there are many other resources online and
below are two places I recommend that you review for additional
information and HTTP best practices:

→ https://developers.google.com/speed/docs/insights/about:
Google page that discusses how to optimize HTTP.
→ http://developer.yahoo.com/performance/rules.html:
Yahoo page that discusses how to optimize HTTP.

Thank you for reading this book. If you found it valuable to your
work as an analyst, engineer or developer, let me know. I’d like to
learn what helped. Email me at [email protected].

And if you have a friend whom you believe this book can help in
their work, please share it with them.

If you have additional tips and recommendations of your own, reach

out to me. I’m always looking to learn more ways to profile and
troubleshoot applications.

In addition to email, you can find me at

https://www.rootperformance.com.

Happy analyzing!

Jean Tunis

50