Exploration of TCP Parameters for Enhanced

Performance in a Datacenter Environment

Mohsin Khalil∗ , Farid Ullah Khan#
∗ National University of Sciences and Technology, Islamabad, Pakistan
# Air University, Islamabad, Pakistan

Abstract—TCP parameters in most of the operating sys- Linux as an operating system is the typical go-to option for
tems are optimized for generic home and office environments network operators due to it being open-source. The param-
arXiv:1905.01194v3 [cs.NI] 5 Sep 2019

having low latencies in their internal networks. However, this eters for Linux distribution are optimized for any range of
arrangement invariably results in a compromized network
performance when same parameters are straight away slotted networks ranging up to 1 Gbps link speed in general. For this
into a datacenter environment. We identify key TCP parameters dynamic range, TCP buffer sizes remain the same for all the
that have a major impact on network performance based on networks irrespective of their network speeds. These buffer
the study and comparative analysis of home and datacenter lengths might fulfill network requirements having relatively
environments. We discuss certain criteria for the modification lesser link capacities, but fall short of being optimum for
of TCP parameters and support our analysis with simulation
results that validate the performance improvement in case of 10 Gbps networks. On the basis of Pareto’s principle, we
datacenter environment. segregate the key TCP parameters which may be tuned for
performance enhancement in datacenters. We have carried out
Index Terms—TCP Tuning, Performance Enhancement, Dat-
acenter Environment. this study for home and datacenter environments separately,
and presented that although the pre-tuned TCP parame-
ters provide satisfactory performance in home environments,
I. I NTRODUCTION
however, the network performance in datacenters would be
Today, majority of the networks being used in home compromized in this case. Therefore, we highlight the impor-
and office environments generally have speeds ranging from tant TCP parameters that can be modified for performance
100 Mbps to 1 Gbps. Although the biggest strength of the enhancement in a datacenter environment and present results
Internet is the TCP/IP hourglass as it hides network and in this regard.
applications from each other, however it also hides the flaws This paper is divided into five sections. Section II of
unfortunately. Appropriate selection of parameters plays a the paper highlights the identification of network parame-
vital role in affecting the network performance [1], because ters whose modification would directly impact the network
their improper configuration may result in compromized performance. The criteria and evaluation methodology for
operation, something that has been proven repeatedly during selecting the appropriate values of network parameters in
the performance evaluation of networks [2]. The impact of Linux operating system has been appended in Section III.
communication growth on economy may be gauged from The results and observations obtained after tuning of these
the fact that Information and Communication Technology network parameters are summarized in Section IV, followed
(ICT) resulted in contributing 1% GDP increase in case of by conclusion and scope for future work in Section V.
Singapore [3].
Many studies have been carried out to ascertain TCP per- II. S ELECTION OF TCP PARAMETERS FOR D IRECT
formance in diverse environments and various improvements I MPACT ON P ERFORMANCE
have been suggested in this regard [4]–[10]. Pontes et al.
[11] have demonstrated the impact of various TCP parameters Various parameters were explored to identify the ones
by evaluating their adjustment for mobile devices. Moreover, which would have a direct effect on network performance.
Saha et al. [12] have carried out evaluation of TCP in office In order to make this paper self-contained, we briefly dilate
environment at 60 GHz. However, these results are not valid upon the key parameters having a significant impact on the
for the case of a datacenter environment as there is no datacenter network performance in ensuing paragraphs.
fit-to-all solution. It has been proved that instead of home
environment, a datacenter environment is more suitable to A. Selection of Performance Metric
be taken as a blueprint for performance enhancement and In our work, throughput was chosen as the metric for
buffer length adjustments in data acquisition networks [13]. assessing the network performance. It is the most widely
used benchmark and is also sometimes used interchangeably the maximum frame size in a transmission, so it may be
with link capacity. So it directly refers to the performance considered as the upper bound to the size of the frame. The
of the network in terms of data rate measured in bits per increase in the size of frame will result in a decrease in
second (bps). Optimum throughput is pivotal to ensure good complexity, because it is similar to the truck transportation
network performance. It can be measured by using any of the analogy; if some cargo from is to be transported from one
large number of tools available online with varying levels of place to another, less number of heavy-trucks would be
accuracy. Since we are identifying the TCP keys on the basis needed to carry more cargo at a time, whereas, if same
of mice-elephant phenomenon, therefore our approach would cargo is carried by large number of mini-trucks, the toll
yield an improved performance instead of an optimal one. plazas at roads would require more processing, wait and
time duration. Therefore, large frames will result in better
B. Adjustment in TCP Socket Buffers data rates compared to smaller ones. It is better to send
large frames only if network quality is good and packet loss
Kernels of operating systems allocate some space to hold probability is low. This is because if network is congested,
TCP data before processing in given socket known as buffers. default packet sizes would be preferred in such situations as
The receiver keeps on storing the data in buffer before large frames will severely impact the network performance
application starts processing on that data and continuously in this case.
tells sender about space in its buffer. If the receiver buffer
is full, sender waits and keeps on storing data in its sending III. E VALUATION M ETHODOLOGY & N ETWORK
buffer during time of wait. Socket buffers are very critical S CENARIOS
in terms of affecting the network throughput. If the socket
buffer is very large it will result in occupying more memory Linux was utilized as an operating system in all the
making the system slow. If it is very less it will increase corresponding hosts which were spawn as server / client
packet loss which results in decrease in data transfer speeds. for analysis of network performance and tweaking of TCP
parameters. However, the network selection for carrying out
the performance analysis was classified into two phases. The
C. TCP Selective Acknowledgements tests / trials during the two phases remained the same with
If Selective Acknowledgement (SACK) is enabled, SACK the exception of network selection and adjustment of relevant
messages with duplicate ACK are also sent to sender with parameters.
original sequence of received packets. Sender can easily
understand the situation after looking at duplicate ACK and A. Phase-I (Home Environment)
sequence of SACK messages and repeated duplicate ACKs
Phase-I of the experimentation comprised of following two
are not needed in this process. Sender sends exactly same
different scenarios:
packet which was missing because whole scenario is known
• The first case was to establish a local area network
by sender.
between two computers and to check the throughput of
the network. The same process was then repeated after
D. TCP Queue Length tweaking of TCP / network parameters.
Maximum value of queue size should be chosen in such a • The second case was to analyze the throughput over the
way so that packet drop may be avoided due to unavailability home network at relatively low internet speeds. Similar
of memory space or a fast transmitting sender. Generally, to first case, the activity was carried out separately
the default size of this queue length is 1000 packet count for default parameters and same process was repeated
for Linux and most of other operating systems. However, the multiple times with varying values of TCP parameters.
actual queue size at any time depends upon the characteristics Thus the analysis was carried out to obtain results in two
of the network. Small value is recommended in case of high networks within home environment: LAN of bandwidth
network latencies, however high value can be chosen for high 1 Gbps and broadband / DSL with bandwidth 10 Mbps.
speed connections in specific cases to facilitate large data
transfers. B. Phase-II (Datacenter Environment)
The Phase-II also involved two different cases, which are
E. Size of Maximum Transmission Unit listed as follows:
Default Maximum Transmission Unit (MTU) in local • The first one involved the virtual machine (VM) deploy-
area networks is generally 1500 bytes, however it may be ment in Datacenter environment. For this purpose, two
increased for environments having link speeds higher than 1 VMs were spawn in East America region on Microsoft
Gbps [14], [15]. Since MTU assists the TCP in determining Azure such that their network interfaces supported
 1 Gbps bandwidth. These VMs were selected from BDP. Zhang et al. in [16] proposed a switch buffer allocation
the DS-2 Category in Microsoft Azure. Subsequently, protocol (SAB) for bandwidth sharing in datacenter networks
throughput of the network was checked separately for which assesses the buffer sizes of all the switches along
default parameters and same process was repeated with a particular path and then adjusts the congestion windows
varying values of TCP parameters. accordingly. They advocate that the full bandwidth utilization
• Similarly, Case-2 also involved the VM deployment in in a network may be ensured by allocating the buffer size
Datacenter environment. However for this case, two greater than its BDP value. In this context, we analyzed
VMs were spawn in East Asia region on Microsoft the networks in home environment as well as datacenter
Azure such that their network interfaces supported 10 environment. The networks having buffer sizes less than BDP
Gbps bandwidth. The VMs were selected from the were assessed for performance after increasing their buffer
Alpha-8 Category. Subsequently, network analysis was lengths higher than the BDP value.
carried out for varying values of TCP parameters.
3) Throughput Measurement: Throughput can be mea-
sured by using any of the available tools with varying levels
C. Linux Keys Resulting in Significant Performance Impact of accuracy. We kept the network in operation with running
upon Modification IPERF for shifts of consecutive 10 hours across several dates
Proper adjustment in TCP buffer can result in increase in for recording the instantaneous throughput at every second in
throughput. Every connection is specified by distinct buffer addition to the cumulative average to cover maximum portion
spaces for input and output. Therefore after thorough study, of the working day for datacenter environment.
values of various TCP keys in Linux OS were modified to
study the impact on network performance. As a result, the IV. R ESULTS AND O BSERVATIONS
following parameters resulted in major impact on network By default, the minimum value of TCP transmit and
performance upon modification: receive memory buffer size is 4096 bytes, while their initial
• tcp moderate rcvbuf : Dynamically updates receiver values correspond to 16384 and 87380 bytes respectively. We
buffer size for generic network. Unique bounds for a will analyze that how deviation from default values impacts
particular network environment the performance over different networks.
• tcp rmem: TCP receive buffer memory size
• tcp wmem: TCP transmit buffer memory size A. Results from Phase-I
• rmem max: TCP maximum receive window
No performance enhancement was observed by modifi-
• wmem max: TCP maximum transmit window
cation of aforementioned TCP parameters in Linux kernel.
• tcp sack: TCP Selective Acknowledgement
Upon analysis, it was revealed that by default, TCP receive
• MTU: Maximum Transmission Unit
buffer size is 85 KB which equals 699 Kbits. For Case-1
• txqueuelen: Transmission Queue Length
(LAN), BDP is 109 bps x 0.12 ms = 120 Kbits and it is
even lesser for Case-2 because of much lower bandwidth
D. Criteria for Modification of TCP Parameters (10 Mbps). So the default buffer size is still higher than the
Miscellaneous criteria were explored on behalf of which BDP and no more improvement in network performance is
optimum values of TCP parameters can be set to ensure possible in both cases of home environment.
maximum throughput for different connection speeds.
B. Results from Phase-II
1) Analyzing the Bandwidth-Delay Product: The
In Case-1 (1 Gbps link), small improvement in perfor-
Bandwidth-delay product (BDP) is amount of data measured
mance was observed for VMs of Microsoft Azure DS-2
in bits or bytes. In fact, it is data that has been transmitted in
category as a result of buffer size variation. The reason was
network but did not receive acknowledgement. It is necessary
investigated and it was found out that the BDP for Case-1
to increase buffer sizes in accordance with bandwidth delay
is 109 bps x 1.49 ms = 1.49 Mbits which is higher than the
product to increase throughput of network. In case of home
default TCP buffer size of 699 Kbits. So increase in buffer
and office environments, latency is generally negligible and
size was carried out and results are appended in Fig. 1. It
bandwidth is very large, which may slightly increase its
is pertinent to mention that the throughput was improved
BDP as compared to other scenarios.
from 501 Mbps for default buffer size to 506 Mbps upon
2) Utilization of Buffer Allocation: It is well known that increasing buffer size. However, since the link capacity was
round-trip times (RTT) within a Datacenter environment are under 1 Gbps, therefore the modification of other parameters
less than other conventional network scenarios. For this rea- such as MTU and Queue length did not contribute any further
son, buffer sizes allocated to switches are much higher than improvement in network performance.
 Fig. 1: Throughput (in Mbps) vs Buffer size for Fig. 3: Throughput (in Gbps) for different MTU
1 Gbps link in Datacenter environment sizes against buffer lengths (in MBs)

Fig. 2: Throughput (in Gbps) vs Buffer size for Fig. 4: Throughput (in Gbps) for two MTU
10 Gbps link in Datacenter environment sizes against varying queue lengths (in packets)

V. C ONCLUSION & F UTURE W ORK

On the other hand, the performance improvement was After carrying out performance analysis of various net-
more significant for Case-2 (10 Gbps link) . It is worth works, we conclude that for LAN speeds up to 1 Gbps and
highlighting that the BDP for Case-1 was higher than default various other internet connections, performance improvement
buffer size, however, the BDP for Case-2 is 1010 bps x 1.58 by modification of TCP parameters is not feasible and the
ms = 15.8 Mbits which is greater than even Case-1 and operating system algorithm is smart enough to meet the
much higher than default buffer size. So the buffer size has changing requirements. This is because the default buffer size
to be increased to maximize link utilization. Same activity is higher than the BDP for home environment due to low
was subsequently carried out and the corresponding results RTT (around 0.1-0.2 ms), whereas it is higher for broadband
are shown in Fig. 2. Throughput (in Gbps) has been plotted case due to low bandwidth. On the other hand, improvement
against various buffer sizes (in MBs) and higher throughput in TCP performance is possible for datacenter environment
for high buffer sizes can be seen. because even for 1 Gbps network interface, the RTT is at
least 10 times higher than that of generic home network
For Case-2 (10 Gbps link), greater MTU size yielded
which makes BDP high. So for 1 Gbps datacenter case, the
higher throughput. At MTU size of 10,000 bytes, maximum
performance improvement can be observed by modification
throughput was observed against any buffer length. Fig.
of buffer size. However, in case of 10 Gbps datacenter
3 depicts the throughput for different MTU sizes against
case, the performance improvement is largely contributed by
varying buffer lengths in MBs.
increased buffer size, high MTU and larger queue lengths.
Queue length was sufficient for 1 Gbps link in Case-1, but Further extensions of this work may involve the consider-
for 10 Gbps it was not enough. So the queue lengths were ation of latency as performance metric, since the networks
increased in order to enable high transfer rate. The activity having frequent speed variation would result in a fluctuating
was carried out for queue lengths of 1000 (default), 2000, BDP, which would render this approach falling short. Since
4000, 8000 and 16000 packets as shown in Fig. 4. Maximum this work focused on achieving improved performance, a
throughput was observed for 8000 queue length for default more comprehensive study may be formulated to achieve
MTU size as well as 10,000 byte MTU size. optimum network performance by consideration of all pa-
rameters as constraints with an aim to maximize the network
performance for optimum network utilization.

R EFERENCES
[1] M. Khalil, A. Khalid, F. U. Khan, and A. Shabbir, “A Review of
Routing Protocol Selection for Wireless Sensor Networks in Smart
Cities,” in 2018 24th Asia-Pacific Conference on Communications
(APCC). IEEE, 2018.
[2] C. Gomez, A. Arcia-Moret, and J. Crowcroft, “TCP in the Internet
of Things: From ostracism to prominence,” IEEE Internet Computing,
vol. 22, no. 1, pp. 29–41, 2018.
[3] M. Khalil, J. Qadir, O. Onireti, M. A. Imran, and S. Younis, “Fea-
sibility, Architecture and Cost Considerations of Using TVWS for
Rural Internet Access in 5G,” in Innovations in Clouds, Internet and
Networks (ICIN), 2017 20th Conference on. IEEE, 2017, pp. 23–30.
[4] E. Atxutegi, F. Liberal, H. K. Haile, K.-J. Grinnemo, A. Brunstrom,
and A. Arvidsson, “On the Use of TCP BBR in Cellular Networks,”
IEEE Communications Magazine, vol. 56, no. 3, pp. 172–179, 2018.
[5] F. Qian, A. Gerber, Z. M. Mao, S. Sen, O. Spatscheck, and W. Will-
inger, “TCP revisited: a fresh look at TCP in the wild,” in Proceedings
of the 9th ACM SIGCOMM conference on Internet measurement.
ACM, 2009, pp. 76–89.
[6] H. Balakrishnan, V. N. Padmanabhan, S. Seshan, and R. H. Katz,
“A comparison of mechanisms for improving TCP performance over
wireless links,” IEEE/ACM Transactions on Networking (ToN), vol. 5,
no. 6, pp. 756–769, 1997.
[7] D. Pacifico, M. Pacifico, C. Fischione, H. Hjalrmasson, and K. H. Jo-
hansson, “Improving TCP performance during the intra LTE handover,”
in Global Telecommunications Conference, 2009. GLOBECOM 2009.
IEEE. IEEE, 2009, pp. 1–8.
[8] N.-C. Wang, Y.-Y. Wang, and S.-C. Chang, “A fast adaptive congestion
control scheme for improving TCP performance during soft vertical
handoff,” in Wireless Communications and Networking Conference,
2007. WCNC 2007. IEEE. IEEE, 2007, pp. 3641–3646.
[9] P. Hurtig and A. Brunstrom, “Enhanced metric caching for short TCP
flows,” in Communications (ICC), 2012 IEEE International Conference
on. IEEE, 2012, pp. 1209–1213.
[10] J. Yao, S. S. Kanhere, and M. Hassan, “An empirical study of
bandwidth predictability in mobile computing,” in Proceedings of
the third ACM international workshop on Wireless network testbeds,
experimental evaluation and characterization. ACM, 2008, pp. 11–18.
[11] J. Pontes, D. Borges, E. Borba, V. Lira, and E. Tavares, “Assessment
of TCP parameters for mobile devices concerning performance and
energy consumption,” in Computer and Information Technology (CIT),
2017 IEEE International Conference on. IEEE, 2017, pp. 135–141.
[12] S. K. Saha, A. Garg, and D. Koutsonikolas, “A first look at TCP per-
formance in indoor IEEE 802.11 ad WLANs,” in Computer Commu-
nications Workshops (INFOCOM WKSHPS), 2015 IEEE Conference
on. IEEE, 2015, pp. 63–64.
[13] G. Jereczek, G. L. Miotto, and D. Malone, “Analogues between tuning
TCP for data acquisition and datacenter networks,” in Communications
(ICC), 2015 IEEE International Conference. IEEE, 2015, pp. 6062–
6067.
[14] “RFC-4638,” [Online] https://tools.ietf.org/html/rfc4638, Accessed: 02
August 2018.
[15] D. Borman, B. Braden, V. Jacobson, and R. Scheffenegger, “TCP
extensions for high performance,” Tech. Rep., 2014.
[16] J. Zhang, F. Ren, X. Yue, R. Shu, and C. Lin, “Sharing bandwidth
by allocating switch buffer in data center networks,” IEEE Journal on
Selected Areas in Communications, vol. 32, no. 1, pp. 39–51, 2014.