MATEC Web of Conferences 232, 04004 (2018) https://doi.org/10.

1051/matecconf/201823204004
EITCE 2018

Dynamic load balancing scheme on massive file transfer system

Hou Weiguang1,a, He Gang2 and Liu Xinwen3
1BeijingUniversity of Posts and Telecommunications, Beijing 100876, China
2BeijingUniversity of Posts and Telecommunications, Beijing 100876, China
3Wisetone Technologies, Beijing 100876, China

Abstract. In this paper, a dynamic load balancing scheme applied to massive file transfer system is proposed.
The scheme is designed to load balance FTP server cluster. Instead of recording connection number, runtime
load information of each server is periodically collected and used in combination of static performance
parameters collected on server startup to calculate the weight of servers. Improved Weighted Round-Robin
algorithm is adopted in this scheme. Importantly, the weight of each server is initialized with static
performance parameters and dynamically modified according to the runtime load. Apache Zookeeper cluster
is used to receive all information and it will inform director of the runtime load variation and offline behavior
of any server. In order to evaluate the effect of this scheme, a contrast experiment with LVS is also conducted.

1 Introduction The static scheduling is poor in adaptability. It never

takes the cluster’s runtime state into consideration for the
The rapid growth of Internet makes the number of access directing scheme is fixed. While the parameters
to multimedia network servers increase rapidly. The traditional dynamic scheduling utilized like connections
server needs to provide service to a large number of number couldn’t precisely reflect the load condition[2].
concurrent accesses. The ability of processing and I/O has The dynamic load balancing scheme proposed in this
become the bottleneck of providing services. To solve this paper is capable of balancing nodes in cluster on the basis
problem, one solution is expensive high-performance of runtime load information and it also takes the hardware
server or SMP, the other is connecting multiple servers to static performance into consideration. It has great
form a cluster so that performance could be improved adaptability, working well in heterogeneous cluster and
through parallel processing and high speed information makes self-adjustment in time when load condition of
exchange between each other. The latter solution, with cluster changes.
high overall performance (such as response time,
throughput), high scalability, high availability, and higher
performance/price ratio, has become the principal method 2 System model
to build high performance information servers.
The cluster system is consisted of a director and several
Load balancing is the core part of the normal work in
file servers that connected by a high-speed network.
cluster system. Its main purpose is to distribute tasks
reasonably to all of the nodes in cluster, achieve the
balanced state of the whole system and ensure the
processing capacity and quality of service of the system.
Load balancing can be achieved directly based on
hardware products or software[1]. The load balancer is
installed between the server and the external network. The
hardware implementation of the load controller is
expensive and not flexible, and it couldn’t support more
improved load balancing strategy and more complex
application protocol.
Figure 1. System Model
In software scheme, the load balancing algorithms
More details of the system model are not described in
could be classified into static and dynamic algorithm.
this figure. Important features left are as follows:
Studies have shown that both the static load balancing
1) There is a load collector on every file server,
scheduling and the dynamic load balancing algorithms
collect static performance parameters on server startup
could improve the performance of the cluster system. But
and periodically collecting runtime load information.
in practice both of the scheduling methods still have some
2) An Apache Zookeeper cluster works as the
limitations.
manager of load information. All load collectors on server

* Corresponding author: a [email protected]

© The Authors, published by EDP Sciences. This is an open access article distributed under the terms of the Creative Commons Attribution License 4.0
(http://creativecommons.org/licenses/by/4.0/).
MATEC Web of Conferences 232, 04004 (2018) https://doi.org/10.1051/matecconf/201823204004
EITCE 2018

have registered a corresponding znode in zookeeper and 3.2 Runtime Load Information
upload load information to it.
3) Director has registered child-watcher on Znodes stored with runtime load information are
servers’ root znode and data-watcher every server’s znode ephemeral znodes. Servers’ online/offline behavior will
that stores load data to monitor all servers’ state. lead to the creation and elimination of corresponding
At time to refresh the load, load collector collects load child znode under “Dynamic” znode. Therefore, director
information and update the corresponding zookeeper has registered child listener to “Dynamic” znode to be
node. Then the director is informed and refresh the load informed of online/offline and data listener to every
table. Weights of servers are recalculated on basis of the server’s corresponding znode to be informed of the load
load table. change[5].
Runtime load information is used to dynamically
modify the weight of server. It includes initial weight,
3 Znode structure usage percentage of CPU, disk I/O, memory and
bandwidth. These information is periodically collected
As a commonly used distributed coordination service and upload to the corresponding znode that attached to
provider, Apache Zookeeper could work as a data “Dynamic” znode. Every time runtime load information
manager and it guarantees data consistency[3]. File updated the director will be informed.
system in Zookeeper is organized as hierarchy tree
structure and every node in it is called znode.
There are two types of znode that a client can create[4]. 4 Load balancing algorithm
Persistent—clients manipulate create and deleting them
explicitly. Ephemeral—client create such znode and they
4.1.Basic algorithm
either delete them explicitly or let the system remove
them automatically when the session that creates them Servers’ runtime usage percentage of hardware resources
terminates. are utilized to measure load condition rather than
Based on znode’s feature, static performance recording connection number in this scheme. Therefore,
parameters and runtime load information are stored at the commonly used algorithm Improved Weighted
different znodes as figure 2 shows. Round-Robin is the most suitable one to be applied in this
scheme.
Assume there are three file servers in cluster and
corresponding weights are W1, W2 and W3. The
implementation of Improved WRR needs a new data
structure “Node” consists of id, weight and current weight
of server. The calculation of the server list to be allocated
is described in pseudocode as follows.
Table 1. Improved WRR Pseudocode

Input: list of Nodes

Output: list of servers to be allocated
length=W1+W2+W3
for i in 1 to length
Figure 2. Znode Structure maxWeight=0
loc=Input[0]
for node in Input
3.1. Static performance parameters if maxWeight <node.currentWeight
then loc=Node
Static parameters are used to measure the carrying Node.currentWeight+=Node.weight
end
capability of server and initialize the weight of server. It loc.currentWeight-=length
includes number of CPU cores, disk I/O speed, memory Output[i]=loc
capacity and bandwidth. At server startup, load collector end
collects all these parameters, store them into serialized Compared with traditional WRR, Improved WRR is
data structure. After data collection, load collector will more suitable to work as the basis of dynamic load
communicate with zookeeper cluster to examine whether balancing scheme. Traditional WRR does not scatter
“Parameters” znode exists under “Static” znode. If it elements of high weight in server list when selecting
doesn’t exist, then the Zookeeper client on this server will element. Therefore, it will give great pressure to servers
create it with the static performance parameters and with higher weight. On the contrary, Improved WRR
record the current server’s weight as 1. Otherwise, the scatters elements in server list so that servers will be
Zookeeper client will get the data “Parameters” znode alternately selected and the weight will be considered at
records and calculate current server’s weight and record the same time[6]. This mechanism will lead to better
the weight locally. balance effect.

2
MATEC Web of Conferences 232, 04004 (2018) https://doi.org/10.1051/matecconf/201823204004
EITCE 2018

4.2.Weight initialization environment there is no special requirement so [Wc,

Wd,Wm,Wb] is set to be [1,1,1,1].
At server start-up, every server collects its own static
performance parameters and download the standard
parameters stored in “/Load/Static/Parameters” znode if 4.3.2. Weight modification
the node exists. Then weighted summation is used to
Every time load factor is updated, the weight server is also
calculate the initial weight. All parameters’ definition
calculated. Define the modified weight as Wm then the
concerning initial weight calculation are presented in
modification of weight is stated as follows:
table 2.
Table 2. Parameters’ Definition
𝑊 = Math. around[ 1 − 𝐹 ∗ 𝑊 ∗ 𝐸] (3)

Parameters Standard Server Weight E in this equation is used to amplify the weight. Both
CPU Cores Number C0 C Wc (1-F) and Wi are smaller float than 1 but the weight
Disk I/O speed D0 D Wd applied into Improved WRR needs to be integer so it’s
Memory Capacity M0 M Wm necessary to set an amplify factor. In experiment it’s set
Bandwidth B0 B Wb E=10. And the final result will be converted to integer.
Define the initial weight as Wi, weighted summation In case all servers in cluster are busy and Wm is below
is used to calculate. or equal to 0, Wm will be set to 1 if the final result is 0.
This strategy guarantees that there will always be
𝑊 =𝑊 ∗ +𝑊 ∗ +𝑊 ∗ +𝑊 ∗ (1) available server to be allocated.

In FTP cluster, the usage of bandwidth is beyond the 4.4.Re-allocate interval

others. After adjusted for many times, weights in this
equation [Wc, Wd,Wm,Wb] is set to be [0.2,0.1,0.1,0.6]. Ftp cluster provides service to a large number of ftp
clients. If client requests for server allocation every time
4.3.Weight dynamic modification establishing connection, the pressure on director will be
increasing rapidly. In this load balancing scheme ftp
client requests for re-allocation at set intervals rather than
4.3.1. Calculation of load factor every time. The time interval is returned by director along
with the server address allocated. It’s also modified
The load collector running on server periodically collects according to the load balancing condition. Assume the
runtime load information. This information will be load factor of servers in cluster is [F1,F2,F3], the ceiling
updated to corresponding znode and the director will use interval, floor interval and initial interval are [Ic,If,Ii].The
it to calculate load factor F. The load factor is also calculation logic of the time interval in pseudocode is
calculated through a weighted calculation formula. All stated as follows.
parameters related to dynamic load factor are defined in
table 3. Table 4. Time Interval Calculation

Table 3. Parameters’ Definition Interval=Ii

max=0, min=1
Parameters Server Weight for f in [F1,F2,F3]
CPU Usage C Wc if f<min
Disk I/O Occupancy D Wd min=f
Memory Allocation M Wm elsif f>max
Bandwidth Occupancy B Wb max=f
The calculation of dynamic load factor is stated as end
follows. if max-min>threshold
Interval=max(interval/2,If)
𝐹 =1− 1−𝑊 ∗𝐶 ∗ 1−𝑊 ∗𝐷 ∗ 1−𝑊 ∗ else
𝑀 ∗ 1−𝑊 ∗𝐵 (2) Interval=min(interval*2,Ic)
The overall logic of time interval modification is to
This formula is chosen rather than weighted increase if the balance condition is good and decrease on
summation for it is more efficient in detecting heavy load the contrary. It helps to balance the load in time and
on a particular hardware. If usage of single hardware reduce the pressure on director.
resource is much higher than others, then the load factor
should be nearly 1 to warn that the server is very busy but
weighted summation couldn’t work out. This formula 5 Contrast experiment
suits the requirement better than weighted summation. To test the performance of dynamic load balancing
All these weights to calculate the factor can be scheme, a contrast experiment with Linux Virtual Server
modified to satisfy special requirements such as setting is conducted.
maximum usage of a particular hardware to make sure it’s
always able to deal with another task. In experimental
5.1.Experimental environment

3
MATEC Web of Conferences 232, 04004 (2018) https://doi.org/10.1051/matecconf/201823204004
EITCE 2018

5.1.1. Dynamic load balancing way could add extra load to server. In experiment multi-
thread SCP clients are launched[9] on one of the servers
The experiment is conducted in a cluster with 7 servers in running FTP clients and transfer files to one particular
it. One server works as director, three servers work as ftp FTP server to supply extra load.
server and the rest work as ftp clients. All these servers’
hardware configuration is the same.
5.3. Experiment result analyzation
Table 5. Servers’ Task
The line chart of bandwidth usage is chosen to be
Server Role Task analyzed because the usage of bandwidth is much higher
Director Director Program than other hardware.
Load Information Recorder In contrast experiment, the duration of each
Apache Zookeeper in standalone mode experiment is about 10 minutes, and the first 3-4 minutes
FTP Servers Server Program
for the FTP balance effect test. After that, additional load
VSFTP
is added and the load changes are observed. After
FTP Clients Concurrent FTP Client
optimizing the weighting coefficients by several
In experiment, load information that collector
experiments, the test results show that the trend of
uploaded is also recorded into files. Compared to
bandwidth occupancy rate of FTP servers in each
connection number and response delay, load information
platform is the same under the different number of FTP
is better to be used to evaluate the balance effect. Gap
clients. The test results of 90 concurrent clients
between different servers’ usage percent directly
transmitting files are analyzed.
represents balance effect.
Server works as ftp client will launch multiple threads
and start an ftp client in every thread to simulate a large 1

number of clients.
0.9

0.8

0.7

0.6

5.1.2. LVS 0.5

0.4

As comparison, the experimental environment is set 0.3

0.2
similar to the environment mentioned above. 0.1

0
1530776771116
1530776777232
1530776785255
1530776791317
1530776797414
1530776805425
1530776811484
1530776817595
1530776825596
1530776831646
1530776837750
1530776845755
1530776851812
1530776857909
1530776865935
1530776872005
1530776878144
1530776886132
1530776892188
1530776898262
1530776906302
1530776912378
1530776918433
1530776926495
1530776932548
1530776938628
1530776946662
1530776952713
1530776958770
1530776966822
1530776972889
1530776978939
1530776986986
1530776993042
1530776999134
1530777007147
1530777013197
1530777019321
1530777027324
1530777033369
1530777039475
1530777047505
1530777053575
1530777061636
1530777067688
1530777073751
1530777081807
1530777087860
1530777093944
1530777101997
1530777108075
1530777116099
1530777122172
1530777130208
1530777136264
1530777142349
1530777150377
1530777156434
1530777162497
1530777170532
1530777176579
1530777182687
1530777190674
1530777196737
1530777202853
1530777210846
1530777216892
1530777223013
1530777230996
1530777237059
1530777243156
1530777251156
1530777257213
1530777263287
1530777271301
1530777277384
1530777283472
1530777291510
1530777297561
1530777303646
1530777311655
1530777317711
1530777323789
1530777331795
1530777337860
1530777343937
1530777351934
1530777357979
1530777364092
1530777372070
1530777378126
1530777384274
1530777392257
Table 6. Servers’ Task
Server1 Server2 Server3
Server Role Task
Director Ipvsadm Figure 3. Bandwidth Usage of LVS SED
Load Information Recorder 0.9

FTP Servers VSFTP 0.8

FTP Clients Concurrent FTP Client 0.7

LVS has already been integrated into Linux kernel so 0.6

0.5
only the administration program—Ipvsadm has to be 0.4

launched on director[7]. Commonly used algorithm— 0.3

SED(Shortest Expected Delay Scheduling) is chosen in 0.2

experiment. 0.1

0
File transition must be finished in FTP PASV mode.
1530841316443
1530841322605
1530841328674
1530841336684
1530841342750
1530841348832
1530841356878
1530841362981
1530841369095
1530841377039
1530841383101
1530841389249
1530841397206
1530841403362
1530841409488
1530841417375
1530841423506
1530841429614
1530841437551
1530841443631
1530841449728
1530841457745
1530841463809
1530841469858
1530841477911
1530841483989
1530841490041
1530841498113
1530841504168
1530841510217
1530841518275
1530841524332
1530841530565
1530841538447
1530841544516
1530841550978
1530841558629
1530841564684
1530841571446
1530841578816
1530841584901
1530841591847
1530841598984
1530841605055
1530841612245
1530841619160
1530841625223
1530841632629
1530841639360
1530841645436
1530841653058
1530841659521
1530841665582
1530841673549
1530841679687
1530841685811
1530841693793
1530841699845
1530841706170
1530841713966
1530841720039
1530841726559
1530841734154
1530841740215
1530841746941
1530841754336
1530841760399
1530841767445
1530841774541
1530841780592
1530841787924
1530841794716
1530841800790
1530841808336
1530841814895
1530841820958
1530841828712
1530841835008
1530841841094
1530841849118
1530841855312
1530841861443
1530841869278
1530841875541
1530841881812
1530841889464
1530841895701
1530841902226
1530841909620
1530841915849
1530841922620
1530841929794

Only in this mode, director doesn’t need to forward all the Server1 Server2 Server3
packets so that its performance won’t be the bottleneck of
cluster. Iptables’ rules are also added on every file server Figure 4. Bandwidth Usage of Dynamic LB Scheme
to solve the address conflict in PASV mode[8]. As can be seen in line charts, in first 3-4 minutes the
bandwidth usage curves of different servers under
dynamic LB scheme almost coincide with each other.
5.2.Experimental scheme This trend is identical with LVS in SED mode, proving
the dynamic scheme could achieve similar balance effect
The main experimental scheme is to test balance effect
to traditional balance scheme.
under different level of load pressure. As max upload
After extra load is added, the bandwidth usage trend
speed of single client is configured to fixed value, the
is different. Under LVS, number of connections is the
number of clients is proportional to the load pressure. The
only indicator to balance the cluster so the server with
purpose of the experiment can be achieved by adjusting
additional load will still be allocated approximately the
the number of concurrent clients.
number of FTP requests similar to other servers[10]. The
Extra load pressure is also added to servers by
consequence is that the server bears additional load will
executing additional tasks on these servers. Through
be much busier than the others.
experimental observation, it’s known that bandwidth
Under dynamic scheme proposed in this paper,
usage percent is much higher than the others. Taken
however, the cluster is able to self-adjust to achieve a
bandwidth usage as the main observation object,
balanced state. Figure 4 shows that about 1 minute later
additional file transmission of other protocol is the best
after the extra load is added, the other two servers have

4
MATEC Web of Conferences 232, 04004 (2018) https://doi.org/10.1051/matecconf/201823204004
EITCE 2018

already been allocated more requests and the cluster has Control and Automation. Springer, Berlin,
achieved a new balanced state. Heidelberg, 2010. 127-134.

6 Conclusion
In practice, clusters are not only responsible for one task.
Sometimes file servers are responsible for some task such
as file integration or file pre-processing. Traditional load
balancing schemes, such as LVS, tend to be more
balanced in a particular business aspect, and cannot be
adjusted according to the actual situation when server's
other business loads are heavy.
The dynamic load balancing scheme proposed in this
paper is proved to be more aware of cluster’s actual load
condition and is effective in self-adjustment. It could
achieve better balance effect than LVS in practice. And
because the load information collecting and weight
calculation is lightweight task, the additional load it
brings is negligible. Therefore, this scheme is ideal for
massive file transfer system.

References
1. M. Katyal, A. Mishra. A comparative study of load
balancing algorithms in cloud computing
environment. arXiv preprint arXiv:1403.6918 (2014).
2. Yichuan Jiang. A survey of task allocation and load
balancing in distributed systems. IEEE Transactions
on Parallel and Distributed Systems 27.2:285-
299.(2016).
3. Saurav Haloi. Apache ZooKeeper Essentials. Packt
Publishing Ltd, 2015.
4. Hunt Patrick, Mahadev Konar, Flavio Paiva
Junqueira, Benjamin Reed. ZooKeeper: Wait-free
Coordination for Internet-scale Systems. USENIX
annual technical conference, vol. 8, no. 9. 2010.
5. Artho, Cyrille, et al. Model-based API testing of
Apache ZooKeeper. Software Testing, Verification
and Validation (ICST), 2017 IEEE International
Conference on. IEEE, 2017.
6. Upstream: smooth weighted round-robin balancing.
https://github.com/phusion/nginx/commit/27e94984
486058d73157038f7950a0a36ecc6e35
7. Wensong Zhang. Linux Virtual Server for Scalable
Network Services. Ottawa Linux Symposium. Vol.
2000(2000).
8. Chen, Yan-sheng, et al. Design and implementation
of FTP service resource sharing platform based on
LVS load balancing technology. CSIA 2014,
Bangkok, Thailand, November 17-18, 2014. CRC
Press, 2015.
9. Andrew van Rooyen. Determining the optimal
protocols for transfer of large data files. (2015).
10. Choi DongJun, Kwang Sik Chung, JinGon Shon. An
improvement on the weighted lest-coonection
scheduling algorithm for load balancing in web
cluster systems. Grid and Distributed Computing,