IEEE SYSTEMS JOURNAL, VOL. 15, NO.

1, MARCH 2021 1445

Packet Scheduling in Multipath TCP:

Fundamentals, Lessons, and Opportunities
Bruno Y. L. Kimura , Demetrius C. S. F. Lima , and Antonio A. F. Loureiro

Abstract—Multipath transmission control protocol (MPTCP) is of various network interfaces. When establishing multiple TCP
a transport protocol that allows the simultaneous use of multiple subflows through different paths between peers, the MPTCP
transmission control protocol subflows across the existing IP ad- provides the following major benefits: better network resource
dresses between peers. Since each subflow undergoes the bottleneck
link condition of its path, selecting the best subflow to schedule utilization; higher throughput from multipath bandwidth aggre-
an outgoing packet plays a key role in the multipath performance. gation; and greater communication failure tolerance when avail-
While good scheduling decisions can significantly improve through- able subflows can replace faulty ones. Such benefits leverage so-
put, wrong decisions prevent users from benefiting the aggregating lutions to different technical challenges, such as enabling hybrid
capacities of available subflows. To deal with this concern, in the access networks, congestion balance in datacenter networks, and
last years, several scheduling solutions were proposed to achieve
different goals and applications. Different from other surveys and seamless IP mobility of mobile devices [3].
tutorials, in this article, we provide a tutorial of packet scheduling in A MPTCP connection consists of a set of different subflows,
the MPTCP that brings not only its fundamentals but also a detailed whose individual performance relies on the condition of its path.
analysis of multipath performance from a consistent experimental The path condition is determined by the state of its bottleneck
methodology. In a multipath network setup, we compare single- link, such as packet loss rate, queue delay, and throughput
criterion and multicriteria schedulers under different bottleneck
conditions and multipath congestion controls. From a comprehen- capacity. To deal with multipath diversity, a scheduler has to
sive experimental analysis that unveils several performance issues, select the best available subflow to send each packet. The concept
we discuss the lessons learned and research opportunities regarding of best subflow depends on the scheduler purpose, and thus,
the multipath throughput improvement as the central motivation. the scheduling decision plays a key role on the improvement
In this way, this article provides a comprehensive analysis of packet of multipath performance. Research efforts in multipath packet
scheduling in the MPTCP, not only in theory but in practice as well.
scheduling have been conducted since the release of the MPTCP
Index Terms—Multipath transmission control protocol
(MPTCP), multicriteria, multipath congestion controls, packet
Linux kernel implementation [4] so that many scheduling solu-
scheduling, single criterion. tions [4]–[17] have been proposed with different approaches
to achieve different goals and applications. In the literature,
I. INTRODUCTION there are surveys that bring comprehensiveness to the knowledge
SER devices are equipped with multiple network inter- domain of multipath transmissions [18]–[21], and there are
U faces, allowing users to access services and resources
on the Internet using different wired and wireless communica-
tutorials devoted to MPTCP [22]–[26] that focus on handling
multiple subflows in a multipath connection and coupling them
tion infrastructures. However, the transmission control protocol in a unique congestion control mechanism. However, packet
(TCP)/IP stack was designed based on the premise that an scheduling has been omitted in review studies albeit the research
end system is a single-homed node only attached to a given effort in proposing new schedulers. Thus, in this article, we
access network. This imposes the packet transmission to flow exploit this subject from the perspective of throughput improve-
throughout at most one end-to-end path between source and ment, and provide contributions in the following three main
destination peers. To overcome such a limitation, Multipath TCP aspects: fundamentals, lessons learned, and opportunities.
(MPTCP) [1], [2] was proposed as an extension of the TCP, Initially, we provide a review on the MPTCP packet schedul-
which enables multihomed devices to make simultaneous use ing, where we discuss several technical aspects about the pro-
tocol, such as: its implementation in the Linux kernel; how
Manuscript received November 12, 2019; accepted December 25, 2019. Date packets are scheduled; what the main scheduling problems are;
of publication January 27, 2020; date of current version March 9, 2021. This and what the main existing schedulers are by grouping them
work was supported by CNPq, and Project Grants 2015/18808-0, 2015/24494-8, into two general categories (single and multicriteria schedulers).
and 2018/23064-8 São Paulo Research Foundation (FAPESP). (Corresponding
author: Bruno Yuji Lino Kimura.) We discuss a consistent experimental methodology suggested
B. Y. L. Kimura is with the Federal University of São Paulo by the literature [27] in order to accomplish depth perfor-
(ICT/UNIFESP), São José dos Campos SP, CEP 12247-014, Brazil (e-mail: mance analysis of the MPTCP. As a consequence, we present a
[email protected]).
D. C. S. F. Lima is with the Federal University of Itajubá (IESTI/UNIFEI), comprehensive experimental performance analysis of MPTCP
Itajubá-MG, CEP 37500-903, Brazil (e-mail: [email protected]). packet schedulers. We deployed a multipath network setup
A. A. F. Loureiro is with the Federal University of Minas Gerais with an MPTCP Linux kernel implementation with up to five
(DCC/UFMG), Belo Horizonte-MG, CEP 31270-901, Brazil (e-mail:
[email protected]). disjoint bottleneck links, where we conducted a number of
Digital Object Identifier 10.1109/JSYST.2020.2965471 experiments considering quantitative and qualitative factors that

1937-9234 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.
1446 IEEE SYSTEMS JOURNAL, VOL. 15, NO. 1, MARCH 2021

most influence the multipath performance. We used two types presented in [23]–[25]. Barré et al. [22] report their experience in
(homogeneous and heterogeneous) and three bottleneck link implementing the MPTCP in the Linux kernel. They show how to
parameters (loss, delay, and capacity) to design eight bottleneck structure the implementation and adapt the buffer management
conditions and evaluate five schedulers under four congestion to deal with multiple subflows, as well as provide performance
controls, resulting in 160 different experiments. As a result, we analysis to demonstrate that coupled congestion control is fairer
share the lessons and insights obtained from these experiments. than the standard one in TCP. Bonaventure [26] presented an
We observed a negligible impact of scheduling decisions on annotated bibliography of the standards and scientific publica-
paths whose bottlenecks are lossy and delayed. While merely tions on MPTCP, regarding its extension, interactions with other
alternating the use of subflows in round-robin could reduce protocols, congestion control, use cases, and support software.
the bottlenecks of heterogeneous delays, the selection of the In the last years, much attention has been paid to MPTCP
lowest-latency subflow showed to be the wrong decision. Under packet scheduling, leading to several scheduling solutions [4]–
bottlenecks of heterogeneous capacities, the congestion controls [17] with different goals, approaches, and applications. How-
imposed a severe multipath performance degradation. Given ever, the available tutorials bring an abstract understanding of
different bottleneck conditions and congestion controls, when MPTCP, typically showing only a single facet of the multipath
considering various subflows’ state variables to make a decision, transmission services: the perspective of the coupled congestion
multicriteria schedulers improved the throughput up to 130% control mechanism. While such a perspective is important to
more than the default MPTCP scheduler of a single criterion. emphasize the fairness design principle when sharing bottleneck
Considering these lessons, we envision that making end systems resources with other nonmultipath flows, surveys are devoted to
aware of the bottleneck link states as well as designing meta- the domain of the multipath congestion control [20], but the
schedulers can be interesting research opportunities to improve crucial role of the packet scheduler mechanism to improve the
the MPTCP performance. multipath performance is omitted. In this sense, there is a lack
The remaining of this article is organized as follows. Sec- of works devoted to individual aspects of the MPTCP packet
tion II discusses the related work, considering the main existing scheduling mechanism, with discussion on performance issues
surveys and tutorials. Section III provides the fundamentals on that arise from bad and good scheduling decisions. Different
the MPTCP packet scheduling that helps to keep this article from the existing surveys and tutorials, our article is not limited
self-contained. Section IV describes the consistent experimen- to a review on the fundamentals of the MPTCP packet schedul-
tal methodology to evaluate MPTCP performance. Section V ing. Using a consistent experimental methodology, we pinpoint
discusses our experience on applying such a methodology to unveiled performance issues and discuss them as new lessons.
evaluate packet schedulers under different bottleneck condi- From the learned lessons, we point out research opportunities.
tions and congestion controls. Section VI discusses the main Given the lack of similar tutorials in the literature, this article
lessons we learned considering the wide spectrum of conducted can be useful for researchers who wish to assimilate both fun-
experiments. Section VII discusses research opportunities that damentals on the MPTCP packet scheduling and performance
only became clear as a result of these experiments. Finally, issues that arise from bad and good scheduling decisions.
Section VIII concludes this article.
III. FUNDAMENTALS ON MPTCP PACKET SCHEDULING
II. RELATED SURVEYS AND TUTORIALS This section provides a background of the MPTCP consid-
ering a general view of its implementation in the Linux kernel,
There are a few comprehensive surveys that provide a mul-
with focus on the packet scheduling and the main scheduling
titude of topics on the multipath transmissions. Li et al. [18]
problems. For simplicity, we use the term MPTCP whenever
discussed taxonomy, problems, and solutions of multipath trans-
referring to the MPTCP Linux kernel implementation.
missions for each layer of the protocol stack. Advances in
transport layer protocols can be found in [19] and [21]. Raiciu
et al. [19] discussed the expected multipath transport services, A. MPTCP Linux Kernel Implementation
the impact of middleboxes on the transport protocol evolvability, Currently, the MPTCP Linux kernel [4] is the best-known
and the extensions of transport protocols such as MPTCP and open source implementation. Fig. 1 illustrates different per-
Minion. Polese et al. [21] introduced multipath capabilities spectives of the MPTCP. The multipath protocol resides at the
and presented research trends on congestion control algorithms, transport layer of the TCP/IP protocol stack, in the kernel space
brand new transport protocols. of Linux operating systems, as shown in Fig. 1(a). Inside the
While survey publications provide comprehensiveness of the MPTCP, as shown in Fig. 1(b), a connection R may have r
multipath domain, tutorials devoted to the MPTCP shed light subflows belonging to it, which are completely transparent to
to its singular aspects. In [23]–[25], the authors who proposed the application layer. The socket is the unique interface that
MPTCP presented general review of the protocol operation hides multipath transport services from the application layer;
(e.g., connection establishment, data transfer, and connection therefore, it keeps the applications unchanged.
release), with discussions on the Internet compatibility and the The connection performance is determined by the cooper-
fairness as a design principle for multipath (coupled) congestion ation of the following three mechanisms: packet scheduler,
controls. Mobility management and load balance in datacenter coupled congestion control, and subflow manager. The subflow
networks are common examples of the use case of MPTCP as manager detects and signals available IP addresses between

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.
Y. L. KIMURA et al.: PACKET SCHEDULING IN MULTIPATH TCP: FUNDAMENTALS, LESSONS, AND OPPORTUNITIES 1447

Fig. 1. General views of the MPTCP Linux kernel implementation regarding a multihomed end system equipped with two network interface cards (NICs).
(a) TCP/IP protocol stack. (b) MPTCP architecture. (c) Modular framework of packet scheduling.

two MPTCP-enabled end systems, as well as establishes and Thus, sRTT is an estimate of RTT, which is determined from
terminates subflows among the existing pairs of addresses. When the exponential moving average, regarding the last estimated
the sublows are established in an MPTCP connection, the packet sRTT(i − 1) and the subsequent RTT measurement i [29]
scheduler breaks user data in the send buffer into packets and
selects the best subflow available to send each packet. Finally, sRTTi = (1 − α) × sRTTi−1 + α × RTTi (1)
the coupled congestion control adapts the congestion window of where α is a weigh of typically 1/8 in the TCP implementa-
each subflow by taking into account the whole connection and tions. Besides being the basis for computing the retransmission
different properties (e.g., fairness, friendliness, responsiveness, timeout (RTO) of the subflows as in TCP, sRTT is also used as
and congestion balance) to share bottlenecks with non-MPTCP single criterion for making a decision of the packet scheduling in
flows. the MPTCP. Fig. 2(a) describes the LL’s algorithm, in which the
current sRTT (τr ) of all the existing subflows r in connection R
B. Scheduling Packets are compared with the minimum known sRTT (τmin ) whenever
a packet has to be scheduled to the best subflow rbest . The LL
MPTCP provides a modular scheduling framework, as de- policy prioritizes faster subflows, tending to leave underused
picted in Fig. 1(c), which allows users to experience QoS based the slowest ones. Based on the experimental results discussed
on different scheduling policies. When sending a packet from in [27], such a policy should offer the best performance com-
the socket buffer, each r subflow belonging to the connection promise [30].
R (1) is checked out. The first verification is the subflow
availability (2), in which three conditions must be satisfied:
C. Main Scheduling Problems
TCP three-way handshake is completed, congestion avoidance
phases out of the loss state, and there is room in the congestion As a result of packet multiplexing across subflows of hetero-
window to accommodate the outgoing packet. If a subflow r geneous characteristics, two scheduling problems may lead to
is unavailable, the next one is checked. When r is available, high delays and goodput degradation [8]: head-of-line blocking
according to a given scheduling policy P (3), r may be selected and bufferbloating. Under head-of-line blocking, the packets
as the best subflow (rbest ) to send the outgoing packet. The scheduled to go through faster subflows reach first the desti-
framework requires to check all the subflows out for every packet nation buffer, which blocks the in-order data delivery until the
to be sent. Upon a successful choice, a subflow is selected as arrival of the remaining out-of-order packets scheduled to slower
rbest (4). Otherwise, there is no available subflow and a null is subflows. The bufferbloating problem occurs due to large buffers
returned. in bottleneck routers, leading the packets to stay enqueued for a
The default MPTCP scheduler performs the lowest- long time in the network when the paths are under congestion.
latency (LL) policy, which selects the best subflow from the When the lowest-latency subflows are always chosen, LL-
lowest smoothed round-trip time (sRTT). Recalling RTT, it is based policies help to mitigate head-of-line blocking and
the elapsed time measured by the sender between sending a bufferbloating [8]. On the other hand, LL may induce behavior
data packet with a particular sequence number and receiving an that affects the multipath performance: blinder decision and
acknowledgment packet that covers that sequence number [28]. biased-feeding phenomenon. Under blinder decision, LL relies

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.
1448 IEEE SYSTEMS JOURNAL, VOL. 15, NO. 1, MARCH 2021

Fig. 2. Default MPTCP scheduler and examples of its blinder scheduling decisions in scenarios where two paths have heterogeneous characteristics. (a) Lowest
latency—LL(default). (b) rbest with less delay, but less capacity. (c) rbest with less delay, but higher loss.

overall goals. Table I summarizes the schedulers, highlighting

the algorithms and their approach to make scheduling decisions.
In bulk transfer applications that have higher throughput re-
quirements, the schedulers LL [4] (default), ESPC [7], LTS [14],
LWS [14], and HSR [14] can improve the multipath per-
formance. For such applications, there are other schedulers
of specific purposes, such as optimizing load balancing in
WRR [15], retransmission reduction, and HoL-block mitigation
in BLEST [12]. For video streaming applications, ECF [13] can
maximize the utilization of the fastest subflows in the connec-
tion, improving quality of service parameters in those streams.
The schedulers LL/RP [8], DAPS-1 [5], and DAPS-2 [6] aim to
mitigate HoL-blocking, with no specific applications. In general
applications, the LL/BM [8] can be applied to lessen the impact
of the bufferbloat problem. On the other hand, RR [4] is of
test academy purposes only, since it leads to poor performance
due to its induced operation of HoL-blocking. For real-time
Fig. 3. Applications, schedulers, and their overall goals. transmission applications, the OTIAS [9] scheduler aims to
decrease jitter and transmission latency. In applications that
require short transmission times (i.e., mice flows), the schedulers
on the path latency as a single criterion to decide the best subflow, FPS [11] and DMPCTP [16] are able to reduce the completion
regardless of other path conditions, such as transmission rate time of these applications. Reduction of Web and interactive
and packet loss. Fig. 2(a) and (b) illustrates two scenarios where application transmission time can be achieved with STTF [17].
two subflows, r0 and r1 , go through heterogeneous bottlenecks, In industrial systems, such as supervisory control and data
b0 and b1 . Even flowing through bottlenecks of lower delays, acquisition (SCADA), the RDR scheduler [10] can improve the
rbest cannot improve the multipath performance because the reliability by reducing the packet transmission latency.
bottlenecks also have a lower capacity or a higher packet loss.
The biased-feeding phenomenon is due to the sticky nature of
E. Two Categories of Packet Scheduling
LL, which makes it to have a systematic preference toward one
subflow [7], i.e., it keeps selecting the same rbest as long as it Usually, schedulers make decisions based on the input metrics
is available. The second best subflow is only chosen after the available in the subflow data structures, e.g., subflow ID, round-
first rbest is no longer available, likely due to a full congestion trip time (RTT), packet loss, congestion window (CWND), max-
window or loss state. As soon as the first rbest becomes available imum segment size (MSS), inflight packets, and data segment
again, LL goes back and starts selecting it again. size. Using such metrics, we can classify the schedulers in two
general categories, as shown in Fig. 4: single criterion, and
multicriteria.
D. Space of Several Solutions
Single-criterion schedulers apply a single input metric belong-
Efforts in multipath packet scheduling investigations have ing to the ongoing subflows in order to make packet scheduling
been intensified since 2013 when the first MPTCP versions decisions, such as LL, FPS, RDR, DAPS-1, and RR. Typically,
have been publicly available in the Linux kernel implemen- RTT is the input metric most used for packet schedulers, since it
tations. Fig. 3 shows the arrangement several schedulers of is easier to measure by the sender along the data packet transmis-
the solution space, regarding applications, schedulers, and their sions. Inherited from TCP to estimate the subflow retransmission

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.
Y. L. KIMURA et al.: PACKET SCHEDULING IN MULTIPATH TCP: FUNDAMENTALS, LESSONS, AND OPPORTUNITIES 1449

TABLE I
SUMMARY OF VARIOUS PACKET SCHEDULING SOLUTIONS PROPOSED FROM THE PUBLIC RELEASES OF MPTCP LINUX KERNEL (2013)

since it is possible to determine the available space for new

outgoing packets within CWND. The data segment size and
MSS are useful for dimensioning a finer subflow workload from
the volume of bytes being transmitted instead of packets.

IV. EXPERIMENTAL METHODOLOGY FOR MPTCP

For a detailed performance analysis of MPTCP packet sched-
ulers, the literature suggests an experimental methodology that
consists of the following three main parts: deployment of mul-
tipath test environments; design of multipath experiments; and
definition of performance metrics. In the following, we provide
details of each part.

A. Multipath Test Environments

Simulations, emulations, and real environment tests have been
the ways of conducting experiments of networking protocols.
Fig. 4. Two general categories, schedulers, and their input metrics.
While simulation tools allow creating rich scenarios to validate
communication algorithms, the algorithm has to be implemented
in simulator-specific frameworks. To deal with real Internet,
timeout (RTO), RTT plays a polyvalent role in MPTCP. Besides however, a rework is needed to reimplement the simulated al-
being used for several packet schedulers, RTT is also an input gorithms into real protocols. MPTCP abstraction models can be
metric for the multipath congestion control, as in wVegas [31]. found in NS3 [33]–[35]. On the other hand, real protocols can be
However, when carrying the noise introduced by every device tested with network emulators. Container-based emulation tools
on the return path to the sender [32], RTT is a path information can provide names-spacing isolation functionality by virtualiz-
that may mislead the decision maker. On the other hand, the ing the operating system software stack, and hence, avoiding the
subflow identifier is used only used by RR, in order to not repeat overhead of virtualizing the hardware as in former virtualization
a scheduled subflow in the round. schemes. While emulated communication scenarios are more
Multicriteria schedulers combine multiple input metrics of the limited than the ones in simulations, container-based emulators
state variable of the subflows to enhance the scheduling decision can validate real protocols with no need of any reimplementation
making. There are several multicriteria schedulers, as shown in to deploy them in real networks. MPTCP can be run in the main
Fig. 4. The congestion window CWND is a commonly used container-based emulators such as Mininet [36], CORE [37],
metric. On noncongested paths, CWND can enable decision- and Kathara [38]. Similarly, one can conduct experiments of
making operation closer to the real-time transmissions, since real protocols in actual network infrastructures or in specific
it is dynamically increased from the acked packets at the RTT network setups. Actual infrastructures allow in situ opportunities
time. On the other hand, under congested paths, CWND induces to test protocols, however, the coexistent cross traffic makes
a reactive operation on the decision maker as it only shrinks after the test environment to be noisy and uncontrollable, and hence,
packets have been lost. Quantifying the packet loss is useful to imposing difficulties to identify, understand, and explain events
determine how congested a subflow is. The metric of inflight that impact the experimental results. On the other hand, net-
packets can provide a notion on how available subflows are, work setups allow to have an isolated network infrastructure

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.
1450 IEEE SYSTEMS JOURNAL, VOL. 15, NO. 1, MARCH 2021

completely controllable and free from noise. The counterpart throughput improvement, one can apply the performance metric
is the cost of acquisitions of such an infrastructure. Due to the of multipath aggregation benefit [42]
reproducible realism capability, emulations and real environ-  g−max(c )
(ci )−max(ci ) , if g ≥ max(ci )
 i
ments are suitable for conducting MPTCP experiments. In the A = g−max(c (2)
i)
max(ci ) , if g < max(ci )
following, we provide our experience in deploying a multipath
network setup of general purpose.
Transferring data quickly is a key objective not only for user where g is the obtained goodput, and ci is the path capacity. In
applications but also for the ones in data centers. To demonstrate our setup, ci corresponds to the capacity of the bottleneck bi .
the potential of the MPTCP in memory-to-memory transfers The aggregation benefit ranges from −1 to 1, being positive
between two high-end servers [25], the MPTCP Linux kernel when it is worth, or negative otherwise. The benefit is 1 when
maintainers proposed a single-hop network setup between two the MPTCP connection perfectly aggregates capacities of all
multihomed nodes [39]. While such a setup allows to identify the available paths. The benefit is 0 when the goodput is equivalent
fastest TCP connections with MPTCP in data center networks, to the throughput of the path of the largest capacity. When the
it is not realistic for other network scenarios. To provide a MPTCP totally fails, the benefit is −1 and the goodput is zero.
general purpose test environment for any network scenario, the To compare the results between a solution a and a baseline b,
single-hop setup has to include bottleneck links. In multihop we applied the relative performance [14]
transmissions on the Internet, the path condition is determined  
g̃a
by the limitation imposed by the existing bottleneck link in RP = − 1 × 100 (3)
g̃b
that path. Thus, by programming the links according to desired
characteristics of bottlenecks, a single-hop setup can provide a where g̃x is the median of the goodput of a solution x. The
fully controlled and flexible test environment to reproduce any relative performance brings results in positive percentage when
network scenario. a overcomes b, or negative otherwise. Instead of summarizing
results by the average, we analyzed the median because goodput
B. Design of Multipath Experiments results exhibit a non-normal statistical distribution.
To overcome the difficulties of having a clear understanding
V. APPLYING THE EXPERIMENTAL METHODOLOGY TO
of how MPTCP behaves in different environments, experimental
EVALUATE MPTCP PACKET SCHEDULERS
design approaches are pivotal to evaluate MPTCP implemen-
tations. The benefits of designing multipath experiments were The use of a unique criterion for scheduling packets may lead
first observed in [27], where an experimental design approach is to bad decisions, as in LL, as shown in Fig. 2. As a result,
proposed as a tool for revealing unknown performance issues in multipath performance cannot achieve its best when aggregating
MPTCP. Such an approach provides the selection of factors that the available capacities of the subflows. On the other hand, when
influence the performance of multipath transmissions, as well considering the state variables available in the subflows as input
as possible domains of values of these factors. Typically, two metrics, scheduling decisions can be based on multicriteria.
types of factors have been sufficient for MPTCP experiments: To further investigate the impacts of bad and good scheduling
quantitative factors related to the scope of the bottleneck link decisions on the throughput performance of MPTCP connec-
parameters, such as throughput capacity, packet loss rate, and tions, in this section, we discuss our experience in applying the
queue delay; and qualitative factors related to the end-system experimental methodology described in the previous section.
configurations, such as packet scheduling, congestion control, We provide details of how we designed a number of experi-
and subflow management. ments to accomplish a comprehensive performance analysis of
The defined sets of factors and their domains of values are packet schedulers of these different categories.
then combined to generate a space of experiments. However,
depending on the number of factors and the range of the domain A. Single-Hop Multipath Network Setup
of values, the experimental space becomes huge, being unfeasi-
We deployed a general purpose setup as depicted in Fig. 5,
ble to conduct all experiments in reasonable time. In that case,
where there are up to five single-hop links to emulate bottlenecks
full space coverage with a reduced amount of experiments can be
of end-to-end disjoint paths. Details of hardware/software com-
generated from two approaches. First, as in [27], using heuristics
ponents are depicted in the right side of Fig. 5. When realistically
such as WSP [40], which randomly generates the experiments
reproducing the path conditions sensed by the MPTCP in general
to cover the dimension spaces. Second, by defining discrete
multihop networks, a single-hop emulation setup can enable
intervals of values for each quantitative factor, which uniformly
multipath diversity required by both schedulers and congestion
generates experiments that cover the dimension spaces, as in [14]
controls to make decisions. The advantages of such a setup are
and [41].
as follows:
1) a fully controlled environment with flexibility for setting
C. Performance Metrics up the selected factors and defined domains for any net-
One of the main goals of the MPTCP is to improve throughput. work scenario;
Thus, the goodput of an MPTCP connection is the basis for the 2) the evaluation of the top performance that a multipath
performance metrics. To verify whether the MPTCP satisfies the transport protocol can obtain.

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.
Y. L. KIMURA et al.: PACKET SCHEDULING IN MULTIPATH TCP: FUNDAMENTALS, LESSONS, AND OPPORTUNITIES 1451

Fig. 5. Multipath network setup and the system architecture used in the experiments.

TABLE II
FACTORS AND DOMAIN OF CHOSEN VALUES FOR THE MPTCP EXPERIMENTS

Each pair of eth interfaces is linked and addressed to a unique transfer. In our setup, the client played the role of an uploader
IP network. With five disjoint bottlenecks, the MPTCP subflow that continuously sends large messages to the server, by keeping
manager full-mesh establishes a complete mesh of 25 different the pressure on the MPTCP to split data across the established
subflows between client and server nodes. Thus, each ethi subflows.
interface and the respective bottleneck bi are shared among up
to five different subflows. The bottleneck emulation is accom-
plished in the sender (client) with tc qdisc netem,1 which B. Factors and Domain of Chosen Values
is the best known tool for traffic control for a Linux kernel [43], The experimental design approach [27] makes possible to
[44]. With such a tool, each bottleneck link can be parame- obtain multipath diversity while reproducing different network
terized according to the desired domain of quantitative values, conditions that would be experienced on the real Internet,
as described in Table II. A simple socket-based client-server thereby, revealing unknown performance issues in the MPTCP.
application can be implemented to generate traffic of bulk data To design the experiments from the main factors that might
influence the MPTCP performance, as in [14] and [41], we
1 http://man7.org/linux/man-pages/man8/tc.8.html defined discrete intervals of values for each quantitative factor,

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.
1452 IEEE SYSTEMS JOURNAL, VOL. 15, NO. 1, MARCH 2021

TABLE III
DESIGNED EXPERIMENTS

by uniformly generating experiments to cover the dimension quantitative factors, we designed eight different experiments, as
spaces. The factors and domain of chosen values we used to described in Table III.
design our experiments are described in Table II. As baseline of multicriteria schedulers, we consider HSR,
To provide realistic multipath diversity and path conditions LWS, and LTS, which are three alternative schedulers for the
(from poor to excellent) that the MPTCP might face in real MPTCP we proposed in [14]. Fig. 6 illustrates the algorithms.
environments, possible quantitative values of bottleneck states The reasoning of HSR is that subflows of higher goodput rates
should regard previous Internet measurement studies [45]–[51]. may be suitable for sending packets, since a larger window wr
Bottleneck capacities of broadband access in both wireless (e.g., and a lower instantaneous RTT (τ¨r ) may indicate a noncongested
WiFi and LTE) and wired (e.g., data centers, FTTH) networks. bottleneck. In LWS, the best subflow is the one that can most
While losses in wired Internet is very low (≤ 0.1%) [47], accommodate outgoing packets in its congestion window. The
higher rates (≤ 10%) are experienced in wireless links [50]. subflow window space ws is the difference between the con-
We observed losses higher than 12 % prevent the MPTCP to gestion window length wr and the number of inflight packets
distinguish subflows. Bottleneck queue delays that allow RTTs fr . The intuition behind LWS is that subflows running on
to vary up to 160 ms, in accordance with the MPTCP latency higher capacity bottlenecks tend to have more space in their
measured in both fixed [49] and mobile networks [48]. congestion windows, since the packets are faster offloaded in
The domain of defined qualitative values consists of the the network; hence, subflows of lager ws tend to have fewer
solutions available across the existing versions of the MPTCP. pending acknowledgment packets in the wr window. Upon a
We consider the current MPTCP subflow manager, full mesh, packet loss, the multiplicative decrease may lead to a window
as well as the four implemented coupled congestion controls, shorter than or equal to the inflight packets, causing a negative
LIA [52] (default), OLIA [53], BALIA [54], and wVegas [31]. or null window space. In such an occasion, LWS reduces its
The operations of these solutions are summarized in Table II. operation to LL. In LTS, the best subflow is one of the lowest ratio
In our experiments to evaluate packet scheduling solutions of between window space ws and latency τr . The intuition behind
different categories, we consider LL (lowest-sRTT) and RR LTS is to merge LWS and LL to achieve a better performance,
as the baselines of single-criterion schedulers. They are offi- since subflows of higher capacity tend to have a larger window
cial schedulers available across the versions of the MPTCP. space and subflows running on less congested bottlenecks have a
While LL aims to improve the performance by aggregating lower latency. Thus, the best subflow must have a lower latency
multipath bandwidth, RR does not. Simply alternating the use and a larger window space by means of the lowest ratio between
of heterogeneous subflows in RR provides poor performance, these criteria. As in LWS, when the window space is negative,
being interesting only for academic or testing purposes. In this LTS reduces its operation to LL.
direction, we assume that RR is possibly the worst baseline for Regarding the space of qualitative factors, we selected five
a given proposed scheduler. In the space of MPTCP scheduling schedulers (RR, LL, HSR, LWS, and LTS), four congestion con-
solutions, LL remains the default scheduler. A common strategy trols (LIA, OLIA, BALIA, and wVegas), and a subflow manager
is to extend LL and apply the RTT of the subflow to complement (full mesh). Since the set of solutions of the MPTCP consists of a
the decision making. Since LL is the best-known scheduler, it is triple (scheduler, congestion control, and subflow manager), we
considered the baseline to overcome. obtained 20 different sets of solutions. For each set, we evaluated
its performance at each one of the eight designed experiments.
This leads to 160 different experiments that we have conducted
C. Designed Experiments in our multipath network emulation setup.
We defined two types of bottleneck link parameterization: It is important to highlight that the goal is not to simulate

0 homogeneous and 
1 heterogeneous, as discussed in [14]. variable traffic patterns with experiments. The designed experi-
The former considers all the five bottlenecks having the same ments take into account two major aspects. First, bottlenecks are
characteristics, with packet loss of 0%, capacity of 10 Mb/s, emulated with realistic but static characteristics of bottlenecks.
and queue delay of 5 ms. The latter considers five bottleneck This allows enabling multipath diversity while distinguishing
links of different characteristics, with packet loss from 0% to precisely in which path condition a given scheduler performs
12%, capacity from 10 to 100 Mb/s, and queue delay from 5 to better than the other one. Second, bottlenecks are emulated with
80 ms. Considering the two types of parameterization and three no traffic competition with non-MPTCP flows such as SPTCP

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.
Y. L. KIMURA et al.: PACKET SCHEDULING IN MULTIPATH TCP: FUNDAMENTALS, LESSONS, AND OPPORTUNITIES 1453

Fig. 6. Alternative multipath scheduling policy algorithms based on multicriteria decisions [14]. (a) Highest Sending Rate—HSR (b) Largest Window Space—
LWS (c) Lowest Time/Space—LTS.

Fig. 7. Experimental results from median of goodput obtained in 160 experiments.

or UDP. This allows to isolate and observe the performance VI. LESSONS LEARNED FROM PERFORMANCE ANALYSIS
of MPTCP schedulers only from the cross traffic of subflows The median of goodput and the multipath aggregation benefit
disputing the bottlenecks, under interpath contention [41]. of each pair of the MPTCP solution (packet scheduler and
To obtain many samples of goodput from each experiment,
congestion control) for each experiment are shown in Figs. 7
the client sent 500 messages of 3 MB each continuously. and 8, respectively. Fig. 9 compares the relative performance
We captured the generated traffic with the packet analyzer between multicriteria schedulers (LTS, LWS, and HSR) and the
tcpdump2 [55]. Using the dump files, we analyzed the
multipath traffic with tshark3 and extracted hundreds of good- 2 https://www.tcpdump.org/
put measurements from each experiment. 3 https://www.wireshark.org/docs/man-pages/tshark.html

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.
1454 IEEE SYSTEMS JOURNAL, VOL. 15, NO. 1, MARCH 2021

Fig. 8. Experimental results from median of aggregation benefit obtained in 160 experiments.

single-criterion scheduler (LL), which is the baseline. In the fol- regardless of the congestion control, as shown in Fig. 8. First,
lowing, we discuss the lessons we learned from the experimental in bottlenecks completely homogeneous (e0 ), when distributing
results. a packet workload equally among subflows of same character-
istics. Second, in delay-divergent bottlenecks (e1 ), RR makes
A. Scheduling Has No Impact on Lossy and Delayed slower subflows to be utilized as much as faster ones. With RR,
Bottlenecks we observed that head-of-line blocking is much more harmful
under bottlenecks of heterogeneous capacities and/or losses than
As shows the macroscopic view of Fig. 7, in Experiments e0 ,
heterogeneous delays.
e1 , e2 , and e3 , when the loss in the bottlenecks is zero, which
means that the loss is only experienced under congestion, the
C. Lowest-Latency Subflows Are Not Suitable for
scheduling decision has a significant impact on the performance
of MPTCP connections. On the other hand, in Experiments e4 , Heterogeneous Delay Bottlenecks
e5 , e6 , and e7 (scenarios of heterogeneous loss rates), there is no In Experiment e1 (bottlenecks of heterogeneous delays), it
significant difference of goodput among the schedulers, except was expected that LL would obtain the best performance when
for RR. We observe that higher loss rates combined with higher prioritizing subflows of lower latency. However, other sched-
delays lead to path conditions so harmful that the scheduling ulers performed better, including RR, under BALIA, LIA, and
decision and congestion control have no influence, regardless OLIA. The biased-feeding in LL makes the congestion windows
of the link capacity. This is observed in Experiments e5 and e7 . of faster subflows prone to be full, underutilizing slower sub-
In such scenarios, the performance is much influenced by the flows. Thus, when reacting to congestion, the window decrease
congestion control. is much more aggressive for fast subflows rather than slow ones,
while the capacity of slower bottlenecks is not leveraged.
B. Equal Load Sharing Is Worth Sometimes
RR showed that having no criterion to distinguish subflows D. Greater the Heterogeneity of Paths’ Capacities, the Lower
leads to performance degradation in most of experiments, mainly the Multipath Performance
when bottlenecks have heterogeneous capacity and/or loss rates In the name of friendliness with non-MPTCP flows sharing
(Experiments e2 , e3 , e4 , e5 , e6 , and e7 ). When no subflow is bottlenecks, the congestion control algorithms have rather con-
prioritized, RR avoids biased feeding with load balance but servative behavior, which leads to adverse performance results.
induces head-of-line blocking in heterogeneous paths. However, Even when the schedulers prioritize the best subflows, the ag-
in two situations, there are aggregation benefits higher than 0.75, gressiveness of the subflows is controlled to limit the congestion

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.
Y. L. KIMURA et al.: PACKET SCHEDULING IN MULTIPATH TCP: FUNDAMENTALS, LESSONS, AND OPPORTUNITIES 1455

we can obtain more robustness when combining multiple cri-

teria to make a scheduling decision. In most of experiments
and congestion controls, the schedulers LTS, LWS, and HSR
provide a greater performance with a higher goodput than LL.
These results suggest that the multipath goodput is not always
inversely proportional to the path latency, which means that
scheduling packets based on LL may not always lead to the
best performance.
Such promising results can be explained by the intuition
behind the multicriteria schedulers we have analyzed. In LTS, a
higher multipath performance can be obtained when scheduling
subflows of both lower latency and larger window space. In
LWS, subflows that share higher performance bottlenecks tend
to have more space in their congestion windows because more
packets can be faster offloaded through the path, which leads to
fewer inflight packets. In HSR, subflows with the highest current
transmission rate may be better for sending packets in certain
conditions, since a larger window and a lower RTT indicate
a moment of the absence of congestion in a higher capacity
bottleneck.
In BALIA, upon a packet loss, the subflow of the highest
goodput presents a more drastic decrease (factor of 1/2) of
its congestion window, while having a less aggressive increase
(factor of 1) upon a packet acknowledgment. Unlike, subflows
of lower goodput have a less drastic window decrease (factor
of 3/4) and a more aggressive window increase (factor of > 1).
Such a strategy attempts to equalize the performance between
subflows of higher and lower goodput. When prioritizing the
lowest-latency subflows under BALIA, LL performs better than
LTS and HSR. However, when prioritizing subflows with a
larger free space in the congestion window, LWS does better use
of higher capacity bottlenecks under BALIA. In Experiments
e1 (heterogeneous delay) and e2 (heterogeneous capability),
Fig. 9. Relative performance between multicriteria schedulers (LTS, LWS,
and HSR) and the baseline scheduler of single criterion (LL). LWS presents a performance 20% and 38% higher than LL,
respectively.
In LIA, the same aggressiveness factor is used to control all
window more than the necessary. This can be verified in the the subflows. The aggressiveness is adapted in such a way that
results of Experiments e2 and e3 , where lossless bottlenecks the multipath aggregated goodput is limited to which a regular
have heterogeneous capacities, whose total multipath aggregate SP-TCP would get in the best available path. This can be seen
capacity is 200 Mb/s. In detriment to the performance, results in Figs. 7 and 8, with the aggregation benefits close to zero in
show that various congestion controls boycott subflows in the Experiment e2 , where the best path flows through the bottleneck
name of a false friendliness that brings no benefit—note that only of higher performance (100 Mb/s). The multicriteria schedulers
MPTCP subflows share bottleneck links, with no contention with are better than LL in most of the experiments under LIA. LTS
non-MPTCP flows. In LIA, a unique aggressive factor limits the and HSR allow performance gains of 20% and 35%, respec-
increase of all the subflows’ congestion windows, preventing the tively, in Experiment e1 , just when the delay is divergent in the
windows from exceeding the one belonging to the subflow on bottlenecks. When bottlenecks are of heterogeneous capacities,
the best path. This makes multipath performance never greater LWS provides performance gains from 10% to 15 % over LL in
than a regular SP-TCP connection in the best available path. Experiments e2 and e3 .
However, we observe similar behavior in OLIA and BALIA, OLIA controls the congestion windows from three sets of
with A ≈ 0 in Experiment e2 in Fig. 8. subflows: a larger congestion window (W); a larger amount
of bytes sent between the last two losses (B); and more sent
bytes but with no larger congestion windows (C). OLIA defines
E. Multicriteria Can Provide a Better Schedule Than a the aggressiveness factor of each subflow in such a way that
Single Criterion the increase of congestion windows is faster for C and slower
Fig. 9 presents the results of relative performance between for W. As a result, traffic is reforwarded from subflows fully
the multicriteria schedulers (LTS, LWS, and HSR) and the utilized in W to paths with free capacity availability in C.
baseline of the single-criterion scheduler (LL). As can be seen, Under OLIA, the multicriteria schedulers perform massively

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.
1456 IEEE SYSTEMS JOURNAL, VOL. 15, NO. 1, MARCH 2021

better than LL in most experiments. LTS was 130% more react efficiently to those conditions. In this direction, an interest
efficient than LL in Experiment e3 . LWS was around 35% research opportunity is to design meta-schedulers to select the
better in Experiment e2 , 20% in e3 , and 13% in e6 . HSR best candidate solution from a poll of available solutions.
obtains 120% more goodput than LL in Experiment e3 , and 10%
in e6 . VIII. CONCLUSION
In wVegas, there are no significant impacts of the schedulers in
Packet scheduling is a key component of the MPTCP. The cor-
the performance. The multicriteria schedulers provide a relative
rect scheduling decision can improve the end-system through-
performance between 1.5% and −1% of LL. We observe that
put, whereas a wrong one takes no advantage of the available
wVegas is the most efficient congestion control because it does
network resources. In this article, we provided a detailed review
not rely on the packet loss event to infer network congestion.
of the MPTCP packet scheduling, supported by a consistent
Instead, it increases and decreases the congestion windows
multipath experimental methodology, which makes this article
of the subflows based on a higher sensitivity strategy, i.e., it
unique in the literature. Such a methodology allowed us to eval-
perceives the bottleneck queuing delays and proactively adjusts
uate solutions in two packet scheduling classes: single criterion
the congestion windows accordingly. As a result, it drains most
and multiple criteria. We conducted well-designed experiments
congested subflows and better distributes the traffic among the
in a network emulation setup that we deployed with an MPTCP
paths, leading to better performance.
Linux kernel implementation. We obtained performance results
using the four existing MPTCP congestion controls that repro-
VII. RESEARCH OPPORTUNITIES duced realistic path conditions MPTCP faces in real scenarios.
Based on the lessons discussed previously, we envision two Based on comprehensive performance analysis, we discussed
research opportunities to improve multipath performance, as new lessons learned and possible research opportunities.
discussed in the following.
REFERENCES
A. Enabling Bottleneck-Aware End Systems [1] A. Ford, C. Raiciu, M. Handley, S. Barre, and J. Iyengar, “Architectural
guidelines for multipath TCP development,” Internet Engineering Task
As a result of the Internet ossification, end systems cannot Force RFC 6182, 2011.
be aware of the bottleneck link state in the end-to-end paths. [2] A. Ford, C. Raiciu, M. Handley, and O. Bonaventure, “TCP Extensions for
To overcome this problem, transmission control protocols at multipath operation with multiple addresses,” Internet Engineering Task
Force RFC 6824, 2013.
the end systems have to estimate the bottleneck conditions to [3] O. Bonaventure and S. Seo, “Multipath TCP deployments,” IETF J.,
try to make better decisions. Explicit congestion notification is vol. 12, no. 2, pp. 24–27, 2016.
the best-known approach to allow the network core to notify [4] C. Paasch and S. Barre, “Multipath TCP in the Linux kernel,” 2013.
[Online]. Available: www.multipath-tcp.org
end systems explicitly if a path is congested. While this is [5] G. Sarwar, R. Boreli, E. Lochin, A. Mifdaoui, and G. Smith, “Mitigating
accomplished without having to discard packets at a bottleneck receiver’s buffer blocking by delay aware packet scheduling in multipath
router, the congestion notification is discrete and carries little data transfer,” in Proc. 27th Int. Conf. Adv. Inf. Netw. Appl. Workshops,
Mar. 2013, pp. 1119–1124.
information. The end systems cannot retrieve in-band the traffic [6] N. Kuhn, E. Lochin, A. Mifdaoui, G. Sarwar, O. Mehani, and R. Boreli,
metrics of the bottleneck of a path, such as queue delay and “DAPS: Intelligent delay-aware packet scheduling for multipath trans-
capacity of the router buffer, transmission rates, packet errors port,” in Proc. IEEE Int. Conf. Commun., Jun. 2014, pp. 1222–1227.
[7] F. Yang, P. Amer, and N. Ekiz, “A Scheduler for multipath TCP,” in Proc.
and losses, and number of flows sharing the router. If the network 22nd Int. Conf. Comput. Commun. Netw., 2013, pp. 1–7.
provides empirical data of the bottleneck link states, both TCP [8] C. Paasch, S. Ferlin, O. Alay, and O. Bonaventure, “Experimental evalu-
and MPTCP could be aware of the current network conditions, ation of multipath TCP schedulers,” in Proc. ACM SIGCOMM Workshop
Capacity Sharing Workshop, 2014, pp. 27–32.
and hence, make accurate transmission decisions. In this sense, [9] F. Yang, Q. Wang, and P. D. Amer, “Out-of-order transmission for in-order
enabling bottleneck-aware end systems show to be an interest arrival scheduling for multipath TCP,” in Proc. 28th Int. Conf. Adv. Inf.
research opportunity to enlarge the space of efficient transport Netw. Appl. Workshops, May 2014, pp. 749–752.
[10] I. Lopez, M. Aguado, C. Pinedo, and E. Jacob, “SCADA systems in the
solutions. railway domain: Enhancing reliability through redundant multipath TCP,”
in Proc. IEEE Intell. Transp. Syst. 18th Int. Conf. 2015, pp. 2305–2310.
[11] J. Hwang and J. Yoo, “Packet scheduling for multipath TCP,” in Proc.
B. Packet Metascheduling IEEE 7th Int. Conf. Ubiquitous Future Netw., 2015, pp. 177–179.
[12] S. Ferlin, O. Alay, O. Mehani, and R. Boreli, “BLEST: Blocking
Ad hoc MPTCP packet schedulers have been proposed to estimation-based MPTCP scheduler for heterogeneous networks,” in Proc.
better perform in different network conditions and/or user ap- IFIP Netw. Conf. Workshops, May 2016, pp. 431–439.
plications. In this article, based on well-designed experiments [13] Y.-s. Lim, E. M. Nahum, D. Towsley, and R. J. Gibbens, “ECF: An MPTCP
path scheduler to manage heterogeneous paths,” in Proc. 13th Int. Conf.
with sets of single-criterion and multicriteria approaches of Emerg. Netw. Exp. Technol., 2017, pp. 147–159.
packet scheduling, we have identified the particular bottleneck [14] B. Y. L. Kimura, D. C. S. F. Lima, and A. A. F. Loureiro, “Alternative
conditions a scheduler can perform better than the other, for scheduling decisions for multipath TCP,” IEEE Commun. Lett., vol. 21,
no. 11, pp. 2412–2415, Nov. 2017.
each multipath congestion control. This is depicted in Fig. 7 that [15] K. W. Choi, Y. S. Cho, Aneta, J. W. Lee, S. M. Cho, and J. Choi, “Op-
shows the best schedulers at the leftmost position in the x-axis. timal load balancing scheduler for MPTCP-based bandwidth aggregation
If somehow, we are able to capture and analyze the bottleneck in heterogeneous wireless environments,” Comput. Commun., vol. 112,
pp. 116–130, 2017.
link empirical data, revealing in which conditions the paths of an [16] P. Dong et al., “Reducing transport latency for short flows with multipath
end system are, then we can select the best available scheduler to TCP,” J. Netw. Comput. Appl., vol. 108, pp. 20–36, 2018.

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.
Y. L. KIMURA et al.: PACKET SCHEDULING IN MULTIPATH TCP: FUNDAMENTALS, LESSONS, AND OPPORTUNITIES 1457

[17] P. Hurtig, K. Grinnemo, A. Brunstrom, S. Ferlin, O. Alay, and N. Kuhn, [46] B. Zhang, T. S. E. Ng, A. Nandi, R. H. Riedi, P. Druschel, and G. Wang,
“Low-latency scheduling MPTCP,” IEEE/ACM Trans. Netw., vol. 27, “Measurement-Based analysis, modeling, and synthesis of the internet
no. 1, pp. 302–315, Feb. 2019. delay space,” IEEE/ACM Trans. Netw., vol. 18, no. 1, pp. 229–242,
[18] M. Li et al., “Multipath transmission for the internet: A survey,” IEEE Feb. 2010.
Commun. Surv. Tuts., vol. 18, no. 4, pp. 2887–2925, Oct.–Dec. 2016. [47] H. X. Nguyen and M. Roughan, “Rigorous statistical analysis of internet
[19] C. Raiciu, J. Iyengar, and O. Bonaventure, “Recent advances in reliable loss measurements,” IEEE/ACM Trans. Netw., vol. 21, no. 3, pp. 734–745,
transport protocols,” in Recent Advances in Networking. ACM SIG- Jun. 2013.
COMM, vol. 1, Aug. 2013, pp. 60–170. [48] Y.-C. Chen, Y.-s. Lim, R. J. Gibbens, E. M. Nahum, R. Khalili, and D.
[20] C. Xu, J. Zhao, and G. Muntean, “Congestion control design for multipath Towsley, “A measurement-based study of MultiPath TCP performance
transport protocols: A survey,” IEEE Commun. Surv. Tuts., vol. 18, no. 4, over wireless networks,” in Proc. ACM Conf. Internet Meas. Conf., 2013,
pp. 2948–2969, Oct.–Dec. 2016. pp. 455–468.
[21] M. Polese, F. Chiariotti, E. Bonetto, F. Rigotto, A. Zanella, and M. Zorzi, [49] B. Hesmans and O. Bonaventure, “Tracing multipath TCP connections,”
“A survey on recent advances in transport layer protocols,” IEEE Commun. ACM SIGCOMM Comput. Commun. Rev., vol. 44, no. 4, pp. 361–362,
Surv. Tuts., vol. 21, no. 4, pp. 3584–3608, Oct.–Dec. 2019. 2015.
[22] S. Barré, C. Paasch, and O. Bonaventure, “Multipath TCP: From the- [50] S. Biswas, J. Bicket, E. Wong, R. Musaloiu-e, A. Bhartia, and D. Aguayo,
ory to practice,” in NETWORKING, J. Domingo-Pascual, P. Manzoni, “Large-scale measurements of wireless network behavior,” in Proc. ACM
S. Palazzo, A. Pont, and C. Scoglio, Eds. Berlin, Germany: Springer, 2011, SIGCOMM Comput. Commun. Rev., vol. 45, no. 4, 2015, pp. 153–165.
pp. 444–457. [51] A. Nikravesh, Y. Guo, F. Qian, Z. M. Mao, and S. Sen, “An in-depth
[23] O. Bonaventure, M. Handley, and C. Raiciu, “An overview of multipath understanding of multipath TCP on mobile devices: Measurement and
TCP,” Usenix Login, vol. 37, no. 5, pp. 17—22, 2012. system design,” in Proc. 22nd Annu. ACM Int. Conf. Mobile Comput.
[24] C. Paasch and O. Bonaventure, “Multipath TCP,” Commun. ACM, vol. 57, Netw., 2016, pp. 189–201.
no. 4, pp. 51–57, Apr. 2014. [52] C. Raiciu and M. H. D. Wischik, “Coupled congestion control for multipath
[25] C. Paasch and O. Bonaventure, “Multipath TCP,” Queue, vol. 12, no. 2, transport protocols,” Internet Engineering Task Force RFC 6356, 2011.
pp. 40–51, Feb. 2014. [53] R. Khalili, N. Gast, M. Popovic, and J.-Y. Le Boudec, “MPTCP is not
[26] O. Bonaventure, “Multipath TCP: An annotated bibliography,” ICTEAM, Pareto-optimal: Performance issues and a possible solution,” IEEE/ACM
Universite Catholique de Louvain, Louvain-la-Neuve Belgium, Tech. Trans. Netw., vol. 21, no. 5, pp. 1651–1665, Oct. 2013.
Rep., 2015. [54] Q. Peng, A. Walid, J. Hwang, and S. H. Low, “Multipath TCP: Analysis,
[27] C. Paasch, R. Khalili, and O. Bonaventure, “On the benefits of applying design, and implementation,” IEEE/ACM Trans. Netw., vol. 24, no. 1,
experimental design to improve multipath TCP,” in Proc. 9th ACM Conf. pp. 596–609, Feb. 2016.
Emerg. Netw. Exp. Technol., 2013, pp. 393–398. [55] V. Jacobson, C. Leres, and S. McCanne, The TCPDUMP Manual Page,
[28] J. Postel et al., “Transmission control protocol,” Internet Engineering Task Lawrence Berkeley Lab., Berkeley, CA, USA, vol. 143, 1989.
Force RFC 793, 1981.
[29] V. Paxson, M. Allman, J. Chu, and M. Sargent, “Computing TCP’s
Bruno Yuji Lino Kimura received the B.Sc. degree
retransmission timer,” Internet Engineering Task Force RFC 6298, 2011.
in computer science from the Pontifical Catholic Uni-
[30] O. Bonaventure and C. Paasch, “Use cases and operational experience with
versity of Minas Gerais, Belo Horizonte, Brazil, in
multipath TCP,” in Internet Engineering Task Force RFC 8041, 2017.
2005, the M.Sc. degree in computer science from the
[31] Y. Cao, M. Xu, and X. Fu, “Delay-based congestion control for multipath
Federal University of São Carlos, São Carlos, Brazil,
TCP,” in Proc. 20th IEEE Int. Conf. Netw. Protocols, 2012, pp. 1–10.
in 2008, and the D.Sc. degree in computer science
[32] S. Ferlin, Ö. Alay, T. Dreibholz, D. A. Hayes, and M. Welzl, “Revisiting
from the University of São Paulo, São Paulo, Brazil,
congestion control for multipath tcp with shared bottleneck detection,” in
in 2012.
Proc. 35th Annu. IEEE Int. Conf. Comput. Commun., 2016, pp. 1–9.
He is currently an Adjunct Professor with the
[33] B. Chihani and C. Denis, “A multipath TCP model for NS-3 simulator,”
Institute of Science and Technology, Federal Univer-
2011, arXiv:1112.1932.
sity of São Paulo (UNIFESP), São Paulo. His main
[34] M. Kheirkhah, I. Wakeman, and G. Parisis, “Multipath-TCP in NS-3,”
research interests include multipath data transport, delay/disruption tolerant
2015, arXiv:1510.07721.
networks, and Internet of Things/Edge networks.
[35] M. Coudron and S. Secci, “An implementation of multipath TCP in NS3,”
Comput. Netw., vol. 116, pp. 1–11, 2017.
[36] B. Lantz, B. Heller, and N. McKeown, “A network in a laptop: Rapid Demetrius Costa Silva Faria Lima received the
prototyping for software-defined networks,” in Proc. 9th ACM SIGCOMM B.Sc. and M.Sc. degrees in computer science from
Workshop Hot Topics Netw., 2010, pp. 19-1–19-6. the Federal University of Itajubá (UNIFEI), Itajubá,
[37] J. Ahrenholz, “Comparison of core network emulation platforms,” in Proc. Brazil, in 2014 and 2017, respectively.
IEEE Mil. Commun. Conf., Oct. 2010, pp. 166–171. He is currently a Software Engineer working on the
[38] G. Bonofiglio, V. Iovinella, G. Lospoto, and G. Di Battista, “Kathará: A development of enterprise systems and web applica-
container-based framework for implementing network function virtual- tions. His research interests include packet schedul-
ization and software defined networks,” in Proc. IEEE/IFIP Netw. Oper. ing, subflow management, and coupled congestion
Manage. Symp., Apr. 2018, pp. 1–9. control for multipath data transport on the Internet.
[39] C. Paasch, G. Detal, S. Barre, F. Duchene, and O. Bonaventure, “The
fastest TCP connection with multiPath TCP,” 2013. [Online]. Available:
http://multipath-tcp.org/pmwiki.php?n=Main.50Gbps
[40] J. Santiago, M. Claeys-Bruno, and M. Sergent, “Construction of space- Antonio A. F. Loureiro received the B.Sc. and
filling designs using WSP algorithm for high dimensional spaces,” Chemo- M.Sc. degrees in computer science from the Federal
metrics Intell. Lab. Syst., vol. 113, pp. 26–31, 2012. University of Minas Gerais (UFMG), Belo Horizonte,
[41] B. Y. L. Kimura, D. C. S. F. Lima, L. A. Villas, and A. A. F. Loureiro, Brazil, in 1983 and 1987, respectively, and the Ph.D.
“Interpath contention in multipath TCP disjoint paths,” IEEE/ACM Trans. degree in computer science from the University of
Netw., vol. 27, no. 4, pp. 1387–1400, Aug. 2019. British Columbia, Vancouver, Canada, in 1995.
[42] D. Kaspar, “Multipath aggregation of heterogeneous access networks,” He is a Full Professor with the UFMG, where he
ACM SIGMultimedia Rec., vol. 4, no. 1, pp. 27–28, 2012. leads the research group on mobile ad hoc networks.
[43] S. Hemminger, “Network emulation with NetEm,” in Proc. Linux Aust. His main research interests include networking, wire-
Conf., 2005, pp. 18–23. less sensor networks, mobile computing, and dis-
[44] A. Jurgelionis, J.-P. Laulajainen, M. Hirvonen, and A. I. Wang, “An tributed systems. In the last 15 years, he has authored
empirical study of NetEM network emulation functionalities,” in Proc. and coauthored regularly in international conferences and journals related to
20th IEEE Int. Conf. Comput. Commun. Netw., 2011, pp. 1–6. those areas and has also presented tutorials at international conferences.
[45] C. Bovy, H. Mertodimedjo, G. Hooghiemstra, H. Uijterwaal, and P. Van Dr. Loureiro was the recipient of the 2015 IEEE Ad Hoc and Sensor Technical
Mieghem, “Analysis of end-to-end delay measurements in internet,” in Achievement Award and the Computer Networks and Distributed Systems In-
Proc. Passive Active Meas. Workshop, vol. 2002, 2002, pp. 1–8. terest Group Technical Achievement Award of the Brazilian Computer Society.

Authorized licensed use limited to: Beijing Jiaotong University. Downloaded on October 02,2022 at 14:54:58 UTC from IEEE Xplore. Restrictions apply.