Please cite this article as: Martino Trevisan, Danilo Giordano, Ali Safari Khatouni.

Measuring HTTP/3: Adoption

and Performance. 19th Mediterranean Communication and Computer Networking Conference (2021). DOI: 1
https://doi.org/10.1109/MedComNet52149.2021.9501274

Measuring HTTP/3:
Adoption and Performance
Martino Trevisan† , Danilo Giordano† , Idilio Drago ‡ , Ali Safari Khatouni?
†
Politecnico di Torino, ‡ University of Turin, ? Shopify
first.last@{polito.it, unito.it, shopify.com}

Abstract—The third version of the Hypertext Transfer Protocol and Facebook2 ). However, currently neither the real state of
(HTTP) is in its final standardization phase by the IETF. Besides its deployment nor the performance benefits of HTTP/3 have
arXiv:2102.12358v2 [cs.NI] 10 Nov 2021

better security and increased flexibility, it promises benefits in been measured yet.
terms of performance. HTTP/3 adopts a more efficient header
compression schema and replaces TCP with QUIC, a transport In this paper we fill this gap by running the first large-scale
protocol carried over UDP, originally proposed by Google and measurement study on HTTP/3 adoption and performance. We
currently under standardization too. Although HTTP/3 early first rely on the HTTPArchive Dataset to study to what extent
implementations already exist and some websites announce its the web ecosystem has already adopted HTTP/3. Then, we
support, it has been subject to few studies. We provide a first run additional campaigns to measure the benefits introduced
measurement study on HTTP/3 adoption and performance. We
testify how it has been adopted by some of the leading Internet by HTTP/3. Considering websites that adopt different versions
companies such as Google, Facebook and Cloudflare in 2020. of the HTTP protocol, we measure several metrics known
We run a large-scale measurement campaign towards thousands to indicate users’ Quality of Experience (QoE). Finally, we
of websites adopting HTTP/3, aiming at understanding to what emulate different network conditions on the paths connecting
extent it achieves better performance than HTTP/2. We find that our measurement platform to assess whether and how HTTP/3
adopting websites often host most web page objects on third-
party servers, which support only HTTP/2 or even HTTP/1.1. improves performance in different scenarios.
As excepted, websites loading objects from a limited set of third- Using the open-source HTTPArchive Dataset,3 we find thou-
party domains (avoiding legacy protocols) are those experiencing sands of websites supporting HTTP/3, most of them hosted by
larger performance gains. Our experiments however show that a handful of Internet hyper-giants, i.e., Facebook, Google, and
HTTP/3 provides sizable benefits only in scenarios with high
Cloudflare. We then automatically revisit websites supporting
latency or poor bandwidth.
HTTP/3 under diverse network conditions to measure the
Index Terms—HTTP/3; Performance; Measurements. performance benefits in terms of QoE-related metrics. We
visit 14 707 websites in total while emulating artificial latency,
I. I NTRODUCTION packet loss, and limiting the bandwidth. We run 2 647 260
visits over a period of one month. We find that HTTP/3
The Hypertext Transfer Protocol (HTTP) is the king of benefits emerge only on particular network conditions and
web protocols and is used to access the vast majority of strongly differ across websites. Our key findings are:
services on the Internet, from websites to social networks
• Google, Facebook and Cloudflare are the early adopters
and collaborative platforms. HTTP was born in the early 90s,
of HTTP/3, hosting almost the totality of currently web-
and its first version 1.1 was standardized in 1997 [1]. Only
sites supporting HTTP/3.
in 2014, HTTP/2 [2] was proposed, with substantial changes
• The majority of web page objects in websites support-
in the framing mechanisms. HTTP/3 is the third version of
ing HTTP/3 are still hosted on non-HTTP/3 third-party
HTTP and is currently in the final standardization phase at
servers.
the IETF [3]. It promises performance benefits and security
• We observe sizable performance benefits only in scenar-
improvements compared to HTTP/2. As a major change,
ios with high latency or low network bandwidth.
HTTP/3 replaces TCP as transport layer in favor of QUIC,
• The performance gains largely depends on the infrastruc-
a UDP-based protocol originally proposed by Google and
ture hosting the website, possibly due to optimizations on
currently being standardized by the IETF [4]. Furthermore,
server-side infrastructure.
it introduces a more effective header compression mechanism
• As expected, the websites relying on fewer connections
and exploits TLS 1.3 [5] (or higher) to improve the level of
to load objects are those benefiting the most.
security.
HTTP/3 is expected to take over the place of HTTP/2 in the The remainder of the paper is organized as follows: Sec-
next years, and some of the leading Internet companies already tion II describes HTTP/3 and illustrates related work. Sec-
announced its support during 2020 (e.g., the CloudFlare CDN1 tion III presents our datasets and how we have collected them.
Section IV illustrates our results on HTTP/3 adoption and
This work has been supported by the EU H2020 research and innovation
programme under grant agreement No. 644399 (MONROE) and by the 2 https://engineering.fb.com/2020/10/21/networking-traffic/
SmartData@PoliTO center on Big Data and Data Science. how-facebook-is-bringing-quic-to-billions/
1 https://blog.cloudflare.com/http3-the-past-present-and-future/ 3 https://httparchive.org/
 2

TABLE I: Description of the employed datasets.

HTTP/1.1 HTTP/2 HTTP/3
Dataset Visits Goal
TLS TLS QUIC HTTPArchive 53 107 185 HTTP/3 Adoption
(optional) (+TLS 1.3)
BrowserTime 2 647 260 HTTP/3 Performance
TCP TCP UDP

IP original servers. Marx et al. [9] compare 15 HTTP/3 imple-

mentations, finding a large heterogeneity in how congestion
Fig. 1: Protocol stack for different HTTP versions. control, prioritization and packetization work. They only run
single file downloads, but their results call for extensive in-
the-wild measurements. Cloudflare benchmarks its own draft
performance. Section V discusses limitations of results and 27 HTTP/3 implementation in [10], finding it to be 1 − 4%
lists the basis for future work, while Section VI concludes the slower than HTTP/2. However, their experiments are limited
paper. to the blog.cloudflare.com website. Guillen et al. [11]
proposed a control algorithm for adaptive streaming tailored
for HTTP/3.
II. BACKGROUND
Google proposed QUIC in 2012 and, as such, it has been
A. HTTP/3 the subject of many studies. Wolsing et al. [12] show that
HTTP/3 is the third version of the well-known Hypertext QUIC outperforms TCP thanks to the fast connection setup.
Transfer Protocol, born in the 90s to transfer multimedia Manzoor et al. [13] show that QUIC performs worse than
content and hyper-textual documents over the Internet. With its TCP in Wireless Mesh Networks thanks to bad interactions of
version 1.1, it has been the king of web protocols for more than the protocol with the WiFi layer in that scenario. Carlucci et
20 years, superseded only by its second version HTTP/2 in al. [14] found that QUIC reduces the overall page retrieval
2014. HTTP/2 implemented several novel features, especially time. Kakhi et al. [15] run a large-scale measurement cam-
to improve how data is framed and transported. It promised paign on QUIC, finding that it outperforms TCP in most cases.
to make the web faster, even if some studies questioned its These works however target Google’s QUIC versions, while
benefits [6], [7]. the current standard proposed at the IETF has made significant
HTTP/3 is currently in the final standardization phase, progresses [16]. Moreover, they focus uniquely on transport
reaching the 34th draft version [3] and making it stable and layer, neglecting the improvement introduced by HTTP/3 in
usable for real deployments. The main improvements from higher layers, which we measure in this work.
version 2 include more efficient header compression, advanced
security features based on TLS 1.3, and, especially, the use III. DATA C OLLECTION
of QUIC at the transport layer. The resulting protocol stack We rely on two datasets to study (i) the adoption of HTTP/3
is thus heavily modified, as we show in Figure 1. QUIC, and (ii) its performance on diverse network conditions. We
initially developed by Google, is a transport protocol based summarize them in Table I.
on UDP and is currently in the standardization phase too [4].
QUIC revisits TCP, moving congestion control in user space
and allowing faster handshakes. Moreover, it solves the long- A. HTTP/3 Adoption – HTTPArchive
standing issue of head-of-line blocking, allowing multiple in- We study the adoption of HTTP/3 using the HTTPArchive,
dependent streams within the same connection. Indeed, QUIC an open dataset available online.4 The dataset contains meta-
allows independent retransmission for sub-streams and decou- data coming from visits to a list of more than 5 million URLs
ples it from congestion control. This operation is expected provided by the Chrome User Experience Report.5 The list
to improve users’ QoE with faster website responsiveness, of URL is built using the navigation data of real Chrome
especially in scenarios with poor network conditions. HTTP/3 users and contains a representative view of the most popular
also mandates the use TLS 1.3 [5], directly incorporated at the website and web services accessed worldwide.6 Each month,
QUIC layer. Finally, it allows 1-RTT handshakes and 0-RTT all URLs are visited using the Google Chrome browser from
resumption, further reducing session setup time. a U.S. data center, and the resulting navigation data is made
public. For each visit, the dataset contains information about
B. Related Work the page characteristics, loading performance, as well as the
HTTP transactions in HAR format,7 including request and
Given its recent conception, few works already targeted response headers.
HTTP/3. Saif et al. [8] run experiments controlling both client
and server accessing a single web page. They study the effect 4 https://httparchive.org/, visited on February 4th 2021.
5 https://developers.google.com/web/tools/chrome-user-experience-report
of delay, packet loss and throughput, without finding any
6 HTTPArchive used to adopt the Alexa top-1M website list, but switched
major impact on performance. In contrast to them, we run a
to the Chrome User Experience Report when Alexa discontinued the rank in
large-scale measurement campaign, controlling only the client July 2018.
and targeting thousands of HTTP/3 websites residing on their 7 http://www.softwareishard.com/blog/har-12-spec/
 3

TABLE II: Network configurations used in the experiments.

Parameter Tested configurations
23 25 27 29
Latency [ms] Native, 50, 100, 200 24 26 28 32
Loss [%] Native, 1, 2, 5 5
Bandwidth [Mbit/s] Native, 5, 2, 1
4

Websites [%]
3
Fundamental for our analyses, the details of HTTP re-
sponses indicate the eventual Alt-Svc header, which is used
2
by servers to announce support to HTTP/3. By setting the
Alt-Svc header, the server has the possibility to inform the 1
client to make subsequent connections using HTTP/3 and may
specify the support to specific draft versions (e.g., 27 or 29). 0
We download the HTTPArchive dataset starting from 0 1 2 1 2 3 4 5 6 7 8 9 0 1 2
9/1 9/1 9/1 0/0 0/0 0/0 0/0 0/0 0/0 0/0 0/0 0/0 0/1 0/1 0/1
November 2019, when we observe the first websites offering 201 201 201 202 202 202 202 202 202 202 202 202 202 202 202
support to HTTP/3. We use the data to study the trend of
HTTP/3 adoption. The data sum up to 6.6 TB. Since we are Fig. 2: Percentage of websites in HTTPArchive that announce
interested in studying the adoption of HTTP/3 on websites, we support to HTTP/3, separately by IETF draft.
discard all visits to internal pages (less than half of the total)
and keep only visits to home pages. We refer to this dataset
as HTTPArchive. protocol. All visits to the same website are run consecutively,
cleaning all state between repetitions, i.e., browser cache, TCP
connections etc. Visits are repeated 5 times to get more reliable
B. HTTP/3 Performance – BrowserTime
results. Hence, we visit each website 4×3×3×5 = 180 times.
We use the most recent snapshot at the time of writ- BrowserTime collects several statistics for each visit, includ-
ing (December 2020) to build the list of websites currently ing details on all HTTP transactions as well as performance
supporting HTTP/3. We find 14 707 websites announcing metrics. We track two metrics that have been shown to be
support to HTTP/3. Next, we visit these websites with three correlated with users’ QoE [17] and can be estimated also at
HTTP versions (HTTP/1.1, HTTP/2, and HTTP/3) to quantify the ISPs [18]:
possible performance improvements. To this end, we rely on • onLoad: The time at which the browser fires the onLoad
BrowserTime, a dockerized tool to run automatic visits to web event – i.e., when all elements of the page, including
pages with a large set of configurable parameters.8 We use images, style sheets and scripts have been downloaded
BrowserTime to instrument Google Chrome to visit websites and parsed;
using a specific HTTP version. Important for our goal, Google 9
• SpeedIndex: Proposed by Google, it represents the time
Chrome allows specifying a set of domains to be contacted at which visible portions of the page are displayed.
with HTTP/3 on the first visit, i.e., without prior indication It is computed by capturing the video of the browser
via Alt-Svc header. We limit ourselves to Chrome, since screen and tracking the visual progress of the page during
we are not aware of similar functionalities in other browsers rendering.
(e.g., Firefox).
In total, we run 2 647 260 visits over a period of one month.
We are interested in studying the impact of HTTP/3 under
The visit metadata account for 189 GB, and we call this dataset
different network conditions. As such, we run our measure-
BrowserTime.
ments enforcing different network configurations. We run our
experiments using two high-end servers connected to the IV. HTTP/3 ADOPTION AND PERFORMANCE
Internet via 1 Gbit/s Ethernet and located in our university
campus. We call this baseline scenario Native, as reported in In this section we first provide an overview of the HTTP/3
Table II. adoption (Section IV-A). Since announcing HTTP/3 support is
Then, we enforce other configurations during the visits not the same as serving content using the protocol, we quantify
relying on the well-known Linux tc tool. Each network the amount of content served over HTTP/3 (Section IV-B).
configuration is defined by changing one of three parameters: Then, we study how HTTP/3 affects QoE-related performance
(i) extra latency, (ii) extra packet loss, or (iii) bandwidth limit. metrics (Section IV-C) and whether identified improvements
For each parameter, we use 4 different configurations, reported can be related to the provider hosting content (Section IV-D)
in Table II. In case of latency, we impose it on the uplink, or website characteristics (Section IV-E).
while packet loss and bandwidth limit are enforced on both up
and down links. For each network configuration, we visit each A. HTTP/3 adoption
website (i) enabling only HTTP/1.1, (ii) enabling HTTP/1.1 We first study to what extent HTTP/3 has been adopted
and HTTP/2, and (iii) enabling all three versions of the since its first proposal. To this end, we profit from the
8 https://www.sitespeed.io/documentation/browsertime/ 9 https://web.dev/speed-index/
 4

to revisit websites to measure their performance, we consider

104 only these 14 707 websites for results that will follow.
The majority of the 14 707 websites supporting HTTP/3
103
are hosted by large companies running their own server
Websites

102 applications. We breakdown these numbers in Figure 3, which

indicates the 10 most popular servers as indicated on the HTTP
101 Server header. CloudFlare, as expected, hosts most of the
websites supporting HTTP/3 (notice the log y-scale). Google
100 is in the second position, with GSE (Google Servlet Engine),
e E d x d s y e y te r
u dflar GS ecifie ngin onten gw Cadd pach Fl ansla Othe Google Frontend and GWS (Google Web Server). Indeed, GSE
clo ts p r A gt r
No eF
o ogl is used on the Blogspot platform, represented by 809 websites
G in our list. For 575 websites, there is no server indication on
HTTP responses, and we find that 445 of these websites belong
Fig. 3: Server in HTTP response (December 2020). to Facebook – e.g., facebook.com and instagram.com
domains. The remaining websites run popular open-source
servers (nginx, Apache) or more peculiar HTTP ones (e.g.,
1.0 Caddy) that offer HTTP/3 support in their earlier versions.
Objects
0.8
Volume
CDF (Websites)

B. Content served over HTTP/3

0.6
Next, we study to what extent objects of enabled websites
0.4 are served using HTTP/3. Indeed, even if a website supports
HTTP/3, not all of its objects are served through HTTP/3. Ob-
0.2 jects may be downloaded from external CDNs, cloud providers
or third-parties not supporting the same protocol. This is the
0.0
0 20 40 60 80 100 case, for example, for ads and trackers typically hosted on
Objects/volume served on HTTP/3 [%] different third-party infrastructure. We use the BrowserTime
dataset, which allows us to observe the protocol used for
Fig. 4: Share of objects/volume served using HTTP/3 on delivering each object composing the visited websites.
enabled websites. In Figure 4 we consider all visits run with HTTP/3 enabled.
For each visit, we compute the share of objects served over
HTTP/3. As each website is accessed multiple times, we aver-
HTTPArchive dataset. The first IETF draft was published on age the values across visits. Clearly, at least the main HTML
in January 2017, but we observe the first websites adopting document is always sent over HTTP/3, but the remaining
HTTP/3 only in late 2019. Since then, the number of websites objects may be served with older HTTP versions. The figure
supporting HTTP/3 has started to grow. Figure 2 shows the shows the distribution of the percentage of objects served
trend for the last months of 2019 and the entire 2020. Looking over HTTP/3 (solid red line) and also depicts the byte-wise
at the Alt-Svc header, we can observe the HTTP/3 draft percentage – i.e., weighting each object by its size. We first
version supported by the server, shown with different colors notice that in 18% of cases, all objects are delivered over
in the figure. In case a website offers more than one version, HTTP/3, meaning that the web page only contains elements
we considered the earliest observed in HTTPArchive. hosted on HTTP/3 enabled servers. The websites having 90%
In the first four months, the number of websites supporting or more of objects (volume) on HTTP/3 are 36 (41) % and
HTTP/3 increased slowly, reaching 0.7 % of the total. At only 9 (28) % have less than 20 % of objects (volume).
that period, only Google and Facebook used to offer HTTP/3 Interestingly, we notice that 51% of websites still have one
for their websites. In February 2020, the number of websites or more objects retrieved using HTTP/1.1.
supporting HTTP/3 exploded. This is due to CloudFlare, which Next, we dissect the above analysis by provider – i.e., the
enabled HTTP/3 on most of the websites it hosts. The share of company/CDN hosting the website. We obtain it by looking
websites supporting HTTP/3 passes 4%, reaching a maximum at the server HTTP header, website name and server IP
in October 2020, with 4.8% of the websites (203 k). In address, which allow easy identification. As discussed for
November, the number of websites suddenly dropped to less Figure 3, we notice that HTTP/3 has been adopted mostly by
than 0.1 % (4 024 in absolute terms). This was caused by (i) Cloudflare CDN, (ii) the Facebook and (iii) Google. The
CloudFlare suspending support to HTTP/3 due to performance remaining 595 websites (i.e., Other) belong mostly to self-
issues, as declared online.10 On December, CloudFlare re- hosted websites running updated versions of the nginx web
enabled HTTP/3 on a subset of websites. On that date, 14 707 server.
websites were announcing support to HTTP/3. Since we need Figure 5 shows the share of objects and volume served over
HTTP/3, separately by provider. Websites hosted by Cloudflare
10 https://community.cloudflare.com/t/community-tip-http-3-with-quic/ tend to be more heterogeneous, with half of the objects
117551, visited on 2/20/2021. retrieved via non-HTTP/3 servers (on median). Moreover, only
 5

100 12
HTTP/1.1
10

Page Load Time [s]

HTTP/2
HTTP/3 Share [%]

75
8 HTTP/3
50 6
4
25 Objects
Volume 2
0 0
Cloudflare Facebook Google Other
(12 455) (445) (1 094) (731)
6
Fig. 5: Share of objects/volume served on HTTP/3, separately 5
by provider.

Speed Index [s]

4
3
24% of the volume is served by using HTTP/3. This is likely
due to the variety of websites relying on the provider: Indeed, 2
Cloudflare offers its hosting service to a very large number 1
of websites. These websites may use complex web pages
composed of several third-party objects stored on external 0
Native 50 100 200
servers that do not rely on HTTP/3 yet. Conversely, Facebook Extra Latency [ms]
and Google show a very different situation. Almost all objects
are served with HTTP/3. This is expected, since Facebook and
Google use their CDNs mostly to offer their own services. Fig. 6: onLoad (top) and SpeedIndex (bottom) with extra
Looking at Google, the long tail of the distribution is due latency, separately for HTTP/1.1, HTTP/2 and HTTP/3.
to Blogspot websites, in which the creator may add content
from external sources. Finally, considering the Other category,
almost all the objects and volumes are served using HTTP/3. We illustrate how the metric values vary when imposing
These websites tend to be simple, composed of a few objects different network conditions, focusing firstly on extra latency
stored in the same self-hosted servers together with the main in Figure 6. Using boxplots, we show the distribution of
HTML document. onLoad (top) and SpeedIndex (bottom), separately by HTTP
version (colored boxes). The boxes span from the first to
C. Performance gains the third quartile, whiskers report the 10th and the 90th
percentiles, while black strokes represent the median. When no
We now study the impact of HTTP/3 on web page per-
extra latency is imposed (native case), we observe that onLoad
formance. To this end, we use the BrowserTime dataset, in
time is in median around 2s, while SpeedIndex around 1s,
which the 14 707 websites have been visited multiple times
without significant differences across HTTP versions. When
under different network conditions. Besides computing the
adding extra latency, the websites load slower as more time is
performance in the native scenario (i.e., 1 gpbs Ethernet on a
needed to download the page objects, requiring in median 6
campus network), we use tc-netem to enforce extra latency,
seconds with 200 ms of additional latency. Not shown here for
packet loss and limit bandwidth. We then contrast page QoE-
brevity, also packet loss and limited bandwidth cause similar
related performance indicators (onLoad and SpeedIndex), (i)
degradation of performance indicators. Figure 6 shows that
showing their absolute value and (ii) computing a metric that
HTTP/1.1 has the worst performance with high latency, while
we call H3 Delta. Given a website and a given network
HTTP/3 shows the greatest benefits. Considering additional
scenario, we obtain the H3 Delta as the relative deviation of
latency of 200 ms, websites onLoad in median in 6.4, 5.8 and
the metric when using HTTP/3 (h3) instead of HTTP/2 (h2).
5.4 s with HTTP versions 1.1, 2 and 3, respectively.
As we always run 5 visits for each case, we consider median
To better catch differences between HTTP/3 and HTTP/2,
values. The H3 Delta for a website w in scenario s is computed
we now study the H3 Delta in Figure 7, where we show the
as follows:
distribution over the 14 707 websites for both onLoad (top
row) and SpeedIndex (bottom row). The three columns refer
median(w, s, h3) − median(w, s, h2) to scenarios with additional latency, limited bandwidth and
H3-Delta(w, s) =
max(median(w, s, h3), median(w, s, h2)) packet loss, respectively. The solid red lines represent the
By definition, H3 Delta(w, s) is bound in [−1, 1] and it is native case. Dashed lines represent scenarios with emulated
negative when a website loads faster under HTTP/3, and network conditions, as indicated in Table II.
positive otherwise. We compute the H3 Delta for both onLoad Starting from latency, we confirm what already emerged
and SpeedIndex. from Figure 6. In the native case, we observe no general trend:
 6

(a) Latency. (b) Bandwidth. (c) Packet loss.

Fig. 7: H3 Delta on different scenarios. onLoad (top) and SpeedIndex (bottom). Negative values indicate that HTTP/3 is faster.

Looking at the solid red lines, we notice that approximately in some other cases. In fact, in several tested cases, some
in 50 % of the cases websites load faster with HTTP/3 and websites can even perform worse when HTTP/3 is enabled.
in the remaining cases HTTP/3 is slower. When latency is
high, HTTP/3 gives sizable benefits compared to HTTP/2. If D. HTTP/3 performance by provider
we impose extra latency of 50 ms, 69 (74) % of websites
have lower onLoad time (SpeedIndex), meaning that they load Next we study whether HTTP/3 performance gains could be
faster. The number of websites loading faster increase to 76 related to the provider hosting the websites. As we observed
(81) % with 100 ms latency. With 200 ms, the number of sizable performance benefit for HTTP/3 only in cases of high
websites loading faster reach 81 (87) %, and the median H3 latency or poor bandwidth, we restrict our analyses to those
Delta is -0.08 (0.12). cases.
Figure 8 shows the distribution of H3 Delta for onLoad,
Focusing on experiments with bandwidth limitation (central separately by provider. We focus on scenarios with 200 ms
plots in Figure 7), different considerations hold. We observe extra latency and 1 Mbit/s bandwidth limit. We observe that
sizable benefits only for onLoad time with the bandwith the H3 Delta largely differs by provider. Focusing on latency
limited to 1 Mbit/s, where 69 % of websites load faster (Figure 8a), Facebook websites show the highest performance
with HTTP/3. Notice that this benefit cannot be introduced gain (H3 Delta −0.13 in median), represented in the figure
by indirect higher latency due to queuing delay (also called with the blue dashed line. Moreover, 95% of websites are
bufferbloat), as we limit the machine queues to 32 KB. In other loaded faster with HTTP/3 than with HTTP/2. Cloudflare (red
cases, no clear trend emerges, but we notice a larger variability solid line) shows the smallest benefits, with only 72% of
of the H3 Delta measure introduced by the constrained setup. websites loading faster. Google and the remaining websites
For example, in case of SpeedIndex, 56, 49, 58 % websites sit in the middle. Similar considerations hold for SpeedIndex,
load faster with HTTP/3 with 5, 2 and 1 Mbit/s bandwidth, not shown here for brevity.
respectively. Similar considerations hold for packet loss (right- With limited bandwidth (Figure 8b), we observe a very
most plots in Figure 7). Despite a larger variability, we cannot different situation. Here, Facebook has in general worst per-
identify any general trend, and the H3 Delta values are equally formance with HTTP/3 with 91% of its websites loading
distributed above and below 0. faster with HTTP/2. Conversely, Google (green dashed line)
In summary, we observe improvements on onLoad time with shows the best figures, with median H3 Delta −0.14 and 79%
poor bandwidth when using HTTP/3. HTTP/3 shows sizable of websites loading faster with HTTP/3. Cloudflare and the
benefits in case of high latency. We do not testify performance remaining websites exhibit no clear trend with roughly half of
benefits of HTTP/3 in scenarios with high packet loss and the websites loading faster with HTTP/3.
 7

1.0 6
H3 Faster
5

Normalized Value
0.8 H3 ≈ H2
4 H2 Faster
0.6
CDF

Cloudflare 3
0.4 Facebook 2
Google
0.2 1
Other
0.0 0
n. main
s n. re ze
−0.5 −0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2 0.3 0.4 0.5 f con t con 3 sha Page si
H3 Delta (onLoad) u m ber o ber of Do on larges HTTP/
N Num Objects
(a) Latency.
(a) Latency.
1.0
Cloudflare 6
0.8 Facebook H3 Faster
5

Normalized Value
H3 ≈ H2
0.6 Google
4 H2 Faster
CDF

Other
0.4 3
2
0.2
1
0.0
−0.5 −0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2 0.3 0.4 0.5 0
n. s n. re e
H3 Delta (onLoad) o f con of Domain rgest con TP/3 sha Page siz
ber a HT
Num Number jects on l
(b) Bandwidth. Ob

Fig. 8: onLoad H3 Delta by website provider for scenarios (b) Bandwidth.

with extra-latency and bandwidth limit.
Fig. 9: Visit characteristics vs. H3 Delta class (normalized
values).
E. Page characteristics
Now we investigate page characteristics and possible cor-
relations to performance when using HTTP/3. To this end, Delta ∈ [−0.1, 0.1].
we compute various metrics describing the web page load • H2 Faster: websites loading faster with HTTP/2, i.e.,
process and contrast them to understand whether they show onLoad H3 Delta > 0.1.
correlations to the H3 Delta. For each visit to the 14 707 In the figure, boxes with different colors represent these
websites in our dataset, in addition to the H3 Delta, we three classes. The y-axis represents the metrics normalized
compute the following metrics: by scaling to unit variance to ease the visualization and the
• Number of connections issued by the browser to load comparison. Again, we study the scenarios with extra latency
the website when using HTTP/3. (top row) and limited bandwidth (bottom row) since they
• Number of domains contacted while loading the web provide the most interesting insights. In case of extra latency,
page, thus including third-party domains). H3 Faster websites are 32%, H2 Faster 11% and H3 ≈ H2 are
• Share of objects on the largest connection, which 57%. With limited bandwidth, they are 38%, 25% and 37%,
measures the share of objects carried over the connection respectively.
where most objects have been requested. Remind that the We first focus on the left-most box group of Figure 9,
best practices of HTTP/3 recommend avoiding domain showing the (normalized) number of connections the browser
sharding to increase performance. issued to load the web page. Green boxes hint that websites
• Share of objects served on HTTP/3, which is used to issuing fewer connections (smaller metric values) are faster
investigate possible correlations of the fraction of objects with HTTP/3 than in HTTP/2. This is true in both scenario,
served over HTTP/3 (shown in Figure 4) and the H3 Delta i.e., low latency and poor bandwidth. Similar considerations
metric. hold if we focus on the second box group, representing the
• Page Size, to breakdown performance for small and large number of contacted domains. Indeed, we notice that the num-
web pages. ber of connections per website and the number of contacted
In Figure 9, we compare the distribution of the aforemen- domains are 0.91-correlated (Pearson correlation). The third
tioned metrics, grouping websites based on classes defined by box group offers a similar perspective, measuring how web
the onLoad H3 Delta: page objects are split on multiple connections/domains. The
• H3 Faster: websites loading faster with HTTP/3, i.e., websites benefiting the most from HTTP/3 are those which
onLoad H3 Delta < −0.1. tend to mass objects on a single connection – see the highest
• H3 ≈ H2: websites having a similar loading position of the green boxes, meaning more objects are on a
time in HTTP/2 and HTTP/3, i.e., onLoad H3 single connection. This is very clear with limited bandwidth
 8

(Figure 9b), rather than with high latency (Figure 9a). scale measurement campaign, we studied the performance of
Serving most objects with HTTP/3 (rather than with HTTP/3 under different network conditions, targeting thou-
HTTP/2) has a positive impact too, as we notice from the sands of websites. We found performance benefits emerging
fourth box group in Figure 9. Again, this is evident especially in scenarios with high latency or poor bandwidth. In the case
with bandwidth limit (Figure 9b), while with extra latency of high packet loss, HTTP/3 and HTTP/2 perform roughly
(Figure 9a) it is hard to find a clear trend. Finally, interesting the same. We found large performance diversity depending on
is the case of the web page size (last box group). In scenarios the infrastructure hosting the website. In general, we observed
with high latency, the websites benefiting from HTTP/3 are that websites taking benefits from HTTP/3 are those loading
small, while large ones typically perform better with HTTP/2. objects from a limited set of third-party domains, thus limiting
Conversely, when bandwidth is scarce, even if moderately, the number of issued connections and avoiding loading content
website loading faster with HTTP/3 are the large ones. using legacy protocols.
In summary, websites taking benefits from HTTP/3 are those
limiting the number of connections and third-party domains,
and fully adopting HTTP/3 on all web page objects. Page size R EFERENCES
has diverse implications depending on the network conditions.
These considerations hold in scenarios with high latency or [1] R. T. Fielding, H. Nielsen, J. Mogul, J. Gettys, and T. Berners-Lee,
limited bandwidth, while we do not observe any clear trend in “Hypertext Transfer Protocol – HTTP/1.1.” RFC 2068, Jan. 1997.
case of optimal network conditions or high packet loss, where [2] M. Belshe, R. Peon, and M. Thomson, “Hypertext Transfer Protocol
Version 2 (HTTP/2).” RFC 7540, May 2015.
metric distributions mostly overlap.
[3] M. Bishop, “Hypertext Transfer Protocol Version 3 (HTTP/3),” Internet-
Draft draft-ietf-quic-http-33, Internet Engineering Task Force, Feb. 2021.
V. D ISCUSSION AND FUTURE WORK Work in Progress.
[4] J. Iyengar and M. Thomson, “QUIC: A UDP-Based Multiplexed and
We dissected the performance of HTTP/3 under diverse net- Secure Transport,” Internet-Draft draft-ietf-quic-transport-33, Internet
work conditions, showing the impact of network latency and Engineering Task Force, Dec. 2020. Work in Progress.
bandwidth across websites. However, we run measurements [5] E. Rescorla, “The Transport Layer Security (TLS) Protocol Version 1.3.”
RFC 8446, Aug. 2018.
using only Google Chrome, as we are not aware of other [6] M. Varvello, K. Schomp, D. Naylor, J. Blackburn, A. Finamore, and
browsers that can be instrumented to use HTTP/3 since the first K. Papagiannaki, “Is the web http/2 yet?,” in International Conference
connection – i.e., without the need to observe the Alt-Svc on Passive and Active Network Measurement, pp. 218–232, Springer,
2016.
header previously. Moreover, we always visit websites with [7] M. Rajiullah, A. Lutu, A. S. Khatouni, M.-R. Fida, M. Mellia, A. Brun-
a fresh browser profile with empty cache and no pre-existing strom, O. Alay, S. Alfredsson, and V. Mancuso, “Web experience in
connections. Clearly, this setup limits the scope of our study mobile networks: Lessons from two million page visits,” in The World
Wide Web Conference, pp. 1532–1543, 2019.
as we cannot measure how HTTP/3 affects performance on
[8] D. Saif, C.-H. Lung, and A. Matrawy, “An early benchmark of quality of
subsequent visits or with a warm HTTP cache, as it will be experience between http/2 and http/3 using lighthouse,” arXiv preprint
the case for future users supporting HTTP/3. arXiv:2004.01978, 2020.
We limited ourselves to a subset of the websites adopting [9] R. Marx, J. Herbots, W. Lamotte, and P. Quax, “Same standards,
different decisions: A study of quic and http/3 implementation diversity,”
HTTP/3. Indeed, we included only a fraction of websites in Proceedings of the Workshop on the Evolution, Performance, and
hosted on the CloudFlare CDN, as its HTTP/3 support is Interoperability of QUIC, pp. 14–20, 2020.
partially disabled at the time of writing. Our measurements [10] S. Tellakula, “Comparing HTTP/3 vs. HTTP/2 Performance.” https://
blog.cloudflare.com/http-3-vs-http-2/, 2020.
will need to be run continuously to observe how the web
[11] L. Guillen, S. Izumi, T. Abe, and T. Suganuma, “Sand/3: Sdn-assisted
ecosystem will react to HTTP/3 in the near future, adopting novel qoe control method for dynamic adaptive streaming over http/3,”
the protocol and modifying the web page structure to optimize Electronics, vol. 8, no. 8, p. 864, 2019.
performance, in particular concerning domain sharding. [12] K. Wolsing, J. Rüth, K. Wehrle, and O. Hohlfeld, “A performance
perspective on web optimized protocol stacks: Tcp+ tls+ http/2 vs. quic,”
Finally, HTTP/3 and QUIC are not yet definitive IETF in Proceedings of the Applied Networking Research Workshop, pp. 1–7,
standards. Although recent modifications to the IETF draft 2019.
only concerned minor protocol features, it will be fundamental [13] J. Manzoor, L. Cerdà-Alabern, R. Sadre, and I. Drago, “On the perfor-
mance of quic over wireless mesh networks,” Journal of Network and
to provide a similar analysis once the final standards are Systems Management, vol. 28, no. 4, pp. 1872–1901, 2020.
approved. Similarly, we did not explore how different endpoint [14] G. Carlucci, L. De Cicco, and S. Mascolo, “Http over udp: An experi-
configurations affect HTTP/3 performance – e.g., the interac- mental investigation of quic,” in Proceedings of the 30th Annual ACM
Symposium on Applied Computing, SAC ’15, (New York, NY, USA),
tions of HTTP/3 and congestion control settings. Moreover, p. 609–614, Association for Computing Machinery, 2015.
whereas we covered different scenarios, it is widely known [15] A. M. Kakhki, S. Jero, D. Choffnes, C. Nita-Rotaru, and A. Mislove,
that network emulation is hard. Similar measurement studies “Taking a long look at quic: an approach for rigorous evaluation of
with actual end-users are still needed to confirm our findings. rapidly evolving transport protocols,” in Proceedings of the 2017 Internet
Measurement Conference, pp. 290–303, 2017.
2020, although most of the early adopters still host the ma- [16] M. Kosek, T. Shreedhar, and V. Bajpai, “Beyond quic v1–a first look
jority of objects on HTTP/2 third-party servers. With a large- at recent transport layer ietf standardization efforts,” arXiv preprint
arXiv:2102.07527, 2021.
VI. C ONCLUSIONS [17] D. N. da Hora, A. S. Asrese, V. Christophides, R. Teixeira, and D. Rossi,
We provided the first study on HTTP/3 adoption and per- “Narrowing the gap between qos metrics and web qoe using above-
the-fold metrics,” in International Conference on Passive and Active
formance, quantifying the performance benefits of HTTP/3 Network Measurement, pp. 31–43, Springer, 2018.
in several network scenarios. We testified how some of the [18] M. Trevisan, I. Drago, and M. Mellia, “Pain: A passive web performance
Internet leading companies started deploying HTTP/3 during indicator for isps,” Computer Networks, vol. 149, pp. 115–126, 2019.