Handling Churn in a DHT

Sean Rhea, Dennis Geels, Timothy Roscoe, and John Kubiatowicz

University of California, Berkeley and Intel Research, Berkeley
{srhea,geels,kubitron}@cs.berkeley.edu, [email protected]

Abstract churn is node session time: the time between when a node
joins the network until the next time it leaves. Median
This paper addresses the problem of churn—the continu- session times observed in deployed networks range from
ous process of node arrival and departure—in distributed as long as an hour to as short as a few minutes.
hash tables (DHTs). We argue that DHTs should perform In this paper we explore the performance of DHTs in
lookups quickly and consistently under churn rates at least such dynamic environments. DHTs may be better able
as high as those observed in deployed P2P systems such to locate rare files than existing unstructured peer-to-peer
as Kazaa. We then show through experiments on an em- networks [18]. Moreover, it is not hard to imagine that
ulated network that current DHT implementations cannot other proposed uses for DHTs will show similar churn
handle such churn rates. Next, we identify and explore rates to file-sharing networks—application-level multicast
three factors affecting DHT performance under churn: re- of a low-budget radio stream, for example. In spite of this
active versus periodic failure recovery, message timeout promise, we show that short session times cause a vari-
calculation, and proximity neighbor selection. We work ety of negative effects on two mature DHT implementa-
in the context of a mature DHT implementation called tions we tested. Both systems exhibit dramatic latency
Bamboo, using the ModelNet network emulator, which growth when subjected to increasing churn, and in one
models in-network queuing, cross-traffic, and packet loss. implementation the network eventually partitions, causing
These factors are typically missing in earlier simulation- subsequent lookups to return inconsistent results. The re-
based DHT studies, and we show that careful attention mainder of this paper is dedicated to determining whether
to them in Bamboo’s design allows it to function effec- a DHT can be built such that it continues to perform well
tively at churn rates at or higher than that observed in P2P as churn rates increase.
file-sharing applications, while using lower maintenance We demonstrate that DHTs can in fact handle high
bandwidth than other DHT implementations. churn rates, and we identify and explore several factors
that affect the behavior of DHTs under churn. The three
most important factors we identify are:
1 Introduction
• reactive versus periodic recovery from failures
The popularity of widely-deployed file-sharing services
has recently motivated considerable research into peer-to- • calculation of message timeouts during lookups
peer systems. Along one line, this research has focused • choice of nearby over distant neighbors
on the design of better peer-to-peer algorithms, especially
in the area of structured peer-to-peer overlay networks or By reactive recovery, we mean the strategy whereby a
distributed hash tables (e.g. [20, 22, 24, 27, 30]), which we DHT node tries to find a replacement neighbor immedi-
will simply call DHTs. These systems map a large iden- ately upon noticing that an existing neighbor has failed.
tifier space onto the set of nodes in the system in a deter- We show that under bandwidth-limited conditions, reac-
ministic and distributed fashion, a function we alternately tive recovery can lead to a positive feedback cycle that
call routing or lookup. DHTs generally perform these overloads the network, causing lookups to have high la-
lookups using only O(log N ) overlay hops in a network tency or to return inconsistent results. In contrast, a DHT
of N nodes where every node maintains only O(log N ) node may recover from neighbor failure at a fixed, pe-
neighbor links, although recent research has explored the riodic rate. We show that this strategy improves perfor-
tradeoffs in storing more or less state. mance under churn by allowing the system to avoid posi-
A second line of research into P2P systems has focused tive feedback cycles.
on observing deployed networks (e.g. [5, 9, 13, 25]). A The manner in which a DHT chooses timeout values
significant result of this research is that such networks are during lookups can also greatly affect its performance un-
characterized by a high degree of churn. One metric of der churn. If a node performing a lookup sends a message
to a node that has left the network, it must eventually time-
111...
out the request and try another neighbor. We demonstrate
that such timeouts are a significant component of lookup
latency under churn, and we explore several methods of
110...
computing good timeout values, including virtual coordi- 0...
nate schemes as used in the Chord DHT.
Finally, we consider proximity neighbor selection
(PNS), where a DHT node with a choice of neighbors
tries to select those that are most nearby itself in net-
work latency. We compare several algorithms for discov-
ering nearby neighbors—including algorithms similar to
10...
those used in the Chord, Pastry, and Tapestry DHTs—to
show the tradeoffs they offer between latency reduction Figure 1: Neighbors in Pastry and Bamboo. A node’s
and added bandwidth. neighbors are divided into its leaf set, shown as dashed
We have augmented the Bamboo DHT [23] such that arrows, and its routing table, shown as solid arrows.
it can be configured to use any of the design choices
described above. As such, we can examine each de-
sign decision independently of the others. Moreover, we implementations under such churn. In Section 4, we study
examine the performance of each configuration by run- each of the factors listed above in isolation, and describe
ning it on a large cluster with an emulated wide-area net- how Bamboo uses these techniques. In Section 5, we sur-
work. This methodology is particularly important with vey related work, and in Section 6 we discuss important
regard to the choice of reactive versus periodic recovery future work. We conclude in Section 7.
as described above. Existing studies of churn in DHTs
(e.g. [7, 8, 16, 19]) have used simulations that—unlike
our emulated network—did not model the effects of net- 2 Introduction to DHT Routing
work queuing, cross traffic, or message loss. In our ex-
perience, these effects are primary factors contributing to In this section we present a brief review of DHT rout-
DHTs’ inability to handle churn. Moreover, our measure- ing, using Pastry [24] as an example. The geometry and
ments are conducted on an isolated network, where the routing algorithm of Bamboo are identical to Pastry; the
only sources of queuing, cross traffic, and loss are the difference (and the main contribution of this paper) lies in
DHTs themselves; in the presence of heavy background how Bamboo maintains the geometry as nodes join and
traffic, we expect that such network realities will exacer- leave the network and the network conditions vary.
bate the ability of DHTs to handle even lower levels of DHTs are structured graphs, and we use the term geom-
churn. etry to mean the pattern of neighbor links in the overlay
Of course, this study has limitations. Building and test- network, independent of the routing algorithms or state
ing a complete DHT implementation on an emulated net- management algorithms used [12].
work is a major effort. Consequently, we have limited our- Each node in Pastry is assigned a numeric identifier in
selves to studying a single DHT on a single network topol- [0, 2160 ), derived either from the SHA-1 hash of the IP
ogy using a relatively simple churn model. Furthermore, address and port on which the node receives packets or
we have not yet studied the effects of some implementa- from the SHA-1 hash of a public key. As such, they are
tion decisions that might affect the performance of a DHT well-distributed throughout the identifier space.
under churn, including the use of alternate routing table In Pastry, a node maintains two sets of neighbors, the
neighbors as in Kademlia and Tapestry, or the use of it- leaf set and the routing table (see Figure 1). A node’s
erative versus recursive routing. Nevertheless, we believe leaf set is the set of 2k nodes immediately preceding and
that the effects of the factors we have studied are dramatic following it in the circular identifier space. We denote this
enough to present them as an important early study in the set by L, and we use the notation Li with −k ≤ i ≤ k to
effort to build a DHT that successfully handles churn. denote the members of L, where L0 is the node itself.
The rest of this paper is structured as follows: in the In contrast, the routing table is a set of nodes whose
next section we review how DHTs perform routing or identifiers share successively longer prefixes with the
lookup, with particular reference to Pastry, whose routing source node’s identifier. Treating each identifier as a se-
algorithm Bamboo also uses. In Section 3, we review ex- quence of digits of base 2b and denoting the routing ta-
isting studies of churn in deployed file-sharing networks, ble entry at row l and column i by Rl [i], a node chooses
describe the way we model such churn in our emulated its neighbors such that the entry at Rl [i] is a node whose
network, and quantify the performance of mature DHT identifier matches its own in exactly l digits and whose
 111...
if (L−k ≤ D ≤ Lk )
next hop = Li s.t. |D − Li | is minimal
else if (Rl [D[l]] 6= null)
00...
next hop = Rl [D[l]]
else
next hop = Li s.t. |D − Li | is minimal
010...

Figure 2: The Bamboo routing algorithm. The code

shown chooses the next routing hop in for a message with 011...
destination D, where D matches the identifier of the local
node in the first l digits. Figure 3: Recursive lookup. To find the node closest to
identifier 011, the node whose identifier starts with 111
sends a lookup message to its neighbor whose first digit
(l + 1)th digit is i. In the experiments in this paper, Bam- is 0. This node then forwards the query to its neighbor
boo uses binary digits (b = 1), though it can be configured whose first two digits are 01, and from there the node is
to use any base. forwarded to the neighbor whose first three digits are 011.
The basic operation of a DHT is to consistently map
identifiers onto nodes from any point in the system, a 111...
function we call routing or lookup. Pastry achieves con-
sistent lookups by directing each identifier to the node
with the numerically closest identifier. Algorithmically,
00...
routing proceeds as shown in Figure 2. To route a message
with key D, a node first checks whether D lies within its
leaf set, and if so, forwards it to the numerically closest
member of that set (modulo 2160 ). If that member is the 010...
local node, routing terminates. If D does not fall within
the leaf set, the node computes the length l of the longest 011...
matching prefix between D and its own identifier. Let
D[i] denote the ith digit of D. If Rl [D[l]] is not empty, Figure 4: Iterative lookup. An iterative lookup involves
the message is forwarded on to that node. If neither of the same nodes as a recursive one, but instead of forward-
these conditions is true, the message is forwarded to the ing the message, each intermediate node responds to the
member of the node’s leaf set numerically closest to D. source with the address of the next hop.
Once the destination node is reached, it sends a message
back to the originating node with its identifier and network
address, and the lookup is complete.
We note that a node can often choose between many network of 65,536 nodes. This resilience is important in
different neighbors for a given entry in its routing ta- handling failures in general and churn in particular, and
ble. For example, a node whose identifier begins with a was the reason we chose the Pastry geometry for use in
1 needs a neighbor whose identifier begins with a 0, but Bamboo. We could also have used a pure ring geome-
such nodes make up roughly half of the total network. In try as in Chord, extending it to account for proximity in
such situations, a node can choose between the possible neighbor selection as described in [12].
candidates based on some metric. Proximity neighbor se- The manner in which we have described routing so
lection is the term used to indicate that nodes in a DHT far is commonly called recursive routing (Figure 3). In
use network latency as the metric by which to choose be- contrast, lookups may also be performed iteratively. As
tween neighbor candidates. shown in Figure 4, an iterative lookup involves the same
Using this design, Pastry and Bamboo perform lookups nodes as a recursive one, but the entire process is con-
in O(log N ) hops [24], while the leaf set allows forward trolled by the source of the lookup. Rather than asking
progress (in exchange for potentially longer paths) in the a neighbor to forward the lookup through the network on
case that the routing table is incomplete. Moreover, the its behalf, the source asks that neighbor for the network
leaf set adds a great deal of static resilience to the geom- address of the next hop. The source then asks the newly-
etry; Gummadi et al. [12] show that with a leaf set of 16 discovered node the same question, repeating the process
nodes, even after a random 30% of the links are broken until no further progress can be made, at which point the
there are still connected paths between all node pairs in a lookup is complete.
 Lifetime First Author Systems Observed Session Time
Saroiu [25] Gnutella, Napster 50% ≤ 60 min.
Session
Chu [9] Gnutella, Napster 31% ≤ 10 min.
Time
time Sen [26] FastTrack 50% ≤ 1 min.
Bhagwan [5] Overnet 50% ≤ 60 min.
Join Leave Join Leave Gummadi [13] Kazaa 50% ≤ 2.4 min.

Figure 5: Metrics of churn. With respect to the routing Table 1: Observed session times in various peer-to-peer
and lookup functionality of a DHT, the session times of systems. The median session time ranges from an hour to
nodes are more relevant than their lifetimes. a minute.

3 The Problem of Churn 3.1 Empirical studies

Elsewhere [23] we have surveyed published studies of de-
There have been very few large-scale DHT-based applica- ployed file-sharing networks. Table 1 shows a summary
tion deployments to date, and so it is hard to derive good of observed session times. At first sight, some of these
requirements on churn-resilience. However, P2P file- values are surprising, and may be due to methodological
sharing networks provide a useful starting point. These problems with the study in question or malfunctioning of
systems provide a simple indexing service for locating the system under observation. However, it is easy to im-
files on those peer nodes currently connected to the net- age a user joining the network, downloading a single file
work, a function which can be naturally mapped onto a (or failing to find it), and leaving, making session times
DHT-based mechanism. For example, the Overnet file- of a few minutes at least plausible. To be conservative,
sharing system uses the Kademlia DHT to store such an then, we would like a DHT to be robust for median ses-
index. While some DHT applications (such as file stor- sion times from as much as an hour to as little as a minute.
age as in CFS [10]) might require greater client availabil-
ity, others may show similar churn rates to file-sharing
networks (such as end-system multicast or a rendezvous
3.2 Experimental Methodology
service for instant messaging). As such, we believe that Our platform for measuring DHT performance under
DHTs should at least handle churn rates observed in to- churn is a cluster of 40 IBM xSeries PCs, each with
day’s file-sharing networks. To that end, in this section we Dual 1GHz Pentium III processors and 1.5GB RAM, con-
survey existing studies of churn in deployed file-sharing nected by Gigabit Ethernet, and running either Debian
networks, describe the way we model such churn in our GNU/Linux or FreeBSD. We use ModelNet [28] to im-
emulated network, and quantify the performance of ma- pose wide-area delay and bandwidth restrictions, and the
ture DHT implementations under such churn. Inet topology generator [3] to create a 10,000-node wide-
Studies of existing file-sharing systems mainly use two area AS-level network with 500 client nodes connected to
metrics of churn (see Figure 5). A node’s session time is 250 distinct stubs by 1 Mbps links. To increase the scale
the elapsed time between it joining the network and subse- of the experiments without overburdening the capacity of
quently leaving it. In contrast, a node’s lifetime is the time ModelNet by running more client nodes, each client node
between it entering the network for the first time and leav- runs two DHT instances, for a total of 1,000 DHT nodes.
ing the network permanently. The sum of a node’s session Our control software uses a set of wrappers which com-
times divided by its lifetime is often called its availabil- municate locally with each DHT instance to send requests
ity. One representative study [5] observed median session and record responses. Running 1000 DHT instances on
times on the order of tens of minutes, median lifetimes on this cluster (12.5 nodes/CPU) produces CPU loads below
the order of days, and median availability of around 30%. one, except during the highest churn rates. Ideally, we
With respect to the lookup functionality of a DHT, we would measure larger networks, but 1000-node systems
argue that session time is the most important metric. Even already demonstrate problems that will surely affect larger
temporary loss of a routing neighbor weakens the cor- ones.
rectness and performance guarantees of a DHT, and un- In an experiment, we first bring up a network of 1000
available neighbors reduce a node’s effective connectivity, nodes, one every 1.5 seconds, each with a randomly as-
forcing it to choose suboptimal routes and increasing the signed gateway node to distribute the load of bootstrap-
destructive potential of future failures. Since nodes are of- ping newcomers. We then churn nodes until the system
ten unavailable for long periods, remembering neighbors performance levels out; this phase normally lasts 20-30
that have failed is of little value in performing lookups. minutes but can take an hour or more. Node deaths are
While remembering neighbors is useful for applications timed by a Poisson process and are therefore uncorrelated
like storage [6], it is of little value for lookup operations. and bursty. A new node is started each time one is killed,
maintaining the total network size at 1000. This model
100
of churn is similar to that described by Liben-Nowell et

Percent of Lookups
all [17]. In a Poisson process, an event rate λ corresponds 80 6.2 h 1.6 h
to a median inter-event period of ln 2/λ. For each event 60 3.1 h 47 min
we select a node to die uniformly at random, so each 40
node’s session time is expected to span N events, where
20 Consistent 23 min
N is the network size. Therefore a churn rate of λ corre- Completed
sponds to a median node session time of 0
0 50 100 150 200
tmed = N ln 2/λ. Time (minutes)

For example, a 1000-node network churning with median Figure 6: FreePastry under churn. The percentage of suc-
session times of one hour will see one node arrive (and cessful lookups in a 1000-node FreePastry network under
one leave) every 5.2 seconds. In our experiments, we used churn. Session times for each 30-minute churn period are
churn rates ranging from 8/second to 4/minute, equal to indicated by arrows, and each churn period is separated
median session times from 1.4 minutes to 3 hours. from the next by 10 minutes of no churn. The churn rate
Each live node continually performs lookups for identi- doubles with each successive period.
fiers chosen uniformly at random, timed by a Poisson pro-
cess with rate 0.1/second, for an aggregate system load of
100 lookups/second. Each lookup is simultaneously per- both systems, it is still possible that alternative configu-
formed by ten nodes, and we report both whether it com- rations could have improved their performance. More-
pletes and whether it is consistent with the others for the over, both systems have seen subsequent development,
same key. If there is a majority among the ten results for a and newer versions may show improved resilience under
given key, all nodes in the majority are said to see a con- churn.
sistent result, and all others are considered inconsistent.
If there is no majority, all nodes are said to see inconsis- FreePastry We tested FreePastry 1.3, the Rice Univer-
tent results. This metric of consistency is more strict than sity implementation of Pastry [1]. Figure 6 shows one
that required by some DHT applications. However, both effect of churn on a network of 1000 FreePastry nodes,
MIT’s Chord and our Bamboo implementation show at which we ran using the default 24-node leaf sets and log-
least 99.9% consistency under 47-minute median session arithm base of 16. We do not enforce proximity between a
times [23], so it does not seem unreasonable. new node and its gateway, as suggested for best FreePas-
There are two ways in which lookups fail in our tests. try performance; this decision only effects the proximity
First, we do not perform end-to-end retries, so a lookup of a node’s neighbors, not the efficiency of its routing.
may fail to complete if a node in the middle of the lookup It is clear from Figure 6 that while successful lookups
path leaves the network before forwarding the lookup re- are mostly consistent, FreePastry fails to complete a ma-
quest to the next node. We observed this behavior primar- jority of lookup requests under heavy churn. A likely
ily in FreePastry as described below. Second, a lookup explanation for this failure is that nodes wait so long on
may return inconsistent results. Such failures occur either lookup requests to time out that they frequently leave the
because a node is not aware of the correct node to forward network with several requests still in their queues. This
the lookup to, or because it erroneously believes the cor- behavior is probably exacerbated by FreePastry’s use of
rect node has left the network (because of congestion or Java RMI over TCP as its message transport, and the way
poorly chosen timeouts). All DHT implementations we that FreePastry nodes handle the loss of their neighbors.
have tested show some inconsistencies under churn, but We present evidence to support these ideas in Section 4.1.
carefully chosen timeouts and judicious bandwidth usage We make a final comment on this graph. FreePas-
can minimize them. try generally recovers well between churn periods, once
again correctly completing all lookups. The difficulty
3.3 Existing DHTs with real systems is that there is no such quiet period; the
network is in a continual state of churn.
In this section we report the results of testing two ma-
ture DHT implementations under churn. Our intent here
is not to place a definitive bound on the performance of MIT Chord We tested MIT’s Chord implementa-
either implementation. Rather, it is to motivate our work tion [4] using a CVS snapshot from 8/4/2003, with the de-
by demonstrating that handling churn in DHTs is both an fault 10-node successor lists and with the location cache
important and a non-trivial problem. While we have dis- disabled (using the -F option), since the cache causes poor
cussed these experiments extensively with the authors of performance under churn.
 5
Chord
4 Handling Churn
4 Bamboo (No PNS)

Mean Latency (s)

Bamboo (PNS)
Having briefly described the way in which DHTs perform
3
lookups, and having given evidence indicating that their
2
ability to do so is hindered under churn, we now turn to
1 the heart of this paper: a study of the factors contribut-
0 ing to this difficulty, and a comparison of solutions that
8 16 32 64 128
can be used to overcome them. In turn, we discuss re-
Median Session Time (min)
active versus periodic recovery from neighbor failure, the
calculation of good timeout values for lookup messages,
Figure 7: Chord under churn. Shown is the mean latency
and techniques to achieve proximity in neighbor selection.
of lookups in a 1000-node MIT Chord network under in-
The remainder of this paper focuses only on the Bamboo
creasing levels of churn. Churn increases to the left.
DHT, in which we have implemented each alternative de-
sign choice studied here. Working entirely within a sin-
gle implementation allows us to minimize the differences
In contrast to FreePastry, almost all lookups in a Chord between experiments comparing one design choice to an-
network complete and return consistent results. Chord’s other.
shortcoming under churn is in lookup latency, as shown
in Figure 7, which shows the result of running Chord un-
4.1 Reactive vs. Periodic Recovery
der the same workload as shown in Figure 6, but where
we have averaged the lookup latency over each churn pe- Early implementations of Bamboo suffered performance
riod. Shown for comparison are two lines representing degradation under churn similar to that of FreePastry.
Bamboo’s performance in the same test, with and with- MIT Chord’s performance, however, does not degrade in
out proximity neighbor selection (PNS). Under all churn the same way. A significant difference in its behavior is a
rates, Bamboo is using slightly under 750 bytes per sec- design choice about how to handle detected node failures.
ond per node, while Chord is using slightly under 2,400. We will call the two alternative approaches reactive and
We discuss in detail the differences that enable Bamboo periodic recovery.
to outperform Chord in Sections 4.2 and 4.3, but some of
the difference in latency between Bamboo and Chord is Reactive recovery In reactive recovery, a node reacts to
due to their routing styles. Bamboo performs lookups re- the loss of one if its existing leaf set neighbors (or the ap-
cursively, whereas Chord routes iteratively. Chord could pearance of a new node that should be added to its leaf
easily be changed to route recursively; in fact, newer ver- set) by sending a copy of its new leaf set to every node
sions of Chord support both recursive routing and PNS. in it. To save bandwidth, a node can only send differ-
Note, however, that Chord’s latency grows more quickly ences from the last message, but the total number of mes-
under increasing churn than does Bamboo’s. In Sec- sages is still O(k 2 ) for a leaf set of k nodes. This algo-
tion 4.2, we will show evidence to support our belief that rithm converges quickly, is used in FreePastry, and was
this growth is due to Chord’s method of choosing time- used in early versions of Bamboo. MSPastry uses a more
outs for lookup messages and is independent of the lookup bandwidth-efficient, but more complex, variant of reactive
style employed. recovery [7].

Periodic recovery In contrast, in periodic recovery a

node periodically shares its leaf set with each of the mem-
bers of that set, each of whom responds in kind with its
3.3.1 Summary
own leaf set. The process takes place independently of
the node detecting changes in its leaf set. As a simple op-
To summarize this section, we note that we have observed timization, a node picks one random member of its leaf
several effects of churn on existing DHT implementa- set to share state with in each period. This change saves
tions. A DHT may fail to complete lookup requests al- bandwidth, but still converges in O(log k) phases, where
together, or it may complete them but return inconsistent k is the size of the leaf set. (Further details can be found
results for the same lookup launched from different source elsewhere [23].) This algorithm is the one currently used
nodes. On the other hand, a DHT may continue to return by Bamboo, and the periodic nature of this algorithm is
consistent results as churn rates increase, but it may suffer shared by Chord’s method of keeping its successor list
from a dramatic increase in lookup latency in the process. correct.
 8 6

95th Percentile Latency (s)

Reactive Reactive

Bandwidth (kB/s/node)
7 Periodic 5 Periodic
6
5 47 min 23 min 4
47 min 23 min
4 3
3 2
2
1 1
0 0
0 10 20 30 40 50 0 10 20 30 40 50
Time (minutes) Time (minutes)

Figure 8: Reactive versus periodic recovery. Without churn, reactive recovery is very efficient, as messages are only
sent in response to actual changes. At reasonable churn rates, however, periodic recovery uses less bandwidth, and
lower contention for the network leads to lower latencies.

4.1.1 Positive feedback cycles 4.1.2 Scalability

Experiments show little difference in correctness between

Reactive recovery runs the risk of creating a positive feed-
periodic and reactive recovery. To see why, consider a
back cycle as follows. Consider a node whose access
node A that joins a network, and let B be the node in the
link to the network is sufficiently congested that timeouts
existing network whose identifier most closely matches
cause it to believe that one of its neighbors has failed. If
that of A. As in Pastry, A retrieves its initial leaf set by
the node is recovering reactively, recovery operations be-
contacting B, and B adds A to its leaf set immediately af-
gin, and the node will add even more packets to its al-
ter confirming its IP address and port (with a probe mes-
ready congested network link. This added congestion will
sage). Until A’s arrival propagates through the network,
increase the likelihood that the node will mistakenly con-
another node C may still route messages that should go
clude that other neighbors have failed. If this process con-
to A to B instead, but B will just forward these messages
tinues, the node will eventually cause congestion collapse
on to A. Likewise, should A fail, B will still be in C’s
on its access link.
leaf set, so once routing messages to A time out, C and
Observations of these cycles in early Bamboo (and ex- other nearby nodes will generally all agree that B is the
amination of the Chord code) originally led us to pro- next best choice.
pose periodic recovery for handling churn. By decou- While both periodic and reactive recovery achieve
pling the rate of recovery from the discovery of failures, roughly identical correctness, there is a large difference in
periodic recovery prevents the feedback cycle described the bandwidth consumed under different churn rates and
above. Moreover, by lengthening the recovery period with leaf set sizes. (A commonly accepted rule of thumb is
the observation of message timeouts, we can introduce a that to provide sufficient resilience to massive node fail-
negative feedback cycle, further improving resilience. ure, the size of a node’s leaf set should be logarithmic
Another way to mitigate the instability associated with in the system size.) Under low churn, reactive recovery
reactive recovery is to be more conservative when detect- is very efficient, as messages are only sent in response
ing node failure. We have found one effective approach to actual changes, whereas periodic recovery is wasteful.
to be to conclude failure only after 15 consecutive mes- As churn increases, however, reactive recovery becomes
sage timeouts to a neighbor. Since timeouts are backed more expensive, and this behavior is exacerbated by in-
off multiplicatively to a maximum of five seconds, it is un- creasing leaf set size. Not only does a node see more fail-
likely that a node will conclude failure due to congestion. ures when its leaf set is larger, but the set of other nodes
One drawback with this technique, however, is that neigh- it must notify about the resulting changes in its own leaf
bors that have actually failed remain in a node’s routing set is larger. In contrast, periodic recovery aggregates all
table for some time. Lookups that would route through changes in each period into a single message.
these neighbors are thus delayed, resulting in long lookup Figure 8 shows this contrast in Bamboo using leaf sets
latencies. To remedy this problem, a node stops routing of 24 nodes, the default leaf set size in FreePastry. In
through a neighbor after seeing five consecutive message this figure, we ran Bamboo using both configurations for
timeouts to that neighbor. We have found these changes two 20-minute churn periods of 47 and 23 minute median
make reactive recovery feasible for small leaf sets and session times separated by five minutes with no churn.
moderate churn. We note that during the periods of the test where there
is no churn, reactive recovery uses less than half of the timeouts such that a late response is indicative of node
bandwidth of periodic recovery. On the other hand, under failure, rather than network congestion or processor load.
churn its bandwidth use jumps dramatically. As discussed
above, Bamboo does not suffer from positive feedback cy- 4.2.1 Techniques
cles on account of this increased bandwidth usage. Never-
theless, the extra messages sent by reactive recovery com- We discuss and study three alternative timeout calculation
pete with lookup messages for the available bandwidth, strategies. In the first, we fix all timeouts at a conser-
and as churn increases we see a corresponding increase vative value of five seconds as a control experiment. In
in lookup latency. Although not shown in the figure, the the second, we calculate TCP-style timeouts using direct
number of hops per lookup is virtually identical between measurement of past response times. Finally, we explore
the two schemes, implying that the growth in bandwidth is using indirect measurements from a virtual coordinate al-
most likely due to contention for the available bandwidth. gorithm to calculate timeouts.
Since our goal is to handle median session times down
to a few minutes with low lookup latency, we do not ex- TCP-style timeouts: If a DHT routes recursively, it
plore reactive recovery further in this work. The remain- rarely communicates with nodes other than its direct
der of the Bamboo results we present are all obtained us- neighbors in the overlay network. Since the number of
ing periodic recovery. these neighbors is logarithmic in the size of the network,
and since each node periodically probes each neighbor
for availability, a node can easily maintain a past history
4.2 Timeout Calculation of each neighbor’s response times for use in calculating
In this section, we discuss the role that timeout calculation timeouts. In Bamboo, we have implemented this strat-
on lookup messages plays in handling churn. egy following the style of the early TCP work [15], where
To understand the relative importance of timeouts in a each node maintains an exponentially weighted mean and
DHT as opposed to a more traditional networked system, variance of the response time for each neighbor. Specifi-
consider a traditional client-server system such as the net- cally, the estimate round-trip timeout (RTO) for a neigh-
worked file system (NFS). In NFS, the server does not bor is calculated as
often fail, and when it does there are generally few op- RTO = AVG + 4 × VAR,
tions for recovery and no alternative servers to fail over
where AVG is the observed average round-trip time and
to. If a response to an NFS request is not received in the
VAR is the observed mean variance of that time.
expected time, the client will usually try again with an ex-
ponentially increasing timeout value.
In a peer-to-peer system under churn, in contrast, re- Timeouts from virtual coordinates: In contrast to re-
quests will be frequently sent to a node that has left the cursive routing, with iterative routing a node must poten-
system, possibly forever. At the same time, a DHT with tially have a good timeout for any other node in the net-
routing flexibility (static resilience) has many alternate work. However, in some scenarios iterative routing does
paths available to complete a lookup. Simply backing have attractive properties. For example, since the source
off the request period is thus a poor response to a request of a lookup request controls the entire process of itera-
timeout; it is often better to retry the request through a tive routing, it is easy to explore several different lookup
different neighbor. paths in parallel. For only a constant increase in band-
width used, this technique prevents a single timeout from
A node should ensure that the timeout for a request was
delaying a lookup ??.
judiciously selected before routing to an alternate neigh-
Virtual coordinates provide one approach to computing
bor. If it is too short, the node to which the original was
timeouts without previously measuring the response time
sent may be yet to receive it, may be still processing it, or
to every node in the system. In this scheme, a distributed
the response may be queued in the network. If so, inject-
machine learning algorithm is employed to assign to each
ing additional requests may result in the use of additional
node coordinates in a virtual metric space such that the
bandwidth without any beneficial result—for example, in
distance between two nodes in the space is proportional
the case that the local node’s access link is congested.
to their latency in the underlying network.
Conversely, if the timeout is too long, the requesting node
Bamboo includes an implementation of the Vivaldi co-
may waste time waiting for a response from a node that
ordinate system employed by Chord [11]. Vivaldi keeps
has left the network. If the request rate is fixed at too low
an exponentially-weighted average of the error of past
a value, these long waits cause unbounded queue growth
round-trip times calculated with the coordinates, and com-
on the request node that might be avoided with shorter
putes the RTO as
timeouts.
For these reasons, nodes should accurately choose RTO = v + 6 × α + 15
 thermore, they perform within a factor of two of TCP-
Fixed 5s style measurements until the median churn rate drops to
2 Vivaldi
Mean Latency (s)
TCP-style 23 minutes. Past this point, their performance quickly di-
1.5 verges, but virtual coordinates continue to provide mean
lookup latencies under two seconds down to twelve-
1 minute median session times.
0.5 Finally, we note the similarity in shape of Figure 9 to
Figure 7, where we compared the performance of Chord
0 to Bamboo, suggesting that the growth in lookup latency
2 4 8 16 32 64 128 256
under Chord at high churn rates is due to timeout calcula-
Median Session Time (min)
tion based on virtual coordinates.
Figure 9: TCP-style versus virtual coordinate-based time-
outs in Bamboo. Timeouts chosen using Vivaldi are com- 4.3 Proximity Neighbor Selection
petitive with TCP-style timeouts for moderate churn rates.
Perhaps one of the most studied aspects of DHT design
has been proximity neighbor selection (PNS), the process
where v is the predicted round-trip time and α is the aver- of choosing among the potential neighbors for any given
age error. The constant term of 15 milliseconds is added to routing table entry according to their network latency to
avoid unnecessary retransmissions when the destination is the choosing node. This research is well motivated. The
the local host. stretch of a lookup operation is defined as the latency of
the lookup divided by the round-trip time between the
4.2.2 Results lookup source and the node discovered by the lookup in
the underlying IP network. Dabek et al. present an argu-
TCP-style timeouts assume a recursive routing algorithm, ment and experimental data that suggest that PNS allows
and a virtual coordinate system is necessary only when a DHT of N nodes to achieve median stretch of only 1.5,
routing iteratively. While we would ideally compare the independent of the size of the network and despite using
two approaches by measuring each in its intended envi- O(log N ) hops [11]. Others have proved that PNS can be
ronment, this would prevent us from isolating the effect used to provide constant stretch in locating replicas un-
of timeouts from the differences caused by routing styles. der a restricted network model [21]. This is the first study
Instead, we study both schemes under recursive rout- of which we are aware, however, to compare methods of
ing. If timeouts calculated with virtual coordinates pro- achieving PNS under churn. We first take a moment to
vide performance comparable to those calculated in the discuss the common philosophy and techniques shared by
TCP-style under recursive routing, we can expect the each of the algorithms we study.
virtual coordinate scheme to not be prohibitively expen-
sive under iterative routing. While other issues may re- 4.3.1 Commonalities
main with iterative routing under churn (e.g. congestion
control—see Section 6), this result would be a useful one. One of the earliest insights in DHT design was the sepa-
Figure 9 shows a direct comparison of the three timeout ration of correctness and performance in the distinction
calculation methods under increasing levels of churn. In between neighbors in the leaf set and neighbors in the
all cases in this experiment, the Bamboo configurations routing table [24, 27]. So long as the leaf sets in the
differed only in choice of timeout calculation method. system are correct, lookups will always return correct re-
Proximity neighbor selection was used, but the latency sults, although they may take O(N ) hops to do so. Leaf
measurements for PNS used separate direct probing and set maintenance is thus given priority over routing table
not the virtual coordinates. maintenance by most DHTs. In the same manner, we note
Even under light levels of churn, fixing all timeouts that so long as each entry in the routing table has some
at five seconds causes lookup timeouts to pull the mean appropriate neighbor (i.e. one with the correct identifier
latency up to more than twice that of the other configu- prefix), lookups will always complete in O(log N ) hops,
rations, confirming our intuition about the importance of even though they make take longer than if the neighbors
good timeout values in DHT routing under churn. More- had been chosen for proximity. We say such lookups are
over, by comparing Figure 9 to Figure 7, we note that un- efficient, even though they may not have low stretch. By
der high churn timeout calculation has a greater effect on this argument, we reason that it is desirable to fill a routing
lookup latency than the use of PNS. table entry quickly, even with a less than optimal neigh-
Virtual coordinate-based timeouts achieve very similar bor; finding a nearby neighbor is a lower priority.
mean latency to TCP-style timeouts at low churn. Fur- There is a further argument to treating proximity as a
lower priority in the presence of churn. Since we expect
our set of neighbors to change over time as part of the
churn process, it makes little sense to work too hard to B
C D
find the absolute closest neighbor at any given time; we A
might expend considerable bandwidth to find them only
to see them leave the network shortly afterward. As such, Figure 10: Sampling neighbors’ neighbors. If A joins us-
our general approach is to run each of the algorithms de- ing D as its gateway, its initial level-0 neighbors are the
scribed below periodically. In the case where churn is same as those of D; assume that these are all within the
high, this technique allows us to retune the routing table dashed line. A contacts a level-0 neighbor, e.g. C, and
as the network changes. When churn is low, rerunning asks it for its level-0 neighbors. A would learn about B in
the algorithms makes up for latency measurement errors this manner. However, there may be no path from the D’s
caused by transient network conditions in previous runs. ideal neighbors to those of A.
Our general approach to finding nearby neighbors thus
takes the following form. First, we use one of the algo-
rithms below to find nodes that may be near to the local There are some cases, however, where global sampling
node. Next, we measure the latency to those nodes. If will take unreasonably long to find the closest possible
we have no existing neighbor in the routing table entry neighbor. For example, consider two nodes separated
that the measured node would fill, or if it is closer than from the core Internet by the same, high latency access
the existing routing table entry, we replace that entry, oth- link, as shown in Figure 11. The relatively high latency
erwise we leave the routing table unchanged. Although seen by these two nodes to all other nodes in the network
the bandwidth cost of multiple measurements is high, the makes them attractive neighbors for each other; if they
storage cost to remember past measurements is low. As a have different first digits in a network with logarithm base
compromise, we perform only a single latency measure- two, they can drastically reduce the cost of the first hop
ment to each discovered node during any particular run of of many routes by learning about each other. However,
an algorithm, but we keep an exponentially weighted av- the time for these nodes to find each other using global
erage of past measurements for each node, and we use sampling is proportional to the size of the total network,
this average value in deciding the relative closeness of and so they may not find each other before their sessions
nodes. This average occupies only eight bytes of mem- end. It is this drawback of global sampling that leads us
ory for each measured node, so we expect this approach to consider other techniques.
to scale comfortably to very large systems.
Neighbors’ neighbors The next technique we consider
4.3.2 Techniques is sampling our neighbors neighbors, a process called
routing table maintenance in the Pastry work [24] or lo-
The techniques for proximity neighbor selection that we cal tuning in our earlier work [23]. In this technique, we
study here are global sampling, sampling of our neigh- contact an existing routing table neighbor at level l of our
bors’ neighbors, and sampling of the nodes that have our routing table and ask for its level l neighbors. Like us,
neighbors as their neighbors. We describe each of these these nodes share a prefix of l − 1 digits with the con-
techniques in turn. tacted neighbor and are thus appropriate for use in our
routing table as well. As in global sampling, having dis-
Global sampling In global sampling (called global tun- covered these new nodes, we probe them for latency and
ing in our earlier work [23]), we use the lookup function- use them if they are closer than our existing neighbors.
ality of the DHT to find new neighbors. For a routing table The motivation for sampling neighbors’ neighbors is
entry that requires a neighbor with prefix p, we perform a illustrated in Figure 10; it relies on the expectation that
lookup for a random identifier with prefix p. The node re- proximity in the network is roughly transitive. If a node
turned by this lookup will almost always have the desired discovers one nearby node, then that node’s neighbors are
prefix. (As an example of why this is not always the case, probably also nearby. In this way, we expect that a node
note that a lookup of identifier 0 may return a node whose can “walk” through the graph of neighbor links to the set
identifier starts with 1 if the node with the largest iden- of nodes most near it.
tifier in the ring is numerically closer to 0 than the node To see one possible shortcoming of sampling our neigh-
with the smallest identifier.) Given enough samples, all bors’ neighbors, consider again Figure 11. While the two
nodes with prefix p will eventually be probed. The moti- isolated nodes would like to discover each other, it is un-
vation for this technique comes from Gummadi et al., who likely that any other nodes in the network would prefer
showed that sampling only around 16 nodes for each rout- them as neighbors; their isolation makes them unattrac-
ing table entry provides almost optimal proximity [12]. tive for routing lookups that originate elsewhere, except
 (1) function nearest neighbors () =
B 100 ms (2) S = highest nonempty rt level ();
C (3) l = longest matching prefix (S);
A (4) while l >= 0
(5) forall n in S
(6) T = n.get inverse rt neighbors (l);
Figure 11: Sampling neighbors’ inverse neighbors. Nodes
(7) S = closest (k, S ∪ T );
A and B are isolated from the remainder of the network
by a long latency, and are initially unaware of each other.
Such a situation is possible if, for example, two Euro- Figure 12: The Tapestry nearest neighbor algorithm.
pean nodes join a network of primarily North American
nodes. As such, they make unattractive neighbors for
other nodes, but they would still like to find each other. If very nearby level-0 neighbors, they will be unlikely to find
they both have C as a neighbor, they can find each other each other among their neighbors’ inverse neighbors.
by asking C for its inverse neighbors. To remedy this final problem, we can perform the sam-
pling of nodes in a manner similar to that used by the
Tapestry nearest neighbor algorithm (and the Pastry op-
in the rare case that they are the result of those lookups. timized join algorithm). Pseudo-code for this technique
As such, since neighbor links in DHTs are rarely symmet- is shown in Figure 12. Starting with the highest level l
ric, it is unlikely that there is a path through the graph of in its routing table, a node contacts the neighbors at that
neighbor links that will lead one isolated node to the other, level and retrieves their neighbors (or inverse neighbors).
despite their relative proximity. The latency to each newly discovered nodes is measured,
and all but the k closest are discarded. The node then
Neighbors’ inverse neighbors The latter argument decrements l and retrieves the level-l neighbors from each
presents an obvious alternative approach. Instead of sam- non-discarded node. This process is repeated until l < 0.
pling our neighbors’ neighbors, why not sample those Along the way, each discovered neighbor is considered as
nodes which have the same neighbors as the local node? a candidate for use in the routing table. To keep the cost of
This technique was originally proposed in the Tapestry this algorithm low, we limit it to having at most three out-
nearest neighbor algorithm [14]; we call it sampling our standing messages (neighbor requests or latency probes)
neighbors’ inverse neighbors. To motivate this technique, at any time.
consider again Figure 11. Although the two remote nodes Note that although this process starts by sampling from
are unlikely to be neighbors of many other nodes, given the routing table, the set of nodes on which it recurses is
that their existing neighbors are mostly nearby, they are not constrained by the prefix-matching structure of that ta-
likely to choose the same neighbors from outside their iso- ble. As such, it does not suffer from the small rendezvous
lated domain. For this reason, they are likely to find each set problem discussed above. In fact, under certain net-
other in the set of their neighbors’ inverse neighbors. work assumptions, it has been proved that this process
Normally, a DHT node would not record the set of finds a node’s nearest neighbor in the underlying network.
nodes that use it as a neighbor. Actively managing such a
list, in fact, requires additional probing bandwidth. Cur-
rently, the Bamboo implementation does actively manage 4.3.3 Results
this set, but it could be easily approximated at each node In order to compare the techniques described above, it is
by keeping track of the set of nodes which have sent it important to consider not only effective they are at finding
liveness probes in the last minute or so. We plan to imple- nearby neighbors, but also at what bandwidth cost they do
ment this optimization in our future work. so. For example, global sampling at a high enough rate
relative to the churn rate would achieve perfect proxim-
Recursive sampling Consider Figure 11 one final time, ity, but at the cost of a very large number of lookups and
and assume that we are using a single-bit digits and that latency probes. To make this comparison, then, we ran
the two remote nodes begin with different digits, i.e. 0 each algorithm (and some combinations of them) at var-
and 1 respectively. The node whose identifier starts with 0 ious periods, then plotted the mean lookup latency under
will have only one neighbor whose identifier begins with 1 churn versus bandwidth used. The results for median ses-
(its level-0 neighbor). Likewise, the node whose identifier sion times of 47 minutes are shown in Figure 13, which is
starts with 1 will have only one neighbor that starts with split into two graphs for clarity.
0. The set of neighbors in whose inverse neighbor sets Figure 13(a) shows several interesting results. First, we
the two isolated neighbors can find each other is thus very note that only a little bit of global sampling is necessary
small. As such, until the two isolated nodes have found to produce a drastic improvement in latency versus the
 500
No PNS 340 Global
NN Rand+NN
Mean Latency (ms)

Mean Latency (ms)

450
NIN 320 Rand+NIN
400 Global Rand+NN Recur
NN Recursive 300 Rand+NIN Recur
350 NIN Recursive 280
300 260

250 240

600 800 1000 1200 1400 600 800 1000 1200 1400
Bandwidth (bytes/s/node) Bandwidth (bytes/s/node)
(a) (b)
Figure 13: Comparison of PNS techniques. “No PNS” is the control case, where proximity is ignored. “Global
Sampling” uses the lookup function to sample all nodes in the DHT. “NN” is sampling our neighbor’s neighbors, and
“NIN” is sampling their inverse neighbors. The recursive versions of “NN” and “NIN” mimic the nearest-neighbor
algorithms of Pastry and Tapestry, respectively. Note that the scales are different between the two figures.

configuration that is not using PNS. With virtually no in- Gummadi et al. [12] present a comprehensive analysis
crease in bandwidth, global sampling drops the mean la- of the static resilience of the various DHT geometries. As
tency from 450 ms to 340 ms. we have argued earlier in this work, static resilience is an
Next, much to our surprise, we find that simple sam- important first step in a DHT’s ability to handle failures in
pling of our neighbor’s neighbors or inverse neighbors is general and churn in particular.
not terribly effective. As we argued above, this result may Liben-Nowell et al. [17] present a theoretical analysis
be in part due to the constraints of the routing table, but of structured peer-to-peer overlays from the point of view
we did not expect the effect to be so dramatic. On the of churn as a continuous process. They prove a lower
other hand, the recursive versions of both algorithms are bound on the maintenance traffic needed to keep such
at least as effective as global sampling, but not much more networks consistent under churn, and show that Chord’s
so. This result agrees with the contention of Gummadi et algorithms are within a logarithmic factor of this bound.
al. that only a small amount of global sampling is neces- This paper, in contrast, has focused more on the systems
sary to achieve near-optimal PNS. issues that arise in handling churn in a DHT. For example,
Figure 13(b) shows several combinations of the vari- we have observed what they call “false suspicions of fail-
ous algorithms. Global sampling plus sampling of neigh- ure”, the appearance that a functioning node has failed,
bors’ neighbors—the combination used in our earlier and shown how reactive failure recovery can exacerbate
work [23]—does well, offering a small decrease in la- such conditions.
tency without much additional bandwidth. However, the
Mahajan et al. [19] present a simulation-based analysis
other combinations offer similar results. At this point, it
of Pastry in which they study the probability that a DHT
seems prudent to say that the most effective technique is to
node will forward a lookup message to a failed node as
combine global sampling with any other technique. While
a function of the rate of maintenance traffic. They also
there may be other differences between the techniques not
present an algorithm for automatically tuning that rate for
revealed by this analysis, we see no clear reason to prefer
a given failure rate. Since this algorithm increases the
one over another as yet.
rate of maintenance traffic in response to losses, we are
concerned that it may cause positive feedback cycles like
5 Related Work those we have observed in reactive recovery. Moreover,
we believe their failure model is pessimistic, as they do
As we noted at the start of this paper, while DHTs have not consider hop-by-hop retransmissions of lookup mes-
been the subject of much research in the last 4 years or sages. By acknowledging lookup messages on each hop,
so, there have been few studies of the resilience of real a DHT can route around failed nodes in the middle of a
implementations at scale, perhaps because of the difficulty lookup path, and in this work we have shown that good
of deploying, instrumenting, and creating workloads for timeout values can be computed to minimize the cost of
such deployments. However, there has been a substantial such retransmissions.
amount of theoretical and simulation-based work. Castro et al. [7] presented a number of optimizations
they have performed in MSPastry, the Microsoft Research is more Gaussian than the distribution of latencies mea-
implementation of Pastry, using simulations. Also, Li et sured on the Internet. Unfortunately for our purposes,
al. [16] performed a detailed simulation-based analysis of these measured latency distributions do not include topol-
several different DHTs under churn, varying their parame- ogy information, and thus cannot be used to simulate the
ters to explore the latency-bandwidth tradeoffs presented. kind of network cross traffic that we have found important
It was their work that inspired our analysis of different in this study. The existence of better topologies would be
PNS techniques. most welcome.
As opposed to the emulated network used in this study, In addition to more realistic network models, we would
simulations do not usually consider such network issues also like to include more realistic models of churn in our
as queuing, packet loss, etc. By not doing so, they either future work. One idea that was suggested to us by an
simulate far larger networks than we have studied here as anonymous reviewer was to scale traces of session times
in [7, 19], or they are able to explore a far larger space collected from deployed networks to produce a range of
of possible DHT configurations as in [16]. On the other churn rates with a more realistic distribution. We would
hand, they do not reveal subtle issues in DHT design, like to explore this approach. Nevertheless, we believe
such as the tradeoffs between reactive and periodic recov- that the effects of the factors we have studied are dramatic
ery. Also, they do not reveal the interactions of lookup enough that they will remain important even as our mod-
traffic and maintenance traffic in competing for network els improve.
bandwidth. We are interested in whether a useful middle Finally, in this work we have only shown the resistance
ground exists between these approaches. of the Bamboo routing layer to churn, an important first
Finally, a number of useful features for handling churn step verifying that DHTs are ready to become the domi-
have been proposed, but are not implemented by Bamboo. nant building block for peer-to-peer systems, but a limited
For example, Kademlia [20] maintains several neighbors one. Clearly other issues remain. Security and possibly
for each routing table entry, ordered by the length of time anonymity are two such issues, but we are unclear about
they have been neighbors. Newer nodes replace existing how they relate to churn. We are currently studying the re-
neighbors only after failure of the latter. This design deci- silience to churn of the algorithms used by the DHT stor-
sion is aimed at mitigating the effects of the high “infant age layer. We hope that the existence of a routing layer
mortality” observed in peer-to-peer networks. that is robust under churn will provide a useful substrate
Another approach to handling churn is to introduce a on which these remaining issues may be studied.
hierarchy into the system, through stable “superpeers” [2,
29]. While an explicit hierarchy is a viable strategy for
handling load in some cases, this work has shown that 7 Conclusion
a fully decentralized, non-hierarchical DHT can in fact
handle high rates of churn at the routing layer. In this work we have summarized the rates of churn ob-
served in deployed peer-to-peer systems and shown that
existing DHTs exhibit less than desirable performance at
6 Future Work the higher end of these churn rates. We have presented
Bamboo and explored various design tradeoffs and their
As discussed in the introduction, there are several other effects on its ability to handle churn.
limitations of this study that we think provide for impor- The design tradeoffs we studied in this work fall into
tant future work. At an algorithmic level, we would like three broad categories: reactive versus periodic recov-
to study the effects of alternate routing table neighbors as ery from neighbor failure, the calculation of timeouts on
in Kademlia and Tapestry. We would also like to con- lookup messages, and proximity neighbor selection. We
tinue our study of iterative versus recursive routing. As have presented the danger of positive feedback cycles in
discussed by others [11], congestion control for iterative reactive recovery and discussed two ways to break such
lookups is a challenging problem. We have implemented cycles. First, we can make the DHT much more cautious
Chord’s STP congestion control algorithm and are cur- about declaring neighbors failed, in order to limit the pos-
rently investigating its behavior under churn, but we do sibility that we will be tricked into recovering a non-faulty
not yet have definitive results about its performance. node by network congestion. Second, we presented the
At a methodological level, we would like to broaden technique of periodic recovery. Finally, we demonstrated
our study to include better models of network topology that reactive recovery is less efficient than periodic recov-
and churn. We have so far used only a single network ery under reasonable churn rates when leaf sets are large,
topology in our work, and so our results should be not as they would be in a large system.
be taken as the last word on PNS. In particular, the dis- With respect to timeout calculation, we have shown that
tribution of internode latencies in our ModelNet topology TCP-style timeout calculation performs best, but argued
that it is only appropriate for lookups performed recur- [9] J. Chu, K. Labonte, and B. N. Levine. Availability and locality
sively. It has long been known that recursive routing pro- measurements of peer-to-peer file systems. In Proc. of ITCom:
Scalability and Traffic Control in IP Networks, July 2002.
vides lower latency lookups than iterative, but this result
[10] F. Dabek, M. F. Kaashoek, D. Karger, R. Morris, and I. Stoica.
presents a further argument for recursive routing where
Wide-area cooperative storage with CFS. In Proc. ACM SOSP,
the lowest latency is important. However, we have also Oct. 2001.
shown that while they are not as effective as TCP-style [11] F. Dabek, J. Li, E. Sit, J. Robertson, M. F. Kaashoek, and R. Mor-
timeouts, timeouts based on virtual coordinates are quite ris. Designing a DHT for low latency and high throughput. In
reasonable under moderate rates of churn. This result in- Proc. NSDI, 2004.
dicates that at least with respect to timeouts, iterative rout- [12] K. Gummadi, R. Gummadi, S. Gribble, S. Ratnasamy, S. Shenker,
ing should not be infeasible under moderate churn. and I. Stoica. The impact of DHT routing geometry on resilience
and proximity. In Proc. ACM SIGCOMM, Aug. 2003.
Concerning proximity neighbor selection, we have
[13] K. P. Gummadi, R. J. Dunn, S. Saroiu, S. D. Gribble, H. M. Levy,
shown that global sampling can provide a 24% reduc- and J. Zahorjan. Measurement, modeling, and analysis of a peer-
tion in latency for virtually no increase in bandwidth used. to-peer file-sharing workload. In Proc. ACM SOSP, Oct. 2003.
By using an additional 40% more bandwidth, a 42% de- [14] K. Hildrum, J. D. Kubiatowicz, S. Rao, and B. Y. Zhao. Distributed
crease in latency can be achieved. Other techniques are object location in a dynamic network. In Proc. SPAA, 2002.
also effective, especially our adaptations of the Pastry and [15] V. Jacobson and M. J. Karels. Congestion avoidance and control.
Tapestry nearest-neighbor algorithms, but not much more In Proc. ACM SIGCOMM, 1988.
so than simple global sampling. Merely sampling our [16] J. Li, J. Stribling, T. M. Gil, R. Morris, and F. Kaashoek. Com-
paring the performance of distributed hash tables under churn. In
neighbors’ neighbors or inverse neighbors is not very ef-
Proc. IPTPS, 2004.
fective in comparison. Some combination of global sam-
[17] D. Liben-Nowell, H. Balakrishnan, and D. Karger. Analysis of
pling an any of the other techniques seems to provide the the evolution of peer-to-peer systems. In Proc. ACM PODC, July
best performance at the least cost. 2002.
[18] B. T. Loo, R. Huebsch, I. Stoica, and J. Hellerstein. The case for a
hybrid P2P search infrastructure. In Proc. IPTPS, 2004.
8 Acknowledgments [19] R. Mahajan, M. Castro, and A. Rowstron. Controlling the cost of
reliability in peer-to-peer overlays. In Proc. IPTPS, Feb. 2003.
We would like to thank a number of people for their help [20] P. Maymounkov and D. Mazieres. Kademlia: A peer-to-peer in-
with this work. Our shepherd, Atul Adya, and the anony- formation system based on the XOR metric. In Proc. IPTPS, 2002.
mous reviewers all provided valuable comments and guid- [21] C. Plaxton, R. Rajaraman, and A. Richa. Accessing nearby copies
ance. Frank Dabek helped us tune our Vivaldi implemen- of replicated objects in a distributed environment. In Proc. of ACM
SPAA, June 1997.
tation, and he and Emil Sit helped us get Chord up and
[22] S. Ratnasamy, P. Francis, M. Handley, R. Karp, and S. Shenker. A
running. Likewise, Peter Druschel provided valuable de- scalable content-addressable network. In Proc. ACM SIGCOMM,
bugging insight for FreePastry. David Becker helped us Aug. 2001.
with ModelNet. Sylvia Ratnasamy, Scott Shenker, and [23] S. Rhea, D. Geels, T. Roscoe, and J. Kubiatowicz. Handling churn
Ion Stoica provided valuable guidance at several stages of in a DHT. Technical Report UCB//CSD-03-1299, University of
this paper’s development. California, Berkeley, December 2003.
[24] A. Rowstron and P. Druschel. Pastry: Scalable, distributed object
location and routing for large scale peer-to-peer systems. In Proc.
References of IFIP/ACM Middleware, Nov. 2001.
[25] S. Saroiu, P. K. Gummadi, and S. D. Gribble. A measurement
[1] Freepastry 1.3. study of peer-to-peer file sharing systems. In Proc. MMCN, Jan.
http://www.cs.rice.edu/CS/Systems/Pastry/. 2002.
[2] Gnutella. http://www.gnutella.com/. [26] S. Sen and J. Wang. Analyzing peer-to-peer traffic across large net-
[3] Inet topology generator. works. In Proc. of ACM SIGCOMM Internet Measurement Work-
http://topology.eecs.umich.edu/inet/. shop, Nov. 2002.
[4] MIT Chord. http://www.pdos.lcs.mit.edu/chord/. [27] I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakr-
ishnan. Chord: A scalable peer-to-peer lookup service for Internet
[5] R. Bhagwan, S. Savage, and G. Voelker. Understanding availabil-
applications. In Proc. ACM SIGCOMM, Aug. 2001.
ity. In Proc. IPTPS, Feb. 2003.
[28] A. Vahdat, K. Yocum, K. Walsh, P. Mahadevan, D. Kostic,
[6] C. Blake and R. Rodrigues. High availability, scalable storage,
J. Chase, and D. Becker. Scalability and accuracy in a large-scale
dynamic peer networks: Pick two. 2003.
network emulator. In Proc. OSDI, Dec. 2002.
[7] M. Castro, M. Costa, and A. Rowstron. Performance and depend-
[29] B. Y. Zhao, Y. Duan, L. Huang, A. D. Joseph, and J. D. Kubiatow-
ability of structured peer-to-peer overlays. Technical Report MSR-
icz. Brocade: Landmark routing on overlay networks. In Proc.
TR-2003-94, Microsoft, 2003.
IPTPS, March 2002.
[8] M. Castro, M. B. Jones, A.-M. Kermarrec, A. Rowstron, [30] B. Y. Zhao, L. Huang, J. Stribling, S. C. Rhea, A. D. Joseph, and
M. Theimer, H. Wang, and A. Wolman. An evaluation of scal- J. D. Kubiatowicz. Tapestry: A resilient global-scale overlay for
able application-level multicast built using peer-to-peer overlays. service deployment. IEEE JSAC, 22(1):41–53, Jan. 2004.
Apr. 2003.