Prefix Hash Tree

An Indexing Data Structure over Distributed Hash Tables

Sriram Ramabhadran ∗ Sylvia Ratnasamy

University of California, San Diego Intel Research, Berkeley

Joseph M. Hellerstein Scott Shenker

University of California, Berkeley International Comp. Science Institute, Berkeley
and and
Intel Research, Berkeley University of California, Berkeley

ABSTRACT this lookup interface has allowed a wide variety of

Distributed Hash Tables are scalable, robust, and system to be built on top DHTs, including file sys-
self-organizing peer-to-peer systems that support tems [9, 27], indirection services [30], event notifi-
exact match lookups. This paper describes the de- cation [6], content distribution networks [10] and
sign and implementation of a Prefix Hash Tree - many others.
a distributed data structure that enables more so-
phisticated queries over a DHT. The Prefix Hash DHTs were designed in the Internet style: scala-
Tree uses the lookup interface of a DHT to con- bility and ease of deployment triumph over strict
struct a trie-based structure that is both efficient semantics. In particular, DHTs are self-organizing,
(updates are doubly logarithmic in the size of the requiring no centralized authority or manual con-
domain being indexed), and resilient (the failure figuration. They are robust against node failures
of any given node in the Prefix Hash Tree does and easily accommodate new nodes. Most impor-
not affect the availability of data stored at other tantly, they are scalable in the sense that both la-
nodes). tency (in terms of the number of hops per lookup)
and the local state required typically grow loga-
Categories and Subject Descriptors rithmically in the number of nodes; this is crucial
since many of the envisioned scenarios for DHTs
C.2.4 [Comp. Communication Networks]: Dis- involve extremely large systems (such as P2P mu-
tributed Systems—distributed applications; E.1 [Data sic file sharing). However, DHTs, like the Internet,
Structures]: distributed data structures; H.3.1 [Info- deliver ”best-effort” semantics; put’s and get’s are
rmation Storage and Retrieval]: Content Anal- likely to succeed, but the system provides no guar-
ysis and Indexing—indexing methods antees. As observed by others [36, 5], this conflict
between scalability and strict semantics appears
General Terms to be inevitable and, for many large-scale Inter-
Algorithms, Design, Performance net systems, the former is deemed more important
than the latter.
Keywords While DHTs have enjoyed some success as a build-
distributed hash tables, data structures, range queries ing block for Internet-scale applications, they are
seriously deficient in one regard: they only directly
1. INTRODUCTION support exact match queries. Keyword queries can
The explosive growth but primitive design of peer- be derived from these exact match queries in a
to-peer file-sharing applications such as Gnutella straightforward but inefficient manner; see [25, 20]
[7] and KaZaa [29] inspired the research community for applications of this to DHTs. Equality joins
to invent Distributed Hash Tables (DHTs) [31, 24, can also be supported within a DHT framework;
see [15]. However, range queries, asking for all ob-
14, 26, 22, 23]. Using a structured overlay network, jects with values in a certain range, are particularly
DHTs map a given key to the node in the network difficult to implement in DHTs. This is because
holding the object associated with that key; this DHTs use hashing to distribute keys uniformly and
lookup operation lookup(key) can be used to sup- so can’t rely on any structural properties of the key
port the canonical put(key,value) and get(key) space, such as an ordering among keys.
hash table operations. The broad applicability of
∗
email [email protected] Range queries arise quite naturally in a number of
 Leaf nodes Keys
000* 000001
0 000100
0 1
000100
00100* 001001
1 001010* 001010
0 1 0 1 001010
001010
2 001011* 001011
0 1 0 1 001011
0011*
01* 010000
3 010101
0 1 10* 100010
101011
4 101111
0 1 110* 110000
110010
5 110011
0 1 110110
111* 111000
111010
6

Figure 1: Prefix Hash Tree

potential application domains: against data loss when nodes go down 1 , the failure
of any given node in the Prefix Hash Tree does not
Databases Peer-to-peer databases [15] need to sup- affect the availability of data stored at other nodes.
port SQL-type relational queries in a distributed
fashion. Range predicates are a key component in But perhaps the most crucial property of PHT is
SQL. that it is built entirely on top of the lookup inter-
face, and thus can run over any DHT. That is, PHT
Distributed computing Resource discovery requires uses only the lookup(key) operation common to
locating resources within certain size ranges in a all DHTs and does not, as in SkipGraph [1] and
decentralized manner. other such approaches, assume knowledge of nor
require changes to the DHT topology or routing
Location-aware computing Many applications want behavior. While designs that rely on such lower-
to locate nearby resources (computing, human or layer knowledge and modifications are appropriate
commercial) based on a user’s current location, for contexts where the DHT is expressly deployed
which is essentially a 2-dimensional range query for the purpose of supporting range queries, we ad-
based on geographic coordinates. dress the case where one must use a pre-existing
DHT. This is particularly important if one wants
Scientific computing Parallel N-body computations to make use of publicly available DHT services,
[34] require 3-dimensional range queries for accu- such as OpenHash [18].
rate approximations.
The remainder of the paper is organized as fol-
In this paper, we address the problem of efficiently lows. Section 2 describes the design of the PHT
supporting 1-dimensional range queries over a DHT. data structure. Section 3 presents the results of an
Our main contribution is a novel trie-based dis- experimental evaluation. Section 4 surveys related
tributed data structure called Prefix Hash Tree (hence- work and section 5 concludes.
forth abbreviated as PHT) that supports such queries.
As a corollary, the PHT can also support heap 2. DATA STRUCTURE
queries (“what is the maximum/minimum ?”), prox-
This section describes the PHT data structure, along
imity queries (“what is the nearest element to X with related algorithms.
?”), and, in a limited way, multi-dimensional ana-
logues of the above, thereby greatly expanding the
querying facilities of DHTs. PHT is efficient, in 2.1 PHT Description
that updates are doubly logarithmic in the size For the sake of simplicity, it is assumed that the do-
of the domain being indexed. Moreover, PHT is main being indexed is {0, 1}D , i.e., binary strings
self-organizing and load-balanced. PHT also toler-
ates failures well; while it cannot by itself protect 1
But PHT can take advantage of any replication or other
data-preserving technique employed by a DHT.
of length D, although the discussion extends nat- in regions of the domain which are sparsely popu-
urally to other domains. Therefore, the data set lated. Finally, property 5 ensures that the leaves
indexed by the PHT consists of some number N of of the PHT form a doubly linked list, which en-
D-bit binary keys. ables sequential traversal of the leaves for answer-
ing range queries.
In essence, the PHT data structure is a binary trie
built over the data set. Each node of the trie is As described this far, the PHT structure is a fairly
labeled with a prefix that is defined recursively: routine binary trie. The novelty of PHT lies in
given a node with label l, its left and right child how this logical trie is distributed among the peers
nodes are labeled l0 and l1 respectively. The root in the network; i.e., in how PHT vertices are as-
is labeled with the attribute being indexed, and signed to DHT nodes. This is achieved by hashing
downstream nodes are labeled as above. the prefix labels of PHT nodes over the DHT iden-
tifier space. A node with label l is thus assigned
The following properties are invariant in a PHT. 4 to the peer to which l is mapped by the DHT,
i.e., the peer whose identifier is closest to HASH(l).
1. (Universal prefix ) Each node has either 0 or 2 This hash-based assignment implies that given a la-
children. bel, it is possible to locate its corresponding PHT
node via a single DHT lookup. This “direct access”
2. (Key storage) A key K is stored at a leaf node property is unlike the successive link traversals as-
whose label is a prefix of K. sociated with typical data structures and results in
the PHT having several desirable features that are
3. (Split) Each leaf node stores atmost B keys. discussed subsequently.
4. (Merge) Each internal node contains atleast 2.2 PHT Operations
(B + 1) keys in its sub-tree. This section describes algorithms for PHT opera-
tions.
5. (Threaded leaves) Each leaf node maintains a
pointer to the leaf nodes on its immediate left and
and immediate right respectively. 2 2.2.1 Lookup
Given a key K, a PHT lookup operation returns
Property 1 guarantees that the leaf nodes of the the unique leaf node leaf (K) whose label is a prefix
PHT form a universal prefix set 3 . Consequently, of K. Because there are (D + 1) distinct prefixes
given any key K ∈ {0, 1}D , there is exactly one leaf of K, there are (D + 1) potential candidates; an
obvious algorithm is to perform a linear scan of
node leaf (K) whose label is a prefix of K. Prop- these (D + 1) nodes until the required leaf node is
erty 2 states that the key K is stored at leaf (K). reached. This is similar to a top-down traversal of
Figure 1 provides an example of a PHT contain- the trie except that a DHT lookup is used to locate
ing N = 20 6-bit keys with B = 4. The table on a PHT node given its prefix label. Pseudocode for
the right in Figure 1 lists the 20 keys and the leaf this algorithm is given below.
nodes they are stored in.
Properties 3 and 4 govern how the PHT adapts Algorithm: PHT-LOOKUP-LINEAR
to the distribution of keys in the data set. Fol- input : A key K
lowing the insertion of a new key, the number of
keys stored at a leaf node may exceed the threshold output: leaf (K)
B, causing property 3 to be violated. To restore for i ← 0 to D do
the invariant, the node splits into two child nodes, /*Pi (K) denotes prefix of K of length
and its keys are redistributed among the children
according to property 2. Conversely, following the i */
deletion of an existing key, the number of keys con- node ← DHT-LOOKUP(Pi (K));
tained in a sub-tree may fall below (B +1), causing if (node is a leaf node) then return node ;
property 4 to be violated. To restore the invari- end
ant, the entire sub-tree is merged into a single leaf return f ailure;
node, where all the keys are aggregated. Notice the
shape of the PHT depends on the distribution of
keys; it is ”deep” in regions of the domain which How can this be improved ? Given a key K, the
are densely populated, and conversely, ”shallow” above algorithm tries different prefix lengths until
2
the required leaf node is reached. Clearly, linear
A pointer here would be the prefixes of neighboring leaves search can be replaced by binary search on prefix
and, as a performance optimization, the cached IP address
of their corresponding DHT nodes. 4
Assignment implies that the peer maintains the state as-
3
A set of prefixes is a universal prefix set if and only if for sociated with the PHT node assigned to it. Henceforth, the
every infinite binary sequence b, there is exactly one element discussion will use PHT node to also refer to the peer as-
in the set that is a prefix of b. signed that node.
lengths. If the current prefix is an internal node
of the PHT, the search tries longer prefixes. Al- 0 1
ternatively, if the current prefix is not an internal
node of the PHT, the search tries shorter prefixes.
The search terminates when the required leaf node 0 1 0 1
is reached. The decision tree to the left in Fig-
ure 1 illustrates the binary search. For example.
consider a lookup for the key 001100. The binary
0 1 0 1
search algorithm first tries the 3-bit prefix 001*
(internal node), then the 5-bit prefix 00110* (not
an internal node), and then finally the 4-bit prefix
0 1
0011*, which is the required leaf node. Pseudocode
for this algorithm is given below.
Parallel
0 1
Algorithm: PHT-LOOKUP-BINARY
input : A key K
output: leaf (K) 0 1
Sequential
lo ← 0;
hi ← D;
while (lo ≤ hi) do
mid ← (lo + hi)/2;
Figure 2: Range queries
/*Pmid (K) denotes prefix of K of
length mid */
node ← DHT-LOOKUP(Pmid (K)); L ≤ K ≤ H. Range queries can be implemented
if (node is a leaf node) then return node ; in a PHT in several ways; we present two simple
else algorithms.
if (node is an internal node) then lo ←
mid + 1; The first algorithm is to locate leaf (L) using the
else hi ← mid- 1; PHT lookup operation. Now the doubly linked list
end of threaded leaves is traversed sequentially until
end the node leaf (H) is reached. All values satisying
the range query are retrieved. This algorithm is
return f ailure; simple and efficient; it initially requires log D DHT
lookups to locate leaf (L). It cannot avoid travers-
Binary search reduces the number of DHT lookups ing the remaining nodes to answer the query. The
from (D + 1) to blog (D + 1)c + 1 ≈ log D. Never- disadvantage of this algorithm is that a sequential
theless, linear search is still significant for atleast scan of the leaf nodes may result in a high latency
two reasons. First, observe that the (D + 1) DHT before the query is completely resolved.
lookups in linear search can be performed in paral-
lel, as opposed to binary search, which is inherently The second algorithm is to parallelize. Using the
sequential. This results in two modes of operation DHT, locate the node whose label corresponds to
viz. low-overhead lookups using binary search, and the smallest prefix range that completely covers the
low-latency lookups using parallel search. Second, specified range. If this is an internal node, then re-
binary search may fail ,i.e., be unable to correctly cursively forward the query onward to those chil-
locate the leaf node, as a result of the failure of dren which overlap with the specified range. This
an internal PHT node 5 . On the other hand, lin- process continuues until the leaf nodes overlapping
ear search is guaranteed to succeed as long as the with the query are reached. If this is not an inter-
leaf node is alive, and the DHT is able to route nal node, the required range query is covered by
to it, and therefore provides a failover mechanism. a single leaf node, which can be located by binary
Note that both algorithms are contingent on the search.
fact that the DHT provides a mechanism to locate Figure 2 shows an example of range search. Con-
any PHT node via a single lookup. sider a query for the range [001001, 001011]. In the
sequential algorithm, a PHT lookup is used to lo-
2.2.2 Range Query cate the node containing the lower endpoint, i.e.,
Given two keys L and H (L ≤ H), a range query node 00100∗. After this a traversal of the linked list
returns all keys K contained in the PHT satisfying forwards the query to the next two leaves 001010∗
5
Binary search will not be able to distinguish between the and 001011∗, which resolves the query. In the par-
failure of an internal node and the absence of an internal allel algorithm, we first identify the smallest prefix
node. range that completely covers the query, which is
0010∗. A single DHT lookup is used to directly DHT being used to distribute B-tree nodes across
jump to this node, after which the query is for- peers in the network. While the tree-based indices
warded in parallel within the sub-tree, until all may be better in traditional indexing applications
leaf nodes that overlap with the search range are like databases, we argue the reverse is true for im-
reached. plementation over a DHT.
Note that in the parallel algorithm, it is sometimes The primary difference between the two approaches
desirable to break the search query into two, and is as follows: a trie partitions the space while a tree
treat these sub-queries independently. For exam- partitions the data set. In other words, a trie node
ple, a very small range that contains the midpoint represents a particular region of space, while a tree
of the space, will result in ∗ being the smallest node represents a particular set of keys. Because
prefix range containing it, thereby potentially over- a trie uses space, which is constant independent of
loading the root. To prevent this, we observe that the actual data set, there is some implicit knowl-
every range is contained in the union of two pre- edge about the location of a key. For example, in
fix ranges that are of roughly the same size as the a trie, a key is always stored at a prefix of the key,
query (within a factor of 2). By handling these which makes it possible to exploit the mechanism
separately, it is possible to ensure a search starts the DHT provides to locate a node via a single
at a level in the PHT that is appropriate for the DHT lookup. In a tree, this knowledge is lacking,
query i.e. smaller queries start lower down in the and it not possible to locate a key without a top-
PHT. down traversal from the root. Therefore, a tree
index cannot use the random access property of
2.2.3 Insert / Delete the DHT in the same manner. This translates into
Insertion and deletion of a key K both require a several key advantages in favor of the PHT when
PHT lookup operation to first locate the leaf node compared to a balanced tree index.
leaf (K). Insertion of a new key can cause this
leaf node to split into two children, followed by a
redistribution of keys. In most cases, the (B + 1) 2.3.1 Efficiency
keys are distributed among the two children such A balanced tree has a height of log N , and therefore
that each of them stores atmost B. However it is a key lookup requires log N DHT lookups. In addi-
possible that all (B + 1) keys are distributed to tion, updates may require the tree to re-balanced.
the same child, necessitating a further split. In the The binary search lookup algorithm in the case of
worst case, an insertion can cause splits to cascade the PHT requires only log D DHT operations, and
all the way to a depth D 6 , making insertion costs updates have the same cost as well. Comparing the
proportional to D. Similarly, in the worst case, cost of lookups in the case of an index consisting
deletion can cause an entire sub-tree of depth D of a million 32-bit keys, a tree index would require
to collapse into a single leaf node, incurring a cost 20 DHT lookups as compared to 6 for the PHT to
proportional to D. retrieve a key. Of course, multiway indexing could
be used to reduce the height of the tree, but this
It is possible to reduce update costs and avoid would also leave the tree more vulnerable to faults
problems of multi-node coordination through stag- in the indexing structure.
gered updates. Only one split operation is allowed
per insertion, and similarly, only one merge oper-
ation is allowed per deletion. While this results in 2.3.2 Load Balancing
update costs reducing to log D DHT lookups (the As mentioned before, every lookup in a tree must
cost of a PHT lookup to locate the leaf node), it goes through the root, creating a potential bottle-
also allows invariants 3 and 4 to be violated. A leaf neck. In the case of a trie, binary search allows the
D
node can now store upto (B + D) keys. This is not load to be spread over 2 2 nodes (assuming uniform
likely to be a problem because in most practical lookups), thus eliminating any bottleneck.
scenarios, B >> D.

2.3 Tries versus Trees 2.3.3 Fault Resilience

This section compares the merits of a trie-based in- In a typical tree-based structure, the loss of an in-
dex, such as the PHT, with balanced tree-based in- ternal node results in the loss of the entire sub-
dices, such as the B-tree, with particular emphasis tree rooted at the failed node. PHT however does
on implementation in a distributed setting. This not require top-down traversals; instead one can
paper has described how the PHT data structure directly “jump” to any node in the PHT. Thus the
can be built over a DHT; it is likewise conceivable failure of any given node in the PHT does not af-
that a B-tree could be built over a DHT, with the fect the availability of data stored at other nodes.
6
This process must terminate because in the worst case, In some sense, the indexing state in the trie is used
all keys are identical, and it is assumed that identical keys only as an optimization. For example, observe that
are distinguished by padding random bits at the end, and correct operation of the PHT is achievable using
appropriately increasing D. only the integrity of the doubly-linked list of leaf
nodes7 . Both updates (through linear search) and 2.4.2 Indexing Known Distributions
range queries (through sequential traversal of the Relative to tree-based indexes, a disadvantage of
list) can be handled without the help of the trie PHTs is that their complexity is expressed in terms
indexing structure. Contrast with a tree where the of the log of the domain size, D, rather than the
indexing structure is indispensible for both updates size of the data set, N . In many scenarios, however,
and queries, and is therefore vulnerable to failures. data is from a known distribution: for example,
keywords in text search follow well-known Zipfian
distributions, and range search queries (e.g. text-
2.4 PHT Enhancements prefix queries like “Cali*”) are quite natural. Here
Until this point, we have discussed the use of PHTs we informally argue that for known distributions,
for satisfying unidimensional range queries. In this the PHT can be modified to run in O(log log N )
section, we describe two refinements: functionality expected hops.
extensions to support multi-dimensional searching,
and performance enhancements for scenarios with We begin by examining the simple uniform distri-
known or relatively static data distributions. bution over D-bit keys. For N data items drawn
from this distribution, the expected depth of a leaf
in the PHT is O(log N ), with low variance. Hence
2.4.1 Multi-dimensional Indexing via Linearization for uniform distributions, the expected number of
There are a plethora of centralized indexing schemes hops with PHT-LOOKUP-BINARY is O(log log N ).
for supporting multidimensional range and near- This search algorithm can be improved further via
neighbor queries; multiple surveys have been pub- a search algorithm that starts at prefix-length log N ,
lished in this area (e.g., [11, 28]). One class of and proceeds upward or downward as necessary.
heuristic multidimensional indexing schemes maps
multidimensional data to a single dimension. This For known but non-uniform distributions, similar
approach is sometimes called linearization, or space- performance can be achieved by “warping” the space
filling curves, and well-known examples include the appropriately, remapping the original distribution
Hilbert, Gray code, and “Z-order” curves [16]. A to a uniform distribution. For each data point
multidimensional query is mapped to a unidimen- drawn from the original distribution, its first bit
sional range query that spans from the lowest to is remapped to be 0 if it is lower than the mid-
highest linearization points of the original query. point of the distribution’s PDF, and 1 otherwise;
In general, linearized queries return a superset of this assignment proceeds recursively through all
the matching data, which has to be post-filtered. D bits. The resulting set of mapped points are
Recent linearization schemes like the Pyramid [3] essentially drawn from the uniform distribution,
and iDistance [35] techniques have been shown em- and a PHT built on these points will have path
pirically to outperform traditional space-filling curves lengths as described above. Queries in the original
as well as popular multidimensional tree structures space are mapped accordingly, and will perform
in high-dimensional scenarios. with O(log log N ) expected hops.
Though work on multidimensional indexing via lin- For globally well-known distributions (e.g. terms
earization schemes is largely heuristic, it has a strong
practical attraction: linearization can be imple- in spoken English), the warping function is more
mented as an overlay upon existing unidimensional or less fixed and can be distributed as part of the
range search structures, which are typically more PHT code. However, in practice many data sets
frequently implemented and carefully debugged than come from distributions that do change, but quite
specialized multidimensional indexes. This argu- slowly. For example, the distribution of terms in
ment holds for PHTs as well as for any distributed filesharing applications will shift slightly as pop-
range search technique. PHTs have the added ad- ularity shifts; some terms may suddenly become
vantage that their underlying substrate, DHTs, are popular when a new popular file is released, but
rapidly emerging as a leading distributed building most terms’ frequency will remain relatively static
block. for long periods of time. For such slowly-changing
distributions, a gossip-based scheme can be used
After mapping a d-dimensional query Q d into a to disseminate compressed representations of the
distribution to all nodes, and any changes to the
unidimensional query Q0 , a PHT finds the answer distribution can be reflected via periodic remap-
set in O(log D + d|Q0 |/Be) network hops, where ping of the data and queries (perhaps as part of
|Q0 | is the size of the result set returned for the soft-state refresh.)
unidimensional query Q0 . Note that we have given
no bound on the difference between |Q 0 | and |Qd |, We close this section by observing a general dual-
and in the worst case |Q0 | = n. This difference ity. A tree index is actually quite analogous to the
captures the inefficacy of the chosen linearization pairing of a trie-based scheme with a known dis-
scheme; it is not particular to PHTs per se. tribution: the “split-keys” in a tree index capture
the data distribution, and the pointers in the tree
7
This is somewhat similar to Chord whose correct operation index serve the same function as the bits in the trie
depends only on the integrity of the successor pointers
encoding. An advantage of the PHT approach is 3.2 Load balance
the ability we noted above to “jump” into any point The next experiment attempts to verify the asser-
of the trie via hashing; global knowledge of the dis- tion that the PHT spreads network load evenly,
tribution provides this uniformity in addressing. A and therefore does not have a bottleneck, unlike a
similar trick could be achieved in tree indexes as binary tree. To test this hypothesis, we generated
well, if every searcher had a fair approximation of 100,000 PHT lookups on uniformly distributed keys
the split keys in the tree and the locations of the and observed the distribution of lookup traffic. By
tree nodes in an identifier space. lookup traffic, we mean the DHT queries generated
by the binary search algorithm, and not the under-
3. EVALUATION lying DHT routing traffic. Figure 4 shows the dis-
This section presents a simulation-based evaluation tribution of lookup traffic over all the nodes in the
of the PHT data structure. Although these simu- DHT. It can be seen that about 80 % of the nodes
lation results are by no means comprehensive, we see less than 400 lookups (out of 100,000). The rest
present them as preliminary experimental evidence of the nodes, which correspond to PHT leaf nodes,
that the PHT is a viable solution. A more complete receive more traffic, but in no case higher than
evaluation, along with gaining experience with de- 1800. Contrast this with a B-tree where each of
ployment of a working prototype, is the focus of the 100,000 messages must necessarily go through
our current efforts. the root. To test the effect of network size, the
experiment was repeated for 1000, 2000 and 3000
Our simulation setup is as follows. A PHT that nodes respectively. As expected, a larger number
indexes 30-bit keys is created on top of a DHT of nodes reduces the amount of per-node traffic, as
consisting of 1000 nodes. Our focus is on evalu- PHT prefixes are distributed among more nodes.
ating the performance of the data structure; for However the actual PHT leaf nodes continue to re-
that reason we abstract away many of the details ceive higher amounts of traffic than the rest of the
of the DHT by using a stripped-down version of nodes.
the Chord protocol. 216 artificially generated keys Range search efficiency
are inserted into the PHT coming from a uniform 1.6
Uniform distribution
distribution over the entire 230 keyspace. We use Gaussian distribution

an artificially low block size of B = 20 in order to 1.55

generate a non-trivial instance of the PHT.

1.5

3.1 Range queries

Sub-optimality factor

Recall that a range query is evaluated by travers- 1.45

ing the leaf nodes of the PHT. The complexity (and

1.4
latency) of the range query operation depends on
the number of such leaves, which is a function of 1.35
the output, i.e., how many keys actually satisfy the
given query. In the ideal case, if the output size is 1.3
O and the block size is B, the number of nodes
traversed should be about d O B e. To see how well 1.25
the PHT distributes keys among the leaf nodes, we 0 0.1 0.2 0.3 0.4 0.5
Fraction of ranges
0.6 0.7 0.8 0.9 1

generate 1000 randomly generated queries of size

varying from 222 to 226 , measured how many leaf Figure 3: Efficiency of range queries
nodes were required to be traversed. The results
O
normalized to the optimal number d B e are shown 4. RELATED WORK
in figure 3. The number of leaf nodes required to Building efficient data structures for searching is
be traversed is roughly the same in all cases: about one of the fundamental problems in computer sci-
1.4 times the optimal value. To evaluate the effect ence; [19, 8] are good references. Our PHT pro-
of skewed distributions on the PHT structure, this posal is particularly reminiscent of Litwin’s Trie
experiment was repeated with a Gaussian distri- Hashing [21], but has an added advantage that the
bution centered at the midpoint of the space to “memory addresses” where buckets of the trie are
generate input keys. For ranges that are close to stored are in fact the DHT keys obtained by hash-
the mean, where keys are densely clustered, the ing the corresponding prefixes.
PHT does well, actually out-performing the uni-
form case. For sparser regions, the PHT does not With respect to the problem of implementing range
do as well, but no worse than 1.6 the optimal value. search over peer-to-peer systems, Aspnes and Shah
These results indicate that the PHT incurs only a [1] have proposed skip graphs, a distributed data
reasonably small constant factor of overhead (in structure based on the skiplist that provides a range
terms of nodes visited) more than the theoretically search solution. However they do not provide a
optimal value. mapping from keys to peers in a network; such a
mapping is provided by Awerbuch and Scheideler
 Distribution of lookup traffic B = 20 5. CONCLUSION
1600
n=1000
n=2000
In their short existence, DHTs have become a widely
1400
n=3000 used tool for building large-scale distributed sys-
tems. While the lookup interface offered by DHTs
1200 is broadly applicable, it does not naturally support
a very common feature in database and other in-
formation processing systems: range queries. Our
Number of lookups

1000

goal was to address this shortcoming but, contrary

800
to early efforts in the field, subject to the constraint
600
that these queries only use the lookup interface and
not rely on changes to or knowledge of the under-
400 lying DHT routing algorithm. This would ensure
that the solution would apply to any DHT, not just
200 those specifically engineered for the task. To this
end, we presented the design and evaluation of Pre-
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 fix Hash Trees (PHT), a data structure designed to
Cumulative distribution of nodes
support range queries. PHT has the properties tra-
ditionally required of large-scale Internet systems:
Figure 4: Cumulative distribution of PHT self-organizing, scalable, and robust in the presence
lookup traffic of failures. While it does not prevent loss of data
due to node outages, such failures do not prevent
it from producing results from the other nodes.
[2] by essentially building a skip graph on top of a
DHT. Lookup times and per data item state in a In short, we believe that PHT will enable general-
skip graph are O(log N ), where N is the number purpose DHTs to support a wider class of queries,
of items. and then broaden the horizon of their applicability.

In recent work, Karger and Ruhl [17] propose a 6. REFERENCES

randomized protocol called item balancing that bal- [1] Aspnes, J., and Shah, G. Skip graphs. In
ances the distribution of items by having DHT Fourteenth Annual ACM-SIAM Symposium
nodes adaptively change their identifiers. While on Discrete Algorithms (Baltimore, MD, Jan
providing excellent theoretical properties, their so- 2003).
lution relies on more than just the hashtable in-
terface of the underlying DHT, which could poten- [2] Awerbuch, B., and Scheideler, C.
tially create a barrier to deployment. A related Peer-to-peer Systems for Prefix Search. In
protocol has been proposed by Ganesan and Bawa ACM Symposium on Principles of Distributed
[12]. Computing (Boston, MA, July 2003).
Other related work includes a DHT-based caching [3] Berchtold, S., Böhm, C., and Kriegel,
scheme [13] and a technique specifically for the H.-P. The pyramid-technique: Towards
CAN DHT based on space-filling curves [32]. breaking the curse of dimensionality. In Proc.
ACM SIGMOD International Conference on
Cone [4] is a trie-based data structure that is used Management of Data (Seattle, WA, June
to evaluate aggregation operators, such as MIN, 1998), pp. 142–153.
MAX and SUM, over keys in a DHT. Although the
PHT is also based on a trie, it differs from Cone [4] Bhagwan, R., Voelker, G., and
in three significant respects. First, Cone builds Varghese, G. Cone: Augmenting DHTs to
a trie over uniformly distributed node identifiers. Support Distributed Resource Discovery.
Second, Cone does not support range queries. Fi- Tech. Rep. UCSD CS2002-0726, Computer
nally, Cone is a DHT augmentation where as the Science Department, University of California,
PHT builds on top of the DHT. San Diego, November 2002.
[5] Blake, C., and Rodrigues, R. High
Waldvogel et al [33] have proposed an IP lookup Availability, Scalable Storage, Dynamic Peer
algorithm based on binary search of prefixes orga- Networks: Pick Two. In HotOS IX (May
nized into hashtables based on prefix length. Al- 2003).
though they are solving longest prefix match, a
different but related problem, their binary search [6] Cabrera, L., Jones, M. B., and
technique is similar to the PHT lookup algorithm. Theimer, M. Herald: Achieving a Global
The key distinguishing characteristic is that the Event Notification Service. In HotOS VIII
PHT operates in a distributed setting, with an (May 2001).
entirely different set of constraints and issues, as
opposed to an IP lookup algorithm that is imple- [7] Clip2. The Gnutella Protocol Specification
mented in hardware in a high-speed router. v.0.4, March 2001.
 [8] Cormen, T., Stein, C., Rivest, R., and [19] Knuth, D. The Art of Computer
Leiserson, C. Introduction to Algorithms. Programming. Addison-Wesley.
McGraw-Hill Higher Education.
[20] Li, J., Loo, B. T., Hellerstein, J. M.,
[9] Dabek, F., Kaashoek, M. F., Karger, Kaashoek, F., Karger, D., and Morris,
D., Morris, R., and Stoica, I. Wide-area R. On the Feasibility of Peer-to-Peer Web
Cooperative Storage with CFS. In Indexing and Search. In Proceedings of the
Proceedings of the 18th ACM Symposium on 2nd International Workshop on Peer-to-Peer
Operating Systems Principles (SOSP 2001) Systems (IPTPS). (Berkeley, CA, Feb. 2003).
(Lake Louise, AB, Canada, October 2001).
[21] Litwin, W. Trie Hashing. In Proceedings of
[10] Freedman, M. J., Freudenthal, E., and ACM SIGMOD (Ann Arbor, MI, 1981).
Mazieres, D. Democratizing Content
Publishing with Coral. In Proceedings of the [22] Malkhi, D., Naor, M., and Ratajczak,
USENIX/ACM Symposium on Networked D. Viceroy: A Scalable and Dynamic
Systems Design and Implementation (NSDI), Emulation of the Butterfly. In ACM
2004 (San Francisco, CA, March 2003). Symposium on Principles of Distributed
Computing (July 2002).
[11] Gaede, V., and Günther, O.
Multidimensional access methods. ACM [23] Maymounkov, P., and Mazières, D.
Computing Surveys 30, 2 (June 1998), Kademlia: A peer-to-peer information system
170–231. based on the xor metric. In 1st International
Workshop on Peer-to-Peer Systems
[12] Ganesan, P., and Bawa, M. Distributed (Cambridge, MA, Mar. 2002).
Balanced Tables: Not Making a Hash of it
All. Tech. rep., Computer Science [24] Ratnasamy, S., Francis, P., Handley,
Department, Stanford University, 2003. M., Karp, R., and Shenker, S. A
Scalable Content-Addressable Network. In
[13] Gupta, A., Agrawal, D., and Abbadi, ACM SIGCOMM (San Diego, CA, August
A. E. Approximate Range Selection Queries 2001), pp. 161–172.
in Peer-to-Peer Systems. In Proceedings of
the First Biennial Conference on Innovative [25] Reynolds, P., and Vahdat, A. Efficient
Data Systems Research (CIDR) (Asilomar, Peer-to-Peer Keyword Searching. In
CA, January 2003). Proceedings of the IFIP International
Conference on Distributed Systems Platforms
[14] Hildrum, K., Kubiatowicz, J. D., Rao, and Open Distributed Processing
S., and Zhao, B. Y. Distributed Object (Middleware) (June 2003).
Location in a Dynamic Network. In 14th
ACM Symposium on Parallel Algorithms and [26] Rowstron, A., and Druschel, P. Pastry:
Architectures (Aug. 2002). Scalable, Distributed Object Location and
Routing for Large-s cale Peer-to-peer
[15] Huebsch, R., Hellerstein, J. M., Systems. In Proceedings of the 18th
Lanham, N., Loo, B. T., Shenker, S., IFIP/ACM International Conference on
and Stoica, I. Querying the Internet with Distributed Systems Platforms (Middleware
PIER. In Proceedings of VLDB 2003 (Berlin, 2001) (Nov. 2001).
Germany, September 2003).
[27] Rowstron, A., and Druschel, P. Storage
[16] Jagadish, H. V. Linear clustering of objects management and caching in PAST, a
with multiple atributes. In Proc. ACM large-scale, persistent peer-to-peer storage
SIGMOD International Conference on utility. In 18th ACM Symposium on
Management of Data (Atlantic City, NJ, Operating Systems Principles (Banff,
May 1990), pp. 332–342. Canada, Oct. 2001).
[17] Karger, D., and Ruhl, M. Simple [28] Salzberg, B., and Tsotras, V.
Efficient Load Balancing Algorithms for Comparison of access methods for
Peer-to-Peer Systems. In Proceedings of the time-evolving data. ACM Computing Surveys
Third International Peer-to-Peer Systems 31, 2 (1999), 158–221.
Workshop (IPTPS) (Feb. 2004).
[29] Sharman Networks Ltd. KaZaA Media
[18] Karp, B., Ratnasamy, S., Rhea, S., and Desktop, 2001.
Shenker, S. Spurring Adoption of DHTs
with OpenHash, a Public DHT service. In [30] Stoica, I., Adkins, D., Zhuang, S.,
Proceedings of the Third International Shenker, S., and Surana, S. Internet
Peer-to-Peer Systems Workshop IPTPS 2004 Indirection Infrastructure. In ACM
(Feb. 2004). SIGCOMM (Pittsburgh, PA, Aug. 2002).
[31] Stoica, I., Morris, R., Liben-Nowell,
D., Karger, D., Kaashoek, M. F.,
Dabek, F., and Balakrishnan, H. Chord:
A Scalable Peer-to-peer Lookup Protocol for
Internet Applications. IEEE/ACM
Transactions on Networking 11, 1 (Feb.
2003), 17–32.
[32] Tang, C., Xu, Z., and Dwarkadas, S.
Peer-to-Peer Information Retrieval using
Self-organizing Semantic Overlay Networks.
In ACM SIGCOMM (Karlsruhe, Germany,
Aug. 2003).
[33] Waldvogel, M., Varghese, G., Turner,
J., and Plattner, B. Scalable Best
Matching Prefix Lookups. In ACM
Symposium on Principles of Distributed
Computing (Puerto Vallarta, Mexico, June
1998).
[34] Warren, M., and Salmon, J.
Astrophysical N-body Simulations Using
Hierarchical Tree Data Structures. In
Proceedings of Supercomputing (Sept. 1992).
[35] Yu, C., Ooi, B. C., Tan, K.-L., and
Jagadish, H. V. Indexing the distance: An
efficient method to knn processing. In Proc.
27th International Conference on Very Large
Data Bases (VLDB) (Roma, Italy, Sept.
2001), pp. 421–430.
[36] Yu, H., and Vahdat, A. The Costs and
Limits of Availability for Replicated Services.
In Proceedings of the 18th ACM Symposium
on Operating Systems Principles (SOSP)
(Lake Louise, AB, Canada, October 2001).