Whitepaper
Whitepaper
March 5, 2018
Table of Contents
1 INTRODUCTION 3
1.1 Definition of the Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Expand on the Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.1 Net Neutrality and Internet Service Providers . . . . . . . . . . . . . . . . . 3
1.2.2 Government Agencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.3 Centralized Server Hosting . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Proposed Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 STORAGE LAYER 5
2.1 InterPlanetary File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Implementation – Shift Storage Cluster . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.1 File Pinning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.2 HTTP API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.3 Consensus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.4 Replication Factor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.5 File Storage and Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 Service Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.1 HAProxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.2 Jenga . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3 BLOCKCHAIN SECURED 8
3.1 Delegated Proof-of-Stake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2 Phantom Sidechain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2.1 Transaction Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.3 External Interactions – Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4 PRESENTATION LAYER 10
4.1 Hydra – Content Management System . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.2 Phantom UI – File Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.3 Content Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
5 SECURITY LAYER 12
5.1 Shift Cluster Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
5.1.1 Preventing Oversubscription . . . . . . . . . . . . . . . . . . . . . . . . . . 12
5.2 Data Privacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
5.3 Illegal Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
6 CONCLUSION 14
REFERENCES 15
1
ABSTRACT
Censorship and the suppression of information has been a common problem throughout human
history. The advent of blockchain and other decentralized technologies has created a technological
revolution that can ensure no content is easily censorable. A combination of storage and presen-
tation layer technologies has the potential to empower end users and operators with the ability to
preserve data free from censorship.
Shift proposes a solution to the problem of censorship and defines how a combination of these
new technologies solve that problem.
2
1 INTRODUCTION
This white paper provides an introduction to Phantom, a unified technology suite used as a platform
for hosting content in a decentralized system. The scope of this document is to specify the general
technical approach Shift is undertaking. This will be done by defining the problems that Shift
is attempting to solve with Phantom as well as describing how the different technologies address
those issues.
Net neutrality was an unwritten concept which governed the Internet since its inception. Freedom
of information was seen as a fundamental need to advance technology for all of humanity’s benefit.
One potential threat to Net neutrality is from the ISP which can block, limit, or slow access to
content they may not agree with. ISPs may choose to charge more for premium access to content.
While bypassing the ISP is generally not possible, content providers can use new technologies to
decentralize their content and spread it in a way that ISPs will not be able to censor easily.
Other examples of Government attacks on the Internet can be seen in the political space. In
September 2017, the Spanish Government seized the top level domain provider for ”.cat” (Morris,
2017), effectively granting them full control of the DNS running under that extension. Further-
more, the Government began to forcibly censor websites supporting the Catalan referendum and
ultimately prevented vote submissions from polling locations. Government censorship is a clear
3
and present danger to the Internet today. The new Internet must evolve past these problems to
keep information freely available.
As the Internet continues to scale, these providers will gather more users requiring hosting and
these issues may occur more often. Mitigating this problem is possible with some newer technolo-
gies like IPFS, but the infrastructure to bring websites into IPFS and present them to the masses
has not been widely available.
Phantom provides the solution to many sources of web censorship. As the name suggests, ser-
vices hosted on Phantom may vanish from one host but remain accessible from another host in the
network. Phantom implements an IPFS backbone to create the storage layer of the Shift network.
End users can submit their files for long term storage using a blockchain based space leasing sys-
tem. Website operators can serve their entire website from the Shift storage cluster using Hydra to
render resources and keep their content live using Jenga, which dynamically updates resources for
discovery, if they are being censored. This technology suite creates a seamless end user experience
that cannot be easily censored.
4
2 STORAGE LAYER
2.1 InterPlanetary File System
The Phantom storage layer consists of a freely available technology called the InterPlanetary File
System. IPFS is a peer-to-peer hypermedia distribution protocol (Benet, 2014) which serves as
the backbone for many decentralized storage applications. It provides a well documented protocol
for implementing a distributed file system suitable for serving web applications and other types
of content. IPFS is described as having no central point of failure and is fully peer-to-peer,
meaning any user can participate in the system. This file system protocol does not rely on any
central authority to govern its operation. IPFS is poised to create a new decentralized Internet
infrastructure that is free from censorship.
In order to store data permanently, IPFS implements a concept called pinning. Pinning con-
tent means that the content will be available permanently (or until it is unpinned). By default
the pinning only applies to a single peer that it is pinned to, but that means if that machine goes
offline, the content can be lost. The way around this is by using an IPFS cluster: a subnet (or
private net) running the IPFS daemon, containing only Shift peers.
The Shift cluster runs as a wrapper around the IPFS daemon. It allows the end user to con-
nect a group of IPFS nodes together so that content can be stored and replicated within the
group. The cluster elects a leader to be in charge of keeping track of which content is available in
which locations.
The Shift cluster provides a modular clustering system for use with IPFS. This clustering sys-
tem works in conjunction with the IPFS daemon in order to accomplish the following tasks:
1. Pin, unpin and repin content to peers
2. Provide an HTTP API for communications
3. Assert and follow cluster consensus
4. Replication factor
5. File storage and retrieval
These functions come together to provide an extensible decentralized storage system that operates
completely independently from the public IPFS network. This is vital to allow users to insert and
persist content within the system.
5
2.2.2 HTTP API
IPFS offers a suite of internal commands and functions to interact with the network. These same
set of functions are mirrored to an HTTP API allowing external software to interact with the
system. These functions allow Phantom to communicate with IPFS remotely without needing the
end user to join the cluster.
2.2.3 Consensus
The IPFS cluster requires consensus between nodes to ensure that all stored information is kept in
sync. This is especially important for pinned content, which must be stored indefinitely. A node
is elected as the leader and operates as such until it goes offline or a new leader is elected.
Files are retrieved from the system by cryptographic hashes. These hashes are generated when the
file is inserted into the system and saved for later use. This is especially important for storing files
using a blockchain as this hash can provide an identifier to link the file address to the transaction
paying for the storage.
2.3.1 HAProxy
Phantom includes HAProxy, which provides discrete addressing and traffic handling for API end-
points for both IPFS and its cluster. In addition to handling the back end traffic, it also handles
any front end requests to the system. It leverages SSL to provide encrypted communications be-
tween client and server. It also takes care of all incoming requests, to be forwarded to either the
daemon or the cluster. With HAProxy acting as a shield, only whitelisted calls will be executed
by the target application. All forbidden requests will be rejected.
2.3.2 Jenga
Since no mainstream browser supports IPFS natively yet, there needs to be a way to map an in-
coming request to a specific server. It could be solved using a Chrome or Firefox browser extension
to pick an IPFS node to serve the content in the browser, but this is not ideal, because end-users
should not have to install third-party software before they can visit a website hosted by Phantom.
6
Another possible solution is using an internal load balancer, but this also has issues. The pri-
mary issue is that it is centralized. If the load balancer fails, all requests to Phantom would fail.
The second issue is found in throughput or performance issues, such as when the system has to
scale to handle more traffic. The system would be required to support throughput of the entire
Phantom network which is not feasible for a worldwide file storage system.
Jenga solves both of these issues by providing a scalable solution that works without requiring
installation of any additional software for the end user viewing the content.
Jenga is a DNS monitoring solution that observes top level DNS entries for changes and records
those changes. When changes relevant to the IPFS cluster are detected, Jenga updates all storage
nodes in the cluster. By maintaining persistent communications with all nodes, Jenga can create
a consensus on healthy DNS state and subsequently evict non conforming nodes from the system.
This prevents attackers from joining the cluster and injecting an improper DNS entry, removing
the DDoS (Distributed Denial-of-Service) attack vector from the system.
Jenga creates a bridge between the decentralized Web and the classical Internet, which relies
solely on DNS records. Jenga creates a dynamic web addressing system and facilitates traffic scal-
ing from one node to many thousands based on the records placed in the system. This functionality
is transparent to the end user, requiring no external interaction for the system to operate. At a
fundamental level, Jenga enables the system to address some of the central points of failure: DNS
censorship and traffic shaping.
In order for Jenga to function, it must maintain contact with all connected peers of the IPFS
cluster. If Jenga detects a change in the cluster, it updates DNS with the new information. In
the event that a node is misbehaving or is offline, Jenga will take action and remove that peers’
allotted CNAME record, thereby evicting it from the cluster. Peers can rejoin the system after
errors are corrected and report a healthy status to Jenga.
When the gateway is called upon by an external actor, one of the peers in the cluster is se-
lected to serve the data to the caller. This process is made seamless by the replication of data
throughout the cluster.
In order for a new site to join the system, that operator must provide a CNAME record for
their domain which is pointed at the gateway. In the event an operator does not want to use the
gateway, the operator can create DNS records pointing to their own gateway. The data will still
be delivered from the cluster, but the site will lose some of the benefits of the Phantom gateway,
mainly the guarantee that all healthy nodes that are available for serving are utilized.
7
3 BLOCKCHAIN SECURED
Content security is of utmost importance in a decentralized system. Any data that goes in must
come out untampered with and the consumer needs to be able to trust its authenticity. Blockchain
technology is leveraged by Phantom to ensure immutability and truthfulness within the system,
as an unfalsifiable ledger removes many trust related issues.
Every user in the system can have one or more private keys at their disposal, and these keys
can be used to provide ownership of tokens within that system. This is especially important when
combined with the blockchain which will host Phantom. Users will need to send some tokens from
the Shift blockchain to the Phantom sidechain.
Within the sidechain for Phantom, there is a separate ledger that tracks end users token bal-
ances. These tokens provide users with the ability to pin or unpin content within the IPFS cluster
and exchange tokens for the ability to store files long term. In order to support the system, end
users can join the system as node operators and lease their excess space to the users seeking storage.
8
to the network. This request is submitted to the network, and upon confirmation, the user is
refunded the locked tokens associated with the storage commitment.
9
4 PRESENTATION LAYER
As with many layered applications, the end user interacts with the top layer, or the presentation
layer. In Phantom, the presentation layer consists of Hydra and works in conjunction with the
storage and service layers.
The custom CMS, called Hydra, is a new technology that operates as a CMS built on IPFS.
Hydra works in conjunction with Phantom’s file manager and is currently capable of handling
most common tasks such as adding, modifying and removing content. The basic feature set will
serve the needs of most users and the codebase is open source to allow developers to customize
and improve the software for their own needs. Hydra was written to be extensible to end users
and its codebase is designed in a modular way. For example, a site page and a blog post share
the same rendering engine but may have different schemas. Creating a new module is performed
programmatically using Node.js by specifying some configuration items. The resulting files can
then be served solely via IPFS, and all rendering occurs client side.
The front end and the back end components are separated. This enables developers to use their
preferred framework such as Vue, React or Angular. The data files and module structures are
rendered into JSON files, which can be used with external systems such as the Wordpress API.
Users are able to manage their stored content in the same fashion as a typical modern oper-
ating system. Actions taken by each user propagate to the cluster. The interface provides an
advanced code editor complete with syntax highlighting for all commonly used file types.
IPFS creates a unique hash for each data file, which prevents hosting or uploading identical con-
tent. This functionality streamlines the upload process, as duplicate files can be identified before
being submitted. Additionally, it enables the system to operate efficiently and reduces bandwidth
usage for end users and operators alike.
Phantom UI contains a DNS wizard to control the addressing of hosted domain names utiliz-
ing the Shift hosted storage cluster. This removes the single point of failure caused by requiring
content submitters to host their own systems. Phantom uses Jenga to populate the list of healthy
nodes that will serve the content for a requested domain.
10
4.3 Content Retrieval
Data stored within the system is done at a content level, rather than a locational level. This
new approach provides many benefits. The primary benefit is that the location of the data is no
longer relevant, allowing many nodes to present the same information and when any of the data
is changed, a new hash is produced. The file system is made more intelligent by combining these
benefits to create a merkle hash (root) and a relative path to a subfolder or file.
Websites are addressed by URL and content is fetched by hash. A mutable hash is used by
the domain resolving system. A mutable hash can be updated by end users using their private key
combined with the immutable hash. This feature allows content updates without requiring domain
record updates.
11
5 SECURITY LAYER
Security is paramount in any system. For a decentralized system, security is a fundamental neces-
sity to create stability and build trust in its operation.
The first layer of authentication is done at the blockchain level. A user will need to complete
the registration process on the blockchain by sending the Type 12 transaction registering their
committed storage amount. Once this is done, the user can then register in Phantom as a cluster
participant and complete the join process.
At the second layer, users are required to launch the application using the private key used to
register with the blockchain and the encrypted join key from the storage cluster. The application
will then search the blockchain for the associated storage commitment and confirm the validity.
Afterwards, the application will decrypt the issued key from registering with Phantom and allow
the user to join the cluster.
Operators will offer their storage in set time frames, with renewals or additions being allowed
at any time before the contract expires. If an operator ceases operation before the end of their
committed time frame, they will lose their stake. This requires operators to maintain high uptimes
to maintain the system. There will be an allowance of a set number of hours offline for each cluster
operator by the blockchain.
As mentioned above, operators are penalized for acting poorly. The loss of their funds is a large
deterrent that should be sufficient for handling operator misbehavior. Users can also act poorly by
oversubscribing or flooding the network. These types of misbehaviors could lead to a temporary
freeze of the staked funds.
Malicious user actors will be handled by limiting the volume of content they are allowed to submit
to the system over a period of time. Users can make a set number of transactions in that time
frame before they begin costing substantially more to initiate after the threshold is exceeded for
the period.
12
Within Phantom, users will be able to encrypt content with specific recipients other than them-
selves in mind. A list of public keys can be provided and used to create encrypted messages that
contain the decrypting key that only the recipient’s private key can decrypt.
13
6 CONCLUSION
Phantom is one of the world’s first decentralized storage applications backed by a blockchain.
It is implemented on top of the Shift blockchain as a sidechain and decentralized application.
By leveraging IPFS and clustering, Jenga and Hydra, Phantom provides a censorship resistant
platform for web hosting and content delivery.
14
REFERENCES
Amazon. (2018). Summary of the amazon s3 service disruption in the northern virginia (us-east-1)
region. Retrieved from https://aws.amazon.com/message/41926/ (accessed January 31,
2018)
Benet, J. (2014). Ipfs - content addressed, versioned, p2p file system. Retrieved from
https://filecoin.io/filecoin.pdf (accessed January 31, 2018)
Chen, C. (2017). Tired of dmca, riaa now seeks isp cooperation in
catching and stopping copyright infringement. Privacy News Online.
(https://www.privateinternetaccess.com/blog/2017/02/tired-dmca-riaa-now-seeks-isp-
cooperation-catching-stopping-copyright-infringement/ (accessed January 31, 2018))
Eckersley, P. (2007). Comcast is also jamming gnutella (and lotus notes?). Retrieved from
https://www.eff.org/deeplinks/2007/10/comcast-also-jamming-gnutella-and-lotus-notes
(accessed January 31, 2018)
Martin, J. (2013). Lost on the silk road: Online drug distribution and the ‘cryptomarket’. SAGE
journals. (http://journals.sagepub.com/doi/abs/10.1177/1748895813505234 (accessed Jan-
uary 31, 2018))
Morris, D. (2017). Spanish polish raid .cat admin offices, threatening the internet’s cutest domain
name. Fortune. (http://fortune.com/2017/09/23/spanish-dot-cat-domain-name/ (accessed
January 31, 2018))
Raphael, J. (2009). Isps join riaa’s fight against piracy: Is your isp one of them? PCWorld .
(https://www.pcworld.com/article/161978/riaa.html (accessed January 31, 2018))
Search.usa.gov. (2018). Seize domain names - immigration and
customs enforcement (ice) search results. Retrieved from
https://search.usa.gov/search?affiliate=ice.gov&query=seize+domain+names&commit=Search
(accessed January 31, 2018)
to Marlene H. Dortch, Z. K. A. (2008). In the matter of formal complaint of free press and
public knowledge against comcast corporation for secretly degrading peer-to-peer applications,
file no. eb-08-ih-1518. Retrieved from https://ecfsapi.fcc.gov/file/6520169715.pdf
(accessed January 31, 2018)
15