Etherpad and EasySync Technical Manual

AppJet, Inc., with modifications by the Etherpad Foundation

March 26, 2011

Contents
1 Documents 2

2 Changesets 2

3 Changeset representation 2

4 Constraints on Changesets 2

5 Notation 3

6 Composition of Changesets 3

7 Changeset Merging 3

8 Follows 4

9 System Overview 5

10 Client State 5

11 Client Operations 5
11.1 New local typing . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
11.2 Submitting changesets to server . . . . . . . . . . . . . . . . . . . 6
11.3 Hear ACK from server . . . . . . . . . . . . . . . . . . . . . . . . 6
11.4 Hear about another client’s changeset . . . . . . . . . . . . . . . 6
11.5 Connect to server . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

12 Server Overview 7

13 Server State 7

14 Server Operations Overview 8

14.1 Respond to client connect . . . . . . . . . . . . . . . . . . . . . . 8
14.2 Respond to client changeset . . . . . . . . . . . . . . . . . . . . . 8

1
1 Documents
• A document is a list of characters, or a string.
• A document can also be represented as a list of changesets.

2 Changesets
• A changeset represents a change to a document.
• A changeset can be applied to a document to produce a new document.
• When a document is represented as a list of changesets, it is assumed that
the first changeset applies to the empty document, [].

3 Changeset representation
(` → `0 )[c1 , c2 , c3 , ...]
where

` is the length of the document before the change,

`0 is the length of the document after the change,
[c1 , c2 , c3 , ...] is an array of `0 characters that described the document after
the change.

Note that ∀ci : 0 ≤ i ≤ `0 is either an integer or a character.

• Integers represent retained characters in the original document.

• Characters represent insertions.

4 Constraints on Changesets
• Changesets are canonical and therefor comparable. When represented in
computer memory, we always use the same representation for the same
changeset. If the memory representation of two changesets differ, they
must be different changesets.
• Changesets are compact. Thus, if there are two ways to represent a change-
set in computer memory, then we always use the representation that takes
up the fewest bytes.

Later we will discuss optimizations to changeset representation (using “strips”

and other such techniques). The two constraints must apply to any representa-
tion of changesets.

2
5 Notation
• We use the algebraic multiplication notation to represent changeset appli-
cation.

• While changesets are defined as operations on documents, documents

themselves are represented as a list of changesets, initially applying to
the empty document.

Example A = (0 → 5)[“hello”] B = (5 → 11)[0 − 4, “ world”]

We can write the document “hello world” as A · B or just AB. Note
that the “initial document” can be made into the changeset (0 → N )[“ <
the document text > ”].
When A and B are changesets, we can also refer to (AB) as “the composi-
tion” of A and B. Changesets are closed under composition.

6 Composition of Changesets
For any two changesets A, B such that

A = (n1 → n2 )[· · ·]

A = (n2 → n3 )[· · ·]

it is clear that there is a third changeset C = (n1 → n3 )[· · ·] such that applying
C to a document X yeilds the same resulting document as does applying A and
then B. In this case, we write AB = C.
Given the representation from Section 3, it is straightforward to compute
the composition of two changesets.

7 Changeset Merging
Now we come to realtime document editing. Suppose two different users make
two different changes to the same document at the same time. It is impossible
to compose these changes. For example, if we have the document X of length
n, we may have A = (n → na )[. . . na characters], B = (n → nb )[. . . nb characters]
where n 6= na 6= nb .
It is impossible to compute (XA)B because B can only be applied to a
document of length n, and (XA) has length na . Similarly, A cannot be appliet
to (XB) because (XB) has length nb .
This is where merging comes in. Merging takes two changesets that apply
to the same initial document (and that cannot be composed), and computes a
single new changeset that presevers the intent of both changes. The merge of
A and B is written as m(A, B). For the Etherpad system to work, we require
that m(A, B) = m(B, A).

3
 Aside from what we have said so far about merging, there aremany different
implementations that will lead to a workable system. We have created one
implementation for text that has the following constraints.

8 Follows
When users A and B have the same document X on their screen, and they
proceed to make respective changesets A and B, it is no use to compute m(A, B),
because m(A, B) applies to document X, but the users are already looking at
document XA and XB. What we really want is to compute B 0 and A0 such
that
XAB 0 = XBA0 = Xm(A, B)
“Following” computes these B 0 and A0 changesets. The definition of the
“follow” function f is such that Af (A, B) = Bf (B, A) = m(A, B) = m(B, A).
When we computer f (A, B).

• Insertions in A become retained characters in f (A, B)

• Insertions in B become insertions in f (A, B)

• Retain whatever characters are retained in both A and B

Example Suppose we have the initial document X = (0 → 8)[“baseball ”] and

user A changes it to “basil” with changeset A, and user B changes it to “below”
with changeset B.
We have X = (0 → 8)[“baseball ”]
A = (8 → 5)[0 − 1, “si ”, 7]
B = (8 → 5)[0, “e”, 6, “ow ”]

First we compute the merge m(A, B) = m(B, A) according to the constraints

m(A, B) = (8 → 6)[0, ”e”, ”si”, ”ow”] = (8 → 6)[0, “esiow ”]

Then we need to compute the follows B 0 = f (A, B) and A0 = f (B, A).

B 0 = f (A, B) = (5 → 6)[0, “e”, 2, 3, “ow ”]

Note that the numbers 0, 2, and 3 are indices into A = (8 → 5)[0, 1, “si ”, 7]
0 1 2 3 4
0 1 s i 7
A0 = f (B, A) = (5 → 6)[0, 1, ”si”, 3, 4]
We can now double check that AB 0 = BA0 = m(A, B) = (8 → 6)[0, “esiow ”].
Now that we have made the mathematical meaning of the preceding pages
complete, we can build a client/server system to support realtime editing by
multiple users.

4
9 System Overview
There is a server that holds the current state of a document. Clients (users) can
connect to the server from their web browsers. The clients and server maintain
state and can send messages to one another in real-time, but because we are in
a web browser scenario, clients cannot send each other messages directly, and
must go through the server always. (This may distinguish from prior art?)
The other critical design feature of the system is that A client must always
be able to edit their local copy of the document, so the user is never blocked from
typing because of waiting to to send or receive data.

10 Client State
At any moment in time, a client maintains its state in the form of 3 changesets.
The client document looks like A · X · Y , where
A is the latest server version, the composition of all changesets committed
to the server, from this client or from others, that the server has informed this
client about. Initially A = (0 → N )[< initial document text >].
X is the composition of all changesets this client has submitted to the server
but has not heard back about yet. Initially X = (N → N )[0, 1, 2, . . . , N − 1], in
other words, the identity, henceforth denoted IN .
Y is the composition of all changesets this client has made but has not yet
submitted to the server yet. Initially Y = (N → N )[0, 1, 2, . . . , N − 1].

11 Client Operations
A client can do 5 things.

1. Incorporate new typing into local state

2. Submit a changeset to the server

3. Hear back acknowledgement of a submitted changeset

4. Hear from the server about other clients’ changesets

5. Connect to the server and request the initial document

As these 5 events happen, the client updates its representation A · X · Y

according to the relations that follow. Changes “move left” as time goes by:
into Y when the user types, into X when change sets are submitted to the
server, and into A when the server acknowledges changesets.

5
11.1 New local typing
When a user makes an edit E to the document, the client computes the com-
position (Y · E) and updates its local state, i.e. Y ← Y · E. I.e., if Y is the
variable holding local unsubmitted changes, it will be assigned the new value
(Y · E).

11.2 Submitting changesets to server

When a client submit its local changes to the server, it transmits a copy of Y
and then assigns Y to X, and assigns the identity to Y . I.e.,

1. Send Y to server,
2. X ← Y
3. Y ← IN (the identity).

This happens every 500ms as long as it receives an acknowledgement. Must

receive ACK before submitting again. Note that X is always equal to the
identity before the second step occurs, so no information is lost.

11.3 Hear ACK from server

When the client hears ACK from server,
A←A·X
X ← IN

11.4 Hear about another client’s changeset

When a client hears about another client’s changeset B, it computes a new A,
X, and Y , which we will call A0 , X 0 , and Y 0 respectively. It also computes a
changeset D which is applied to the current text view on the client, V . Because
AXY must always equal the current view, AXY = V before the client hears
about B, and A0 X 0 Y 0 = V D after the computation is performed.
The steps are:

1. Compute A0 = AB
2. Compute X 0 = f (B, X)
3. Compute Y 0 = f (f (X, B), Y )
4. Compute D = f (Y, f (X, B))
5. Assign A ← A0 , X ← X 0 , Y ← Y 0 .
6. Apply D to the current view of the document displayed on the user’s
screen.

In steps 2,3, and 4, f is the follow operation described in Section 8.

6
Proof that AXY = V ⇒ A0 X0 Y0 = VD. Substituting A0 X 0 Y 0 = (AB)(f (B, X))(f (f (X, B), Y )),
we recall that merges are commutative. So for any two changesets P and Q,

m(P, Q) = m(Q, P ) = Qf (Q, P ) = P f (P, Q)

Applying this to the relation above, we see

A0 X 0 Y 0 = ABf (B, X)f (f (X, B), Y )

= AXf (X, B)f (f (X, B), Y )
= AXY f (Y, f (X, B))
= AXY D
= VD

As claimed.

11.5 Connect to server

When a client connects to the server for the first time, it first generates a random
unique ID and sends this to the server. The client remembers this ID and sends
it with each changeset to the server.
The client receives the latest version of the document from the server, called
HEADTEXT. The client then sets

A ← HEADTEXT
X ← IN
Y ← IN

And finally, the client displays HEADTEXT on the screen.

12 Server Overview
Like the client(s), the server has state and performs operations. Operations are
only performed in response to messages from clients.

13 Server State
The server maintains a document as an ordered list of revision records. A
revision record is a data structure that contains a changeset and authorship
information.

RevisionRecord = {
ChangeSet,
Source (unique ID),
Revision Number (consecutive order, starting at 0)
}

7
 For efficiency, the server may also store a variable called HEADTEXT, which
is the composition of all changesets in the list of revision records. This is an
optimization, because clearly this can be computed from the set of revision
records.

14 Server Operations Overview

The server does two things in addition to maintaining state representing the set
of connected clients and remembering what revision number each client is up to
date with:

1. Respond to a client’s connection requesting the initial document.

2. Respond to a client’s submission of a new changeset.

14.1 Respond to client connect

When a server recieves a connection request from a client, it receives the client’s
unique ID and stores that in the server’s set of connected clients. It then sends
the client the contents of HEADTEXT, and the corresponding revision number.
Finally the server notes that this client is up to date with that revision number.

14.2 Respond to client changeset

When the server receives information from a client about the client’s changeset
C, it does five things:

1. Notes that this change applies to revision number rc (the client’s latest
revision).

2. Creates a new changeset C 0 that is relative to the server’s most recent revi-
sion number, which we call rH (H for HEAD). C 0 can be computed using
follows (Section 8). Remember that the server has a series of changesets,

S0 → S1 → . . . Src → Src +1 → . . . → SrH

C is relative to Src , but we need to compute C 0 relative to SrH . We can

compute a new C relative to Src +1 by computing f (Src +1 , C). Similarly
we can repeat for Src +2 and so forth until we have C 0 represented relative
to SrH .

3. Send C 0 to all other clients

4. Send ACK back to original client

5. Add C 0 to the server’s list of revision records by creating a new revision

record out of this and the client’s ID.

8
Additional topics
(a) Optimizations (strips, more caching, etc.)
(b) Pseudocode for composition, merge, and follow
(c) How authorship information is used to color-code the document based
on who typed what
(d) How persistent connections are maintained between client and server