Flexible Wide Area Consistency

Management

Sai Susarla

PhD Proposal Defense

 The Problem
Wide area services are not alike
Different consistency & availability requirements
But many common issues & mechanisms
Concurrency control, update propagation, conflict detection, etc.
Ideal: Reuse mechanisms from existing systems
Problem: Each system designs monolithic solution
Suited to its own particular needs
Hard to isolate individual mechanisms and reuse differently

Managing consistency of wide area replicated data

requires significant development effort

PhD Proposal Defense

 Thesis
"A consistency management system that provides
a small set of customizable coherence mechanisms
can efficiently satisfy the data coherence needs of a
wide variety of distributed applications."

PhD Proposal Defense

 Thesis
"A consistency management system that provides
a small set of customizable coherence mechanisms
can efficiently satisfy the data coherence needs of a
wide variety of distributed applications."

PhD Proposal Defense

 Thesis
"A consistency management system that provides a
small set of customizable coherence mechanisms
can efficiently satisfy the data coherence needs of a
wide variety of distributed applications."

PhD Proposal Defense

 Thesis
"A consistency management system that provides
a small set of customizable coherence mechanisms
can efficiently satisfy the data coherence needs of a
wide variety of distributed applications."

PhD Proposal Defense

 Contributions
Identify policies, mechanisms that together satisfy a
variety of data coherence needs
Implement them in a wide area framework, retaining
Reusability: by application-independence of mechanisms
Efficiency: by letting applications customize system behavior

Evaluate its effectiveness and scalability for a variety

of applications

PhD Proposal Defense

Wide Area Consistency Management
Many common policy issues & design choices
When is access allowed to a replica? (optimistic vs. pessimistic)
Where are updates issued? (single/multiple masters)
How are updates propagated? (push vs. pull)
What triggers update transfers? (timer, staleness, eager, lazy)
Many apps hardwire a combination of above choices
Hard to isolate & reuse in other combinations

PhD Proposal Defense

 Exploit Commonality

Pull updates System files, static web content

Single-master
Push updates Scoreboard, Active Dir. schema
Optimistic config files, multimedia streaming
last-writer-wins,
append-only Pull updates Bayou, Coda, calendar,
sensor logging, mobile file access
Multi-master
Push updates Active Directory data, chat,
whiteboard

PhD Proposal Defense

 Exploit Commonality

Pull updates System files, static web content

Single-master
Push updates Scoreboard, Active Dir. schema
Optimistic config files, multimedia streaming
last-writer-wins,
append-only Pull updates Bayou, Coda, calendar,
sensor logging, mobile file access
Multi-master
Push updates Active Directory data, chat,
whiteboard

Current Approach:
• Implement one combination effectively to suit one app.
• Restricts reusability

PhD Proposal Defense

 Exploit Commonality

Pull updates System files, static web content

Single-master
Push updates Scoreboard, Active Dir. schema
Optimistic config files, multimedia streaming
last-writer-wins,
append-only Pull updates Bayou, Coda, calendar,
sensor logging, mobile file access
Multi-master
Push updates Active Directory data, chat,
whiteboard

Approach:
• Implement common mechanisms in app-independent manner
• Provide hooks to let applications customize their behavior

PhD Proposal Defense

 Evaluation Platform
Khazana: configurable caching infrastructure
File-like data abstraction, page-grained consistency
Performs caching, concurrency control
Asynchronous update notifications
Application controls per-file behavior
Consistency policy (last-writer-wins, append-only, or strict)
Control over per-replica quality
Optimistic vs. pessimistic access
Push vs. pull updates, single vs. multiple masters
Per-file dynamic cache hierarchy

PhD Proposal Defense

 ... Replicated Objects
in Khazana
P
S1 S2
S3 P: Primary replica
Sn: Khazana server

S4
S5
P

...
Object O1: multi-master, pull-updates optimistic consistency (e.g., shared file)
Object O2: single-master, push-updates strict consistency (e.g., shared password db)
Parent-child relationship between object replicas
Client accesses replica

PhD Proposal Defense

 ... Replicated Objects
in Khazana
P S2 Crashes
S1 S2
S3 P: Primary replica
Link down
O2 copy inaccessible Sn: Khazana server
Broken link
S4
S5
P

...
Object O1: multi-master, pull-updates optimistic consistency (e.g., shared file)
Object O2: single-master, push-updates strict consistency (e.g., shared password db)
Parent-child relationship between object replicas
Client accesses replica

PhD Proposal Defense

 ... Replicated Objects
in Khazana
P
S1 S2
S3 P: Primary replica
Link Up
Sn: Khazana server
Broken link
S4
S5
P

...
Object O1: multi-master, pull-updates optimistic consistency (e.g., shared file)
Object O2: single-master, push-updates strict consistency (e.g., shared password db)
Parent-child relationship between object replicas
Client accesses replica

PhD Proposal Defense

 ... Replicated Objects
in Khazana
P S2 Recovers
S1 S2
S3 P: Primary replica
Sn: Khazana server
Broken link
S4
S5
P

...
Object O1: multi-master, pull-updates optimistic consistency (e.g., shared file)
Object O2: single-master, push-updates strict consistency (e.g., shared password db)
Parent-child relationship between object replicas
Client accesses replica

PhD Proposal Defense

 Thesis
"A consistency management system that provides
a small set of customizable coherence mechanisms
can efficiently satisfy the data coherence needs of a
wide variety of distributed applications."

PhD Proposal Defense

 Representative Applications
DataStations: Distributed file store that supports multiple
file sharing patterns
Chat room: Concurrent appends to a transcript file
Scoreboard: Broadcasts game state to registered
listeners
Directory Service: Hash table implemented in a shared
file

PhD Proposal Defense

 A Khazana-based Chat Room

3 callback(handle, newdata)
P {
4 display(newdata);
}
main()
2 {
1 handle = kh_open(kid, "a+");
kh_snoop(handle, callback);
while (! done) {
read(&newdata);
display(newdata);
kh_write(handle, newdata);
}
kh_close(handle);
}
Sample Chat client code
Chat transcript: multi-master, push updates, optimistic append-only consistency
Update propagation path

PhD Proposal Defense

 Evaluation
Evaluate
Scalability, resilience to wide-area conditions (DataStations)
Performance
» Microbenchmarks (latency, bandwidth utilization)
» Per-application macrobenchmarks
Demonstrate
Application control over system mechanisms
Scalable real-time data exchange (Chat room)
Control over per-replica data quality (Scoreboard)
Compare performance
Chat room with IRC
Directory service with MS Active directory

PhD Proposal Defense

 Related Work

Support for multiple consistency policies

Munin, WebFS, Fluid Replication, TACT
Wide area issues in consistency management
Coda, Ficus, Bayou
Applications with specific consistency semantics
Active Directory, Thor
Peer-to-peer data sharing systems
Napster, PAST, Farsite, Freenet

PhD Proposal Defense

 Open Issues
Optimal replica placement
Efficient replica topologies reflecting Internet routing
Security

PhD Proposal Defense

 Timeline
Sep Dec
Khazana on WAN, last-writer, Async updates, update triggers
push updates, DataStations (timer, staleness ...)
Defend proposal Scoreboard & chat
Oct Experiments 5 and 6
Other policies, push, multi- Jan
master (req. file versioning) Directory service
Experiments 1 & 2 Experiment 4
Nov Feb
Op. timeouts, node failures Start writing thesis
Experiment 3
April/May
Finish thesis and defend.

PhD Proposal Defense

 Conclusion
Managing consistency of replicated data is hard
Many mechanisms available, but hard to reuse efficiently
Current approach: reinvent the wheel
Proposed a solution: generic consistency framework
Offloads this complexity from applications
Reusable, customizable, scalable
Contributions
Identify policies, mechanisms that serve a variety of coherence needs
Design a wide area framework to leverage them efficiently
Evaluate its effectiveness & scalability for a variety of useful
applications

PhD Proposal Defense