Grokking #9: Building a real-time and offline editing service with Couchbase
6 likes867 views
The document discusses building an offline and real-time editing service using Couchbase, highlighting the approach to architecture design for such systems. It evaluates various real-time frameworks and databases, shares experiences with Firebase and Couchbase Mobile, and outlines the challenges of conflict resolution in collaborative editing. The author concludes that creating a custom solution with Couchbase may better fit specific application needs, while still leveraging Couchbase's strengths in synchronization and conflict management.
![How Couchbase handle changes?
GET /<db>/_changes?since=900
{
{"results":[
{"seq":909,"id":"_user/1NEzeQ","changes":[]},
{"seq":1317,"id":"mIM0Uuscg0MC","deleted":true,
"changes":[{"rev":"1-96307afa"}]},
{"seq":1318,"id":"mQQ0NwKUM0Dy","changes":[{"rev":"1-5fdcaba"}]},
{"seq":1319,"id":"sLX0SKTpM41X","changes":[{"rev":"1-d15ee59"}]}
}
}](https://image.slidesharecdn.com/grokking-160330082801/85/Grokking-9-Building-a-real-time-and-offline-editing-service-with-Couchbase-51-320.jpg)
![How Couchbase handle changes?
{
"_deleted": true,
"_sync": {
"rev": "1-91fe6048079942fb279c6c51cfd039f0",
"sequence": 10392,
"history": {
"revs": [
"1-91fe6048079942fb279c6c51cfd039f0"
],
}
}](https://image.slidesharecdn.com/grokking-160330082801/85/Grokking-9-Building-a-real-time-and-offline-editing-service-with-Couchbase-52-320.jpg)
![Sample DCP receiver
func (r *Receiver) OnError(err error)
func (r *Receiver) DataUpdate(vbucketId uint16, key []byte, seq uint64,
req *gomemcached.MCRequest) error
func (r *Receiver) DataDelete(vbucketId uint16, key []byte, seq uint64,
req *gomemcached.MCRequest) error
func (r *Receiver) SetMetaData(vbucketId uint16, value []byte) error
func (r *Receiver) GetMetaData(vbucketId uint16)
func (r *Receiver) Rollback(vbucketId uint16, rollbackSeq uint64) error](https://image.slidesharecdn.com/grokking-160330082801/85/Grokking-9-Building-a-real-time-and-offline-editing-service-with-Couchbase-62-320.jpg)
![How do we structure data in Couchbase
// Tree
{
_id: “…”,
_rev: “…”,
shares: { … },
items: []
}
// Node
{
_id: “…”,
_rev: “…”,
content: []
}
<tree1>
<tree2>
<tree1>/<node1>
<tree2>/<node2>](https://image.slidesharecdn.com/grokking-160330082801/85/Grokking-9-Building-a-real-time-and-offline-editing-service-with-Couchbase-63-320.jpg)
![How do we handle changes?
GET /change?since=524
{
trees: [
{ _id: “…”, _rev: “…” }
],
nodes: [
{ _id: “…”, _rev: “…” }
],
current_seq: “…”
}
<tree1>
<tree2>
<tree1>/<node1>
<tree2>/<node2>](https://image.slidesharecdn.com/grokking-160330082801/85/Grokking-9-Building-a-real-time-and-offline-editing-service-with-Couchbase-64-320.jpg)
Grokking #9: Building a real-time and offline editing service with Couchbase
- 1. Grokking #9 Building an offline & real-time editing service with Couchbase Vu Nguyen [email protected]
- 2. In this talk You’ll walk away with • Our approach for an unknown architecture • How do we design architecture for real-time & offline • Couchbase and other real-time solutions
- 3. Talk structure 1. Problems of a real-time & offline editing service 2. Evaluate real-time frameworks and databases 3. First prototype with Firebase 4. Second version with Couchbase Mobile 5. Time to make our own one! 6. Why Couchbase?
- 4. A real-time and offline editing service What is it?
- 6. A real-time and offline editing services 1. Users can edit at the same time 2. Can work without the internet 3. Synchronize as soon as online 4. Resolve conflict between editing sessions
- 7. Evernote does not resolve conflict!
- 8. Hackpad disables editing while offline!
- 9. And Asana!
- 10. First problem: How to handle conflict?
- 11. A real-time and offline editing service Google Docs
- 14. Operational transformation 0 , 0 1 , 0 0 , 1 1 , 1 2 , 1 1 , 2 2 , 2 2 , 0 0 , 2 client
- 15. Operational transformation 0 , 0 1 , 0 0 , 1 1 , 1 2 , 1 1 , 2 2 , 2 2 , 0 0 , 2 client server
- 16. Operational transformation 0 , 0 1 , 0 0 , 1 1 , 1 2 , 1 1 , 2 2 , 2 2 , 0 0 , 2 client server
- 17. Operational transformation 0 , 0 1 , 0 0 , 1 1 , 1 2 , 1 1 , 2 2 , 2 2 , 0 0 , 2 client server
- 18. Operational transformation 0 , 0 1 , 0 0 , 1 1 , 1 2 , 1 1 , 2 2 , 2 2 , 0 0 , 2 client server Conflict!
- 19. Operational transformation 0 , 0 1 , 0 0 , 1 1 , 1 2 , 1 1 , 2 2 , 2 2 , 0 0 , 2 client server Conflict!
- 20. Example for operational transformation client server “ABCDE” “ABCDE” delete 2 delete 4 “ACDE” “ABCE”
- 21. Example for operational transformation client server “ABCDE” “ABCDE” delete 2 delete 4 “ACDE” “ABCE” “ACD” “ACE”
- 22. Example for operational transformation client server “ABCDE” “ABCDE” delete 2 delete 4 “ACDE” “ABCE” “ACD” “ACE”
- 23. Example for operational transformation client server “ABCDE” “ABCDE” delete 2 delete 4 “ACDE” “ABCE” “ACE” “ACE” delete 3
- 24. Example for operational transformation client server “ABCDE” “ABCDE” delete 2 delete 4 “ACDE” “ABCE” “ACE” “ACE” delete 3 “delete 4” must be transformed to “delete 3”
- 25. Google Docs 1. Apply Operation Transformation 2. Client and server exchange “change action” 3. Same code runs on both client and server • Google Web Toolkit
- 26. A real-time and offline editing service Git
- 27. Commit & Revision 1 2 3 3A 3B 4 client server Conflict!
- 28. Git 1. Change unit is “line” 2. Commit and revision 3. Possible actions • Create new file / line • Edit a file / line • Delete a file / line • Move a file 4. Manual conflict resolving
- 29. Second problem: Which real-time & offline solution?
- 30. Real-time frameworks and databases 1. Firebase 2. Deepstream.io 3. RethinkDB 4. CouchDB 5. PostgreSQL, PinelineDB, … (?)
- 31. Real-time frameworks and databases 1. Firebase 2. Deepstream.io 3. RethinkDB 4. CouchDB 5. PostgreSQL, PinelineDB, … (?)
- 32. Other problems 1. Work on web and mobile (Android, iOS) 2. Keep client in sync with server • Can resume from a specific point in time 3. Edit history • Can rollback to a specific version
- 34. Why Firebase? 1. Real-time database (as a service) 2. Work on web and mobile (Android, iOS) 3. Store value as a big JSON 4. Permission on sub-tree 5. Quickly make a prototype
- 35. Sample code var ref = new Firebase("https://<MY-FIREBASE-APP>.firebaseio.com"); ref.set({ name: "Alex Wolfe" }); ref.on("value", function(data) { var name = data.val().name; alert("My name is " + name); });
- 37. Data as a tree
- 38. How do we structure data in Firebase { nodes: { “u12345”: { “a3f0”: { parent_id: “b02c”, order: 18 }, “b02c”: { parent_id: “”, order: 1 } } } nodes/<user>/<id1> nodes/<user>/<id2> nodes/<user>/<id3>
- 39. How do we structure data in Firebase { users: { “u12345”: { … }, }, nodes: { “u12345”: { “a3f0”: … , “b41c”: … } }, shares: { “s98765”: { “2abf”: … } } } nodes/<user>/<id1> nodes/<user>/<id2> nodes/<user>/<id3>
- 40. When we want to share a sub-tree { users: { “u12345”: { … }, }, nodes: { “u12345”: { “a3f0”: … , “b41c”: … } }, shares: { “s98765”: { “2abf”: … } } } nodes/<user>/<id1> shares/<sid>/<id2> nodes/<user>/<id3>
- 41. When we want to share a sub-tree { users: { “u12345”: { … }, }, nodes: { “u12345”: { “a3f0”: … , “b41c”: … } }, shares: { “s98765”: { “2abf”: … } } } nodes/<user>/<id1> shares/<sid>/<id2> nodes/<user>/<id3> - Firebase does not support “move” action - Permission is defined on sub-tree - We have to copy data from “nodes” to “shares”
- 42. Why not Firebase? 1. Firebase does not support “move” action 2. Permission is defined on sub-tree • When sharing some nodes, we have to copy them from “nodes” to “shares” • This defeats synchronization! 3. Does not resolve conflict • Latest update win. • Workaround solutions like hack!
- 44. Couchbase Mobile? 1. Couchbase Server 2. Couchbase Sync Gateway 3. Couchbase Lite on Device - Use CouchDB protocol - PouchDB for offline JS database
- 45. Sample code in PouchBD var db = new PouchDB('dbname'); db.put({ _id: '[email protected]', name: 'David', age: 69 }); db.changes().on('change', function() { console.log('Ch-Ch-Changes'); }); db.replicate.to('http://example.com/mydb');
- 46. Why Couchbase Mobile? 1. Real-time database (self-deployment) 2. Work on web and mobile (Android, iOS) 3. Store value as a JSON documents 4. Permission using channels 5. It does handle conflict!
- 47. Why Couchbase Mobile? 1. Real-time database (self-deployment) 2. Work on web and mobile (Android, iOS) 3. Store value as a JSON documents 4. Permission using channels 5. It does handle conflict!
- 48. How Couchbase handle conflict? 1 2A 2_ 2B db.put({ _id: '[email protected]', _rev: '1-af2345c1', name: 'David', age: 69, }); db.replicate.to(“http://...”)
- 49. How Couchbase handle conflict? 1 2A 2_ 2B 1. Each change must include _rev 2. When two changes conflict 1. Choose arbitrary winner based on a “deterministic algorithm” 2. The document has two _rev numbers 3. Client know that a document is in conflicting status and can decide to resolve. 4. Deleted documents are kept in db { _id: “”, _rev: “”, _deleted: true }
- 50. Couchbase Mobile Permission Master DB User1 DB User2 DB User3 DB UserN DB User1 DB User2 DB User3 DB UserN DB Couchbase DB Sync Gateway … … Couchbase Lite / Pouch DB Replication
- 51. How Couchbase handle changes? GET /<db>/_changes?since=900 { {"results":[ {"seq":909,"id":"_user/1NEzeQ","changes":[]}, {"seq":1317,"id":"mIM0Uuscg0MC","deleted":true, "changes":[{"rev":"1-96307afa"}]}, {"seq":1318,"id":"mQQ0NwKUM0Dy","changes":[{"rev":"1-5fdcaba"}]}, {"seq":1319,"id":"sLX0SKTpM41X","changes":[{"rev":"1-d15ee59"}]} } }
- 52. How Couchbase handle changes? { "_deleted": true, "_sync": { "rev": "1-91fe6048079942fb279c6c51cfd039f0", "sequence": 10392, "history": { "revs": [ "1-91fe6048079942fb279c6c51cfd039f0" ], } }
- 53. How Couchbase handle changes? 1. Each update include a sequence number 2. Client can request changes by ?since=seq • Only latest document revisions are returned 3. Client can request by long-polling or websocket
- 54. Why Couchbase Mobile? 1. Real-time database (self-deployment) 2. Work on web and mobile (Android, iOS) 3. Store value as a JSON documents 4. Permission using channels 5. It does handle conflict!
- 55. How do we structure data in Couchbase Mobile { _id: “a3f0”, _rev: “…” parent_id: “b02c”, order: 18 shares: { “u231b”: … } } <Id1> <id2> <id3>
- 56. How do we structure data in Couchbase Mobile { _id: “a3f0”, _rev: “…” parent_id: “b02c”, order: 18 shares: { “u231b”: … } } <id1> <id3> - Couchbase Mobile does not understand parent-child relationship - We have to update “shares” property on all child nodes => All child nodes will be updated and synchronized to all clients <id2>
- 57. Why not Couchbase Mobile? 1. Permission based on each document • It does not understand parent-child relationship 2. We can not easily share a sub-tree • Have to update all children nodes • This defeats synchronization purpose!
- 58. Third version Build our own solution with Couchbase
- 59. What did we learn? 1. We need our own data structure and API • Pre-built solutions are nice. But they do not fit to our app. 2. We learned how Couchbase Sync Gateway implement synchronization and conflict resolving 3. We learned from Couchbase Sync Gateway • It is written in Go!
- 60. What do we have to do? 1. Our own data structure and API 2. Sync service 3. SDK for JavaScript and mobiles (Android and iOS) 4. Synchronization and conflict solution 5. Sharing solution
- 61. Why still Couchbase? 1. Both database and memcached 2. Data Change Protocol for replication (DCP) • Can resume at a specific time • Can be used to replicate whole database content
- 62. Sample DCP receiver func (r *Receiver) OnError(err error) func (r *Receiver) DataUpdate(vbucketId uint16, key []byte, seq uint64, req *gomemcached.MCRequest) error func (r *Receiver) DataDelete(vbucketId uint16, key []byte, seq uint64, req *gomemcached.MCRequest) error func (r *Receiver) SetMetaData(vbucketId uint16, value []byte) error func (r *Receiver) GetMetaData(vbucketId uint16) func (r *Receiver) Rollback(vbucketId uint16, rollbackSeq uint64) error
- 63. How do we structure data in Couchbase // Tree { _id: “…”, _rev: “…”, shares: { … }, items: [] } // Node { _id: “…”, _rev: “…”, content: [] } <tree1> <tree2> <tree1>/<node1> <tree2>/<node2>
- 64. How do we handle changes? GET /change?since=524 { trees: [ { _id: “…”, _rev: “…” } ], nodes: [ { _id: “…”, _rev: “…” } ], current_seq: “…” } <tree1> <tree2> <tree1>/<node1> <tree2>/<node2>
- 65. How do we share? <tree1> <tree2> <tree1>/<node1> <tree2>/<node2> 1. Only define permission on trees 2. Split tree when needed 3. Tree has it own revisions 4. Nodes are used to store data (without explicit permission)
- 66. Remaining problems 1. Change set & history 2. Moving & tree conflict 3. Optimize loading: load important data first 4. Improve caching 5. More tests 6. …
- 68. Grokking #9 Building an offline & real-time editing service with Couchbase Vu Nguyen [email protected]