Scalable Event Analytics with MongoDB & Ruby on Rails
Download as PPTX, PDF117 likes28,406 views
The document discusses scaling event analytics applications using Ruby on Rails and MongoDB. It describes how the author's startup initially used a standard Rails architecture with MySQL, but ran into performance bottlenecks. It then explores solutions like replication, sharding, key-value stores and Hadoop, but notes the development challenges with each approach. The document concludes that using MongoDB provided scalable writes, flexible reporting and the ability to keep the application as "just a Rails app". MongoDB's sharding allows scaling to high concurrency levels with increased storage and transaction capacity across shards.
Scalable Event Analytics with MongoDB & Ruby on Rails
- 1. Scalable Event Analytics with Ruby on Rails & MongoDBRuby Conf China 2010Jared Rosoff (@forjared) [email protected]
- 3. OverviewRuby at ScaleWhat is Event Analytics? What are the different ways you could do it? How we did it
- 5. Event AnalyticsData SourceEventUserEventEventQueryEvent AnalyticsData SourceReportEventEventEventEventUserQueryData SourceEventEventReportEventEventEventData Source
- 7. Oh and we are a startup
- 9. Rails default architectureCollection ServerData SourceMySQLUserReporting Server
- 10. Rails default architectureCollection ServerData SourceMySQLUserReporting Server“Just” a Rails App
- 11. Rails default architecture Performance Bottleneck: Too much loadCollection ServerData SourceMySQLUserReporting Server“Just” a Rails App
- 12. Let’s add replication!MySQLMasterCollection ServerData SourceReplicationMySQLMasterUserReporting ServerMySQLMasterMySQLMaster
- 13. Let’s add replication!MySQLMasterCollection ServerData SourceReplicationMySQLMasterUserReporting ServerMySQLMasterMySQLMasterOff the shelf!Scalable Reads!
- 14. Let’s add replication! Performance Bottleneck: Still can’t scale writesMySQLMasterCollection ServerData SourceReplicationMySQLMasterUserReporting ServerMySQLMasterMySQLMasterOff the shelf!Scalable Reads!
- 15. What about sharding?Collection ServerData SourceShardingMySQLMasterMySQLMasterMySQLMasterUserReporting ServerSharding
- 16. What about sharding?Collection ServerData SourceShardingMySQLMasterMySQLMasterMySQLMasterUserReporting ServerShardingScalable Writes!
- 17. What about sharding? Development Bottleneck:Need to write custom codeCollection ServerData SourceShardingMySQLMasterMySQLMasterMySQLMasterUserReporting ServerShardingScalable Writes!
- 18. Key Value stores to the rescue?Collection ServerData SourceMySQLMasterMySQLMasterCassandra orVoldemortUserReporting Server
- 19. Key Value stores to the rescue?Collection ServerData SourceMySQLMasterMySQLMasterCassandra orVoldemortUserReporting ServerScalable Writes!
- 20. Key Value stores to the rescue? Development Bottleneck:Reporting is limited / hardCollection ServerData SourceMySQLMasterMySQLMasterCassandra orVoldemortUserReporting ServerScalable Writes!
- 21. Can I Hadoop my way out of this?MySQLMasterMySQLMasterCassandra orVoldemortCollection ServerData SourceHadoopMySQLMasterMySQLMasterUserReporting ServerMySQLMasterMySQLSlave
- 22. Can I Hadoop my way out of this?MySQLMasterMySQLMasterCassandra orVoldemortCollection ServerData SourceHadoopMySQLMasterScalable Writes!MySQLMasterUserReporting ServerMySQLMasterMySQLSlave
- 23. Can I Hadoop my way out of this?MySQLMasterMySQLMasterCassandra orVoldemortCollection ServerData SourceHadoopMySQLMasterScalable Writes!Flexible Reports!MySQLMasterUserReporting ServerMySQLMasterMySQLSlave
- 24. Can I Hadoop my way out of this?MySQLMasterMySQLMasterCassandra orVoldemortCollection ServerData SourceHadoopMySQLMasterScalable Writes!Flexible Reports!“Just” a Rails AppMySQLMasterUserReporting ServerMySQLMasterMySQLSlave
- 25. Can I Hadoop my way out of this? Development Bottleneck:Too many systems!MySQLMasterMySQLMasterCassandra orVoldemortCollection ServerData SourceHadoopMySQLMasterScalable Writes!Flexible Reports!“Just” a Rails AppMySQLMasterUserReporting ServerMySQLMasterMySQLSlave
- 26. MongoDB! Collection ServerData SourceMySQLMasterMySQLMasterMongoDBUserReporting Server
- 27. MongoDB! Collection ServerData SourceMySQLMasterMySQLMasterMongoDBUserReporting ServerScalable Writes!
- 28. MongoDB! Collection ServerData SourceMySQLMasterMySQLMasterMongoDBUserReporting ServerScalable Writes!Flexible Reporting!
- 29. MongoDB! Collection ServerData SourceMySQLMasterMySQLMasterMongoDBUserReporting ServerScalable Writes!“Just” a rails appFlexible Reporting!
- 32. MongoDApp ServerData SourceCollectionMongoDLoadBalancerPassengerNginxMongosReportingUserMongoDSharding!High Concurrency
- 33. MongoDApp ServerData SourceCollectionMongoDLoadBalancerPassengerNginxMongosReportingUserMongoDSharding!High ConcurrencyScale-Out
- 34. MongoDBSharding
- 35. MongoDBShardingReplica Sets let us scale storage & transaction capacity for each shard
- 36. MongoDBShardingReplica Sets let us scale storage & transaction capacity for each shardMongos routes transactions to shards based on “shard key”
- 37. MongoDBShardingReplica Sets let us scale storage & transaction capacity for each shardMongos routes transactions to shards based on “shard key”Config servers store information about which shards exist
- 38. Inserting32Shard key == namebob Shard 21insert { ‘name’ : bob }Insert { ‘name’ : bob }
- 39. Querying32Shard key == namebob Shard 21Query { ‘name’ : bob }Query { ‘name’ : bob }
- 40. Map Reduce22221Map-reduce( … ) Map-reduce(…)
- 41. Working with MongoMongoMapper makes it look like ActiveRecordDocuments are more natural than rows in many casesMap-Reduce rocks (but needs better support in rails)http://www.flickr.com/photos/elhamalawy/2526783078/
- 43. RubyMongo
- 44. Runs over all the objects in the views table, counting how many times a page was viewedAdds up all the counts for a unique url / date combinationRun the map reduce job and return a collection containing the results
- 45. ResultsVersion 1 of our analytics system took 2 weeks with 1 engineer We have since added a lot more complexity, but we did it incrementallyWe replaced MySQL entirely with MongoDBNo need for joins, transactions Every table is now a document collectionIt’s fast! 63ms – Average response time for sending data to server93ms – Average response time for displaying reports