Building LinkedIn's Learning Platform with MongoDB
5 likes•10,585 views
This document discusses how LinkedIn's internal learning portal LearnIn was developed using Node.js, MongoDB, and Elasticsearch. It covers using Mongoose to model schemas and build APIs in MongoDB. To enable full-text search, an Elasticsearch river was configured to replicate data from MongoDB to Elasticsearch for improved search capabilities like relevancy scoring. The document provides code examples of setting up Mongoose schemas, queries, and configuring the Elasticsearch river.
![var AssetSchema = new BaseModel.BaseSchema({
title: {
type: String,
required: true,
trim: true
},
type: {
type: String,
default: 'general',
enum: typeEnums
},
relatedAssets: [
{
type: ObjectId,
ref: 'Asset'
}
]
}, {
collection: 'assets'
});
31](https://image.slidesharecdn.com/zwon4cpxsfchxbaq7i79-signature-41934cbc7379ca067d76322c42d073c0dc50c41dfb1c7e975c9c0301f0bb145e-poli-140707141118-phpapp01/85/Building-LinkedIn-s-Learning-Platform-with-MongoDB-31-320.jpg)
![CourseSchema = new BaseModel.BaseSchema({
groups: [
{
title: {
type: String
},
items: [
{
asset: {
type: ObjectId,
ref: 'Asset'
},
task: {
type: ObjectId,
ref: 'Task'
}
}
]
}
]
}, {
collection: 'courses'
});
// nested population
function getCourseById(id, callback) {
Model
.findById(id)
.populate('groups.items.asset groups.items.task’)
.exec(callback);
}
32](https://image.slidesharecdn.com/zwon4cpxsfchxbaq7i79-signature-41934cbc7379ca067d76322c42d073c0dc50c41dfb1c7e975c9c0301f0bb145e-poli-140707141118-phpapp01/85/Building-LinkedIn-s-Learning-Platform-with-MongoDB-32-320.jpg)
![{ "_id" : ObjectId("51c0b755d292ad5a7a000039"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"),
"description" : "We'll discuss what is required to work in a global context at LinkedIn and how managers can become effective global leaders? ", "image_url" : "/media/images/comingsoon/
thumbnails_coming-soon3.jpg", "longdescription" : "What is required to work in a global context at LinkedIn? What are the personal capabilities that enable a global manager to succeed? How
do managers begin to develop a global perspective that will enable them to become effective global leaders? Please come prepared to discuss the following questions:nn(a) In your job, what
will you be doing differently to reflect the global nature of our business?n(b) What are some things we do as a company that still take a US-centric approach, and what would you recommend
we do to change that?", "state" : "draft", "tag_suggest" : { "input" : [ "Management & Leadership" ] }, "tags" : [ "Management & Leadership" ], "title" : "Think Global with Arvind Rajan", "type" :
"slideshow" }
{ "_id" : ObjectId("51c0b755d292ad5a7a00003a"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"),
"description" : "We'll discuss how to handle difficult conversations with employees regarding performance or not meeting expectations", "image_url" : "/media/images/comingsoon/
thumbnails_coming-soon3.jpg", "longdescription" : "We'll discuss how to handle difficult conversations with employees regarding performance or not meeting expectations", "state" : "draft",
"tag_suggest" : { "input" : [ "Management & Leadership" ] }, "tags" : [ "Management & Leadership" ], "title" : "Handling Difficult Conversations with Cliff Rosenberg", "type" : "slideshow" }
{ "_id" : ObjectId("51c0b755d292ad5a7a00003b"), "academy" : "Leadership", "adder" : "", "applyContent" : "", "author" : "", "clicks" : 8, "dateCreated" : ISODate("2013-06-18T19:39:01Z"),
"date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "Containing vital performance and career lessons for managers at every level; this book presents the remarkable
findings of Gallup's massive in-depth study of great managers across a wide variety of situations.", "enableEnrichedAsset" : false, "image_url" : "/media/images/learningassets/leadership/
12.jpg", "longdescription" : "(b) What are some things we do as a company that still take a US-centric approach, and what would you recommend we do to change that?", "practiceContent" : "",
"state" : "live", "tag_suggest" : { "input" : [ "Editor", "Leadership & Management", "Books & Articles", "Transitioning to Management", "Management & Leadership" ] },
"tags" : [ "Editor", "Leadership & Management", "Books & Articles", "Transitioning to Management", "Management & Leadership" ], "thinkContent" : "", "title" : "First, Break All
the Rules: What the World's Greatest Managers Do Differently", "type" : "book", "url" : "https://linkedin.okta.com/app/template_saml_2_0/k2bnhvkoMCFSHKCKIJUA/sso/saml?
RelayState=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Flinkedin.skillport.com%2Fskillportfe%2Fcustom%2Flogin%2Flinkedin%2Flogin.action%3Fcourseaction%3DLaunch%26assetid%3D_ss_book%3A15501" }
{ "_id" : ObjectId("51c0b755d292ad5a7a00003c"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"),
"description" : "We'll discuss the FCS model for managing hyper-growth. Fewer things done better. Communicating the right information to the right person at the right time.", "image_url" : "/
media/images/comingsoon/thumbnails_coming-soon3.jpg", "longdescription" : "We'll discuss the FCS model for managing hyper-growth. Fewer things done better. Communicating the right
information to the right person at the right time.", "state" : "draft", "tag_suggest" : { "input" : [ "Management & Leadership" ] }, "tags" : [ "Management & Leadership" ], "title" : "Managing Hyper-
growth with Jeff Weiner", "type" : "slideshow" }
{ "_id" : ObjectId("51c0b755d292ad5a7a00003d"), "academy" : "Leadership", "adder" : "", "applyContent" : "", "author" : "", "clicks" : 24, "dateCreated" : ISODate("2013-06-18T19:39:01Z"),
"date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "David shares learnings from the leaders that have inspired him and examples from his own career. ",
"enableEnrichedAsset" : false, "image_url" : "/media/images/learningassets/leadership/43.jpg", "longdescription" : "David shares learnings from the leaders that have inspired him and
examples from his own career. ", "practiceContent" : "", "recommendations" : [ ObjectId("521e6fd66f13a29715000044") ], "state" : "live", "tag_suggest" : { "input" : [ "Editor", "Leadership &
Management", "Videos", "Leadership 101", "Leaders Teaching Leaders", "LTL", "Management & Leadership" ] }, "tags" : [ "Editor", "Leadership & Management",
"Videos", "Leadership 101", "Leaders Teaching Leaders", "LTL", "Management & Leadership" ], "thinkContent" : "", "title" : "Leadership 101 with David Henke", "type" :
"video", "url" : "http://innertube.linkedin.biz/video/Leadership101_2013_0118/index.html" }
{ "_id" : ObjectId("51c0b755d292ad5a7a00003e"), "academy" : "Professional", "author" : "EMEA", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" :
ISODate("2014-02-07T16:35:00.173Z"), "description" : "Learn how to apply tips and tools for using your time more efficiently, meeting deadlines and increasing productivity, including learning
how to use MS Outlook more effectively.nn", "image_url" : "/media/images/learningassets/professional/36_time_management.jpg", "longdescription" : "long description", "options" :
[ "hidden" ], "state" : "draft", "tag_suggest" : { "input" : [ "Professional Development", "Learning Programs & Classes" ] }, "tags" : [ "Professional Development", "Learning Programs &
Classes" ], "title" : "Time Management ", "type" : "article", "url" : "https://linkedin.okta.com/app/template_saml_2_0/k166h5vsMZMIUWGSDZSJ/sso/saml?RelayState=%2fdeeplink
%2fssodeeplink.aspx%3fmodule%3dtranscript%26loid%3d919c0d74-4d51-43bd-87c9-546896287d49" }
{ "_id" : ObjectId("51c0b755d292ad5a7a000039"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"),
"description" : "We'll discuss what is required to work in a global context at LinkedIn and how managers can become effective global leaders? ", "image_url" : "/media/images/comingsoon/
thumbnails_coming-soon3.jpg", "longdescription" : "What is required to work in a global context at LinkedIn? What are the personal capabilities that enable a global manager to succeed? How
do managers begin to develop a global perspective that will enable them to become effective global leaders? Please come prepared to discuss the following questions:nn(a) In your job, what
will you be doing differently to reflect the global nature of our business?n(b) What are some things we do as a company that still take a US-centric approach, and what would you recommend
How do we search?
36](https://image.slidesharecdn.com/zwon4cpxsfchxbaq7i79-signature-41934cbc7379ca067d76322c42d073c0dc50c41dfb1c7e975c9c0301f0bb145e-poli-140707141118-phpapp01/85/Building-LinkedIn-s-Learning-Platform-with-MongoDB-36-320.jpg)
![Configuring the River
$ curl -XPUT “localhost:9200/_river/<RIVER_NAME>/_meta” -d ‘
{
"type": "mongodb",
"mongodb": {
"servers": [
{
"host": "< HOST >",
"port": "< PORT >"
}
],
"db": "learnin",
"options": {
"import_all_collections": "true"
}
},
"index": {
"name": "prodindexnew"
}
}’
49](https://image.slidesharecdn.com/zwon4cpxsfchxbaq7i79-signature-41934cbc7379ca067d76322c42d073c0dc50c41dfb1c7e975c9c0301f0bb145e-poli-140707141118-phpapp01/85/Building-LinkedIn-s-Learning-Platform-with-MongoDB-49-320.jpg)
Building LinkedIn's Learning Platform with MongoDB
- 1. 1
- 2. Jacob Dejno Web Developer Ryan Seamons Product Manager 2
- 3. What we will not cover: • Scaling MongoDB • High load volume and QPS • Using MongoDB for analytics • MongoDB Production/Development Deployments • Big Data. 3
- 4. We will cover: • LearnIn, LinkedIn’s Internal Learning Portal • Using Node.js and MongoDB • Mongoose ODM • Move to Elasticsearch 4
- 5. Quick API development using MongoDB 5
- 6. What’s your next play? 6
- 7. 7
- 8. 8
- 9. 9
- 10. 10
- 11. What technologies should we use? 11
- 12. How do we leverage our current knowledge? 12
- 13. What server and database? 13
- 14. How did we build this… quickly? 14
- 15. Let’s start with server selection… which one? 15
- 16. 16
- 17. Node.js • Server-side JavaScript • Lightweight and quick to setup • NPM package support • Extensive documentation and community • Allows for data-driven JSON template rendering • Easy REST API creation using Express.js 17
- 19. LearnIn’s Database Needs • Support 5,000+ employees, globally • Minimal data storage needs • Many more reads that writes • Able to scale with company growth • Flexibility 19
- 20. 20
- 21. MongoDB • Easy to setup • NoSQL with JSON structure • Advanced querying • Flexible schema, schema-less • Extensive documentation • High Single instance thresholds 21
- 22. 22
- 24. mongoose 24
- 25. Mongoose • Object Document Mapper with easy type casting • Quick to setup, replaces Mongo JS driver • Easy document modeling and field validation API • Business logic hooks, Custom Middleware • Mongo _id reference population 25
- 26. var mongoose = require(‘mongoose’), ! connection = mongoose.connect('mongodb://localhost:27017/test'); 26
- 27. var mongoose = require(‘mongoose’), Schema = mongoose.Schema ! ! function BaseSchema() { Schema.apply(this, arguments); this.add({ publisher: { type: ObjectId, ref: 'User' } }); } Document Modeling & Validation var AssetSchema = new BaseModel.BaseSchema({ title: { type: String, required: true, trim: true }, description: { type: String, required: true, trim: true } },{ collection: 'assets' }); 27
- 28. this.pre('save', function (next) { if ( !this.publisher ) { this.publisher = this.lastUpdatedByUser; } // Continue on to next function in queue next(); }); Business Logic Hooks 28
- 29. function queryAssets(query, callback) { model .find(query) // query is JSON object .populate(‘publisher’) // populate publisher from ‘users’ collection .exec(function (error, resultSet) { return callback && callback(error, resultSet); }); } NoSQL Reference Population 29
- 30. Our Schema Design with Mongoose Normalized Data Modeling Many-to-many relationships Collection per data type 30
- 31. var AssetSchema = new BaseModel.BaseSchema({ title: { type: String, required: true, trim: true }, type: { type: String, default: 'general', enum: typeEnums }, relatedAssets: [ { type: ObjectId, ref: 'Asset' } ] }, { collection: 'assets' }); 31
- 32. CourseSchema = new BaseModel.BaseSchema({ groups: [ { title: { type: String }, items: [ { asset: { type: ObjectId, ref: 'Asset' }, task: { type: ObjectId, ref: 'Task' } } ] } ] }, { collection: 'courses' }); // nested population function getCourseById(id, callback) { Model .findById(id) .populate('groups.items.asset groups.items.task’) .exec(callback); } 32
- 33. function getAsset(req, res) { if (!req.params.id) { return res.send(500, 'Please add a asset ID.'); } assetModel.model.findOne({ _id: req.params.id }).populate('relatedAssets').exec(function (err, asset) { if(err){ res.send(500, err); return; } res.json(200, asset); }); } app.get('/api/asset/:id', controller.getAsset); Route: Controller: 33
- 34. 34
- 35. 35
- 36. { "_id" : ObjectId("51c0b755d292ad5a7a000039"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "We'll discuss what is required to work in a global context at LinkedIn and how managers can become effective global leaders? ", "image_url" : "/media/images/comingsoon/ thumbnails_coming-soon3.jpg", "longdescription" : "What is required to work in a global context at LinkedIn? What are the personal capabilities that enable a global manager to succeed? How do managers begin to develop a global perspective that will enable them to become effective global leaders? Please come prepared to discuss the following questions:nn(a) In your job, what will you be doing differently to reflect the global nature of our business?n(b) What are some things we do as a company that still take a US-centric approach, and what would you recommend we do to change that?", "state" : "draft", "tag_suggest" : { "input" : [ "Management & Leadership" ] }, "tags" : [ "Management & Leadership" ], "title" : "Think Global with Arvind Rajan", "type" : "slideshow" } { "_id" : ObjectId("51c0b755d292ad5a7a00003a"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "We'll discuss how to handle difficult conversations with employees regarding performance or not meeting expectations", "image_url" : "/media/images/comingsoon/ thumbnails_coming-soon3.jpg", "longdescription" : "We'll discuss how to handle difficult conversations with employees regarding performance or not meeting expectations", "state" : "draft", "tag_suggest" : { "input" : [ "Management & Leadership" ] }, "tags" : [ "Management & Leadership" ], "title" : "Handling Difficult Conversations with Cliff Rosenberg", "type" : "slideshow" } { "_id" : ObjectId("51c0b755d292ad5a7a00003b"), "academy" : "Leadership", "adder" : "", "applyContent" : "", "author" : "", "clicks" : 8, "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "Containing vital performance and career lessons for managers at every level; this book presents the remarkable findings of Gallup's massive in-depth study of great managers across a wide variety of situations.", "enableEnrichedAsset" : false, "image_url" : "/media/images/learningassets/leadership/ 12.jpg", "longdescription" : "(b) What are some things we do as a company that still take a US-centric approach, and what would you recommend we do to change that?", "practiceContent" : "", "state" : "live", "tag_suggest" : { "input" : [ "Editor", "Leadership & Management", "Books & Articles", "Transitioning to Management", "Management & Leadership" ] }, "tags" : [ "Editor", "Leadership & Management", "Books & Articles", "Transitioning to Management", "Management & Leadership" ], "thinkContent" : "", "title" : "First, Break All the Rules: What the World's Greatest Managers Do Differently", "type" : "book", "url" : "https://linkedin.okta.com/app/template_saml_2_0/k2bnhvkoMCFSHKCKIJUA/sso/saml? RelayState=https%3A%2F%2Ffanyv88.com%3A443%2Fhttps%2Flinkedin.skillport.com%2Fskillportfe%2Fcustom%2Flogin%2Flinkedin%2Flogin.action%3Fcourseaction%3DLaunch%26assetid%3D_ss_book%3A15501" } { "_id" : ObjectId("51c0b755d292ad5a7a00003c"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "We'll discuss the FCS model for managing hyper-growth. Fewer things done better. Communicating the right information to the right person at the right time.", "image_url" : "/ media/images/comingsoon/thumbnails_coming-soon3.jpg", "longdescription" : "We'll discuss the FCS model for managing hyper-growth. Fewer things done better. Communicating the right information to the right person at the right time.", "state" : "draft", "tag_suggest" : { "input" : [ "Management & Leadership" ] }, "tags" : [ "Management & Leadership" ], "title" : "Managing Hyper- growth with Jeff Weiner", "type" : "slideshow" } { "_id" : ObjectId("51c0b755d292ad5a7a00003d"), "academy" : "Leadership", "adder" : "", "applyContent" : "", "author" : "", "clicks" : 24, "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "David shares learnings from the leaders that have inspired him and examples from his own career. ", "enableEnrichedAsset" : false, "image_url" : "/media/images/learningassets/leadership/43.jpg", "longdescription" : "David shares learnings from the leaders that have inspired him and examples from his own career. ", "practiceContent" : "", "recommendations" : [ ObjectId("521e6fd66f13a29715000044") ], "state" : "live", "tag_suggest" : { "input" : [ "Editor", "Leadership & Management", "Videos", "Leadership 101", "Leaders Teaching Leaders", "LTL", "Management & Leadership" ] }, "tags" : [ "Editor", "Leadership & Management", "Videos", "Leadership 101", "Leaders Teaching Leaders", "LTL", "Management & Leadership" ], "thinkContent" : "", "title" : "Leadership 101 with David Henke", "type" : "video", "url" : "http://innertube.linkedin.biz/video/Leadership101_2013_0118/index.html" } { "_id" : ObjectId("51c0b755d292ad5a7a00003e"), "academy" : "Professional", "author" : "EMEA", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "Learn how to apply tips and tools for using your time more efficiently, meeting deadlines and increasing productivity, including learning how to use MS Outlook more effectively.nn", "image_url" : "/media/images/learningassets/professional/36_time_management.jpg", "longdescription" : "long description", "options" : [ "hidden" ], "state" : "draft", "tag_suggest" : { "input" : [ "Professional Development", "Learning Programs & Classes" ] }, "tags" : [ "Professional Development", "Learning Programs & Classes" ], "title" : "Time Management ", "type" : "article", "url" : "https://linkedin.okta.com/app/template_saml_2_0/k166h5vsMZMIUWGSDZSJ/sso/saml?RelayState=%2fdeeplink %2fssodeeplink.aspx%3fmodule%3dtranscript%26loid%3d919c0d74-4d51-43bd-87c9-546896287d49" } { "_id" : ObjectId("51c0b755d292ad5a7a000039"), "academy" : "Leadership", "dateCreated" : ISODate("2013-06-18T19:39:01Z"), "date_es_indexed" : ISODate("2014-02-07T16:35:00.173Z"), "description" : "We'll discuss what is required to work in a global context at LinkedIn and how managers can become effective global leaders? ", "image_url" : "/media/images/comingsoon/ thumbnails_coming-soon3.jpg", "longdescription" : "What is required to work in a global context at LinkedIn? What are the personal capabilities that enable a global manager to succeed? How do managers begin to develop a global perspective that will enable them to become effective global leaders? Please come prepared to discuss the following questions:nn(a) In your job, what will you be doing differently to reflect the global nature of our business?n(b) What are some things we do as a company that still take a US-centric approach, and what would you recommend How do we search? 36
- 37. > db.collection.find({ title: new RegExp(‘<search query>’, ‘i’) }); 37
- 39. 39 > db.collection.ensureIndex({ <field>: “text" })
- 40. 40 > db.collection.find({ $text: { $search: <string> } });
- 41. • Easy searching by creating MongoDB index • Relevancy scoring • Stemming and multi-language support What text search gives us: 41
- 42. • Single tokenizer/analyzer • Simple relevancy scoring, but not using Lucene • No completion suggestion • No fuzzy matching • No related item search Some limitations… 42
- 43. What we needed: • Lucene index relevancy scoring and performance • Custom field analyzers for tokenization and stemming • ‘Related to’ or ‘More like this’ querying • Quick completion suggestions • Complicated wildcard searching • Easy Node.js integration 43
- 44. 44
- 45. 45 ?
- 47. How does it work? 47
- 48. Replica Set OpLog Primary Secondary Replica Index ES River 48
- 49. Configuring the River $ curl -XPUT “localhost:9200/_river/<RIVER_NAME>/_meta” -d ‘ { "type": "mongodb", "mongodb": { "servers": [ { "host": "< HOST >", "port": "< PORT >" } ], "db": "learnin", "options": { "import_all_collections": "true" } }, "index": { "name": "prodindexnew" } }’ 49
- 53. 53
- 55. 55
- 56. Contact linkedin.com/in/rseamons ! @ryanseamons linkedin.com/in/dejno ! @dejno Jacob Dejno Web Developer Ryan Seamons Product Manager 56
- 57. Questions? 57
- 58. Appendix • MongoDB NoSQL Database • Mongoose ODM • Node.js server • Express.js WebApp Framework • MongoDB Full-Text Search • Elasticsearch • Elasticsearch-River-MongoDB • Elasticsearch Rivers • These Slides 58
- 59. ©2014 LinkedIn Corporation. All Rights Reserved. 59