Scribd
Upload
9 views34 pages

Dokumen - Tips - Mmds 2014 Talk Distributing ML Algorithms From Gpus To The Cloud

This document summarizes distributing large-scale machine learning algorithms from GPUs to the cloud. It discusses Netflix's interest in high quality recommendations and improving rating prediction accuracy. It outlines the scale of Netflix's data and types of models used. It then discusses three levels of distributing and parallelizing algorithms - by population subset, hyperparameters, and training data. Finally, it provides an example of distributing artificial neural network training over GPUs and AWS, addressing questions of data and computation distribution and latency requirements.

Uploaded by

juan aguirre
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
9 views34 pages

Dokumen - Tips - Mmds 2014 Talk Distributing ML Algorithms From Gpus To The Cloud

This document summarizes distributing large-scale machine learning algorithms from GPUs to the cloud. It discusses Netflix's interest in high quality recommendations and improving rating prediction accuracy. It outlines the scale of Netflix's data and types of models used. It then discusses three levels of distributing and parallelizing algorithms - by population subset, hyperparameters, and training data. Finally, it provides an example of distributing artificial neural network training over GPUs and AWS, addressing questions of data and computation distribution and latency requirements.

Uploaded by

juan aguirre
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 34

Distributing Large-scale ML

Algorithms: from GPUs to the


Cloud

MMDS 2014
June, 2014

Xavier Amatriain
Director - Algorithms Engineering @xamat
Outline
■ Introduction
■ Emmy-winning Algorithms
■ Distributing ML Algorithms in Practice
■ An example: ANN over GPUs & AWS Cloud
What we were interested in:
■ High quality recommendations
Proxy question:
■ Accuracy in predicted rating
■ Improve by 10% = $1million!
Data size:
■ 100M ratings (back then “almost massive”)
2006 2014
Netflix Scale
▪ > 44M members
▪ > 40 countries
▪ > 1000 device types
▪ > 5B hours in Q3 2013
▪ Plays: > 50M/day
▪ Searches: > 3M/day
▪ Ratings: > 5M/day
▪ Log 100B events/day
▪ 31.62% of peak US downstream
traffic
Smart Models ■ Regression models (Logistic,
Linear, Elastic nets)
■ GBDT/RF
■ SVD & other MF models
■ Factorization Machines
■ Restricted Boltzmann Machines
■ Markov Chains & other graphical
models
■ Clustering (from k-means to
modern non-parametric models)
■ Deep ANN
■ LDA
■ Association Rules
■ …
“Emmy Winning”

Netflix Algorithms
Rating Prediction
2007 Progress Prize
▪ Top 2 algorithms
▪ MF/SVD - Prize RMSE: 0.8914
▪ RBM - Prize RMSE: 0.8990

▪ Linear blend Prize RMSE: 0.88


▪ Currently in use as part of Netflix’ rating prediction component
▪ Limitations
▪ Designed for 100M ratings, we have 5B ratings
▪ Not adaptable as users add ratings
▪ Performance issues
Ranking
Ranking
Page composition
Similarity
Search Recommendations
Postplay
Gamification
Distributing ML algorithms in practice
1. Do I need all that data?

2. At what level should I distribute/parallelize?

3. What latency can I afford?


Do I need all that data?
Really?

Anand Rajaraman: Former Stanford Prof. &


Senior VP at Walmart
Sometimes, it’s not
about more data
[Banko and Brill, 2001]

Norvig: “Google does not


have better Algorithms,
only more Data”

Many features/
low-bias models
Sometimes, it’s not
about more data
At what level should I parallelize?
The three levels of Distribution/Parallelization

1. For each subset of the population (e.g.


region)
2. For each combination of the
hyperparameters
3. For each subset of the training data

Each level has different requirements


Level 1 Distribution
■ We may have subsets of the
population for which we need to
train an independently optimized
model.
■ Training can be fully distributed
requiring no coordination or data
communication
Level 2 Distribution
■ For a given subset of the population we
need to find the “optimal” model
■ Train several models with different
hyperparameter values
■ Worst-case: grid search
■ Can do much better than this (E.g. Bayesian
Optimization with Gaussian Process Priors)
■ This process *does* require coordination
■ Need to decide on next “step”
■ Need to gather final optimal result
■ Requires data distribution, not sharing
Level 3 Distribution
■ For each combination of
hyperparameters, model training may
still be expensive
■ Process requires coordination and
data sharing/communication
■ Can distribute computation over
machines splitting examples or
parameters (e.g. ADMM)
■ Or parallelize on a single multicore
machine (e.g. Hogwild)
■ Or… use GPUs
ANN Training over GPUS and AWS
ANN Training over GPUS and AWS
■ Level 1 distribution: machines over different AWS
regions
■ Level 2 distribution: machines in AWS and same AWS
region
■ Use coordination tools
■ Spearmint or similar for parameter optimization
■ Condor, StarCluster, Mesos… for distributed cluster coordination
■ Level 3 parallelization: highly optimized parallel CUDA
code on GPUs
What latency can I afford?
3 shades of latency
▪ Blueprint for multiple
algorithm services
▪ Ranking
▪ Row selection
▪ Ratings
▪ Search
▪ …

▪ Multi-layered Machine
Learning
Matrix Factorization Example
Xavier Amatriain (@xamat)
[email protected]

Thanks!
(and yes, we are hiring)

You might also like