Scribd
Upload
79 views236 pages

How A Small Team Scales Instagram

Mike Krieger describes Instagram's scaling efforts over its first 3 years with a small team. In year 1, they scaled up the database and caches to handle growth. In year 2, they scaled out the app tier across multiple servers. In years 3+, they focused on stability, rolled out new features like video, and integrated with Facebook while maintaining their core principles of prioritizing users and moving quickly.

Uploaded by

atkawa7
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
79 views236 pages

How A Small Team Scales Instagram

Mike Krieger describes Instagram's scaling efforts over its first 3 years with a small team. In year 1, they scaled up the database and caches to handle growth. In year 2, they scaled out the app tier across multiple servers. In years 3+, they focused on stability, rolled out new features like video, and integrated with Facebook while maintaining their core principles of prioritizing users and moving quickly.

Uploaded by

atkawa7
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 236

A Brief, Rapid History

of Scaling Instagram
(with a tiny team)
Mike Krieger
QConSF 2013

'_
' . _
,. ..
' _
,..

'
',
' ' ,
_ ' '' '
, , '
__ '
' '

_
Core principles
' ' '_ '
, _ ' '
' , ' ' _
' '
,'_
'
Early days
Year 1: Scaling Up
Year 2: Scaling Out
Year 3-present: Stability, Video, FB
Getting Started
.
. _,
'


, '
Django + Apache mod_wsgi
Postgres
Redis
Gearman
Memcached
Nginx
',
Django + uWSGI
Postgres
Redis
Celery
Memcached
HAproxy
' '
_ _'

'
Year 1: Scaling Up
_,
_

' _
_' '

'


.
' '_
. '' _
' _ ' _'
' _
,
'
' , ,
_ '

Scaling up DB
' ''

' ' '

'_ '
' ''_
_ '
''
Media
Likes
Comments
Everything else
' '
''
_' some '
' ' _ _
,' ' , ,
,
_' , , '
Scaling up Redis
,
,
' ''_ ,
' ',
, _' ',
', '_
'_ '_
' _''
_ '

Scaling Memcached
'' _
'
'' '
, '
_
'
Cong Management
& Deployment
_'
,, '
'
'
'
''

'_
'
Ending the year
__ '
'
'
'_ '_ '
' '
'_ '

Year 2: Scaling Out


App tier
'' ' '
,
, ''
' '
,

_
' '

' '
'' ' '' ''
Database Scale Out
' _
,
__' '


'_
, ''_ _
' _' _'
' ' '
_ ,. ' '
'
'

'_ ' , ,

' _'
'

' '
'
, ' ,

' ,
_
_ '_
' '
'
, _
_ ''_
' '
' '_
''
'
''
' '
. ' '
'_
'

'



',

',
.
',

',


',

',
.
',

',


',

',
.
',

',


',

',
.
',

',


',

',
.
',

',
'
',
. _ '
,
_'
'
No extra moving parts
64 bits max
Time ordered
Containing partition key
41 ' ' , ,
13 ' _
10 ' ''_
.
_' .
'

Lesson learned
'
''
'', ' ,
'_ '
_ '
Scaling congs/host
discovery


' ' '
''
.'_
'''
,_' ,
' '
'
' ' _'
' , ' _
Monitoring
' '
' ''
'' '
'
statsd.timer
statsd.incr
' _
''' ' ''
'_ '
' _ _
580 '' '
164 '' '
Ending the year

,_ '
',
'
'
, , _
'
Year 3+: Stability,
Video, FB
' ' '
'
Deployment &
Cong Management
,
'_ '

,

,
, ' ,
' '_'
' '


'
'' _ ,'


' _ _
,' '_
''_
Hit issue
Develop manual approach
Build tools to improve manual / hands on
approach
Replace manual with automated system
Monitoring
,
_ __
'_
,''_
''' '
___ _
'' _'_'
' '_
s = time.time()
# do work
statsd.incr("waittime.VIEWNAME.C
OMPONENT", time.time() - s)
',
Feeds and Inboxes

, '
, '
'

'
_
''
' . '

,' '

Knobs
,
_
,
,
,
, '
'
,
knobs.get(feature_name,
default)

Incremental feature rollouts


Dynamic page sizing (shedding load)
Feature killswitches
'
''
,
' '
Video
' ' ,
' ' _
' . '
'
_

'

_
'_

''
,, ' ' '
,
. '_
', ' '
_ '
', ''
'' '
_
___
' ,

,' ' '
, ' ' '
FB

,
,'
_'_
re.compile(f[o0][1l][o0]w)
' '_ ' '
'
_
'
',
, _
. '_
- ' '_
'

Wrap up
Core principles
' ' '_ '
, _ ' '
' , ' ' _
' '
,'_
Takeaways
' , '
, '
' ,
, ,
' _
,' '
_ ' '
_
,'_

You might also like