Django Doesn't Scale

Transcript

1. Django Doesn’t Scale! (and what you can do about it)

   Jacob Kaplan-Moss [email protected] 1

2. 2

3. Rails doesn’t scale. 3

4. Django doesn’t scale. 4

5. Frameworks don’t scale! 5

6. 6 DjangoCon 2008 70 Cal Henderson, Why I hate Django

   http://www.youtube.com/watch?v=i6Fr65PFqfk

7. 7 Cal Henderson, Why I hate Django http://www.youtube.com/watch?v=i6Fr65PFqfk DjangoCon 2008

   71

8. 8 Cal Henderson, Why I hate Django http://www.youtube.com/watch?v=i6Fr65PFqfk DjangoCon 2008

   72

9. 9 “Measure twice, cut once.”

10. 10 front-end rendering time database load resource contention ddos attack

    network latency bugs in your software bugs in third party software slow template rendering misconfigured servers not enough threads “Our site’s too slow!”

11. Data collection 11 •logging http://docs.python.org/library/logging •Sentry http://sentry.readthedocs.org/ •python-statsd http://packages.python.org/python-statsd/ •mmstats

    http://mmstats.readthedocs.org/ https://github.com/schmichael/django-mmstats/ •Metrology http://metrology.readthedocs.org

12. import time from metrology import Metrology http_ok = Metrology.counter('http.ok') http_err

    = Metrology.counter('http.err') response_time = Metrology.histogram('request.time') class RequestMetricsMiddleware(object): def process_request(self, request): request._start_time = time.time() def process_response(self, request, response): response_time.update(time.time() - request._start_time) if 200 <= response.status_code < 400: http_ok.increment() else: http_err.increment() return response def process_exception(self, request, exception): http_err.increment() 12

13. Graphite http://graphite.readthedocs.org/ 13 http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/

14. 14

15. Be lazy. 15

16. Required viewing: “Cache rules everything around me” Jacob Burch, Noah

    Silas http://pyvideo.org/video/679 16

17. 17 “But my social pinstafacegramiqus isn’t cachable!

18. 18 cacheable cacheable

19. 19 not cacheable not cacheable

20. 20 <div id="header"> Hello, {{ user.name }} </div>

21. 21 Technique: resource decomposition.

22. •Edge-side includes http://en.wikipedia.org/wiki/Edge_Side_Includes https://github.com/mrfunyon/django-esi •Two-phased template rendering http://www.holovaty.com/writing/django-two-phased-rendering/ http://django-phased.readthedocs.org/ •Client-side

    composition 22 Decomposition options

23. 23 <div id="header"> <esi:include src="/user/username" /> </div> Edge-side includes

24. Two-phased rendering <div id="header"> {% load phased_tags %} {% phased

    with user %} Hello, {{ user.name }} {% endphased %} </div> 24

25. Client-side composition <div id="header">Hello</div> <script type="text/javascript"> $(function() { $.get("/user/username", function(data)

    { $("#header").html(data); }); }); </script> 25

26. “ ” There are only two hard things in computer

    science: cache invalidation, naming things, and off-by-one errors. 26

27. 27 from books.models import Book from django.shortcuts import render def

    book_list(request): qs = Book.objects.all() return render(request, 'books.html', {'books': qs})

28. 28 from books.models import Book from django.shortcuts import render from

    django.views.decorators.cache import cache_page @cache_page(600) def book_list(request): qs = Book.objects.all() return render(request, 'books.html', {'books': qs})

29. from books.models import book from django.core.cache import cache def create_book():

    Book.objects.create(...) cache.delete("U MAD?") 29

30. “ ” There are only two hard things in computer

    science: cache invalidation, naming things, and off-by-one errors. 30

31. Technique: serve everything from cache. 31

32. 32 Cache cache outdated? View Regenerate cache data key:arg:arg  

         -­‐>  (ttl,  data) username:jacob        -­‐>  (1342710010,  "JKM") Data changed

33. Procrastinate. 33

34. 34 Cache cache outdated? View Regenerate cache data Hang on,

    this might take a while...

35. “Eventual consistency” 35

36. “Perceived performance” 36

37. 37 http://celeryproject.org/

38. 38 def message_list(request): messages = Message.objects.filter(recipient=request.user) messages.update(received=True) return render(...)

39. 39 Rule of thumb: Writes are 10x as expensive as

    reads* * This isn’t actually true, but you should pretend it is anyway.

40. 40 @task def mark_received(message_ids): Message.objects.filter(id__in=message_ids) \ .update(received=True) def message_list(request): messages

    = Message.objects.filter(recipient=request.user) mark_received.delay(messages.values_list('id', flat=True)) return render(...)

41. Watch that query count! 41

42. If you can only monitor one metric, make it query

    count. 42

43. 43 request response query query query

44. 44 network latency query time network latency

45. 45 Your new BFFs: http://django.me/select_related http://django.me/prefetch_related https://django.me/raw

46. Don’t fear the DB. 46

47. “ORM is the Vietnam of Computer Science” 47 http://blogs.tedneward.com/2006/06/26/The+Vietnam+Of+Computer+Science.aspx

48. ORM inefficiencies •Queryset cloning Copying a queryset requires cloning a

    heavy structure. Model.objects.filter(...).filter(...).order_by(...) •Model instantiation ~40k __init__s per second (see http://bit.ly/Muepgo). •Saving models Readable, but slow: m.foo  =  bar;  m.save() Much much faster: Model.objects.filter(id=instance.id).update(foo=bar) 48

49. Bulk inserts • Horrifically slow: for  title  in  big_title_list:  

         Book.objects.create(title=title) • Faster, but still terrible: with  transaction.commit_on_success():        for  title  in  big_title_list:                Book.objects.create(title=title) • Fast: bl  =  [Book(title=t)  for  t  in  big_title_list] Book.objects.bulk_create(bl) • But COPY  FROM wins, hands down. http://www.postgresql.org/docs/9.1/static/sql-copy.html http://initd.org/psycopg/docs/usage.html#using-copy-to-and-copy-from 49

50. Django needs to be database-agnostic. You don’t. 50

51. Technique: managers and raw queries 51

52. 52 class BookManager(models.Manager): def inventory(self): return self.raw("... BIG-ASS QUERY ...")

53. Modern databases are incredible. 53 See, e.g. Schemaless SQL, Craig

    Kerstiens: http://klewel.com/conferences/djangocon-2012/index.php?talkID=29

54. 54 1. Measure everything. 2. Cache carefully. 3. Procrastinate. 4.

    Count queries. 5. Hug your DBA.

55. Thanks! [email protected] http://lanyrd.com/swqxb Office hour: 1:40pm, Expo Hall 55