Celery

http://www.celeryproject.org/

( A python Library )
@ronaldlcheung

Tuesday, February 25, 14

 What is Celery?
• Celery is Python Library that performs:
• Task Queuing
• Asynchronously
• Distributed across many machines
• Works with Django
• Celery is just Python
Tuesday, February 25, 14
 When should we use
Celery?
• For web / real time applications, Celery can
be used for any processing that is better
executed Asychonously or in a Non
Blocking fashion

• Periodic Tasks ( more Pythonic than cron! )

• Expensive or heavy processing you want to
do on a separate machine or instance

Tuesday, February 25, 14

 Take a use case
• Photo sharing app allows users to upload
an album of photos
• The photos need to be postprocessed by
the app ( resized, cropped, filtered,
persisted to storage)
• This post processing takes a long time to
complete

Tuesday, February 25, 14

 Approach #1
Make the user wait
• Http request kept open until processing
complete
• Bad because the user experience slow
response time
• Blocks the web server thread - with
sufficient users web server will run out of
threads

Tuesday, February 25, 14

 Approach #2
Use AJAX

• Use a AJAX to call expensive operation

while initial response returns quickly
• User has a better experience but threads
are still blocked

Tuesday, February 25, 14

 Approach #3
Cron Jobs
• The photo post processing is persisted
somewhere to be processed by a cron job
that runs every x minutes
• Better, because user has a fast response,
threads are not blocked
• However user’s input is not processed until
the next cycle

Tuesday, February 25, 14

 Approach #4
Celery
• Post processing are celery tasks
placed on distributed messaging
queues
• User has fast response, threads not blocked
• Post processing starts as soon as user input
received
• Can be scaled as traffic increases by
Tuesday, February 25, 14
 How does Celery
work?

Tuesday, February 25, 14

 Distributed

Python / AMQP Broker Results

Django [RabbitMQ] Storage
Celery Worker
Nodes

Tuesday, February 25, 14

 Distributed
Architecture

Python / Django Results

Storage

Results returned to Python

Tuesday, February 25, 14

 Making a Celery Task

from celery.task import task

@task(ignore_result=True)
def generate_resized_page_images(self,page):
page.get_picture_thumbnail_url()
page.get_picture_large_url()
page.get_picture_medium_url()
page.get_picture_normal_url()
return 0

Tuesday, February 25, 14

 Making a Celery Task
from celery.task import Task
from celery.registry import tasks

Class ProcessImages(Task):
def run(self,title,**kwargs):
page.get_picture_thumbnail_url()
page.get_picture_large_url()
return 0

tasks.register(ProcessImages)
Tuesday, February 25, 14
 Run the task!

generate_resized_page_images.delay(‘Cat picture’)

from someapp.task import ProcessImages

result = ProcessImages.delay(‘Cat picture’)

Tuesday, February 25, 14

 How to Install Celery
• Install an AMQP Broker ( RabbitMQ )
• add AMQP config to settings.py
AMQP_SERVER = ‘localhost’
AMQP_PORT = ‘5672’
AMQP_USER = ‘myuser’
AMQP_PASSWORD = ‘password’
AMQP_VHOST = ‘myvhost’

• add celery to INSTALLED_APPS

manage.py synchdb
• Start Celery:
manage.py celeryd

Tuesday, February 25, 14

 Try Celery yourself!

• Twitter:
@ronaldlcheung
• Email:
[email protected]

Tuesday, February 25, 14