Celery

Celery is a Python library that allows tasks to be executed asynchronously (in the background) and distributed across multiple machines. It uses message passing with RabbitMQ as the broker to assign tasks to worker nodes. This allows long-running or expensive tasks to not block the web server threads, improving response times for users. Celery can be used for periodic tasks, processing user uploads asynchronously, and scaling heavy workloads across many servers.

Uploaded by

Eric Chua

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

17 views16 pages

Celery

Uploaded by

Eric Chua

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 16

Celery

http://www.celeryproject.org/

( A python Library )
@ronaldlcheung

Tuesday, February 25, 14

What is Celery?
• Celery is Python Library that performs:
• Task Queuing
• Asynchronously
• Distributed across many machines
• Works with Django
• Celery is just Python
Tuesday, February 25, 14
When should we use
Celery?
• For web / real time applications, Celery can
be used for any processing that is better
executed Asychonously or in a Non
Blocking fashion

• Periodic Tasks ( more Pythonic than cron! )

• Expensive or heavy processing you want to
do on a separate machine or instance

Tuesday, February 25, 14

Take a use case
• Photo sharing app allows users to upload
an album of photos
• The photos need to be postprocessed by
the app ( resized, cropped, filtered,
persisted to storage)
• This post processing takes a long time to
complete

Tuesday, February 25, 14

Approach #1
Make the user wait
• Http request kept open until processing
complete
• Bad because the user experience slow
response time
• Blocks the web server thread - with
sufficient users web server will run out of
threads

Tuesday, February 25, 14

Approach #2
Use AJAX

• Use a AJAX to call expensive operation

while initial response returns quickly
• User has a better experience but threads
are still blocked

Tuesday, February 25, 14

Approach #3
Cron Jobs
• The photo post processing is persisted
somewhere to be processed by a cron job
that runs every x minutes
• Better, because user has a fast response,
threads are not blocked
• However user’s input is not processed until
the next cycle

Tuesday, February 25, 14

Approach #4
Celery
• Post processing are celery tasks
placed on distributed messaging
queues
• User has fast response, threads not blocked
• Post processing starts as soon as user input
received
• Can be scaled as traffic increases by
Tuesday, February 25, 14
How does Celery
work?

Tuesday, February 25, 14

Distributed

Python / AMQP Broker Results

Django [RabbitMQ] Storage
Celery Worker
Nodes

Tuesday, February 25, 14

Distributed
Architecture

Python / Django Results

Storage

Results returned to Python

Tuesday, February 25, 14

Making a Celery Task

from celery.task import task

@task(ignore_result=True)
def generate_resized_page_images(self,page):
page.get_picture_thumbnail_url()
page.get_picture_large_url()
page.get_picture_medium_url()
page.get_picture_normal_url()
return 0

Tuesday, February 25, 14

Making a Celery Task
from celery.task import Task
from celery.registry import tasks

Class ProcessImages(Task):
def run(self,title,**kwargs):
page.get_picture_thumbnail_url()
page.get_picture_large_url()
return 0

tasks.register(ProcessImages)
Tuesday, February 25, 14
Run the task!

generate_resized_page_images.delay(‘Cat picture’)

from someapp.task import ProcessImages

result = ProcessImages.delay(‘Cat picture’)

Tuesday, February 25, 14

How to Install Celery
• Install an AMQP Broker ( RabbitMQ )
• add AMQP config to settings.py
AMQP_SERVER = ‘localhost’
AMQP_PORT = ‘5672’
AMQP_USER = ‘myuser’
AMQP_PASSWORD = ‘password’
AMQP_VHOST = ‘myvhost’