15/02/2021 How To Use Celery with RabbitMQ to Queue Tasks on an Ubuntu VPS | DigitalOcean

TUTORIAL

How To Use Celery with RabbitMQ to Queue Tasks on an

Ubuntu VPS
Ubuntu Python Scaling Messaging

By Justin Ellingwood
Published on December 19, 2013  186.6k

Introduction
Asynchronous, or non-blocking, processing is a method of separating the execution
of certain tasks from the main flow of a program. This provides you with several
advantages, including allowing your user-facing code to run without interruption.

Message passing is a method which program components can use to communicate

and exchange information. It can be implemented synchronously or asynchronously
and can allow discrete processes to communicate without problems. Message
passing is often implemented as an alternative to traditional databases for this type
of usage because message queues often implement additional features, provide
increased performance, and can reside completely in-memory.

Celery is a task queue that is built on an asynchronous message passing system. It

can be used as a bucket where programming tasks can be dumped. The program
S C R O L L TO TO P
that passed the task can continue to execute and function responsively, and then

https://www.digitalocean.com/community/tutorials/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps 1/10
15/02/2021 How To Use Celery with RabbitMQ to Queue Tasks on an Ubuntu VPS | DigitalOcean

later on, it can poll celery to see if the computation is complete and retrieve the
data.

While celery is written in Python, its protocol can be implemented in any language. It
can even function with other languages through webhooks.

By implementing a job queue into your program’s environment, you can easily
offload tasks and continue to handle interactions from your users. This is a simple
way to increase the responsiveness of your applications and not get locked up while
performing long-running computations.

In this guide, we will install and implement a celery job queue using RabbitMQ as the
messaging system on an Ubuntu 12.04 VPS.

Install the Components

Install Celery
Celery is written in Python, and as such, it is easy to install in the same way that we
handle regular Python packages.

We will follow the recommended procedures for handling Python packages by

creating a virtual environment to install our messaging system. This helps us keep
our environment stable and not effect the larger system.

Install the Python virtual environment package from Ubuntu’s default repositories:

sudo apt-get update

sudo apt-get install python-virtualenv

We will create a messaging directory where we will implement our system:

mkdir ~/messaging
cd ~/messaging

We can now create a virtual environment where we can install celery by using the
following command:
S C R O L L TO TO P

https://www.digitalocean.com/community/tutorials/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps 2/10
15/02/2021 How To Use Celery with RabbitMQ to Queue Tasks on an Ubuntu VPS | DigitalOcean

virtualenv --no-site-packages venv

With the virtual environment configured, we can activate it by typing:

source venv/bin/activate

Your prompt will change to reflect that you are now operating in the virtual
environment we made above. This will ensure that our Python packages are
installed locally instead of globally.

If at any time we need to deactivate the environment (not now), you can type:

deactivate

Now that we have activated the environment, we can install celery with pip:

pip install celery

Install RabbitMQ
Celery requires a messaging agent in order to handle requests from an external
source. This agent is referred to as a “broker”.

There are quite a few options for brokers available to choose from, including
relational databases, NoSQL databases, key-value stores, and actual messaging
systems.

We will be configuring celery to use the RabbitMQ messaging system, as it provides

robust, stable performance and interacts well with celery. It is a great solution
because it includes features that mesh well with our intended use.

We can install RabbitMQ through Ubuntu’s repositories:

sudo apt-get install rabbitmq-server

The RabbitMQ service is started automatically on our server upon installation.

S C R O L L TO TO P

https://www.digitalocean.com/community/tutorials/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps 3/10
15/02/2021 How To Use Celery with RabbitMQ to Queue Tasks on an Ubuntu VPS | DigitalOcean

Create a Celery Instance

In order to use celery’s task queuing capabilities, our first step after installation must
be to create a celery instance. This is a simple process of importing the package,
creating an “app”, and then setting up the tasks that celery will be able to execute in
the background.

Let’s create a Python script inside our messaging directory called tasks.py where
we can define tasks that our workers can perform.

sudo nano ~/messaging/tasks.py

The first thing we should do is import the Celery function from the celery package:

from celery import Celery

After that, we can create a celery application instance that connects to the default
RabbitMQ service:

from celery import Celery

app = Celery('tasks', backend='amqp', broker='amqp://')

The first argument to the Celery function is the name that will be prepended to
tasks to identify them.

The backend parameter is an optional parameter that is necessary if you wish to

query the status of a background task, or retrieve its results.

If your tasks are simply functions that do some work and then quit, without
returning a useful value to use in your program, you can leave this parameter out. If
only some of your tasks require this functionality, enable it here and we can disable
it on a case-by-case basis further on.

The broker parameter specifies the URL needed to connect to our broker. In our
case, this is the RabbitMQ service that is running on our server. RabbitMQ operates
S C R O L L TO
using a protocol called “amqp”. If RabbitMQ is TO P
operating under its default

https://www.digitalocean.com/community/tutorials/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps 4/10
15/02/2021 How To Use Celery with RabbitMQ to Queue Tasks on an Ubuntu VPS | DigitalOcean

configuration, celery can connect with no other information other than the amqp://
scheme.

Build Celery Tasks

Still in this file, we now need to add our tasks.

Each celery task must be introduced with the decorator @app.task . This allows
celery to identify functions that it can add its queuing functions to. After each
decorator, we simply create a function that our workers can run.

Our first task will be a simple function that prints out a string to console.

from celery import Celery

app = Celery('tasks', backend='amqp', broker='amqp://')

@app.task
def print_hello():
print 'hello there'

Because this function does not return any useful information (it instead prints it to
the console), we can tell celery to not use the backend to store state information
about this task. This is less complicated under the hood and requires fewer
resources.

<pre>
from celery import Celery

app Celery(‘tasks’, backend='amqp’, broker='amqp://’)

@app.task<span class=“highlight”>(ignoreresult=True)</span>
def printhello():
print 'hello there’
</pre>

Next, we will add another function that will generate prime numbers (taken from
RosettaCode). This can be a long-running process, so it is a good example for how
we can deal with asynchronous worker processes when we are waiting for a result.

S C R O L L TO TO P

https://www.digitalocean.com/community/tutorials/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps 5/10
15/02/2021 How To Use Celery with RabbitMQ to Queue Tasks on an Ubuntu VPS | DigitalOcean

from celery import Celery

app = Celery('tasks', backend='amqp', broker='amqp://')

@app.task(ignore_result=True)
def print_hello():
print 'hello there'

@app.task
def gen_prime(x):
multiples = []
results = []
for i in xrange(2, x+1):
if i not in multiples:
results.append(i)
for j in xrange(i*i, x+1, i):
multiples.append(j)
return results

Because we care about what the return value of this function is, and because we
want to know when it has completed (so that we may use the results, etc), we do
not add the ignore_result parameter to this second task.

Save and close the file.

Start Celery Worker Processes

We can now start a worker processes that will be able to accept connections from
applications. It will use the file we just created to learn about the tasks it can
perform.

Starting a worker instance is as easy as calling out the application name with the
celery command. We will include a “&” character at the end of our string to put our
worker process in the background:

celery worker -A tasks &

This will start up an application, and then detach it from the terminal, allowing you
to continue to use it for other tasks.
S C R O L L TO TO P

https://www.digitalocean.com/community/tutorials/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps 6/10
15/02/2021 How To Use Celery with RabbitMQ to Queue Tasks on an Ubuntu VPS | DigitalOcean

If you want to start multiple workers, you can do so by naming each one with the -n
argument:

celery worker -A tasks -n one.%h &

celery worker -A tasks -n two.%h &

The %h will be replaced by the hostname when the worker is named.

To stop workers, you can use the kill command. We can query for the process id and
then eliminate the workers based on this information.

ps auxww | grep 'celery worker' | awk '{print $2}' | xargs kill

This will allow the worker to complete its current task before exiting.

If you wish to shut down all workers without waiting for them to complete their
tasks, you can execute:

ps auxww | grep 'celery worker' | awk '{print $2}' | xargs kill -9

Use the Queue to Handle Work

We can use the worker process(es) we spawned to complete work in the
background for our programs.

Instead of creating an entire program to demonstrate how this works, we will

explore the different options in a Python interpreter:

python

At the prompt, we can import our functions into the environment:

from tasks import print_hello

from tasks import gen_prime

S C R O L L TO TO P

https://www.digitalocean.com/community/tutorials/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps 7/10
15/02/2021 How To Use Celery with RabbitMQ to Queue Tasks on an Ubuntu VPS | DigitalOcean

If you test these functions, they appear to not have any special functionality. The
first function prints a line as expected:

print_hello()

hello there

The second function returns a list of prime numbers:

primes = gen_prime(1000)
print primes

If we give the second function a larger range of numbers to check, the execution
hangs while it calculates:

primes = gen_prime(50000)

Stop the execution by typing “CTRL C”. This process is clearly not computing in the
background.

To access the background worker, we need to use the .delay method. Celery wraps
our functions with additional capabilities. This method is used to pass the function
to a worker to execute. It should return immediately:

primes = gen_prime.delay(50000)

This task is now being executed by the workers we started earlier. Because we
configured a backend parameter for our application, we can check the status of the
computation and get access to the result.

To check whether the task is complete, we can use the .ready method:

primes.ready()

False S C R O L L TO TO P

https://www.digitalocean.com/community/tutorials/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps 8/10
15/02/2021 How To Use Celery with RabbitMQ to Queue Tasks on an Ubuntu VPS | DigitalOcean

A value of “False” means that the task is still running and a result is not available yet.
When we get a value of “True”, we can do something with the answer.

primes.ready()

True

We can get the value by using the .get method.

If we have already verified that the value is computed with the .ready method, then
we can use that method like this:

print primes.get()

[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 9
. . .

If, however, you have not used the .ready method prior to calling .get , you most
likely want to add a “timeout” option so that your program isn’t forced to wait for the
result, which would defeat the purpose of our implementation:

print primes.get(timeout=2)

This will raise an exception if it times out, which you can handle in your program.

Conclusion
Although this is enough information to get you started on using celery within your
programs, it is only scratching the surface on the full functionality of this library.
Celery allows you to string background tasks together, group tasks, and combine
functions in interesting ways.

Although celery is written in Python, it can be used with other languages through
webhooks. This makes it incredibly flexible for moving tasks into the background,
S C R O L L TO TO P
regardless of your chosen language.

https://www.digitalocean.com/community/tutorials/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps 9/10