Python - Some extra libraries

and features and tips and

tricks of python programming
language you must know

(useful-python-tricks.html)

Python - Django Best Practices

For Model Design

(django-best-practices-models-design.html)

Python - Simple Tricks to Level

Up Your Python Coding

(python-simple-tricks-to-level-up-your-python-coding.html)

Python - Python Pandas tips to

make data analysis faster

(python-pandas-tips-to-make-data-analysis-faster.html)

Python - How to Create

Custom Django Management
Commands

(create-custom-django-management-commands.html)

Python - How to Setup a SSL

Certificate on Nginx for a
Django Application

(setup-ssl-certificate-on-nginx-for-django-application.html)
 Python - How to Use Celery
and RabbitMQ with Django

(use-celery-with-django.html)

Python - How to Use JWT

Authentication with Django
REST Framework

(use-jwt-authentication-with-django-rest-framework.html)

Python - Django Tips for

Designing Better Models

(django-tips-for-designing-better-models.html)

Python - Django Query Tips

select & prefetch_related

(django-query-prefetch-related.html)

Python - Django Tips for

Working with Databases

(django-tips-for-working-with-databases.html)

Python - Django REST

Framework: adding additional
field to ModelSerializer

(adding-additional-field-to-modelserializer.html)

Python - How to Implement

Dependent/Chained
Dropdown List with Django

(how-to-Implement-dependent-chained-dropdown-list-with-django.html)

How to Use Celery and RabbitMQ with Django

Celery is an asynchronous task queue based on distributed message passing.
Task queues are used as a strategy to distribute the workload between
threads/machines. In this tutorial I will explain how to install and setup Celery +
RabbitMQ to execute asynchronous in a Django application.

To work with Celery, we also need to install RabbitMQ because Celery requires
an external solution to send and receive messages. Those solutions are called
message brokers. Currently, Celery supports RabbitMQ, Redis, and Amazon
SQS as message broker solutions.
Table of Contents
Why Should I Use Celery?
Installation
Installing RabbitMQ on Ubuntu 16.04
Installing RabbitMQ on Mac
Installing RabbitMQ on Windows and Other OSs
Celery Basic Setup
Creating Our First Celery Task
Starting The Worker Process
Managing The Worker Process in Production with Supervisord
Further Reading

Why Should I Use Celery?

Web applications works with request and response cycles. When the user
access a certain URL of your application the Web browser send a request to
your server. Django receive this request and do something with it. Usually it
involves executing queries in the database, processing data. While Django
does his thing and process the request, the user have to wait. When Django
finalize its job processing the request, it sends back a response to the user who
finally will see something.

Ideally this request and response cycle should be fast, otherwise we would
leave the user waiting for way too long. And even worse, our Web server can
only serve a certain number of users at a time. So, if this process is slow, it can
limit the amount of pages your application can serve at a time.

For the most part we can work around this issue using cache, optimizing
database queries, and so on. But there are some cases that theres no other
option: the heavy work have to be done. A report page, export of big amount of
data, video/image processing are a few examples of cases where you may
want to use Celery.

We don’t use Celery through the whole project, but only for specific tasks that
are time-consuming. The idea here is to respond to the user as quick as
possible, and pass the time-consuming tasks to the queue so to be executed
in the background, and always keep the server ready to respond to new
requests.

Installation
The easiest way to install Celery is using pip:

pip install Celery

Now we have to install RabbitMQ.

Installing RabbitMQ on Ubuntu 16.04

To install it on a newer Ubuntu version is very straightforward:

apt-get install -y erlang

apt-get install rabbitmq-server
Then enable and start the RabbitMQ service:

systemctl enable rabbitmq-server

systemctl start rabbitmq-server

Check the status to make sure everything is running smooth:

systemctl status rabbitmq-server

Installing RabbitMQ on Mac

Homebrew is the most straightforward option:

brew install rabbitmq

The RabbitMQ scripts are installed into /usr/local/sbin . You can add it to your
.bash_profile or .profile .

vim ~/.bash_profile

Then add it to the bottom of the file:

export PATH=$PATH:/usr/local/sbin

Restart the terminal to make sure the changes are in effect.

Now you can start the RabbitMQ server using the following command:

rabbitmq-server
Celery Basic Setup
First, consider the following Django project named mysite with an app named
core:

mysite/
|-- mysite/
| |-- core/
| | |-- migrations/
| | |-- templates/
| | |-- apps.py
| | |-- models.py
| | +-- views.py
| |-- templates/
| |-- __init__.py
| |-- settings.py
| |-- urls.py
| +-- wsgi.py
|-- manage.py
+-- requirements.txt

Add the CELERY_BROKER_URL configuration to the settings.py file:

settings.py

CELERY_BROKER_URL = 'amqp://localhost'
Alongside with the settings.py and urls.py files, let’s create a new file named
celery.py.

celery.py

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')

app = Celery('mysite')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

Now edit the __init__.py file in the project root:

__init__.py

from .celery import app as celery_app

__all__ = ['celery_app']

This will make sure our Celery app is important every time Django starts.

Creating Our First Celery Task

We can create a file named tasks.py inside a Django app and put all our
Celery tasks into this file. The Celery app we created in the project root will
collect all tasks defined across all Django apps listed in the INSTALLED_APPS
configuration.

Just for testing purpose, let’s create a Celery task that generates a number of
random User accounts.

core/tasks.py
 import string

from django.contrib.auth.models import User

from django.utils.crypto import get_random_string

from celery import shared_task

@shared_task
def create_random_user_accounts(total):
for i in range(total):
username = 'user_{}'.format(get_random_string(10, string.ascii_lette
rs))
email = '{}@example.com'.format(username)
password = get_random_string(50)
User.objects.create_user(username=username, email=email, password=pa
ssword)
return '{} random users created with success!'.format(total)

The important bits here are:

from celery import shared_task

@shared_task
def name_of_your_function(optional_param):
pass # do something heavy

Then I defined a form and a view to process my Celery task:

forms.py

from django import forms

from django.core.validators import MinValueValidator, MaxValueValidator

class GenerateRandomUserForm(forms.Form):
total = forms.IntegerField(
validators=[
MinValueValidator(50),
MaxValueValidator(500)
]
)

This form expects a positive integer field between 50 and 500. It looks like this:
Then my view:

views.py

from django.contrib.auth.models import User

from django.contrib import messages
from django.views.generic.edit import FormView
from django.shortcuts import redirect

from .forms import GenerateRandomUserForm

from .tasks import create_random_user_accounts

class GenerateRandomUserView(FormView):
template_name = 'core/generate_random_users.html'
form_class = GenerateRandomUserForm

def form_valid(self, form):

total = form.cleaned_data.get('total')
create_random_user_accounts.delay(total)
messages.success(self.request, 'We are generating your random users!
Wait a moment and refresh this page.')
return redirect('users_list')

The important bit is here:

create_random_user_accounts.delay(total)
Instead of calling the create_random_user_accounts directly, I’m calling
create_random_user_accounts.delay() . This way we are instructing Celery to
execute this function in the background.

Then Django keep processing my view GenerateRandomUserView and returns

smoothly to the user.

But before you try it, check the next section to learn how to start the Celery
worker process.

Starting The Worker Process

Open a new terminal tab, and run the following command:

celery -A mysite worker -l info

Change mysite to the name of your project. The result is something like this:

Now we can test it. I submitted 500 in my form to create 500 random users.

The response is immediate:

Meanwhile, checking the Celery Worker Process:

[2017-08-20 19:11:17,485: INFO/MainProcess] Received task:

mysite.core.tasks.create_random_user_accounts[8799cfbd-deae-41aa-afac-95ed4c
c859b0]

Then after a few seconds, if we refresh the page, the users are there:

If we check the Celery Worker Process again, we can see it completed the
execution:

[2017-08-20 19:11:45,721: INFO/ForkPoolWorker-2] Task

mysite.core.tasks.create_random_user_accounts[8799cfbd-deae-41aa-afac-95ed4c
c859b0] succeeded in
28.225658523035236s: '500 random users created with success!'

Managing The Worker Process in Production with Supervisord

If you are deploying your application to a VPS like DigitalOcean
(https://m.do.co/c/074832454ff1) you will want to run the worker process in the
background. In my tutorials I like to use Supervisord to manage the Gunicorn
workers, so it’s usually a nice fit with Celery.

First install it (on Ubuntu):

 sudo apt-get install supervisor

Then create a file named mysite-celery.conf in the folder:

/etc/supervisor/conf.d/mysite-celery.conf:

[program:mysite-celery]
command=/home/mysite/bin/celery worker -A mysite --loglevel=INFO
directory=/home/mysite/mysite
user=nobody
numprocs=1
stdout_logfile=/home/mysite/logs/celery.log
stderr_logfile=/home/mysite/logs/celery.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.

; Increase this if you have very long running tasks.
stopwaitsecs = 600

stopasgroup=true

; Set Celery priority higher than default (999)

; so, if rabbitmq is supervised, it will start first.
priority=1000

In the example below, I’m considering my Django project is inside a virtual

environment. The path to my virtual environment is /home/mysite/.

Now reread the configuration and add the new process:

sudo supervisorctl reread

sudo supervisorctl update

Calling Tasks (http://docs.celeryproject.org/en/latest/getting-started/next-steps.html#calling-

tasks)
Celery Configuration and Defaults (http://docs.celeryproject.org/en/latest/getting-
started/next-steps.html#calling-tasks)
Celery GitHub Repository (https://github.com/celery/celery)
DigitalOcean® Developer Cloud -
Simple, Powerful Cloud Hosting
Ad try.digitalocean.com

Python Standard
Library - Learn the
Hidden Gems
Ad pythonstandardlibrarybook.c…

Angularjs - Create a
Real-Time App with
Socket.IO, Angular,…
designmycodes.com

Find Python Courses

Ad Coursary

CODE - ReactJS
Environment Setup
designmycodes.com

CODE - Vuejs
Tutorial With
Example From…
designmycodes.com

Company
Home (../index.html)
About Us (../about.html)
Privacy Policy (../privacy-policy.html)

Help and Support

 Contact Us (../contact.html)

Follow us on

FACEBOOK (HTTPS://WWW.FACEBOOK.COM/DESIGNMYCODE-287393808601218/?REF=NF&HC_REF=

© 2021 designmycodes, made with love