A case of Django, Circus, Chaussette, Celery, and Nginx

circus_load

TLDR; a short story on how I deployed a Django app on Circus, Celery and Nginx. 

I have an internal (closed source) web app called “inspectr”; what it does is not important for this discussion. For various reasons, I wanted to deploy it in “full” production environment instead of just using the traditional “python manage.py runserver“.

Django

Inspectr is a normal multi-page Django app.

Celery

Celery is a “background task processor”. In views (request handler functions), Django app enqueues long running tasks for Celery to execute between normal requests. This avoids blocking for a long time in views.

Circus and Chaussette

Circus is a process manager that (re)launches and monitors processes. Chaussette is a WSGI web server that allows using file descriptors opened by parent process; this means that Circus opens and logs activity in all the sockets, while the sockets are actually served by Chaussette and the Django app. Chaussette receives the file descriptors to listen to from the Circus process as command line arguments.

Circus provides a handy web interface where you can increase/decrease the amount of worker processes, kill runaway processes etc.

NGINX

NGINX is a web server like apache, but lighter/faster/more modern/easier/whatever.

Role of the NGINX in this layout is to

– Serve all static files (js, css, images) directly.
– Forward everyhing else to Circus, where it ends up being served by Django.

Workflow

My Django app has a directory called SITE_ROOT/siteconf

ville@ville-tp:~/p/inspectr/siteconf$ ls -F
circus_inspectr.ini nginx.conf static/

All the static files are dumped under static/ by “python manage.py collectstatic“. This is needed because Django only server static files if you are using Django’s built in web server (manage.py runserver). For production, you have to serve them separately (for various good reasons).

They are served by following stanza in nginx.conf:

location /static {
 root /home/ville/p/inspectr/siteconf;
}

The rest are followed to circus by the following nginx config stanza:

location / {
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_redirect off;
  proxy_pass http://127.0.0.1:8080;
 }

Note that 8080 is the port where Circus is listening.

In order to “activate” the ngix configuration, you need to symlink to it from global nginx configuration:

ville@ville-tp:/etc/nginx/sites-enabled$ ls -l /etc/nginx/sites-enabled/ 
total 0
lrwxrwxrwx 1 root root 42 Dec 27 09:11 inspectr.conf -> /home/ville/p/inspectr/siteconf/nginx.conf

I launch circus using an application specific config file:

circusd circus_inspectr.ini

After this, nginx can serve using circus.

Circus has the following stanza (in circus_inpectr.ini) for Celery process:

[watcher:dcelery]
working_dir = /home/ville/p/inspectr
cmd = /usr/bin/python 
args = manage.py celery worker
stdout_stream.class = StdoutStream
stderr_stream.class = StdoutStream

and this for the Chaussette processes (that hand over the requests for Django):

[socket:dwebapp]
host = 127.0.0.1 
port = 8080
[watcher:dwebworker]
cmd = /usr/local/bin/chaussette --backend gevent --fd $(circus.sockets.dwebapp) inspectr.wsgi.application 
use_sockets = True 
numprocesses = 2
stdout_stream.class = StdoutStream
stderr_stream.class = StdoutStream

[env:dwebworker] 
PYTHONPATH = /home/ville/p/inspectr

The streams are needed for debugging phase. Once the app is known to work, they can be removed.

Advertisements
This entry was posted in Uncategorized and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s