A case of Django, Circus, Chaussette, Celery, and Nginx

TLDR; a short story on how I deployed a Django app on Circus, Celery and Nginx. 

I have an internal (closed source) web app called “inspectr”; what it does is not important for this discussion. For various reasons, I wanted to deploy it in “full” production environment instead of just using the traditional “python manage.py runserver“.

Django

Inspectr is a normal multi-page Django app.

Celery

Celery is a “background task processor”. In views (request handler functions), Django app enqueues long running tasks for Celery to execute between normal requests. This avoids blocking for a long time in views.

Circus and Chaussette

Circus is a process manager that (re)launches and monitors processes. Chaussette is a WSGI web server that allows using file descriptors opened by parent process; this means that Circus opens and logs activity in all the sockets, while the sockets are actually served by Chaussette and the Django app. Chaussette receives the file descriptors to listen to from the Circus process as command line arguments.

Circus provides a handy web interface where you can increase/decrease the amount of worker processes, kill runaway processes etc.

NGINX

NGINX is a web server like apache, but lighter/faster/more modern/easier/whatever.

Role of the NGINX in this layout is to

– Serve all static files (js, css, images) directly.
– Forward everyhing else to Circus, where it ends up being served by Django.

Workflow

My Django app has a directory called SITE_ROOT/siteconf

ville@ville-tp:~/p/inspectr/siteconf$ ls -F
circus_inspectr.ini nginx.conf static/

All the static files are dumped under static/ by “python manage.py collectstatic“. This is needed because Django only server static files if you are using Django’s built in web server (manage.py runserver). For production, you have to serve them separately (for various good reasons).

They are served by following stanza in nginx.conf:

location /static {
 root /home/ville/p/inspectr/siteconf;
}

The rest are followed to circus by the following nginx config stanza:

location / {
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Host $http_host;
  proxy_redirect off;
  proxy_pass http://127.0.0.1:8080;
 }

Note that 8080 is the port where Circus is listening.

In order to “activate” the ngix configuration, you need to symlink to it from global nginx configuration:

ville@ville-tp:/etc/nginx/sites-enabled$ ls -l /etc/nginx/sites-enabled/ 
total 0
lrwxrwxrwx 1 root root 42 Dec 27 09:11 inspectr.conf -> /home/ville/p/inspectr/siteconf/nginx.conf

I launch circus using an application specific config file:

circusd circus_inspectr.ini

After this, nginx can serve using circus.

Circus has the following stanza (in circus_inpectr.ini) for Celery process:

[watcher:dcelery]
working_dir = /home/ville/p/inspectr
cmd = /usr/bin/python 
args = manage.py celery worker
stdout_stream.class = StdoutStream
stderr_stream.class = StdoutStream

and this for the Chaussette processes (that hand over the requests for Django):

[socket:dwebapp]
host = 127.0.0.1 
port = 8080
[watcher:dwebworker]
cmd = /usr/local/bin/chaussette --backend gevent --fd $(circus.sockets.dwebapp) inspectr.wsgi.application 
use_sockets = True 
numprocesses = 2
stdout_stream.class = StdoutStream
stderr_stream.class = StdoutStream

[env:dwebworker] 
PYTHONPATH = /home/ville/p/inspectr

The streams are needed for debugging phase. Once the app is known to work, they can be removed.

QML: How to not crash if QtMobility.location is not available

Python has a handy pattern to avoid crashes if, say, a module is not available:

try:
    import foobar
except ImportError:
    foobar_not_available = True

QML has no such feature, at least yet. That’s why the whole file “fails” at once when a module is missing, leaving you with a black screen and no error message in sight (unless you are launching the application from console). This can be a drag, especially when you want to publish both on platforms that have the feature (Symbian, Maemo, Harmattan, Qt Simulator) as well as platforms that don’t have it yet (Ubuntu desktop, MeeGo 1.1, …).

However, QML *can* instantiate objects dynamically, so instead of having this directly in your .qml file:

PositionSource {
    id: positionSource
    updateInterval: 1000
    active: true
}

move the potentially offending code above segregated to another file, “MaybePositionSource.qml”:

import QtMobility.location 1.1

PositionSource {
    id: positionSource
    updateInterval: 1000
    active: true
}

Now, you can create positionSource dynamically, and replace it with a dummy placeholder object if it’s not available:

    Component.onCompleted: {
        var co = Qt.createComponent("MaybePositionSource.qml")
        if (co.status == Component.Ready) {
            var o = co.createObject(statusDialogItem)
            positionSource = o            
        } else {
            console.log("QtMobility.location 1.1 not available, using placeholder. Error= " + co.errorString())
            positionSource = { position : { coordinate : {} } }
         }
    }

Now, reference to, say, positionSource.position.latitude evaluated to ‘undefined’ instead of causing an exception. You could use your own placeholder values if it seems more appropriate.

Intel’s Qt Quick Components available for Ubuntu

Yesterday, Intel announced the open source release of MeeGo Tablet UX, and along with it their version of Qt Quick Components:

http://lists.meego.com/pipermail/meego-dev/2011-March/482351.html

http://wiki.meego.com/MeeGo_UX_Components

Without doubt, you want to kick the tires of these components. If you don’t have a place to install MeeGo Tablet UX image (highly recommended e.g. for the popular Lenovo Ideapad), you can now (thanks to Antonio Aloisio, who did the packaging) install the components on your trusty old Ubuntu workhorse (currently tested on Maverick).

Do this to add the Forum Nokia Qt PPA:

sudo add-apt-repository ppa:forumnokia/fn-ppa

sudo apt-get update

Then do

sudo apt-get install meegoux-components

If everything goes as planned, you should be able try the widgetgallery example:

Either get it from git:

git clone git://gitorious.org/meego-ux/meego-ux-components.git

cd meego-ux-components/examples/meego-ux-widgetgallery

qmlviewer ./main.qml

Or from the repo:

sudo apt-get install meegoux-components-examples

qmlviewer /usr/share/meego-ux-widgetgallery/main.qml

Note that if you get it from git, you will need to edit the example code a bit (comment out some offending lines) because of the old Qt version in Maverick. Required modifications should be obvious.

Qt Quick Components – survival while you wait

MeeGo Touch (MTF) has been communicated as being a deprecated UI solution for both “real MeeGo” and Nokia-specific Harmattan. In practice, it’s not likely that it will be removed from Harmattan in the middle of Harmattan lifetime, but there is no official promise of your application continuing to work after Harmattan.

 
Qt Quick Components

A lot has been said about Qt Quick Components already:

http://labs.qt.nokia.com/2010/09/10/building-the-future-reintroducing-the-qt-quick-components/

http://blogs.forum.nokia.com/blog/kate-alholas-forum-nokia-blog/2010/11/14/how-to-make-modern-mobile-applications-with-qt-quick-components

However, this may lead some developers to crawl in a hole, incapable of getting any development done because they are waiting for the Components release (as they harbor the misconception that QML alone can’t get the job done).

This is the wrong attitude. Even when using Qt Quick Components, you need to understand QML anyway – there is no way around creating custom list delegates, for example. It’s a better idea to write your application completely in raw QML right now, and when official components become available, you can easily change your own button’s for the official, theme-following buttons. In the meantime, you can implement page transitions using this nifty helper component:

http://juhaturunen.com/blog/2011/01/eine-kleine-view-framework-with-qtquick-overture

http://juhaturunen.com/blog/2011/01/eine-kleine-view-framework-with-qtquick-part-ii

If you wish to avoid writing your own button / checkbox / slider / whatever, you can take a snapshot of “custom” branch from Qt Quick Components repository. Here’s a minimal example that just copies the components (pure QML) to a subdirectory. Note that it is work in progress, so do not expect anything out of the quality or features:

http://gitorious.org/qml-components-placeholder

Lauri’s blog provides easy-to-steal examples for modern “conventional” mobile UI look, instead of the usual bouncing rectangles and exploding marbles:

http://qtsource.wordpress.com/