blog

Image of channels by Hans Braxmeier on Pixabay

Django Channels: From The Ground Up – Part 1

by

You stare mournfully into the mass of code you’ve inherited. At some point, it’s clear, the requirements called for the server to push information to the client, because there’s an unholy mix of Server-Side Events, long-polling, hidden iframes and even a Java applet in there, all supporting some level of long-term connectivity with the server. It’s almost fascinating in its barely functional hideousness, and you would be inclined to leave well enough alone… except for the new feature specifications you’ve been assigned, which require the client to be able to send data back to the server in response to the received events, in as close to real-time as you can get.

It’s time for a major overhaul. Only Websockets can save us now. And, as it so happens, one of your favourite frameworks just added support for real-time, bi-directional communication…

Adding More Batteries

If you’ve been paying attention to the Django project, you’ve probably heard about Django Channels. A new server gateway interface (ASGI), a new server (Daphne) and, as a result, new functionality – including support for Websockets and (at some near-future point) HTTP2.

Support for websockets isn’t new – Node.js (and thus, Express, Meteor, Sails, etc.) had them pretty much out of the box, and its event-based model is well suited to them. Even on the Python side of things, if you needed websockets, you could have them – Gevent, Tornado/Cyclone and Twisted had your back, amongst others. In fact, the preferred server for Django Channels, Daphne, is based on Twisted.

So, the big news here isn’t OMG Websockets, but rather Websockets + Django. Pyramid and Flask fans may scoff, but Django’s batteries-included approach echoes that of Python itself, and enables some solid work that tends to be easier to maintain than, for instance, the special snowflakes that a large Flask project can end up as (your mileage may vary).

So, if you’re sold on Django, there’s good reason to be excited about Channels. Although its inclusion in Django core has been deferred for now, chances are good that it will see integration with the Django core project at some near-future point, making now a good time to get up to speed.

For this short series of articles, we’ll be looking at doing just that. While the Django Channels documentation lives up to the high standard set by the core Django documentation, it doesn’t spend much time describing what a production-ready setup might look like, or how to achieve it. Additionally, like most tutorials for websockets, it focuses on a simple echo or chat server. We’re going to go a step beyond, and take a look at what we need to setup real-time streaming video in the browser.

For the first two articles, though, we’ll focus on how to get our shiny new Django Channels installation up and running. We’re going to cover a lot of ground, so we won’t be able to focus too closely on the details. If you run into trouble, a quick search might do it for you – if not, I suggest you bring your question to the good people at Stack Overflow; and if you note any mistakes in the article, or just have something to say, the comment section below awaits you.

Let’s get started, shall we?

Penguins or Daemons?

It’s true, penguins are cuter, and I’ve never had to kill -9 a penguin.

One small additional wrinkle to the following documentation is that we’re going to add some details for this same setup with FreeBSD. I’ve had some recent projects that have caused me to dip my toes into the BSD pool, and I noticed there’s a relative lack of documentation for this sort of thing there, so we’re going to help rectify that a little.

Don’t worry if you’re planning to use a Linux flavour – in most cases, the instructions are either identical, or the differences will be called out.

From the Ground Up

Assumption: You have a relatively modern unix-like operating system. Windows folks, as ever in server-land, I’m afraid you’re on your own (maybe now’s a good time to look into Ubuntu on Windows?).

We’re going to be re-treading some ground that is likely very familiar for some of you, so bear with me – we’re going to start up from scratch here.

Now, let’s discuss our stack. We’re going to go with a set of software that should look familiar to anyone who’s set up a basic, fairly production-worthy server with Django:

  • Python (2.7+ or 3.4+);
  • Virtualenv;
  • Pip;
  • Nginx;
  • Postgresql;
  • Supervisord;
  • Django (1.8+).

And adding some new elements to support Channels:

  • Redis;
  • Daphne;
  • channels.

Let’s go through the setup step-by-step, and discuss the how and, in some cases, the why of our software stack.

For this first part, we’re going to go get us far enough to run Django using our shiny new ASGI server, via the usual development invocation; and in part 2, we’ll be getting our stack somewhere approaching production ready.

Package Installation

This step is nice and straightforward. Ask your friendly neighbourhood package manager for more details.

Debian/Ubuntu one-liner:

sudo apt-get -y -q install python python-dev python-pip python-virtualenv libpq-dev postgresql postgresql-contrib nginx supervisor python-software-properties redis-server

FreeBSD example:

cd /usr/ports/databases/postgresql95-server/ && make install clean

OR

pkg install postgresql95-server

and repeat for python27 (e.g. pkg install python27), nginx, supervisord, etc. I suggest searching FreshPorts for the specific package names.

If you’re on FreeBSD, you’ll also want to add the appropriate enable commands to rc.conf to allow these applications to start with the server.

User & VirtualEnv setup

Now that we have our system packages installed, let’s set up our python virtual environment in preparation for getting our python packages into place.

First, we’ll create a new, low-priviledged user who we want to run our server software – running web-facing processes as root can be a security faux pas.

sudo adduser web (If using FreeBSD, drop the sudo and execute this (and other commands requiring sudo) as root by suing into root account, or else install sudo package).

Let’s su in as this user… su web

We’re now going to drop our virtualenv directory (and, in a moment, our deploy directory) into this users home dir – we don’t have to do it this way, of course, but this avoids some bothersome chmod’ing and chown’ing we’d have to do otherwise.

mkdir -p /home/web/venv/
virtualenv --no-site-packages /home/web/venv/

Python Package Installation

Now that we have our virtualenv in place, we can install our python packages via pip. If we follow the git deploy setup, below, these can also be automatically installed/upgraded for us by processing a requirements.txt file.

Let’s make a requirements.txt file that looks like the following:

asgi-redis==0.12.0
asgiref==0.13.0
autobahn==0.14.1
channels==0.14.0
daphne==0.12.1
Django>=1.9,=2.6,<2.7
redis==2.10.5
six==1.10.0
Twisted==16.2.0
txaio==2.5.1
zope.interface==4.1.3

Now, we can activate the virtual environment and install these requirements with pip like so:

source /home/web/venv/bin/activate
pip install -r requirements.txt
deactivate

FreeBSD (csh) example:

source /home/web/venv/bin/activate.csh
pip install -r requirements.txt
deactivate

Django Project Setup

Now, we’re going to get a bare minimal Django project structure in place. Thankfully, Django has some built-in ability to generate it’s own boilerplate, which we’ll take advantage of. For this, I refer you to the official Django docs.

For our purposes, the only file of interest is settings.py in your app directory. Open that up in your favourite editor, and find the INSTALLED_APPS tuple. Installing channels into Django is as simple as adding 'channels' to the end of the tuple.

You may also need/want to add your ‘app_name’ to this tuple, especially if you need static file discovery. You can take a look at the Channels Installation documentation for a little more detail here, but it’s nicely straightforward.

You should also copy that requirements.txt file we made earlier into the root of your app directory. If nothing else, it serves as a reference as to what packages you’re using.

FreeBSD Note:

At this point we’ll be relying on SQLite to provide our database solution – in many distributions, this is built-in to Python, but on FreeBSD we’ll also need to install the py-sqlite3 package: pkg install py27-sqlite3

Deploy Setup [Optional]

This step is purely optional, but I like it as an easy way to deploy via git push. That link has more info, so we’ll just sketch out the implementation here:

Make the necessary directories:

mkdir -p /home/web/www.git/
mkdir -p /home/web/www/

Setup a bare git repository:

cd /home/web/www.git/
git config core.sharedRepository group

And create a post-receive hook that will install/update our packages, run migrations for us, and kill and restart the development server:

Ubuntu (bash) example:

#!/bin/bash
GIT_WORK_TREE=/home/web/www/ git checkout -f
source /home/web/venv/bin/activate
pushd /home/web/www/
# Install python libs via pip and perform database migrations
pip install --upgrade -r requirements.txt
python manage.py migrate
kill ps aux | grep runserver | grep -v grep | awk '{print $2}'
python manage.py runserver
popd
deactivate

FreeBSD (csh) example:

#!/bin/csh
env GIT_WORK_TREE=/home/web/www/ git checkout -f
source /home/web/venv/bin/activate.csh
pushd /home/web/www/
pip install --upgrade -r requirements.txt
python manage.py migrate
kill ps aux | grep runserver | grep -v grep | awk '{print $2}'
python manage.py runserver
popd
deactivate

Save this file as post-receive, and then make it executable: chmod +x post-receive.

Now, git init in your local directory, and add a new remote with git remote add your.server.addr.octets/home/web/www.git. When pushing, do so as your web user, or you may run into permissions issues.

As noted before, more details on this can be found in this article.

Are we there yet?

Hah, no, but we are done for today. We’re now at a point where you could push your code into your server, and use django’s runserver command to run the development server. With the channels app installed, this well-known invocation will not start up the old wsgi server, but a shiny new ASGI server instead, with workers and interface living inside the same process as separate threads.

Next time, we’ll go the rest of the way – getting your server something resembling production ready, with postgresql setup, supervisord managing our server workers and interface, and we’ll start figuring out what we need to actually use this setup for something cool.

+ more

Accurate Timing

Accurate Timing

In many tasks we need to do something at given intervals of time. The most obvious ways may not give you the best results. Time? Meh. The most basic tasks that don't have what you might call CPU-scale time requirements can be handled with the usual language and...

read more