This site runs best with JavaScript enabled.

Setting up a Django Server with PostgreSQL, Nginx, Celery & RabbitMQ


Step-by-step guide to setting up a Django server with all the works.

Disclaimer: I am not a sysadmin. I'm just a developer. I welcome and encourage comments to improve this process!

I have set up a couple of Django servers lately and taken copious notes that I have extracted from various sources. Below are the commands I issue to a fresh Ubuntu server install to get Django up and running. This puts everything on one server (PostgreSQL, Celery, RabbitMQ, etc) so it's nice for a small starter project but don't expect it to scale.

Log in as root and add a non-root user. Add the user to the sudoers group. Log out and log back in as 'username'.

1adduser username
2adduser username sudo
3exit

Update the local package index. Upgrade all the packages that can be upgraded. Remove packages that are no longer needed and then reboot for good measure.

1sudo apt update
2sudo apt dist-upgrade
3sudo apt autoremove
4sudo reboot

Install libraries for Python, PIP, PIL/Pillow, PostgreSQL, libevent for gevent, memcached server and library, RabbitMQ, git, nginx, & supervisor

1sudo apt install build-essential python2-dev libjpeg8-dev libfreetype6-dev \
2 zlib1g-dev postgresql postgresql-contrib libpq-dev libevent-dev memcached \
3 libmemcached-dev rabbitmq-server git nginx supervisor

Install virtualenv and virtualenvwrapper. To enable it, we need to add a line to our .bashrc file and log out and back in.

1sudo pip install virtualenv virtualenvwrapper
2echo "" >> .bashrc
3echo "source /usr/local/bin/virtualenvwrapper.sh" >> .bashrc
4source .bashrc

Make a virtualenv

1mkvirtualenv project_env

Install postgres adminpack

1sudo -u postgres psql
2CREATE EXTENSION "adminpack";
3\q

Change postgres password & create database

1sudo passwd postgres
2sudo su - postgres
3psql -d template1 -c "ALTER USER postgres WITH PASSWORD 'changeme';"
4createdb projectdb
5createuser username --pwprompt
6psql -d template1 -U postgres
7GRANT ALL PRIVILEGES ON DATABASE projectdb to username;
8\q
9exit

Install RabbitMQ

1sudo rabbitmqctl add_user username username_pw
2sudo rabbitmqctl add_vhost username_vhost
3sudo rabbitmqctl set_permissions -p username_vhost username ".<em>" ".</em>" ".*"
4sudo rabbitmqctl clear_permissions -p username_vhost guest

Generate ssh key to upload to Github, Bitbucket, or wherever you host your code.

1ssh-keygen -t rsa -C you@sample.com
2cat ~/.ssh/id_rsa.pub

Create some /var/www dirs & set permissions on these directories.

1sudo mkdir -p /var/www/static
2sudo mkdir /var/www/media
3sudo chown -R username:www-data /var/www

Clone your repository to your home directory and install the packages in your requirements file.

1git clone git@bitbucket.org:yourusername/project.git
2cd project/requirements
3pip install -r prod.txt

Remove the default symbolic link for Nginx. Create a new blank config, and make a symlink to it. Edit the new configuration file.

1sudo rm /etc/nginx/sites-enabled/default
2sudo touch /etc/nginx/sites-available/project
3cd /etc/nginx/sites-enabled
4sudo ln -s ../sites-available/project
5sudo vim /etc/nginx/sites-available/project

Add the following content to nginx config:

define an upstream server named gunicorn on localhost port 8000

1upstream gunicorn { server localhost:8000; }

make an nginx server

1server {
2# listen on port 80
3listen 80;
4
5# for requests to these domains
6server_name yourdomain.com www.yourdomain.com;
7
8# look in this directory for files to serve
9root /var/www/;
10
11# keep logs in these files
12access_log /var/log/nginx/project.access.log;
13error_log /var/log/nginx/project.error.log;
14
15# You need this to allow users to upload large files
16# See http://wiki.nginx.org/HttpCoreModule#client_max_body_size
17# I'm not sure where it goes, so I put it in twice. It works.
18client_max_body_size 0;
19
20# THIS IS THE IMPORTANT LINE
21# this tries to serve a static file at the requested url
22# if no static file is found, it passes the url to gunicorn
23try_files $uri @gunicorn;
24
25# define rules for gunicorn
26location @gunicorn {
27 # repeated just in case
28 client_max_body_size 0;
29
30 # proxy to the gunicorn upstream defined above
31 proxy_pass http://gunicorn;
32
33 # makes sure the URLs don't actually say http://gunicorn
34 proxy_redirect off;
35
36 # If gunicorn takes > 5 minutes to respond, give up
37 # Feel free to change the time on this
38 proxy_read_timeout 5m;
39
40 # make sure these HTTP headers are set properly
41 proxy_set_header Host $host;
42 proxy_set_header X-Real-IP $remote_addr;
43 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
44}
45
46} server {
47listen 443 ssl;
48# start mine
49ssl on;
50ssl_certificate /etc/ssl/localcerts/yourdomain_com.crt;
51ssl_certificate_key /etc/ssl/localcerts/yourdomain.com.key;
52ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
53ssl_ciphers HIGH:!aNULL:!MD5:!kEDH;
54server_name yourdomain.com www.yourdomain.com;
55
56# look in this directory for files to serve
57root /var/www/;
58
59# keep logs in these files
60access_log /var/log/nginx/project.access.log;
61error_log /var/log/nginx/project.error.log;
62
63# You need this to allow users to upload large files
64# See http://wiki.nginx.org/HttpCoreModule#client_max_body_size
65# I'm not sure where it goes, so I put it in twice. It works.
66client_max_body_size 0;
67
68# THIS IS THE IMPORTANT LINE
69# this tries to serve a static file at the requested url
70# if no static file is found, it passes the url to gunicorn
71try_files $uri @gunicorn;
72
73# define rules for gunicorn
74location @gunicorn {
75 # repeated just in case
76 client_max_body_size 0;
77
78 # proxy to the gunicorn upstream defined above
79 proxy_pass http://gunicorn;
80
81 # makes sure the URLs don't actually say http://gunicorn
82 proxy_redirect off;
83
84 # If gunicorn takes > 5 minutes to respond, give up
85 # Feel free to change the time on this
86 proxy_read_timeout 5m;
87
88 # make sure these HTTP headers are set properly
89 proxy_set_header Host $host;
90 proxy_set_header X-Real-IP $remote_addr;
91 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
92}
93}

Restart nginx

1sudo service nginx restart

Set up database

1cd /home/username/project
2python manage.py syncdb --settings=project.settings.prod
3python manage.py migrate --settings=project.settings.prod

Run collectstatic command

1python manage.py collectstatic -l --noinput --settings=project.settings.prod
2sudo /etc/init.d/nginx restart

Configure supervisor

Add the following contents to /etc/supervisor/conf.d/celeryd.conf

1sudo vim /etc/supervisor/conf.d/celeryd.conf

Contents:

1# the name of this service as far as supervisor is concerned</h1>
2[program:celeryd]
3
4# the command to start celery
5command = /home/username/.virtualenvs/project_env/bin/python /home/username/project/manage.py celeryd -B -E --settings=project.settings.prod</pre>
6# the directory to be in while running this
7directory = /home/username/project</pre>
8
9# the user to run this service as
10user = username</pre>
11
12# start this at boot, and restart it if it fails
13autostart = true autorestart = true<
14
15# take stdout and stderr of celery and write to these log files
16stdout_logfile = /var/log/supervisor/celeryd.log stderr_logfile =
17/var/log/supervisor/celeryd_err.log

Now we will create CeleryCam in /etc/supervisor/conf.d/celerycam.conf

1sudo vim /etc/supervisor/conf.d/celerycam.conf

Contents:

1[program:celerycam]
2command = /home/username/.virtualenvs/project_env/bin/python /home/username/project/manage.py celerycam --settings=project.settings.prod
3directory = /home/username/project
4user = username
5autostart = true
6autorestart = true
7stdout_logfile = /var/log/supervisor/celerycam.log
8stderr_logfile = /var/log/supervisor/celerycam_err.log

Create Gunicorn script in /etc/supervisor/conf.d/gunicorn.conf

1sudo vim /etc/supervisor/conf.d/gunicorn.conf

Contents:

1[program:gunicorn]
2command = /home/username/.virtualenvs/project_env/bin/python /home/username/project/manage.py run_gunicorn -w 4 -k gevent --settings=project.settings.prod
3directory = /home/username/project
4user = username
5autostart = true
6autorestart = true
7stdout_logfile = /var/log/supervisor/gunicorn.log
8stderr_logfile = /var/log/supervisor/gunicorn_err.log

Restart supervisor

1sudo service supervisor restart

Restart/stop/start all services managed by supervisor

1sudo supervisorctl restart all sudo supervisorctl stop all sudo
2supervisorctl start all

Or restart just celeryd

1sudo supervisorctl restart celeryd

Or, start just gunicorn

1sudo supervisorctl start gunicorn

Reboot and make sure everything starts up

1sudo reboot

Bonus: set up ssl

1sudo mkdir /etc/ssl/localcerts
2cd /etc/ssl/localcerts
3sudo openssl req -new -nodes -days 365 -keyout yourdomain.com.key -out yourdomain.com.csr
4sudo chmod 400 /etc/ssl/localcerts/yourdomain.com.key
5sudo chmod 400 /etc/ssl/localcerts/yourdomain.com.crt

A little extra...

Use vim as the default editor:

1sudo update-alternatives --config editor

Don't require password for sudo user:

1sudo visudo

Add the following line at the bottom of sudoers file:

1username ALL=(ALL) NOPASSWD: ALL

Discuss on TwitterEdit post on GitHub

Share article
Dustin Davis

Dustin Davis is a software engineer, people manager, hacker, and entreprenuer. He loves to develop systems and automation. He lives with his wife and five kids in Utah.

Join the Newsletter



Dustin Davis