Setting up a Django Server with PostgreSQL, Nginx, Celery & RabbitMQ
Photo by Kevin Horvat on Unsplash
Step-by-step guide to setting up a Django server with all the works.
Disclaimer: I am not a sysadmin. I'm just a developer. I welcome and encourage comments to improve this process!
I have set up a couple of Django servers lately and taken copious notes that I have extracted from various sources. Below are the commands I issue to a fresh Ubuntu server install to get Django up and running. This puts everything on one server (PostgreSQL, Celery, RabbitMQ, etc) so it's nice for a small starter project but don't expect it to scale.
Log in as root and add a non-root user. Add the user to the sudoers group. Log out and log back in as 'username'.
1adduser username2adduser username sudo3exit
Update the local package index. Upgrade all the packages that can be upgraded. Remove packages that are no longer needed and then reboot for good measure.
1sudo apt update2sudo apt dist-upgrade3sudo apt autoremove4sudo reboot
Install libraries for Python, PIP, PIL/Pillow, PostgreSQL, libevent for gevent, memcached server and library, RabbitMQ, git, nginx, & supervisor
1sudo apt install build-essential python2-dev libjpeg8-dev libfreetype6-dev \2 zlib1g-dev postgresql postgresql-contrib libpq-dev libevent-dev memcached \3 libmemcached-dev rabbitmq-server git nginx supervisor
Install virtualenv and virtualenvwrapper. To enable it, we need to add a line to our .bashrc file and log out and back in.
1sudo pip install virtualenv virtualenvwrapper2echo "" >> .bashrc3echo "source /usr/local/bin/virtualenvwrapper.sh" >> .bashrc4source .bashrc
Make a virtualenv
1mkvirtualenv project_env
Install postgres adminpack
1sudo -u postgres psql2CREATE EXTENSION "adminpack";3\q
Change postgres password & create database
1sudo passwd postgres2sudo su - postgres3psql -d template1 -c "ALTER USER postgres WITH PASSWORD 'changeme';"4createdb projectdb5createuser username --pwprompt6psql -d template1 -U postgres7GRANT ALL PRIVILEGES ON DATABASE projectdb to username;8\q9exit
Install RabbitMQ
1sudo rabbitmqctl add_user username username_pw2sudo rabbitmqctl add_vhost username_vhost3sudo rabbitmqctl set_permissions -p username_vhost username ".<em>" ".</em>" ".*"4sudo rabbitmqctl clear_permissions -p username_vhost guest
Generate ssh key to upload to Github, Bitbucket, or wherever you host your code.
1ssh-keygen -t rsa -C you@sample.com2cat ~/.ssh/id_rsa.pub
Create some /var/www
dirs & set permissions on these directories.
1sudo mkdir -p /var/www/static2sudo mkdir /var/www/media3sudo chown -R username:www-data /var/www
Clone your repository to your home directory and install the packages in your requirements file.
1git clone git@bitbucket.org:yourusername/project.git2cd project/requirements3pip install -r prod.txt
Remove the default symbolic link for Nginx. Create a new blank config, and make a symlink to it. Edit the new configuration file.
1sudo rm /etc/nginx/sites-enabled/default2sudo touch /etc/nginx/sites-available/project3cd /etc/nginx/sites-enabled4sudo ln -s ../sites-available/project5sudo vim /etc/nginx/sites-available/project
Add the following content to nginx config:
define an upstream server named gunicorn on localhost port 8000
1upstream gunicorn { server localhost:8000; }
make an nginx server
1server {2# listen on port 803listen 80;45# for requests to these domains6server_name yourdomain.com www.yourdomain.com;78# look in this directory for files to serve9root /var/www/;1011# keep logs in these files12access_log /var/log/nginx/project.access.log;13error_log /var/log/nginx/project.error.log;1415# You need this to allow users to upload large files16# See http://wiki.nginx.org/HttpCoreModule#client_max_body_size17# I'm not sure where it goes, so I put it in twice. It works.18client_max_body_size 0;1920# THIS IS THE IMPORTANT LINE21# this tries to serve a static file at the requested url22# if no static file is found, it passes the url to gunicorn23try_files $uri @gunicorn;2425# define rules for gunicorn26location @gunicorn {27 # repeated just in case28 client_max_body_size 0;2930 # proxy to the gunicorn upstream defined above31 proxy_pass http://gunicorn;3233 # makes sure the URLs don't actually say http://gunicorn34 proxy_redirect off;3536 # If gunicorn takes > 5 minutes to respond, give up37 # Feel free to change the time on this38 proxy_read_timeout 5m;3940 # make sure these HTTP headers are set properly41 proxy_set_header Host $host;42 proxy_set_header X-Real-IP $remote_addr;43 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;44}4546} server {47listen 443 ssl;48# start mine49ssl on;50ssl_certificate /etc/ssl/localcerts/yourdomain_com.crt;51ssl_certificate_key /etc/ssl/localcerts/yourdomain.com.key;52ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;53ssl_ciphers HIGH:!aNULL:!MD5:!kEDH;54server_name yourdomain.com www.yourdomain.com;5556# look in this directory for files to serve57root /var/www/;5859# keep logs in these files60access_log /var/log/nginx/project.access.log;61error_log /var/log/nginx/project.error.log;6263# You need this to allow users to upload large files64# See http://wiki.nginx.org/HttpCoreModule#client_max_body_size65# I'm not sure where it goes, so I put it in twice. It works.66client_max_body_size 0;6768# THIS IS THE IMPORTANT LINE69# this tries to serve a static file at the requested url70# if no static file is found, it passes the url to gunicorn71try_files $uri @gunicorn;7273# define rules for gunicorn74location @gunicorn {75 # repeated just in case76 client_max_body_size 0;7778 # proxy to the gunicorn upstream defined above79 proxy_pass http://gunicorn;8081 # makes sure the URLs don't actually say http://gunicorn82 proxy_redirect off;8384 # If gunicorn takes > 5 minutes to respond, give up85 # Feel free to change the time on this86 proxy_read_timeout 5m;8788 # make sure these HTTP headers are set properly89 proxy_set_header Host $host;90 proxy_set_header X-Real-IP $remote_addr;91 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;92}93}
Restart nginx
1sudo service nginx restart
Set up database
1cd /home/username/project2python manage.py syncdb --settings=project.settings.prod3python manage.py migrate --settings=project.settings.prod
Run collectstatic command
1python manage.py collectstatic -l --noinput --settings=project.settings.prod2sudo /etc/init.d/nginx restart
Configure supervisor
Add the following contents to /etc/supervisor/conf.d/celeryd.conf
1sudo vim /etc/supervisor/conf.d/celeryd.conf
Contents:
1# the name of this service as far as supervisor is concerned</h1>2[program:celeryd]34# the command to start celery5command = /home/username/.virtualenvs/project_env/bin/python /home/username/project/manage.py celeryd -B -E --settings=project.settings.prod</pre>6# the directory to be in while running this7directory = /home/username/project</pre>89# the user to run this service as10user = username</pre>1112# start this at boot, and restart it if it fails13autostart = true autorestart = true<1415# take stdout and stderr of celery and write to these log files16stdout_logfile = /var/log/supervisor/celeryd.log stderr_logfile =17/var/log/supervisor/celeryd_err.log
Now we will create CeleryCam in /etc/supervisor/conf.d/celerycam.conf
1sudo vim /etc/supervisor/conf.d/celerycam.conf
Contents:
1[program:celerycam]2command = /home/username/.virtualenvs/project_env/bin/python /home/username/project/manage.py celerycam --settings=project.settings.prod3directory = /home/username/project4user = username5autostart = true6autorestart = true7stdout_logfile = /var/log/supervisor/celerycam.log8stderr_logfile = /var/log/supervisor/celerycam_err.log
Create Gunicorn script in /etc/supervisor/conf.d/gunicorn.conf
1sudo vim /etc/supervisor/conf.d/gunicorn.conf
Contents:
1[program:gunicorn]2command = /home/username/.virtualenvs/project_env/bin/python /home/username/project/manage.py run_gunicorn -w 4 -k gevent --settings=project.settings.prod3directory = /home/username/project4user = username5autostart = true6autorestart = true7stdout_logfile = /var/log/supervisor/gunicorn.log8stderr_logfile = /var/log/supervisor/gunicorn_err.log
Restart supervisor
1sudo service supervisor restart
Restart/stop/start all services managed by supervisor
1sudo supervisorctl restart all sudo supervisorctl stop all sudo2supervisorctl start all
Or restart just celeryd
1sudo supervisorctl restart celeryd
Or, start just gunicorn
1sudo supervisorctl start gunicorn
Reboot and make sure everything starts up
1sudo reboot
Bonus: set up ssl
1sudo mkdir /etc/ssl/localcerts2cd /etc/ssl/localcerts3sudo openssl req -new -nodes -days 365 -keyout yourdomain.com.key -out yourdomain.com.csr4sudo chmod 400 /etc/ssl/localcerts/yourdomain.com.key5sudo chmod 400 /etc/ssl/localcerts/yourdomain.com.crt
A little extra...
Use vim as the default editor:
1sudo update-alternatives --config editor
Don't require password for sudo user:
1sudo visudo
Add the following line at the bottom of sudoers file:
1username ALL=(ALL) NOPASSWD: ALL