Debugging Django Management Commands in PyCharm

Photo by Jill Heyer

My favorite editor for Python projects is PyCharm. Besides editing code, it allows you to inspect the database, work with Git repositories, run management commands, execute bash commands and Python scripts, and debug code just in the same window. In this article, I will show you how to set breakpoints and debug Django management commands visually in PyCharm.

Django management commands are scripts that can be executed on your Django project to do something with the project database, media files, or code. Django itself comes with a bunch of commands like: migrate, runserver, collectstatic, makemessages, and clearsessions. Management commands can be executed like this:

(myproject_env)$ python manage.py clearsessions

If you want to create a custom management command in your project, you can find how to do that in the official Django documentation. Also you can find some practical examples in the Chapter 9, Data Import and Export of the Web Development with Django Cookbook - Second Edition.

In this example, I won't create any new management command, but will debug the clearsessions command that is coming from Django and is located at django/contrib/sessions/management/commands/clearsessions.py.

First of all, let's click on "Edit Configurations..." in the top toolbar just before the Run button (with the Play icon). In the opened dialog box "Run/Debug Configurations" click on the "Add New Configuration" button (with the Plus icon) and choose "Python".

Let's fill in the configuration with these values:

Name: Clear Outdated Sessions
Script: /Users/me/DjangoProjects/myproject_env/project/myproject/manage.py
Script paramethers: clearsessions --traceback --verbosity=2
Python interpreter: Python 2.7.6 virtualenv at ~/DjangoProjects/myproject_env
Working directory: /Users/me/DjangoProjects/myproject_env/project/myproject/

Then open the file with the definition of the management command django/contrib/sessions/management/commands/clearsessions.py. Click on the left padding of the editor to add a breakpoint (marked with a red circle) where the script should stop executing for inspection.

Normally to run this script, you could click on the Run button (with the Play icon). But as we want to debug the script, we will click on the Debug button (with the Bug icon) in the toolbar.

The script will start executing and will stop temporarily at the breakpoint you made. You will be able to inspect all local variables in the debug panel that is opened at the bottom of your window by default.

You can navigate through code execution with the arrow buttons "Step Over", "Step Into", etc. To evaluate local or global variables or values, click on the "Evaluate Expression" button (with the Calculator icon) and enter some Python code.

Click on the "Resume Program" button (with the Fast Play icon) to continue execution of the script, when you are ready.

Analogously, you can debug your models, views, forms, or middlewares by running a development server ("Django server") in debug mode.


Deploying a Django Website on Heroku

Photo by Frances Gunn

Once you have a working project, you have to host it somewhere. One of the most popular deployment platforms nowadays is Heroku. Heroku belongs to a Platform as a Service (PaaS) category of cloud computing services. Every Django project you host on Heroku is running inside a smart container in a fully managed runtime environment. Your project can scale horizontally (adding more computing machines) and you pay for what you use starting with a free tier. Moreover, you won't need much of system administrator's skills to do the deployment - once you do the initial setup, the further deployment is as simple as pushing Git repository to a special heroku remote.

However, there are some gotchas to know before choosing Heroku for your Django project:

  • One uses PostgreSQL database with your project. MySQL is not an option.
  • You cannot store your static and media files on Heroku. One should use Amazon S3 or some other storage for that.
  • There is no mailing server associated with Heroku. One can use third-party SendGrid plugin with additional costs, GMail SMTP server with sent email amount limitations, or some other SMTP server.
  • The Django project must be version-controlled under Git.
  • Heroku works with Python 2.7. Python 3 is not yet supported.

Recently I deployed a small Django project on Heroku. To have a quick reference for the future, I summed up the process here providing instructions how to do that for future reference.

1. Install Heroku Toolbelt

Sign up for a Heroku account. Then install Heroku tools for doing all the deployment work in the shell.

To connect your shell with Heroku, type:

$ heroku login

When asked, enter your Heroku account's email and password.

2. Prepare Pip Requirements

Activate your project's virtual environment and install Python packages required for Heroku:

(myproject_env)$ pip install django-toolbelt

This will install django, psycopg2, gunicorn, dj-database-url, static3, and dj-static to your virtual environment.

Install boto and Django Storages to be able to store static and media files on an S3 bucket:

(myproject_env)$ pip install boto
(myproject_env)$ pip install django-storages

Go to your project's directory and create the pip requirements that Heroku will use in the cloud for your project:

(myproject_env)$ pip freeze -l > requirements.txt

3. Create Heroku-specific Files

You will need two files to tell Heroku what Python version to use and how to start a webserver.

In your project's root directory create a file named runtime.txt with the following content:


Then at the same location create a file named Procfile with the following content:

web: gunicorn myproject.wsgi --log-file -

4. Configure the Settings

As mentioned in the "Web Development with Django Cookbook - Second Edition", we keep the developmnent and production settings in separate files both importing the common settings from a base file.

Basically we have myproject/conf/base.py with the settings common for all environments.

Then myproject/conf/dev.py contains the local database and dummy email configuration as follows:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .base import *

    "default": {
        "CONN_MAX_AGE": 0,
        "ENGINE": "django.db.backends.postgresql",
        "HOST": "localhost",
        "NAME": "myproject",
        "PASSWORD": "",
        "PORT": "",
        "USER": "postgres"

EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"

Lastly for the production settings we need myproject/conf/prod.py with special database configuration, non-debug mode, and unrestrictive allowed hosts as follows:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .base import *
import dj_database_url

    "default": dj_database_url.config()


DEBUG = False

Now let's open myproject/settings.py and add the following content:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .conf.dev import *

Finally, open the myproject/wsgi.py and change the location of the DJANGO_SETTINGS_MODULE there:

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.conf.prod")

5. Set Up Amazon S3 for Static and Media Files

Create an Amazon S3 bucket myproject.media at the AWS Console (web interface for Amazon Web Services). Go to the properties of the bucket, expand "Permissions" section, click on the "add bucket policy" button and enter the following:

    "Version": "2008-10-17",
    "Statement": [
            "Sid": "AllowPublicRead",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::myproject.media/*"

This ensures that files on the S3 bucket will be accessible publicly without any API keys.

Go back to your Django project and add storages to the INSTALLED_APPS in myproject/conf/base.py:

    # ...

Media files and static files will be stored on different paths under S3 bucket. To implement that, we need to create two Python classes under a new file myproject/s3utils.py as follows:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from storages.backends.s3boto import S3BotoStorage

class StaticS3BotoStorage(S3BotoStorage):
    Storage for static files.

    def __init__(self, *args, **kwargs):
        kwargs['location'] = 'static'
        super(StaticS3BotoStorage, self).__init__(*args, **kwargs)

class MediaS3BotoStorage(S3BotoStorage):
    Storage for uploaded media files.

    def __init__(self, *args, **kwargs):
        kwargs['location'] = 'media'
        super(MediaS3BotoStorage, self).__init__(*args, **kwargs)

Finally, let's edit the myproject/conf/base.py and add AWS settings:

AWS_S3_SECURE_URLS = False       # use http instead of https
AWS_QUERYSTRING_AUTH = False                # don't add complex authentication-related query parameters for requests
AWS_S3_ACCESS_KEY_ID = "..."                # Your S3 Access Key
AWS_S3_SECRET_ACCESS_KEY = "..."            # Your S3 Secret
AWS_STORAGE_BUCKET_NAME = "myproject.media"
AWS_S3_HOST = "s3-eu-west-1.amazonaws.com"  # Change to the media center you chose when creating the bucket

STATICFILES_STORAGE = "myproject.s3utils.StaticS3BotoStorage"
DEFAULT_FILE_STORAGE = "myproject.s3utils.MediaS3BotoStorage"

# the next monkey patch is necessary to allow dots in the bucket names
import ssl
if hasattr(ssl, '_create_unverified_context'):
   ssl._create_default_https_context = ssl._create_unverified_context

Collect static files to the S3 bucket:

(myproject_env)$ python manage.py collectstatic --noinput

6. Set Up Gmail to Send Emails

Open myproject/conf/prod.py and add the following settings:

EMAIL_HOST = "smtp.gmail.com"
EMAIL_HOST_USER = "myproject@gmail.com"
EMAIL_HOST_PASSWORD = "mygmailpassword"

7. Push to Heroku

Commit and push all the changes to your Git origin remote. Personally I prefer using SourceTree to do that, but you can also do that in the command line, PyCharm, or another software.

In your project directory type the following:

(myproject_env)$ heroku create my-unique-project

This will create a Git remote called "heroku", and a new Heroku project "my-unique-project" which can be later accessed at http://my-unique-project.herokuapp.com.

Push the changes to heroku remote:

(myproject_env)$ git push heroku master

8. Transfer Your Local Postgres Database To Heroku

Create local database dump:

(myproject_env)$ PGPASSWORD=mypassword pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydb > mydb.dump

Upload the database dump temporarily to some server, for example, S3 bucket: http://myproject.media.s3-eu-west-1.amazonaws.com/mydb.dump. Then import that dump into the Heroku database:

(myproject_env)$ heroku pg:backups restore 'http://myproject.media.s3-eu-west-1.amazonaws.com/mydb.dump' DATABASE_URL

Remove the database dump from S3 server.

9. Set Environment Variables

If your Git repository is not private, put your secret values in environment variables rather than in the Git repository directly.

(myproject_env)$ heroku config:set AWS_S3_ACCESS_KEY_ID=ABCDEFG123
$ heroku config:set AWS_S3_SECRET_ACCESS_KEY=aBcDeFg123

To read out the environment variables you can type:

(myproject_env)$ heroku config

To read out the environment variables in the Python code open myproject/conf/base.py and type:

import os
AWS_S3_ACCESS_KEY_ID = os.environ.get("AWS_S3_ACCESS_KEY_ID", "")

10. Set DNS Settings

Open your domain settings and set CNAME to "my-unique-project.herokuapp.com".

At last, you are done! Drop in the comments if I missed some part. For the new updates, see the next section.

*. Update Production

Push the changes to heroku remote:

(myproject_env)$ git push heroku master

If you have changed something in the static files, collect them again:

(myproject_env)$ python manage.py collectstatic --noinput

Collecting static files to S3 bucket takes quite a long time, so I do not recommend to do that automatically every time when you want to deploy to Heroku.

Further Reading

You can read more about Django on Heroku in the following resources:


Special Offer for the Readers of DjangoTricks Blog

Packt Publishing, the company that published my Django book, has a special offer for enthusiast and professional developers reading this blog. For two weeks you can get the eBook "Web Development with Django Cookbook - Second Edition" for half price. The eBook is available in PDF, ePub, Mobi, and Kindle formats. Also you will get access to download the related code files.

Use the discount code DJGTRK50 at the Packt Publishing bookstore.
The discount is valid until the 24th of February, 2016.


Fresh Book for Django Developers

This week the post office delivered a package that made me very satisfied. It was a box with three paper versions of my "Web Development with Django Cookbook - Second Edition". The book was published at the end of January after months of hard, but fulfilling work in the late evenings and at weekends.

The first Django Cookbook was dealing with Django 1.6. Unfortunately, the support for that version is over. So it made sense to write an update for a newer Django version. The second edition was adapted for Django 1.8 which has a long-term support until April 2018 or later. This edition introduces new features added to Django 1.7 and Django 1.8, such as database migrations, QuerySet expressions, or System Check Framework. Most concepts in this new book should also be working with Django 1.9.

My top 5 favourite new recipes are these:

  • Configuring settings for development, testing, staging, and production environments
  • Using database query expressions
  • Implementing a multilingual search with Haystack
  • Testing pages with Selenium
  • Releasing a reusable Django app

The book is worth reading for any Django developer, but will be best understood by those who already know the basics of web development with Django. You can learn more about the book and buy it at the Packt website or Amazon.

I thank the Packt Publishing very much for long cooperation in the development of this book. I am especially thankful to acquisition editor Nadeem N. Bagban, content development editors Arwa Manasawala and Sumeet Sawant, and technical editor Bharat Patil. Also I am grateful for insightful feedback from the reviewer Jake Kronika.

What 5 recipes do you find the most useful?


How to Find the Performance Bottlenecks in Your Django Views?

Once you have your Django projects running, you come to situations, when you need to optimize for performance. The rule of thumb is to find the bottlenecks and then to take action to eliminate them by more idiomatic Python code, database denormalization, caching, or other techniques.

What is a bottleneck? Literally it refers to the top narrow part of a bottle. In engineering, bottleneck is a case where the performance or capacity of an entire system is limited by a single or small number of components or resources.

How to find these parts of your code? The most trivial way is to check the current time before specific code execution and after that code execution, and then count the time difference:

from datetime import datetime
start = datetime.now()
# heavy execution ...
end = datetime.now()
d = end - start  # datetime.timedelta object
print d.total_seconds()  # prints something like 7.861985

However, measuring code performance for Django projects like this is inefficient, because you need a lot of such wrappers for your code until you find which part is the most critical. Also you need a lot of manual computation to find the critical parts.

Recently I found line_profiler module that can inspect the performance of the code line by line. By default, to use line_profiler for your functions, you should decorate them with @profile decorator and then to execute the script:

$ kernprof -l some_script_to_profile.py

This script will execute your script, analize the decorated function, and will save results to a binary file that can later be inspected with:

$ python -m line_profiler some_script_to_profile.py.lprof

That's quite complicated, but to use line_profiler for Django views, you can install django-devserver which replaces the original development server of Django and will output the performance calculations immediately in the shell like this:

[30/Jan/2015 02:26:40] "GET /quotes/json/ HTTP/1.1" 200 137
    [sql] 1 queries with 0 duplicates
    [profile] Total time to render was 0.01s
    [profile] Timer unit: 1e-06 s

          Total time: 0.001965 s
          File: /Users/archatas/Projects/quotes_env/project/inspirational/quotes/views.py
          Function: quote_list_json at line 27

          Line #      Hits         Time  Per Hit   % Time  Line Contents
              27                                           def quote_list_json(request):
              28         1            2      2.0      0.1      quote_dict_list = []
              29         2         1184    592.0     60.3      for quote in InspirationQuote.objects.all():
              30         1            1      1.0      0.1          quote_dict = {
              31         1            1      1.0      0.1              'author': quote.author,
              32         1            1      1.0      0.1              'quote': quote.quote,
              33         1          363    363.0     18.5              'picture': quote.get_medium_picture_url(),
              34                                                   }
              35         1            1      1.0      0.1          quote_dict_list.append(quote_dict)
              37         1           42     42.0      2.1      json_data = json.dumps(quote_dict_list)
              38         1          370    370.0     18.8      return HttpResponse(json_data, content_type="application/json")

The most interesting data in this table is the "% Time" column, giving an overview in percentage which lines of the Django view function are the most time-consuming. For example, here it says that I should pay the most attention to the QuerySet, the method get_medium_picture_url() and the HttpResponse object.

To setup line profiling, install line_profiler and django-devserver to you virtual environment:

(myproject_env)$ pip install line_profiler
(myproject_env)$ pip install django-devserver

Then make sure that you have the following settings in your settings.py or local_settings.py:

# settings.py
    # ...

    # ...


    # Modules not enabled by default

DEVSERVER_AUTO_PROFILE = True  # profiles all views without the need of function decorator

When you execute

(myproject_env)$ python manage.py runserver

it will run the development server from django-devserver and for each visited view, it will show the analysis of code performance. I have tested this setup with Django 1.7, but it should work since Django 1.3.

Do you know any more useful tools to check for performance bottlenecks?


Win Free Copies of “Web Development with Django Cookbook”

Readers would be pleased to know that I have teamed up with Packt Publishing to organize a Giveaway of my book Web Development with Django Cookbook.

Three lucky winners stand a chance to win an e-copy of the book. Keep reading to find out how you can be one of the lucky ones.

Book Overview

  • Improve your skills by developing models, forms, views, and templates
  • Create a rich user experience using Ajax and other JavaScript techniques
  • A practical guide to writing and using APIs to import or export data

How to Enter?

All you need to do is head on over to the book page and look through the product description of the book and drop a line via the comments below this post to let us know what interests you the most about this book. It’s that simple.


Winners will get an e-copy of the Book.


The contest will close on the 1st of December, 2014 at 00:00 GMT. Winners will be contacted by email, so be sure to use your real email address when you comment!

...And the winners are...

Out of the 21 participants I finally chose these winners: Mark Phillips, Faria Chowdhury, and Mario Gudelj. The guys from Packt Publishing will contact you by email and will give you the free copies of "Web Development with Django Cookbook". Have a nice reading!


Contributing Back to the Community - Django Cookbook

In the early beginning of year 2014, the IT book publishing company "Packt Publishing" contacted me with an interesting offer: to share my Django experience in a form of a book. I thought it might be a good challenge for me and also value for the Django community, as I had been working with Django for 7 years or so, and during that time there was quite a lot of knowledge gathered and used practically. So for the next 9 months on late evenings and weekends I was adapting some of the most useful code snippets and describing them in the book. The great staff from the Packt Publishing helped me to structure the content, make everything clear and understandable, and get the grammar correct. Finally, the book was released and it's called "Web Development with Django Cookbook".

Word cloud of the Web Development with Django Cookbook

This book is written for intermediate or advanced Django web developers. When writing the book, my own purpose was not to fully cover every possible web development task, but rather to have enough useful bits of knowledge for those who seek information about web development with Django. The book was written in the format of recipes. There are over 70 recipes giving you instructions how to deal with different challenges of web development. The code mentioned in the book is optimized for Django 1.6, but most of it should also work with older Django versions as well as with Django 1.7.

The cookbook consists of 10 chapters:

  1. Getting started with Django 1.6. This chapter will guide you through the basic configuration which is necessary to start any Django project. It will cover topics like virtual environment, version control, and project settings.
  2. Database Structure. When you create a new app, the first thing to do is to define your models. In this chapter you will learn how to write reusable pieces of code to use in your models. You will learn how to deal with multilingual data in your database. Also you will be told how to manage database schema changes using South migrations.
  3. Forms and Views. To show your data or let one create it, you need views and forms. This chapter will show you some patterns for creating them.
  4. Templates and JavaScript. Information is presented to the user by rendered templates. In modern websites, JavaScript is a must for richer user experience. This chapter shows practical examples of using templates and JavaScript together.
  5. Custom Template Filters and Tags. The default Django template system is quite extensive, but there are more things to add for different cases. This chapter shows you how to create and use own template tags and filters.
  6. Model Administration. Django framework comes with a handy pre-build administration. This chapter shows how to extend the default administration with your own functionality.
  7. Django CMS. Django CMS is the most popular open source content management system made in Django. This chapter deals with best practices using Django CMS and extending it for your needs.
  8. Hierarchical Structures. When you need to create a tree-like structure in Django, django-mptt module comes to hand. This chapter shows you how to use it and how to set administration for hierarchical structures.
  9. Data Import and Export. Very often there are cases when you need to transfer data from and to different formats, retrieve it from and provide it to different sources. This chapter deals with management commands for data import and also APIs for data export.
  10. Bells and Whistles. There is more to Django. This chapter shows additional snippets and tricks useful in web development with Django. Also you will learn about deployment of Django projects.

Get a copy of this book at Packt Publishing and tell me what you think about it in the comments below. Which recipes did you find the most useful? What would you like to read more in this blog or another edition of the cookbook?

Web Development with Django Cookbook


How to Export Your Data as CSV, XLS, or XLSX

There are times, when you need to export the data from your database to different formats. For example, you want to create some diagrams in Office program for a presentation. In this post I will show you how to create admin actions which export selected items as files for a spreadsheet application (like MS Excel, OpenOffice Calc, LibreOffice Calc, Gnumeric, or Numbers). I will cover the mostly used formats: Comma Separated Values (CSV), Binary Excel (XLS), and Office Open XML (XLSX).

First of all, have a look at the model we will be dealing with. It's a simple model with title, description, and - of course - the id.

# models.py
from django.db import models
class MyModel(models.Model):
    title = models.CharField(max_length=100)
    description = models.TextField(blank=True)

In the admininstration options, we'll define three admin actions: export_csv, export_xls, and export_xlsx.

# admin.py
from django.contrib import admin
from models import MyModel

# ... export functions will go here ...

class MyModelAdmin(admin.Admin):
    actions = [export_csv, export_xls, export_xlsx]

admin.site.register(MyModel, MyModelAdmin)

Now let's create functions for each of those actions!

Comma-Separated Values Format

CSV is the most common import and export format for spreadsheets and databases. It's a textual format which one could easily create or parse himself, but there is also a python built-in library csv for handy data manipulation.

def export_csv(modeladmin, request, queryset):
    import csv
    from django.utils.encoding import smart_str
    response = HttpResponse(mimetype='text/csv')
    response['Content-Disposition'] = 'attachment; filename=mymodel.csv'
    writer = csv.writer(response, csv.excel)
    response.write(u'\ufeff'.encode('utf8')) # BOM (optional...Excel needs it to open UTF-8 file properly)
    for obj in queryset:
    return response
export_csv.short_description = u"Export CSV"

As you can see, HttpResponse is a file-like object and we used it to write data to.

Excel Binary File Format

XLS is the main spreadsheet format which holds data in worksheets, charts, and macros. We are going to use xlwt library to create a spreadsheet. There is analogous library xlrd to read XLS files. Note, that this format allows to have only 256 columns.

def export_xls(modeladmin, request, queryset):
    import xlwt
    response = HttpResponse(mimetype='application/ms-excel')
    response['Content-Disposition'] = 'attachment; filename=mymodel.xls'
    wb = xlwt.Workbook(encoding='utf-8')
    ws = wb.add_sheet("MyModel")
    row_num = 0
    columns = [
        (u"ID", 2000),
        (u"Title", 6000),
        (u"Description", 8000),

    font_style = xlwt.XFStyle()
    font_style.font.bold = True

    for col_num in xrange(len(columns)):
        ws.write(row_num, col_num, columns[col_num][0], font_style)
        # set column width
        ws.col(col_num).width = columns[col_num][1]

    font_style = xlwt.XFStyle()
    font_style.alignment.wrap = 1
    for obj in queryset:
        row_num += 1
        row = [
        for col_num in xrange(len(row)):
            ws.write(row_num, col_num, row[col_num], font_style)
    return response
export_xls.short_description = u"Export XLS"

Here we created one worksheet, filled it with data, marked the first row in bold, and made the lines in the other cells wrapped. Also we set the width for each column. We'll do the same in the next format too.

Office Open XML Format

XLSX (a.k.a. OOXML or OpenXML) is a zipped, XML-based file format developed by Microsoft. It is fully supported by Microsoft Office 2007 and newer versions. OpenOffice 4.0, for example, can only read it. There is a python library openpyxl for reading and writing those files. This format is great when you need more than 256 columns and text formatting options.

def export_xlsx(modeladmin, request, queryset):
    import openpyxl
    from openpyxl.cell import get_column_letter
    response = HttpResponse(mimetype='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
    response['Content-Disposition'] = 'attachment; filename=mymodel.xlsx'
    wb = openpyxl.Workbook()
    ws = wb.get_active_sheet()
    ws.title = "MyModel"

    row_num = 0

    columns = [
        (u"ID", 15),
        (u"Title", 70),
        (u"Description", 70),

    for col_num in xrange(len(columns)):
        c = ws.cell(row=row_num + 1, column=col_num + 1)
        c.value = columns[col_num][0]
        c.style.font.bold = True
        # set column width
        ws.column_dimensions[get_column_letter(col_num+1)].width = columns[col_num][1]

    for obj in queryset:
        row_num += 1
        row = [
        for col_num in xrange(len(row)):
            c = ws.cell(row=row_num + 1, column=col_num + 1)
            c.value = row[col_num]
            c.style.alignment.wrap_text = True

    return response

export_xlsx.short_description = u"Export XLSX"


So whenever you need to get your Django project data to some spreadsheet application, there are several ways to do that. If you are planning to import the data to some other database, CSV is probably the best, as it is simple, straightforward, and requires no third-party libraries. However, if you need your data with nice formatting and maybe some statistical formulas, you should export XLS or XLSX format. The maximum amount of columns in XLS format is limited to 256, whereas XLSX format allows more columns, but is not fully supported by all spreadsheet applications.


How to Store Your Media Files in Amazon S3 Bucket

In this article, I will show you how to use Amazon Simple Storage Service (S3) to store your media files in the cloud. S3 is known and widely used for its scalability, reliability, and relatively cheap price. It is free to join and you only pay the hosting and bandwidth costs as you use it. The service is provided by Amazon.com. S3 tends to be attractive for start-up companies looking to minimize costs.

S3 uses a concept of buckets which is like a storage database. Each bucket has its own url. Inside the buckets you have folders and under that you have files. In fact, directories don't actually exist within S3 buckets. The entire file structure is actually just one flat single-level container of files. The illusion of directories is actually created based on having the file names like dirA/dirB/file.

If you want to browse the files in a folder-like structure, you can use Transmit FTP client on Mac OS X. It supports S3 services. Amazon browser-based console also has interface for browsing or uploading files.

OK. Now let's have a look how to set up a Django project which will use S3 for media files.

1. Create a bucket

At first you will need to create a bucket at S3 and make it accessible for all visitors. Login to your Amazon Web Services console. Click on "Services" in the menu, then on the "S3". Click on the button "Create bucket" and enter your bucket name like "mywebsite.media" or even better without dots like "mywebsitemedia". Choose a region there which is the closest to your target audience, for example, if your website is for Europeans, choose "Ireland". Go to the properties of the bucket, expand "Permissions" section, click on the "add bucket policy" button and enter the following:

    "Version": "2008-10-17",
    "Statement": [
            "Sid": "AllowPublicRead",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::mywebsite.media/*"

2. Install boto and django-storages

Amazon Web Services provide a python library called boto for accessing the API. There is a django app called django-storages which allows to use AWS S3 as the main storage. So your next step is to activate your virtual environment and install latest versions of boto and django-storages.

pip install boto==2.19.0
pip install django-storages==1.1.8

3. Set up django-storages for your project

Add the following to the settings.py:

    # ...
    # ...
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_S3_SECURE_URLS = False       # use http instead of https
AWS_QUERYSTRING_AUTH = False     # don't add complex authentication-related query parameters for requests
AWS_S3_ACCESS_KEY_ID = '...'     # enter your access key id
AWS_S3_SECRET_ACCESS_KEY = '...' # enter your secret access key
AWS_STORAGE_BUCKET_NAME = 'mywebsite.media'

# the next monkey patch is necessary if you use dots in the bucket name
import ssl
if hasattr(ssl, '_create_unverified_context'):
   ssl._create_default_https_context = ssl._create_unverified_context

4. Create your models with FileField or ImageField

Let's create a Profile model with avatar field.

def upload_avatar_to(instance, filename):
    import os
    from django.utils.timezone import now
    filename_base, filename_ext = os.path.splitext(filename)
    return 'profiles/%s%s' % (

class Profile(models.Model):
    # ...
    avatar = models.ImageField(_("Avatar"), upload_to=upload_avatar_to, blank=True)
    # ...

Whenever you save an instance of the Profile with the new avatar picture, avatar will be uploaded to S3 bucket. To show it in a template, you will need something like <img src="{{ profile.avatar.url }}" alt="" /> where the image source will look like "http://mywebsite.media.s3.amazonaws.com/profiles/20140214203012.jpg".

5. Use the storage to manipulate file versions

If you need to create a thumbnail version of your image, it's probably best to overwrite the save method of the model and trigger the generation of the thumbs there. Let's add some methods to the Profile class:

class Profile(models.Model):
    # ...

    def save(self, *args, **kwargs):
        super(Profile, self).save(*args, **kwargs)

    def create_avatar_thumb(self):
        import os
        from PIL import Image
        from django.core.files.storage import default_storage as storage
        if not self.avatar:
            return ""
        file_path = self.avatar.name
        filename_base, filename_ext = os.path.splitext(file_path)
        thumb_file_path = "%s_thumb.jpg" % filename_base
        if storage.exists(thumb_file_path):
            return "exists"
            # resize the original image and return url path of the thumbnail
            f = storage.open(file_path, 'r')
            image = Image.open(f)
            width, height = image.size

            if width > height:
                delta = width - height
                left = int(delta/2)
                upper = 0
                right = height + left
                lower = height
                delta = height - width
                left = 0
                upper = int(delta/2)
                right = width
                lower = width + upper

            image = image.crop((left, upper, right, lower))
            image = image.resize((50, 50), Image.ANTIALIAS)

            f_thumb = storage.open(thumb_file_path, "w")
            image.save(f_thumb, "JPEG")
            return "success"
            return "error"

    def get_avatar_thumb_url(self):
        import os
        from django.core.files.storage import default_storage as storage
        if not self.avatar:
            return ""
        file_path = self.avatar.name
        filename_base, filename_ext = os.path.splitext(file_path)
        thumb_file_path = "%s_thumb.jpg" % filename_base
        if storage.exists(thumb_file_path):
            return storage.url(thumb_file_path)
        return ""

As you might have guessed, the avatar can be placed in the template using something like

{% if profile.get_avatar_thumb_url %}
    <img src="{{ profile.get_avatar_thumb_url }}" alt="" />
{% endif %}

where the image source will look like "http://mywebsite.media.s3.amazonaws.com/profiles/20140214203012_thumb.jpg".


When you have a basic overview about Amazon Simple Storage Service, it is quite easy to use it in Django projects with existing third-party libraries. For flexibility, if you need to modify uploaded files, storage object should be used instead of the default os methods. That way, you can simply switch to the default file storage for local development.


How to Create a File Upload Widget

Ajaxified file uploads are becoming de facto standard on the web. People want to see what they chose right after selecting a file instead of during submit of the form. Also if a form has validation errors, nobody wants to select the files again; the selection made should be still available in the form. Therefore, we need a solution that works by Ajax.

Luckily there is a Django app that can help. It's called django-ajax-uploader. Ajax uploader allows to create asynchronous single and multiple file uploads. All uploads are done into the uploads directory. In addition to the default functionality we need a possibility to show the preview of the temporary uploaded file, translate all messages and buttons, delete a file, move a file to a specific directory. I created a demo project which implements all that.

If you have a ModelForm with a FileField or ImageField, you need to not show this field in the form, but instead to add a hidden field for uploaded file path. When you choose a file, it will be uploaded to a temporary uploads directory and its path will be set in the hidden field. Also you need a hidden BooleanField for the state of file deletion. When you submit the form, the file will be moved from the temporary location to the correct location defined by the FileField or ImageField.

In the given example, besides django-ajax-uploader, I am using django-crispy-forms with bootstrap 3.0.0 support. It allows you to define the structure for the forms without creating repetitive HTML markup.

This would be a quick workflow of integrating ajax uploads into your project:

  • Install django-ajax-uploader and django-crispy-forms.
  • Put 'crispy_forms' and 'ajaxuploader' into INSTALLED_APPS.
  • Define CRISPY_TEMPLATE_PACK = 'bootstrap3' in the settings.
  • Define the url rule for ajax uploader
    from ajaxuploader.views import AjaxFileUploader
    uploader = AjaxFileUploader()
    urlpatterns = patterns('',
        # ...
        url(r'^helper/ajax-upload/$', uploader, name="ajax_uploader"),
        # ...
  • Create a form for your model which will have the ajax uploader.
  • Create the view which will handle your model and the chosen files and call it in your urls.
  • Create the template with appropriate javascript.

For a working example, please check image uploads project.