2017-03-01

Tracking the Results of Cron Jobs

Every Django website needs some automatic background tasks to execute regularly. The outdated sessions need to be cleaned up, search index needs to be updated, some data needs to be imported from RSS feeds or APIs, backups need to be created, you name it. Usually, if not all the time, those regular tasks are being set as cron jobs. However, when some task is run in the background, by default, you don't get any feedback whether it was successfully completed, or whether it crashed on the way. In this post I will show you how I handle the results of cron jobs.

In a Django project, all those tasks are usually implemented as management commands. For each such command I write a short bash script, that will call the management command with specific parameters and will print the verbose output to a log file.

Let's say my project structure is like this on a remote server:

/home/myproject
├── bin
├── include
├── lib
├── public_html
├── backups
├── project
│   └── myproject
├── scripts
└── logs

A virtual environment is created in the home directory of myproject linux user. The Django project itself is kept under project directory. The scripts directory is for my bash scripts. And the logs directory is for the verbose output of the bash scripts.

For example, for the default clearsessions command that removes outdated sessions, I would create scripts/cleanup.sh bash script as follows:

#!/usr/bin/env bash
SECONDS=0
PROJECT_PATH=/home/myproject
CRON_LOG_FILE=${PROJECT_PATH}/logs/cleanup.log

echo "Cleaning up the database" > ${CRON_LOG_FILE}
date >> ${CRON_LOG_FILE}

cd ${PROJECT_PATH}
source bin/activate
cd project/myproject    
python manage.py clearsessions --verbosity=2 --traceback >> ${CRON_LOG_FILE}  2>&1

echo "Finished." >> ${CRON_LOG_FILE}
duration=$SECONDS
echo "$(($duration / 60)) minutes and $(($duration % 60)) seconds elapsed." >> ${CRON_LOG_FILE}

To run this command every night at 1 AM, you could create a file myproject_crontab with the following content:

MAILTO=""
00 01 * * * /home/myproject/scripts/cleanup.sh

Then register the cron jobs with:

$ crontab myproject_crontab

By such a bash script, I can track:

  • At what time the script was last executed.
  • What is the verbose output of the management command.
  • If the management command broke, what was in the traceback.
  • Whether the command finish executing or hung up.
  • How long it took to run the command.

In addition, this gives me information whether the crontab was registered and whether the cron service was running at all. As I get the total time of execution in minutes and seconds, I can decide how often I can call the cron job regularly so that it doesn't clash with another cron job.

When you have multiple Django management commands, you can group them thematically into single bash script, or you can wrap them into individual bash scripts. After putting them into the crontab, the only thing left is manually checking the logs from time to time.

If you have any suggestions how I could even improve this setup, I would be glad to hear your opinion in the comments.

Here is the Gist of the scripts in this post. To see some examples of custom Django management commands, you can check Chapter 9, Data Import and Export in my book Web Development with Django Cookbook - Second Edition.

2016-12-16

Django Administration: Inlines for Inlines

The default Django model administration comes with a concept of inlines. If you have a one-to-many relationship, you can edit the parent and its children in the same form. However, you are limited in a way that you cannot have inlines under inlines at nested one-to-many relations. For example, you can't show models Painter, Picture, and Review in the same form if one painter may have drawn multiple pictures and each picture may have several reviews.

In this article I would like to share a workaround allowing you to quickly access the inlines of an inline model. The idea is that for every inline you can provide a HTML link leading to the separate form where you can edit the related model and its own relations. It's as simple as that.

For example, in the form of Painter model, you have the instances of Picture listed with specific links "Edit this Picture separately":

When such a link is clicked, the administrator goes to the form of the Picture model which shows the instances of Review model listed underneath:

Let's have a look, how to implement this.

First of all, I will create a gallery app and define the three models there. Nothing fancy here. The important part there is just that the Picture model has a foreign key to the Painter model and the Review model has a foreign key to the Picture model.

# gallery/models.py
# -*- coding: UTF-8 -*-
from __future__ import unicode_literals

import os

from django.db import models
from django.utils.encoding import python_2_unicode_compatible
from django.utils.translation import ugettext_lazy as _
from django.utils.text import slugify


@python_2_unicode_compatible
class Painter(models.Model):
    name = models.CharField(_("Name"), max_length=255)

    class Meta:
        verbose_name = _("Painter")
        verbose_name_plural = _("Painters")

    def __str__(self):
        return self.name


def upload_to(instance, filename):
    filename_base, filename_ext = os.path.splitext(filename)
    return "painters/{painter}/{filename}{extension}".format(
        painter=slugify(instance.painter.name),
        filename=slugify(filename_base),
        extension=filename_ext.lower(),
    )

@python_2_unicode_compatible
class Picture(models.Model):
    painter = models.ForeignKey(Painter, verbose_name=_("Painter"), on_delete=models.CASCADE)
    title = models.CharField(_("Title"), max_length=255)
    picture = models.ImageField(_("Picture"), upload_to=upload_to)

    class Meta:
        verbose_name = _("Picture")
        verbose_name_plural = _("Pictures")

    def __str__(self):
        return self.title


@python_2_unicode_compatible
class Review(models.Model):
    picture = models.ForeignKey(Picture, verbose_name=_("Picture"), on_delete=models.CASCADE)
    reviewer = models.CharField(_("Reviewer name"), max_length=255)
    comment = models.TextField(_("Comment"))

    class Meta:
        verbose_name = _("Review")
        verbose_name_plural = _("Reviews")

    def __str__(self):
        return self.reviewer

Then I will create the administration definition for the models of the gallery app. Here I will set two types of administration for the Picture model:

  • By extending admin.StackedInline I will create administration stacked as inline.
  • By extending admin.ModelAdmin I will create administration in a separate form.

In Django model administration besides usual form fields, you can also include some computed values. This can be done by your fields (or fieldsets) and readonly_fields attributes referring to a callable or a method name.

You can set a translatable label for those computed values by defining short_description attribute for the callable or method. If you want to render some HTML, you can also set the allow_tags attribute to True (otherwise your HTML string will be escaped).

# gallery/admin.py
# -*- coding: UTF-8 -*-
from django.contrib import admin
from django.core.urlresolvers import reverse
from django.utils.translation import ugettext_lazy as _
from django.utils.text import force_text

from .models import Painter, Picture, Review

def get_picture_preview(obj):
    if obj.pk:  # if object has already been saved and has a primary key, show picture preview
        return """<a href="{src}" target="_blank"><img src="{src}" alt="{title}" style="max-width: 200px; max-height: 200px;" /></a>""".format(
            src=obj.picture.url,
            title=obj.title,
        )
    return _("(choose a picture and save and continue editing to see the preview)")
get_picture_preview.allow_tags = True
get_picture_preview.short_description = _("Picture Preview")


class PictureInline(admin.StackedInline):
    model = Picture
    extra = 0
    fields = ["get_edit_link", "title", "picture", get_picture_preview]
    readonly_fields = ["get_edit_link", get_picture_preview]

    def get_edit_link(self, obj=None):
        if obj.pk:  # if object has already been saved and has a primary key, show link to it
            url = reverse('admin:%s_%s_change' % (obj._meta.app_label, obj._meta.model_name), args=[force_text(obj.pk)])
            return """<a href="{url}">{text}</a>""".format(
                url=url,
                text=_("Edit this %s separately") % obj._meta.verbose_name,
            )
        return _("(save and continue editing to create a link)")
    get_edit_link.short_description = _("Edit link")
    get_edit_link.allow_tags = True


@admin.register(Painter)
class PainterAdmin(admin.ModelAdmin):
    save_on_top = True
    fields = ["name"]
    inlines = [PictureInline]


class ReviewInline(admin.StackedInline):
    model = Review
    extra = 0
    fields = ["reviewer", "comment"]


@admin.register(Picture)
class PictureAdmin(admin.ModelAdmin):
    save_on_top = True
    fields = ["painter", "title", "picture", get_picture_preview]
    readonly_fields = [get_picture_preview]
    inlines = [ReviewInline]

In this administration setup, the get_edit_link() method creates a HTML link between the inline and the separate administration form for the Picture model. As you can see, I also added the get_picture_preview() function as a bonus. It is included in both administration definitions for the Picture model and its purpose is to show a preview of the uploaded picture after saving it.

To recap, nested inlines are not supported by Django out of the box. However, you can have your inlines edited in a separate page with the forms linked to each other. For the linking you would use some magic of the readonly_fields attribute.

What if you really need to have inlines under inlines in your project? In that case you might check django-nested-admin and don't hesitate to share your experience with it in the comments.

2016-05-26

Debugging Django Management Commands in PyCharm

My favorite editor for Python projects is PyCharm. Besides editing code, it allows you to inspect the database, work with Git repositories, run management commands, execute bash commands and Python scripts, and debug code just in the same window. In this article, I will show you how to set breakpoints and debug Django management commands visually in PyCharm.

Django management commands are scripts that can be executed on your Django project to do something with the project database, media files, or code. Django itself comes with a bunch of commands like: migrate, runserver, collectstatic, makemessages, and clearsessions. Management commands can be executed like this:

(myproject_env)$ python manage.py clearsessions

If you want to create a custom management command in your project, you can find how to do that in the official Django documentation. Also you can find some practical examples in the Chapter 9, Data Import and Export of the Web Development with Django Cookbook - Second Edition.

In this example, I won't create any new management command, but will debug the clearsessions command that is coming from Django and is located at django/contrib/sessions/management/commands/clearsessions.py.

First of all, let's click on "Edit Configurations..." in the top toolbar just before the Run button (with the Play icon). In the opened dialog box "Run/Debug Configurations" click on the "Add New Configuration" button (with the Plus icon) and choose "Python".

Let's fill in the configuration with these values:

Name: Clear Outdated Sessions
Script: /Users/me/DjangoProjects/myproject_env/project/myproject/manage.py
Script paramethers: clearsessions --traceback --verbosity=2
Python interpreter: Python 2.7.6 virtualenv at ~/DjangoProjects/myproject_env
Working directory: /Users/me/DjangoProjects/myproject_env/project/myproject/

Then open the file with the definition of the management command django/contrib/sessions/management/commands/clearsessions.py. Click on the left padding of the editor to add a breakpoint (marked with a red circle) where the script should stop executing for inspection.

Normally to run this script, you could click on the Run button (with the Play icon). But as we want to debug the script, we will click on the Debug button (with the Bug icon) in the toolbar.

The script will start executing and will stop temporarily at the breakpoint you made. You will be able to inspect all local variables in the debug panel that is opened at the bottom of your window by default.

You can navigate through code execution with the arrow buttons "Step Over", "Step Into", etc. To evaluate local or global variables or values, click on the "Evaluate Expression" button (with the Calculator icon) and enter some Python code.

Click on the "Resume Program" button (with the Fast Play icon) to continue execution of the script, when you are ready.

Analogously, you can debug your models, views, forms, or middlewares by running a development server ("Django server") in debug mode.

2016-05-21

Deploying a Django Website on Heroku

Once you have a working project, you have to host it somewhere. One of the most popular deployment platforms nowadays is Heroku. Heroku belongs to a Platform as a Service (PaaS) category of cloud computing services. Every Django project you host on Heroku is running inside a smart container in a fully managed runtime environment. Your project can scale horizontally (adding more computing machines) and you pay for what you use starting with a free tier. Moreover, you won't need much of system administrator's skills to do the deployment - once you do the initial setup, the further deployment is as simple as pushing Git repository to a special heroku remote.

However, there are some gotchas to know before choosing Heroku for your Django project:

  • One uses PostgreSQL database with your project. MySQL is not an option.
  • You cannot store your static and media files on Heroku. One should use Amazon S3 or some other storage for that.
  • There is no mailing server associated with Heroku. One can use third-party SendGrid plugin with additional costs, GMail SMTP server with sent email amount limitations, or some other SMTP server.
  • The Django project must be version-controlled under Git.
  • Heroku works with Python 2.7. Python 3 is not yet supported.

Recently I deployed a small Django project on Heroku. To have a quick reference for the future, I summed up the process here providing instructions how to do that for future reference.

1. Install Heroku Toolbelt

Sign up for a Heroku account. Then install Heroku tools for doing all the deployment work in the shell.

To connect your shell with Heroku, type:

$ heroku login

When asked, enter your Heroku account's email and password.

2. Prepare Pip Requirements

Activate your project's virtual environment and install Python packages required for Heroku:

(myproject_env)$ pip install django-toolbelt

This will install django, psycopg2, gunicorn, dj-database-url, static3, and dj-static to your virtual environment.

Install boto and Django Storages to be able to store static and media files on an S3 bucket:

(myproject_env)$ pip install boto
(myproject_env)$ pip install django-storages

Go to your project's directory and create the pip requirements that Heroku will use in the cloud for your project:

(myproject_env)$ pip freeze -l > requirements.txt

3. Create Heroku-specific Files

You will need two files to tell Heroku what Python version to use and how to start a webserver.

In your project's root directory create a file named runtime.txt with the following content:

python-2.7.11

Then at the same location create a file named Procfile with the following content:

web: gunicorn myproject.wsgi --log-file -

4. Configure the Settings

As mentioned in the "Web Development with Django Cookbook - Second Edition", we keep the developmnent and production settings in separate files both importing the common settings from a base file.

Basically we have myproject/conf/base.py with the settings common for all environments.

Then myproject/conf/dev.py contains the local database and dummy email configuration as follows:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .base import *

DATABASES = {
    "default": {
        "CONN_MAX_AGE": 0,
        "ENGINE": "django.db.backends.postgresql",
        "HOST": "localhost",
        "NAME": "myproject",
        "PASSWORD": "",
        "PORT": "",
        "USER": "postgres"
    }
}

EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"

Lastly for the production settings we need myproject/conf/prod.py with special database configuration, non-debug mode, and unrestrictive allowed hosts as follows:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .base import *
import dj_database_url

DATABASES = {
    "default": dj_database_url.config()
}

ALLOWED_HOSTS = ["*"]

DEBUG = False

Now let's open myproject/settings.py and add the following content:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from .conf.dev import *

Finally, open the myproject/wsgi.py and change the location of the DJANGO_SETTINGS_MODULE there:

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.conf.prod")

5. Set Up Amazon S3 for Static and Media Files

Create an Amazon S3 bucket myproject.media at the AWS Console (web interface for Amazon Web Services). Go to the properties of the bucket, expand "Permissions" section, click on the "add bucket policy" button and enter the following:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "AllowPublicRead",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::myproject.media/*"
        }
    ]
}

This ensures that files on the S3 bucket will be accessible publicly without any API keys.

Go back to your Django project and add storages to the INSTALLED_APPS in myproject/conf/base.py:

INSTALLED_APPS = [
    # ...
    "storages",
]

Media files and static files will be stored on different paths under S3 bucket. To implement that, we need to create two Python classes under a new file myproject/s3utils.py as follows:

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from storages.backends.s3boto import S3BotoStorage

class StaticS3BotoStorage(S3BotoStorage):
    """
    Storage for static files.
    """

    def __init__(self, *args, **kwargs):
        kwargs['location'] = 'static'
        super(StaticS3BotoStorage, self).__init__(*args, **kwargs)


class MediaS3BotoStorage(S3BotoStorage):
    """
    Storage for uploaded media files.
    """

    def __init__(self, *args, **kwargs):
        kwargs['location'] = 'media'
        super(MediaS3BotoStorage, self).__init__(*args, **kwargs)

Finally, let's edit the myproject/conf/base.py and add AWS settings:

AWS_S3_SECURE_URLS = False       # use http instead of https
AWS_QUERYSTRING_AUTH = False                # don't add complex authentication-related query parameters for requests
AWS_S3_ACCESS_KEY_ID = "..."                # Your S3 Access Key
AWS_S3_SECRET_ACCESS_KEY = "..."            # Your S3 Secret
AWS_STORAGE_BUCKET_NAME = "myproject.media"
AWS_S3_HOST = "s3-eu-west-1.amazonaws.com"  # Change to the media center you chose when creating the bucket

STATICFILES_STORAGE = "myproject.s3utils.StaticS3BotoStorage"
DEFAULT_FILE_STORAGE = "myproject.s3utils.MediaS3BotoStorage"

# the next monkey patch is necessary to allow dots in the bucket names
import ssl
if hasattr(ssl, '_create_unverified_context'):
   ssl._create_default_https_context = ssl._create_unverified_context

Collect static files to the S3 bucket:

(myproject_env)$ python manage.py collectstatic --noinput

6. Set Up Gmail to Send Emails

Open myproject/conf/prod.py and add the following settings:

EMAIL_USE_TLS = True
EMAIL_HOST = "smtp.gmail.com"
EMAIL_HOST_USER = "myproject@gmail.com"
EMAIL_HOST_PASSWORD = "mygmailpassword"
EMAIL_PORT = 587

7. Push to Heroku

Commit and push all the changes to your Git origin remote. Personally I prefer using SourceTree to do that, but you can also do that in the command line, PyCharm, or another software.

In your project directory type the following:

(myproject_env)$ heroku create my-unique-project

This will create a Git remote called "heroku", and a new Heroku project "my-unique-project" which can be later accessed at http://my-unique-project.herokuapp.com.

Push the changes to heroku remote:

(myproject_env)$ git push heroku master

8. Transfer Your Local Postgres Database To Heroku

Create local database dump:

(myproject_env)$ PGPASSWORD=mypassword pg_dump -Fc --no-acl --no-owner -h localhost -U myuser mydb > mydb.dump

Upload the database dump temporarily to some server, for example, S3 bucket: http://myproject.media.s3-eu-west-1.amazonaws.com/mydb.dump. Then import that dump into the Heroku database:

(myproject_env)$ heroku pg:backups restore 'http://myproject.media.s3-eu-west-1.amazonaws.com/mydb.dump' DATABASE_URL

Remove the database dump from S3 server.

9. Set Environment Variables

If your Git repository is not private, put your secret values in environment variables rather than in the Git repository directly.

(myproject_env)$ heroku config:set AWS_S3_ACCESS_KEY_ID=ABCDEFG123
$ heroku config:set AWS_S3_SECRET_ACCESS_KEY=aBcDeFg123

To read out the environment variables you can type:

(myproject_env)$ heroku config

To read out the environment variables in the Python code open myproject/conf/base.py and type:

import os
AWS_S3_ACCESS_KEY_ID = os.environ.get("AWS_S3_ACCESS_KEY_ID", "")
AWS_S3_SECRET_ACCESS_KEY = os.environ.get("AWS_S3_SECRET_ACCESS_KEY", "")

10. Set DNS Settings

Open your domain settings and set CNAME to "my-unique-project.herokuapp.com".

At last, you are done! Drop in the comments if I missed some part. For the new updates, see the next section.

*. Update Production

Push the changes to heroku remote:

(myproject_env)$ git push heroku master

If you have changed something in the static files, collect them again:

(myproject_env)$ python manage.py collectstatic --noinput

Collecting static files to S3 bucket takes quite a long time, so I do not recommend to do that automatically every time when you want to deploy to Heroku.

Further Reading

You can read more about Django on Heroku in the following resources:

2016-02-10

Special Offer for the Readers of DjangoTricks Blog

Packt Publishing, the company that published my Django book, has a special offer for enthusiast and professional developers reading this blog. For two weeks you can get the eBook "Web Development with Django Cookbook - Second Edition" for half price. The eBook is available in PDF, ePub, Mobi, and Kindle formats. Also you will get access to download the related code files.

Use the discount code DJGTRK50 at the Packt Publishing bookstore.
The discount is valid until the 24th of February, 2016.

2016-02-06

Fresh Book for Django Developers

This week the post office delivered a package that made me very satisfied. It was a box with three paper versions of my "Web Development with Django Cookbook - Second Edition". The book was published at the end of January after months of hard, but fulfilling work in the late evenings and at weekends.

The first Django Cookbook was dealing with Django 1.6. Unfortunately, the support for that version is over. So it made sense to write an update for a newer Django version. The second edition was adapted for Django 1.8 which has a long-term support until April 2018 or later. This edition introduces new features added to Django 1.7 and Django 1.8, such as database migrations, QuerySet expressions, or System Check Framework. Most concepts in this new book should also be working with Django 1.9.

My top 5 favourite new recipes are these:

  • Configuring settings for development, testing, staging, and production environments
  • Using database query expressions
  • Implementing a multilingual search with Haystack
  • Testing pages with Selenium
  • Releasing a reusable Django app

The book is worth reading for any Django developer, but will be best understood by those who already know the basics of web development with Django. You can learn more about the book and buy it at the Packt website or Amazon.

I thank the Packt Publishing very much for long cooperation in the development of this book. I am especially thankful to acquisition editor Nadeem N. Bagban, content development editors Arwa Manasawala and Sumeet Sawant, and technical editor Bharat Patil. Also I am grateful for insightful feedback from the reviewer Jake Kronika.

What 5 recipes do you find the most useful?

2015-01-30

How to Find the Performance Bottlenecks in Your Django Views?

Once you have your Django projects running, you come to situations, when you need to optimize for performance. The rule of thumb is to find the bottlenecks and then to take action to eliminate them by more idiomatic Python code, database denormalization, caching, or other techniques.

What is a bottleneck? Literally it refers to the top narrow part of a bottle. In engineering, bottleneck is a case where the performance or capacity of an entire system is limited by a single or small number of components or resources.

How to find these parts of your code? The most trivial way is to check the current time before specific code execution and after that code execution, and then count the time difference:

from datetime import datetime
start = datetime.now()
# heavy execution ...
end = datetime.now()
d = end - start  # datetime.timedelta object
print d.total_seconds()  # prints something like 7.861985

However, measuring code performance for Django projects like this is inefficient, because you need a lot of such wrappers for your code until you find which part is the most critical. Also you need a lot of manual computation to find the critical parts.

Recently I found line_profiler module that can inspect the performance of the code line by line. By default, to use line_profiler for your functions, you should decorate them with @profile decorator and then to execute the script:

$ kernprof -l some_script_to_profile.py

This script will execute your script, analize the decorated function, and will save results to a binary file that can later be inspected with:

$ python -m line_profiler some_script_to_profile.py.lprof

That's quite complicated, but to use line_profiler for Django views, you can install django-devserver which replaces the original development server of Django and will output the performance calculations immediately in the shell like this:

[30/Jan/2015 02:26:40] "GET /quotes/json/ HTTP/1.1" 200 137
    [sql] 1 queries with 0 duplicates
    [profile] Total time to render was 0.01s
    [profile] Timer unit: 1e-06 s

          Total time: 0.001965 s
          File: /Users/archatas/Projects/quotes_env/project/inspirational/quotes/views.py
          Function: quote_list_json at line 27

          Line #      Hits         Time  Per Hit   % Time  Line Contents
          ==============================================================
              27                                           def quote_list_json(request):
              28         1            2      2.0      0.1      quote_dict_list = []
              29         2         1184    592.0     60.3      for quote in InspirationQuote.objects.all():
              30         1            1      1.0      0.1          quote_dict = {
              31         1            1      1.0      0.1              'author': quote.author,
              32         1            1      1.0      0.1              'quote': quote.quote,
              33         1          363    363.0     18.5              'picture': quote.get_medium_picture_url(),
              34                                                   }
              35         1            1      1.0      0.1          quote_dict_list.append(quote_dict)
              36
              37         1           42     42.0      2.1      json_data = json.dumps(quote_dict_list)
              38         1          370    370.0     18.8      return HttpResponse(json_data, content_type="application/json")

The most interesting data in this table is the "% Time" column, giving an overview in percentage which lines of the Django view function are the most time-consuming. For example, here it says that I should pay the most attention to the QuerySet, the method get_medium_picture_url() and the HttpResponse object.

To setup line profiling, install line_profiler and django-devserver to you virtual environment:

(myproject_env)$ pip install line_profiler
(myproject_env)$ pip install django-devserver

Then make sure that you have the following settings in your settings.py or local_settings.py:

# settings.py
INSTALLED_APPS = (
    # ...
    'devserver',
)

MIDDLEWARE_CLASSES = (
    # ...
    'devserver.middleware.DevServerMiddleware',
)

DEVSERVER_MODULES = (
    'devserver.modules.sql.SQLRealTimeModule',
    'devserver.modules.sql.SQLSummaryModule',
    'devserver.modules.profile.ProfileSummaryModule',

    # Modules not enabled by default
    'devserver.modules.profile.LineProfilerModule',
)

DEVSERVER_AUTO_PROFILE = True  # profiles all views without the need of function decorator

When you execute

(myproject_env)$ python manage.py runserver

it will run the development server from django-devserver and for each visited view, it will show the analysis of code performance. I have tested this setup with Django 1.7, but it should work since Django 1.3.

Do you know any more useful tools to check for performance bottlenecks?

2014-11-04

Win Free Copies of “Web Development with Django Cookbook”

Readers would be pleased to know that I have teamed up with Packt Publishing to organize a Giveaway of my book Web Development with Django Cookbook.

Three lucky winners stand a chance to win an e-copy of the book. Keep reading to find out how you can be one of the lucky ones.

Book Overview

  • Improve your skills by developing models, forms, views, and templates
  • Create a rich user experience using Ajax and other JavaScript techniques
  • A practical guide to writing and using APIs to import or export data

How to Enter?

All you need to do is head on over to the book page and look through the product description of the book and drop a line via the comments below this post to let us know what interests you the most about this book. It’s that simple.

Prize

Winners will get an e-copy of the Book.

Deadline

The contest will close on the 1st of December, 2014 at 00:00 GMT. Winners will be contacted by email, so be sure to use your real email address when you comment!


...And the winners are...

Out of the 21 participants I finally chose these winners: Mark Phillips, Faria Chowdhury, and Mario Gudelj. The guys from Packt Publishing will contact you by email and will give you the free copies of "Web Development with Django Cookbook". Have a nice reading!

2014-10-27

Contributing Back to the Community - Django Cookbook

In the early beginning of year 2014, the IT book publishing company "Packt Publishing" contacted me with an interesting offer: to share my Django experience in a form of a book. I thought it might be a good challenge for me and also value for the Django community, as I had been working with Django for 7 years or so, and during that time there was quite a lot of knowledge gathered and used practically. So for the next 9 months on late evenings and weekends I was adapting some of the most useful code snippets and describing them in the book. The great staff from the Packt Publishing helped me to structure the content, make everything clear and understandable, and get the grammar correct. Finally, the book was released and it's called "Web Development with Django Cookbook".

Word cloud of the Web Development with Django Cookbook

This book is written for intermediate or advanced Django web developers. When writing the book, my own purpose was not to fully cover every possible web development task, but rather to have enough useful bits of knowledge for those who seek information about web development with Django. The book was written in the format of recipes. There are over 70 recipes giving you instructions how to deal with different challenges of web development. The code mentioned in the book is optimized for Django 1.6, but most of it should also work with older Django versions as well as with Django 1.7.

The cookbook consists of 10 chapters:

  1. Getting started with Django 1.6. This chapter will guide you through the basic configuration which is necessary to start any Django project. It will cover topics like virtual environment, version control, and project settings.
  2. Database Structure. When you create a new app, the first thing to do is to define your models. In this chapter you will learn how to write reusable pieces of code to use in your models. You will learn how to deal with multilingual data in your database. Also you will be told how to manage database schema changes using South migrations.
  3. Forms and Views. To show your data or let one create it, you need views and forms. This chapter will show you some patterns for creating them.
  4. Templates and JavaScript. Information is presented to the user by rendered templates. In modern websites, JavaScript is a must for richer user experience. This chapter shows practical examples of using templates and JavaScript together.
  5. Custom Template Filters and Tags. The default Django template system is quite extensive, but there are more things to add for different cases. This chapter shows you how to create and use own template tags and filters.
  6. Model Administration. Django framework comes with a handy pre-build administration. This chapter shows how to extend the default administration with your own functionality.
  7. Django CMS. Django CMS is the most popular open source content management system made in Django. This chapter deals with best practices using Django CMS and extending it for your needs.
  8. Hierarchical Structures. When you need to create a tree-like structure in Django, django-mptt module comes to hand. This chapter shows you how to use it and how to set administration for hierarchical structures.
  9. Data Import and Export. Very often there are cases when you need to transfer data from and to different formats, retrieve it from and provide it to different sources. This chapter deals with management commands for data import and also APIs for data export.
  10. Bells and Whistles. There is more to Django. This chapter shows additional snippets and tricks useful in web development with Django. Also you will learn about deployment of Django projects.

Get a copy of this book at Packt Publishing and tell me what you think about it in the comments below. Which recipes did you find the most useful? What would you like to read more in this blog or another edition of the cookbook?

Web Development with Django Cookbook

2013-12-18

How to Export Your Data as CSV, XLS, or XLSX

There are times, when you need to export the data from your database to different formats. For example, you want to create some diagrams in Office program for a presentation. In this post I will show you how to create admin actions which export selected items as files for a spreadsheet application (like MS Excel, OpenOffice Calc, LibreOffice Calc, Gnumeric, or Numbers). I will cover the mostly used formats: Comma Separated Values (CSV), Binary Excel (XLS), and Office Open XML (XLSX).

First of all, have a look at the model we will be dealing with. It's a simple model with title, description, and - of course - the id.

# models.py
from django.db import models
class MyModel(models.Model):
    title = models.CharField(max_length=100)
    description = models.TextField(blank=True)

In the admininstration options, we'll define three admin actions: export_csv, export_xls, and export_xlsx.

# admin.py
from django.contrib import admin
from models import MyModel

# ... export functions will go here ...

class MyModelAdmin(admin.Admin):
    actions = [export_csv, export_xls, export_xlsx]

admin.site.register(MyModel, MyModelAdmin)

Now let's create functions for each of those actions!

Comma-Separated Values Format

CSV is the most common import and export format for spreadsheets and databases. It's a textual format which one could easily create or parse himself, but there is also a python built-in library csv for handy data manipulation.

def export_csv(modeladmin, request, queryset):
    import csv
    from django.utils.encoding import smart_str
    response = HttpResponse(mimetype='text/csv')
    response['Content-Disposition'] = 'attachment; filename=mymodel.csv'
    writer = csv.writer(response, csv.excel)
    response.write(u'\ufeff'.encode('utf8')) # BOM (optional...Excel needs it to open UTF-8 file properly)
    writer.writerow([
        smart_str(u"ID"),
        smart_str(u"Title"),
        smart_str(u"Description"),
    ])
    for obj in queryset:
        writer.writerow([
            smart_str(obj.pk),
            smart_str(obj.title),
            smart_str(obj.description),
        ])
    return response
export_csv.short_description = u"Export CSV"

As you can see, HttpResponse is a file-like object and we used it to write data to.

Excel Binary File Format

XLS is the main spreadsheet format which holds data in worksheets, charts, and macros. We are going to use xlwt library to create a spreadsheet. There is analogous library xlrd to read XLS files. Note, that this format allows to have only 256 columns.

def export_xls(modeladmin, request, queryset):
    import xlwt
    response = HttpResponse(mimetype='application/ms-excel')
    response['Content-Disposition'] = 'attachment; filename=mymodel.xls'
    wb = xlwt.Workbook(encoding='utf-8')
    ws = wb.add_sheet("MyModel")
    
    row_num = 0
    
    columns = [
        (u"ID", 2000),
        (u"Title", 6000),
        (u"Description", 8000),
    ]

    font_style = xlwt.XFStyle()
    font_style.font.bold = True

    for col_num in xrange(len(columns)):
        ws.write(row_num, col_num, columns[col_num][0], font_style)
        # set column width
        ws.col(col_num).width = columns[col_num][1]

    font_style = xlwt.XFStyle()
    font_style.alignment.wrap = 1
    
    for obj in queryset:
        row_num += 1
        row = [
            obj.pk,
            obj.title,
            obj.description,
        ]
        for col_num in xrange(len(row)):
            ws.write(row_num, col_num, row[col_num], font_style)
            
    wb.save(response)
    return response
    
export_xls.short_description = u"Export XLS"

Here we created one worksheet, filled it with data, marked the first row in bold, and made the lines in the other cells wrapped. Also we set the width for each column. We'll do the same in the next format too.

Office Open XML Format

XLSX (a.k.a. OOXML or OpenXML) is a zipped, XML-based file format developed by Microsoft. It is fully supported by Microsoft Office 2007 and newer versions. OpenOffice 4.0, for example, can only read it. There is a python library openpyxl for reading and writing those files. This format is great when you need more than 256 columns and text formatting options.

def export_xlsx(modeladmin, request, queryset):
    import openpyxl
    from openpyxl.cell import get_column_letter
    response = HttpResponse(mimetype='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
    response['Content-Disposition'] = 'attachment; filename=mymodel.xlsx'
    wb = openpyxl.Workbook()
    ws = wb.get_active_sheet()
    ws.title = "MyModel"

    row_num = 0

    columns = [
        (u"ID", 15),
        (u"Title", 70),
        (u"Description", 70),
    ]

    for col_num in xrange(len(columns)):
        c = ws.cell(row=row_num + 1, column=col_num + 1)
        c.value = columns[col_num][0]
        c.style.font.bold = True
        # set column width
        ws.column_dimensions[get_column_letter(col_num+1)].width = columns[col_num][1]

    for obj in queryset:
        row_num += 1
        row = [
            obj.pk,
            obj.title,
            obj.description,
        ]
        for col_num in xrange(len(row)):
            c = ws.cell(row=row_num + 1, column=col_num + 1)
            c.value = row[col_num]
            c.style.alignment.wrap_text = True

    wb.save(response)
    return response

export_xlsx.short_description = u"Export XLSX"

Conclusion

So whenever you need to get your Django project data to some spreadsheet application, there are several ways to do that. If you are planning to import the data to some other database, CSV is probably the best, as it is simple, straightforward, and requires no third-party libraries. However, if you need your data with nice formatting and maybe some statistical formulas, you should export XLS or XLSX format. The maximum amount of columns in XLS format is limited to 256, whereas XLSX format allows more columns, but is not fully supported by all spreadsheet applications.