How to create a virtual environment that also uses global modules

Last weekend I upgraded my Mac to Mountain Lion. I installed Python 2.7 with PIL and MySQLdb and virtualenv using MacPorts.

When I created and activated a virtual environment with

cd myproject
virtualenv .
. bin/activate

the PIL and MySQLdb modules were unrecognizible in that environment. What I needed was just different Django versions in my virtual environments, but MySQLdb and PIL libraries had to be shared.

After a quick search I found an additional parameter for the creation of virtual environment:

virtualenv --system-site-packages .

With this parameter I could use my global modules within the virtual environment.

How to Set Hijax with Django and jQuery

The main principle of progressive enhancement is that the markup, styling, and Javascript are separated. The site should be functional and browsable without Javascript, but when Javascript is activated, it can add additional functionality to the elements of a page. For example, it can hijack the default behavior of normal links replacing it with Ajax loading of just a specific part of the page. This type of progressive enhancement was called Hijax by Jeremy Keit.

Hijax has the following benefits:

• Visitors can browse the content on any browser and on any platform.
• The content is indexable by search engines for each page.
• Open graph lets you share each page on Facebook with specific title, description and image.

Let me show you an example of Hijax. Recently during my spare time I developed a simple collective storytelling game 1000 Words. Every participant at any point of a story has a chance to choose the continuation previously entered by someone else or to continue by his own words. It's like threaded comments where you see only one thread at a time.

All continuations are browsable without Javascript and each of them has Open Graph set, so Google can index all variations of the stories and any page can be shared on Facebook with custom title, description, and image. When Javascript is enabled (which is the usual case for human visitors), clicking on one of the given continuation choices doesn't refresh the full page, but rather loads the piece of the story and further possible continuations by Ajax and injects into a proper place. This way, I don't need to load all the history with each request and the visitor has more pleasant experience.

Django HttpRequest has a method to check if the call to a page was made by Ajax or by the normal browsing. It's the request.is_ajax() method. Depending on the call, the view of a page can provide different templates: one containing the full HTML with head and body, and another one just with the required snippet of HTML.

There are two ways to organize your templates for Hijax in Django without repeating yourself. One way is to include the Ajax snippet into the template of non-Ajax version.

# views.py
def my_view(request, ...):
    # ...
    template = "myapp/story.html"
    if request.is_ajax():
        template = "myapp/story_ajax.html"
    # ...   
    return render(request, template, context_dict)

{# story.html #}
{% extends "base.html" %}

{% block content %}
    {% include "myapp/story_ajax.html" %}
{% endblock %}

{# story_ajax.html #}
<div class="dynamic-content">
    ...
</div>

The other way to organize your templates for Hijax is extend different bases for normal and Ajax cases. The normal base would have all usual HTML and body, whereas the base Ajax template would just have empty block.

# views.py
def my_view(request, ...):
    # ...
    if request.is_ajax():
        context_dict['base_template'] = "base_ajax.html"
    # ...   
    return render(request, "myapp/story.html", context_dict)

{# story.html #}
{% extends base_template|default:"base.html" %}

{% block content %}
    <div class="dynamic-content">
        ...
    </div>
{% endblock %}

{# base_ajax.html #}
{% block content %}
{% endblock %}

Last but not least, it's time to add unobtrusive Javascript which replaces the default behavior of the links with Ajax calls. With jQuery it should be something like this:

// hijax.html
(function($, undefined) {
    $(document).ready(function() {
        // hijack all links with the css class "dynamic"
        $('a.dynamic').live('click', function() {
            $('#content').load($(this).attr('href'));
            return false;
        });
    });
})(jQuery);

This is not the exact code from the game, but it illustrates the concept of progressive enhancement very well.

Quick Tip on Setuptools

You are probably used to the default usage of Setuptools, for example the following command installs the latest Django version into your python installation or currently active virtual environment:

easy_install django

Sometimes you might need to install a specific version of a module. This is an example of installing Django 1.3.3:

easy_install django==1.3.3

If you want to upgrade the existing installation to the latest version, you can do that with:

easy_install -U django

Initial Data Using South Migrations

Usually when starting a new project with commonly used apps, you will need some default data for the database, for example, default categories. Django provides fixtures for that. A fixture is a collection of data that Django knows how to import into a database.

They say:

If you create a fixture named initial_data.[xml/yaml/json], that fixture will be loaded every time you run syncdb.

There is one problem with this approach. It is very likely that you will need to run syncdb in the future when adding new apps (without south migrations). In such cases the modified data will be overwritten with the initial data from fixtures.

But there is a solution for that problem. You need to name your fixtures differently and then import them by south migrations. If you don't use south, you should start doing that, because it is a very handy way to modify database schema and data automatically. :)

So this is how you can prepare south migration with the default fixture:

1. Create a fixture from existing models.

python manage.py dumpdata --format=json --indent=4 myapp.Category > /path/to/myapp/fixtures/myapp_categories.json

2. Create data migration

python manage.py datamigration myapp load_default_data

3. Edit data migration

class Migration(DataMigration):
    
    def forwards(self, orm):
        from django.core.management import call_command
        call_command("loaddata", "myapp_categories.json")
    
    def backwards(self, orm):
        pass

4. Migrate

python manage.py migrate myapp

The Usage of the only() Method

Recently I found the only() method of a queryset which can significantly improve the loading speed of forms with ModelChoiceField or ModelMultipleChoiceField. Let's say we have a big Event model with thousands of instances:

class Event(models.Model):
    title = models.CharField(_("Title"), max_length=200)
    # ... lots of other fields ...
    def __unicode__(self):
        return self.title

Then we have a form with ModelMultipleChoiceField:

class EventForm(forms.Form):
    related_events = forms.ModelMultipleChoiceField(
        queryset=Event.objects.all().only("id", "title"),
        label=_("Related Events"),
        required=False,
        )
    # ... other fields of the form ...

The used only() method loads only the primary key and the title for the selection widget.

Custom Admin Filter Specification

Let's say we have a model Project with age groups as boolean fields:

class Project(models.Model):
    # ...
    age_group_0_5 = models.BooleanField(u"From 0 to 5 years")
    age_group_6_12 = models.BooleanField(u"From 6 to 12 years")
    age_group_13_18 = models.BooleanField(u"From 13 to 18 years")
    age_group_19_27 = models.BooleanField(u"Fron 19 to 27 years")
    # ...

In order to list them as one selection in the admin list filter, we need a specific filter specification. Let's create it in the admin.py:

from django.contrib.admin.filterspecs import FilterSpec

class AgeChoicesFilterSpec(FilterSpec):
    def title(self):
        return u"Age group"
        
    def __init__(self, f, request, params, model, model_admin,
                 field_path=None):
        super(AgeChoicesFilterSpec, self).__init__(f, request, params, model,
                                                model_admin,
                                                field_path=field_path)
        self.request = request
        
    def choices(self, cl):
        yield {'selected': not(
                    self.request.GET.get('age_group_0_5', None) or 
                    self.request.GET.get('age_group_6_12', None) or 
                    self.request.GET.get('age_group_13_18', None) or 
                    self.request.GET.get('age_group_19_27', None) 
                    ),
               'query_string': cl.get_query_string(
                               {},
                               ['age_group_0_5', 'age_group_6_12', 'age_group_13_18', 'age_group_19_27']),
               'display': u'All'}
        yield {'selected': self.request.GET.get('age_group_0_5', None) == 'True',
               'query_string': cl.get_query_string(
                               {'age_group_0_5': 'True'},
                               ['age_group_6_12', 'age_group_13_18', 'age_group_19_27']),
               'display': u"From 0 to 5 years"}
        yield {'selected': self.request.GET.get('age_group_6_12', None) == 'True',
               'query_string': cl.get_query_string(
                               {'age_group_6_12': 'True'},
                               ['age_group_0_5', 'age_group_13_18', 'age_group_19_27']),
               'display': u"From 6 to 12 years"}
        yield {'selected': self.request.GET.get('age_group_13_18', None) == 'True',
               'query_string': cl.get_query_string(
                               {'age_group_13_18': 'True'},
                               ['age_group_0_5', 'age_group_6_12', 'age_group_19_27']),
               'display': u"From 13 to 18 years"}
        yield {'selected': self.request.GET.get('age_group_19_27', None) == 'True',
               'query_string': cl.get_query_string(
                               {'age_group_19_27': 'True'},
                               ['age_group_0_5', 'age_group_6_12', 'age_group_13_18']),
               'display': u"From 19 to 27 years"}

FilterSpec.filter_specs.insert(0, (lambda f: f.name == 'age_group_0_5',
                               AgeChoicesFilterSpec))

Here the most important method of FilterSpec is choices. It defines the choices for the filter by dictionary with three keys:

• selected is the condition when this choice should be selected.
• query_string is the URL query string that will be formed to filter the list by. The query string is formed by get_query_string method of ChangeList instance. It takes two parameters: a dictionary of new parameters to add to the current query, and a list of parameters to remove from the current query.
• display is the human-readable string which will be shown for this choice in the filter.

The new filter specification should be attached to a field. Here it will be applied to all fields with the name "age_group_0_5" and we have just one such field.

The last thing to do is to define admin options for the Project model:

from django.contrib import admin
from models import Project

class ProjectAdmin(admin.ModelAdmin):
    # ...
    list_filter = ("age_group_0_5", )
    # ...
    
admin.site.register(Project, ProjectAdmin)

Now you know how to create a custom admin filter specification for your model. :)

Django CMS, Haystack, and Custom Plugins

When you need a full-text search for Django-CMS-based website, you can use Haystack and django-cms-search. The latter module ensures that all CMS Pages get indexed.

One important thing to mention is that if you use any custom Plugins, search_fields need to be defined for them, so that the Pages using them are indexed properly. For example, here is an Overview plugin which makes its title and description searchable:

from django.db import models
from django.utils.translation import ugettext_lazy as _
from cms.models import CMSPlugin

class Overview(CMSPlugin):
    title = models.CharField(_('Title'), max_length=200)
    description = models.TextField(_('Description'))
    url = models.URLField(_('url'), max_length=200, blank=True)
    
    search_fields = ("title", "description")
    
    def __unicode__(self):
        return self.title
        

For more information check the documentation online:

• Haystack
• Django CMS
• django-cms-search

Deployment Using Fabric

Yesterday just a night before EuroDjangoCon 2010, I was writing deployment scripts for my personal project. I have been using analogous scripts at least for a year for my work projects of different scopes. And I can prove that it's really worth to have such scripts, because they save a lot of time and let you avoid confusion and mistakes.

Most of the projects are continuously improved, so the main purpose of deployment in my case is not the initial installation, but handy and fast updating.

For each website that I am managing, there are two environments set up under different domains, where one of them is production environment and another one is staging/testing environment. The configuration of staging environment mirrors the production environment. Both have their own webserver configurations, codebases, databases, and uploaded media files. At first the newest features are tested in the staging environment with the copy of live data, and then they are published in the production environment.

I am using subversion for version control with third-party modules defined as svn:externals and put under the same codebase. Unfortunately, some of the modules have to be copied if they were originally taken from Git, Bazaar, or other places. Anyway, keeping all the codebase under SVN makes the initial installation easier and ensures that you can easily update the whole combination of module dependencies.

In addition, I am using south migrations, to make the changes to the database schema automagically.

Earlier the process of updating the staging environment consisted of these manual steps:

• Set up under-construction page for the staging site either by changing .htaccess file or httpd.conf and restarting webserver (depending on the server configuration).
• Export the dump of production database and import it into the staging database.
• Copy the media uploads from production environment to staging environment.
• Update code from subversion trunk.
• Migrate database schema and/or do any database changes.
• Restart apache and memcached.
• Test if everything is working as expected.
• Unset under-construction page.

Then some similar instructions should be executed in the production environment:

• Set up under-construction page for the production site.
• Backup database in case if anything goes wrong, so that you could recover.
• Update code from subversion.
• Run database migrations.
• Test everything.
• Unset under-construction page.

And that's quite a long routine to do manually and also sometimes when the changes were small, I was always tempted to skip some steps which increased the risk to fail. But then I started using Fabric to automate the process. Fabric is python-based deployment tool providing simple API for FTP and SSH connections as well as for local shell calls.

To use Fabric, you have to install it in your local machine and then write scripts called fabfiles for each website, you are updating. For example, updating the staging environment in all my cases is as simple as running

# cd path/to/directory/with/fabfile.py/
# fab staging deploy

and then entering user passwords and answering questions like

Backup database ([y]/n)? _

The fabfiles could also be without user interaction, but I made those dialogs to be able to monitor the process and stop it at any point in case if I need some specific additional manual work.

For security reasons, you should avoid keeping passwords under subversion or in the deployment script. Much better it is to write some bash scripts in the remote servers which would do the work and then to address them from your fabfile.

Here is an example of possible fabfile. Keep in mind that different apache configurations and some bash scripts are something that you have to write yourself depending on your server configuration. Also the fabfile itself might slightly differ depending on the server specifics.

According to Thilo Fromm with whom I talked today at EuroDjangoCon, another way to create deployment scripts is to create linux-distribution-related packages which you could install using installation tools within the distribution, for example apt-get for Debian or rpm for Suse. However, in my opinion, that kind of deployment fits better to those cases when you don't have control of the production server and need to deliver a final product to corporate clients, or where you need to install the same project/product to multiple servers.

Weather App on Github

As promised, I put the code of the climate_change app online. For that reason, I created an account on github trying to use Git for the first time. This is also the first time I am using a distributed version control system, i.e. you can create branches of the project, develop the forks separately on different computers, and merge them in any order. There is no main centralized repository. The code on github that I uploaded is just one initial copy.

I am still new to Git commands and workflows. You can share your knowledge with me. For example, maybe you know some good cheat sheet or writings about Git. Or you can try to fork the project I committed, change it somehow and then push for merging (not sure if I'm using correct jargon). I will share with you what I find interesting and useful about Git while learning it.

Weather App Tutorial. Part 5 of 5. Graph View

I had some problems with the internet, so I am posting this entry a little later that expected. But anyway, this is it - the final part of the tutorial about the weather app. You can read the first four parts here: 1, 2, 3, 4. Today I will show you how to display a chart with statistical data from the collected weather details.

I googled a little to find out what charting system to use for this small project. And I found Open Flash Chart as an interesting solution. It's a flash-based charting system which uses Ajax requests to get the data for the graphs. That means that we'll need a view which will show the graphs and another view, which will provide the data. Each location will have its own page displaying a chart, so it would be nice to have URLs where each location would be accessed by a slug. We already have a field sysname which could be used here, but my best practice is to have separate fields for url slugs which might be changed at anytime and sysnames which should be defined once and not changed at all because the sysname binds an object with a template.

So I add this new field to the Location model in models.py:

class Location(models.Model):
    #...
    slug = models.SlugField(
        _("slug for urls"),
        max_length=200,
        unique=True,
        blank=True,
        default="",
        )
    #...

Then I create and apply a migration for it:

python manage.py startmigration add_slug --auto
python manage.py migrate climate_change

Let's run the webserver once again. Go to http://127.0.0.1:8000/admin/climate_change/location/1/, add "berlin" as a slug and save the record.

Next I need to define a view which will display the graphs. This looks as simple as this:

# -*- coding: UTF-8 -*-
from django.shortcuts import get_object_or_404, render_to_response

Location = models.get_model("climate_change", "Location")

def display_charts(request, slug):
    location = get_object_or_404(Location, slug=slug)
    return render_to_response(
        "climate_change/charts.html",
        {'location': location},
        context_instance=RequestContext(request),
        )

Whereas the template created at templates/climate_change/charts.html includes this code:

{% extends "base.html" %}

{% load i18n %}

{% block title %}{{ location.name }}{% endblock %}

{% block header %}
    </code><h1>{{ location.name }}</h1>
{% endblock %}

{% block content %}
    <div class="chart-temperature">
        <h2>{% trans "Temperature" %}</h2>
        <object type="application/x-shockwave-flash" data="/media/flash/open-flash-chart.swf?data-file={% url climate_change_temperature slug=location.slug %}" height="250" width="500">
            <param name="movie" value="/media/flash/open-flash-chart.swf?data-file={% url climate_change_temperature slug=location.slug %}">
            <param name="allowScriptAccess" value="sameDomain">
            <param name="quality" value="high">
        </object>
    </div>
    <div class="chart-humidity">
        <h2>{% trans "Humidity" %}</h2>
        <object type="application/x-shockwave-flash" data="/media/flash/open-flash-chart.swf?data-file={% url climate_change_humidity slug=location.slug %}" height="250" width="500">
            <param name="movie" value="/media/flash/open-flash-chart.swf?data-file={% url climate_change_humidity slug=location.slug %}">
            <param name="allowScriptAccess" value="sameDomain">
            <param name="quality" value="high">
        </object>
    </div>
    <div class="chart-wind-speed">
        <h2>{% trans "Wind speed" %}</h2>
        <object type="application/x-shockwave-flash" data="/media/flash/open-flash-chart.swf?data-file={% url climate_change_wind_speed slug=location.slug %}" height="250" width="500">
            <param name="movie" value="/media/flash/open-flash-chart.swf?data-file={% url climate_change_wind_speed slug=location.slug %}">
            <param name="allowScriptAccess" value="sameDomain">
            <param name="quality" value="high">
        </object>
    </div>
    <div class="chart-visibility">
        <h2>{% trans "Visibility" %}</h2>
        <object type="application/x-shockwave-flash" data="/media/flash/open-flash-chart.swf?data-file={% url climate_change_visibility slug=location.slug %}" height="250" width="500">
            <param name="movie" value="/media/flash/open-flash-chart.swf?data-file={% url climate_change_visibility slug=location.slug %}">
            <param name="allowScriptAccess" value="sameDomain">
            <param name="quality" value="high">
        </object>
    </div>
{% endblock %}

Also let's not forget to download the Open Flash Chart flash file itself which is in the archive of this package. I placed the file open-flash-chart.swf at media/flash.

Originally, Open Flash Chart uses PHP on the server side to provide data to the flash-based client side. But fortunately there is a port to python which I will be using for forming the data to the charts. I don't want to get dirty fingers by creating all this manually forming JSON object using simplejson, especially because there is not complete reference of Open-Flash-Chart API (just examples and tutorials). So I downloaded python port, extracted the content and copied OpenFlashChart.py to climate_change_env/lib/python2.5/site-packages/ which is the location of python libraries which will be installed a.k.a. under python path for the virtual environment "climate_change_env".

Using the examples found at a tutorial, I created a view for providing chart data. The views.py looks now like this:

# -*- coding: UTF-8 -*-
from datetime import datetime, timedelta
from random import random

# http://open-flash-chart-python.googlecode.com/files/Python%202.10.zip
from OpenFlashChart import Chart

from django.db import models
from django.http import HttpResponse
from django.template import RequestContext
from django.shortcuts import get_object_or_404, render_to_response
from django.utils.translation import ugettext as _
from django.utils.encoding import force_unicode

Location = models.get_model("climate_change", "Location")
WeatherLog = models.get_model("climate_change", "WeatherLog")

def display_charts(request, slug):
    location = get_object_or_404(Location, slug=slug)
    return render_to_response(
        "climate_change/charts.html",
        {'location': location},
        context_instance=RequestContext(request),
        )

def rgb_to_html_color(r, g, b):
    """ convert an R, G, B to #RRGGBB """
    return '#%02x%02x%02x' % (r, g, b)

def json_get_statistics(request, slug, field):
    location = get_object_or_404(Location, slug=slug)
    extreme_values = location.weatherlog_set.aggregate(
        models.Min(field),
        models.Max(field),
        models.Min("timestamp"),
        models.Max("timestamp"),
        )
    elements = []

    for year in range(
        extreme_values['timestamp__min'].year,
        extreme_values['timestamp__max'].year + 1,
        ):
        element = Chart()
        monthly_values = []
        for month in range (1, 13):
            monthly = location.weatherlog_set.filter(
                timestamp__year=year,
                timestamp__month=month,
                ).aggregate(models.Avg(field))
            monthly_values.append(monthly['%s__avg' % field])
        element.values = monthly_values
        element.type = "line"
        element.dot_style.type = "dot"
        element.dot_style.dot_size = 5
        random_html_color = rgb_to_html_color(
            r=63+int(random() * 128),
            g=63+int(random() * 128),
            b=63+int(random() * 128),
            )
        element.dot_style.colour = random_html_color
        element.width = 4
        element.colour = random_html_color
        element.text = str(year)
        element.font_size = 10
        elements.append(element)
    
    # Create chart
    chart = Chart()
    chart.y_axis.min = float(str(extreme_values['%s__min' % field]))
    chart.y_axis.max = float(str(extreme_values['%s__max' % field]))
    chart.y_axis.font_size = 10
    chart.title.text = force_unicode(WeatherLog._meta.get_field(field).verbose_name)
    chart.x_axis.labels.labels = [
        _("Jan"), _("Feb"), _("Mar"), _("Apr"),
        _("May"), _("Jun"), _("Jul"), _("Aug"),
        _("Sep"), _("Oct"), _("Nov"), _("Dec"),
        ]

    #
    # here we add our data sets to the chart:
    #
    chart.elements = elements
    
    return HttpResponse(chart.create())

If I run the webserver again and go to http://127.0.0.1:8000/climate-change/berlin/, I see a single dot in each graph.



We need some more data. There is Time Machine app in the Applications directory in my computer, so let's use it to move forward to year 2012 to check weather statistics. Damn it! It didn't function properly. Then I'll need to generate statistical data myself. Anyway, according to Mark Twain, there are three kinds of lies: lies, damned lies, and statistics.

I'll use python interactive development environment a.k.a. shell to prepopulate that data.

python manage.py shell
>>> from django.db import models
>>> Location = models.get_model("climate_change", "Location")
>>> location = Location.objects.get(sysname="berlin")
>>> extremes = (
...     (-5, 4), (-4, 5), (-2, 10), (2, 15),
...     (6, 20), (9, 24), (11, 25), (10, 25),
...     (7, 20), (4, 15), (0, 9), (-2, 5),
...     )
>>> from random import random
>>> from datetime import datetime
>>> for year in range(2009, 2012):
...     for month in range(1, 13):
...         temp_range = extremes[month-1][1] - extremes[month-1][0]
...         t = extremes[month-1][0] + int(random() * temp_range)
...         location.weatherlog_set.create(
...             timestamp=datetime(year, month, 1),
...             temperature=t,
...             humidity=int(random()*100),
...             wind_speed="%.2f" % (random()*372),,
...             visibility="%.2f" % (random()*100),
...             )

I defined minimal and maximal temperatures for each month in a tuple of tuples. Then I went through three years and for each month of that year I added random temperatures which would be in the range of minimal and maximal temperatures for that month. Humidity, wind speed and visibility are randomized without monthly ranges. The graph after data population looks like this:



That's it. Finally, I completed this tutorial. You had a chance to watch the process of Django app development. You saw, how virtualmin, migrations and third-party libraries can be used. Hopefully, you got some inspirations to create something cool by yourself just for fun.

I will publish the full code soon somewhere and you'll have a chance to download it and modify.

If something is too confusing, you don't understand some part of the tutorial, or you think that it could be improved somehow, don't hesitate to write comments.