Improving Page Speed with Incremental Loading

Summary: you can use django-include-by-ajax to improve the performance and usability of your website by forcing some parts of the Django website page to be loaded and shown before other parts of the page.

---

Web browsers load and render traditional HTML pages from top to down, from left to right and as a developer you have little control over what will be shown first, second, and last. However, sometimes you need a different loading sequence to improve user experience and usability. Let's examine a couple of cases when it is advantageous to have primary content showing up immediately and secondary content loading in a moment.

Case 1. Above the Fold vs. Below the Fold

People want speed. 47% of visitors expect the website to be loaded in less than 2 seconds. If the website takes more than 3 seconds to show up, it's a big chance, that you will lose 40% of visitors. If you sell something on your website, every one-second delay causes 7% fewer visitors becoming buyers.

One technique to improve the perception of the speed of the website is to display the visible part of the screen as soon as possible, and then load the rest of the website in another go. Usually, the website pages are long vertically scrollable areas. The part of it that fits in the screen is called "above the fold" and the part underneath is called "below the fold".

It is recommended to load the part above the fold in 6 requests, including all your HTML, CSS, JavaScript, images and fonts. It's only 6 requests for a reason - that's the maximal number of requests that most browsers keep to the same HTTP/1.1 server at the same time. With HTTP/2 there is no such limitation.

You can only achieve this minimal load if you bundle and minimize your CSS and JavaScript to single files, and use only a couple of images and fonts. Going one step further you can split your CSS and JavaScript into parts that are used above the fold, and the ones that are used below the fold.

Case 2. Main Content vs. Navigation

For the users to have best user experience and smooth loading, you could display the content of articles or blog post first, and then load and display the website navigation in the header, sidebars, or footer.

If the visitor navigated to a specific page of your website, they most likely want to see the content of that page rather than navigate out to other pages.

If you have extensive nested navigation, you can also save some milliseconds of its loading at the first request, by skipping it there, but loading it by Ajax at the next go.

Additionally, if visitor disables JavaScript in their browser, they will still be able to read the content.

Case 3. Own Content vs. Third-party Content

Wouldn't you agree, websites that show ads before their own content are pretty annoying? One way to improve the user experience is to show the main content at first and show the ads or third-party widgets after several seconds.

The primary content will be correctly indexed by search engines, whereas the included widgets might be skipped, depending on implementation, which we'll examine next.

Solution 1. Iframes

One way to load the delayed secondary content is to use iframes.

Pros:

• Works without JavaScript.

Cons:

• For each iframed section, you need a separate HTML with custom CSS.
• You have to predefine and hardcode all heights of each secondary section, so it wouldn't work well with increased or decreased font size or different amounts of content.
• You cannot have interactive elements like tooltips that would go outside the boundaries of the iframe.

Solution 2. API Calls by Ajax

The page would load with empty placeholders for the secondary content and then some JavaScript function would load content for the missing sections in HTML, JSON, or XML format by Ajax, parse them, and include into the placeholders. This approach has been used by Facebook.

Pros:

• You can use the same global CSS for everything.
• The amount of content is flexible, so the designs would look good with different variations.

Cons:

• For each secondary section, you need to define a separate API endpoint.
• There are many extra requests (unless you use GraphQL for that).

Solution 3. A Second Request to the Same Page with Specific Query Parameters

The page loads with empty placeholders for the secondary content. A JavaScript function uses Ajax to load the HTML of the same page this time containing all rendered primary and secondary content. Then another JavaScript function goes through all placeholders and fills the content from the second load.

Pros:

• You can use the same global CSS for everything.
• The amount of content is flexible, so the designs could look good with different variations.
• Each page uses a single data endpoint.
• Only one extra request is necessary for the full HTML.

Cons:

• If there is a lot of primary content and not so much of secondary content, it might take too long to load and parse the secondary content.

Implementation for a Django Website using django-include-by-ajax

You can implement the third solution in a Django website using my open-source Django app django-include-by-ajax. It is meant to be understandable and simple to use for frontend Django developers, who don't touch Python code but need to work on the layouts and styling.

The idea is that instead of including different sections of a template with the {% include template_name %} template tag, you do the same using {% include_by_ajax template_name %} template tag. This template tag renders as an empty placeholder unless you access the page from a search crawler or if you access the page with a specific query parameter. Otherwise, it works more-or-less the same as the {% include %} template tag.

By adding jQuery and one jQuery-based JavaScript file to your page template, you enable the magic that does all the loading and parsing. Since version 1.0, CSS and JavaScript files can also be included in those delayed sections.

You can see django-include-by-ajax in action at the start page of my personal project 1st things 1st. There I use the above-the-fold case with the visible content coming to the screen almost immediately and the offscreen content loading in several more seconds.

Installation

It should be trivial to convert any standard heavy website page to a page loading the secondary content dynamically. There are mainly these steps to follow:

1. Install the app with your Python package manager:

   (venv)$ pip install django-include-by-ajax==1.0.0

2. Put the app into the INSTALLED_APPS in your Django project settings:

   # settings.py
   INSTALLED_APPS = [
       # ...
       # Third-party apps
       'include_by_ajax',
       # ...
   ]

3. Put these in your base.html:

   {% load staticfiles %}
   <script src="https://code.jquery.com/jquery-3.4.0.min.js" crossorigin="anonymous"></script>
   <script src="{% static 'include_by_ajax/js/include_by_ajax.min.js' %}" defer></script>

   It should also work with older or newer jQuery versions.

4. Use the new template tag in any template where you need it:

   {% load include_by_ajax_tags %}
   {% include_by_ajax "blog/includes/latest_blog_posts.html" %}

   You can even define the content for the placeholder that will be shown while the main content is loading:

   {% load include_by_ajax_tags %}
   {% include_by_ajax "blog/includes/latest_blog_posts.html" placeholder_template_name="utils/loading.html" %}

5. If you need some JavaScript action to be called after all content is loaded, you can use the custom include_by_ajax_all_loaded event on the document like this:

   $(document).on('include_by_ajax_all_loaded', function() {
       console.log('Now all placeholders are loaded and replaced with content. Hurray!');
   });

I would be glad if you tried it on some of your projects and see how it improved user experience and loading times. If you use it in production, please mention it in the comments of this blog post.

More Information

The app on Github: A Django App Providing the {% include_by_ajax %} Template Tag

The practical usage example: Strategic planner - Prioritizer - 1st things 1st.

An article on performance and usability: 5 Reasons Visitors Leave Your Website

---

Cover photo by Thomas Tucker.

Thanks to Adam Johnson for reviewing this post.

Fresh Book for Django Developers

This week the post office delivered a package that made me very satisfied. It was a box with three paper versions of my "Web Development with Django Cookbook - Second Edition". The book was published at the end of January after months of hard, but fulfilling work in the late evenings and at weekends.

The first Django Cookbook was dealing with Django 1.6. Unfortunately, the support for that version is over. So it made sense to write an update for a newer Django version. The second edition was adapted for Django 1.8 which has a long-term support until April 2018 or later. This edition introduces new features added to Django 1.7 and Django 1.8, such as database migrations, QuerySet expressions, or System Check Framework. Most concepts in this new book should also be working with Django 1.9.

My top 5 favourite new recipes are these:

• Configuring settings for development, testing, staging, and production environments
• Using database query expressions
• Implementing a multilingual search with Haystack
• Testing pages with Selenium
• Releasing a reusable Django app

The book is worth reading for any Django developer, but will be best understood by those who already know the basics of web development with Django. You can learn more about the book and buy it at the Packt website or Amazon.

I thank the Packt Publishing very much for long cooperation in the development of this book. I am especially thankful to acquisition editor Nadeem N. Bagban, content development editors Arwa Manasawala and Sumeet Sawant, and technical editor Bharat Patil. Also I am grateful for insightful feedback from the reviewer Jake Kronika.

What 5 recipes do you find the most useful?

Contributing Back to the Community - Django Cookbook

In the early beginning of year 2014, the IT book publishing company "Packt Publishing" contacted me with an interesting offer: to share my Django experience in a form of a book. I thought it might be a good challenge for me and also value for the Django community, as I had been working with Django for 7 years or so, and during that time there was quite a lot of knowledge gathered and used practically. So for the next 9 months on late evenings and weekends I was adapting some of the most useful code snippets and describing them in the book. The great staff from the Packt Publishing helped me to structure the content, make everything clear and understandable, and get the grammar correct. Finally, the book was released and it's called "Web Development with Django Cookbook".

This book is written for intermediate or advanced Django web developers. When writing the book, my own purpose was not to fully cover every possible web development task, but rather to have enough useful bits of knowledge for those who seek information about web development with Django. The book was written in the format of recipes. There are over 70 recipes giving you instructions how to deal with different challenges of web development. The code mentioned in the book is optimized for Django 1.6, but most of it should also work with older Django versions as well as with Django 1.7.

The cookbook consists of 10 chapters:

1. Getting started with Django 1.6. This chapter will guide you through the basic configuration which is necessary to start any Django project. It will cover topics like virtual environment, version control, and project settings.
2. Database Structure. When you create a new app, the first thing to do is to define your models. In this chapter you will learn how to write reusable pieces of code to use in your models. You will learn how to deal with multilingual data in your database. Also you will be told how to manage database schema changes using South migrations.
3. Forms and Views. To show your data or let one create it, you need views and forms. This chapter will show you some patterns for creating them.
4. Templates and JavaScript. Information is presented to the user by rendered templates. In modern websites, JavaScript is a must for richer user experience. This chapter shows practical examples of using templates and JavaScript together.
5. Custom Template Filters and Tags. The default Django template system is quite extensive, but there are more things to add for different cases. This chapter shows you how to create and use own template tags and filters.
6. Model Administration. Django framework comes with a handy pre-build administration. This chapter shows how to extend the default administration with your own functionality.
7. Django CMS. Django CMS is the most popular open source content management system made in Django. This chapter deals with best practices using Django CMS and extending it for your needs.
8. Hierarchical Structures. When you need to create a tree-like structure in Django, django-mptt module comes to hand. This chapter shows you how to use it and how to set administration for hierarchical structures.
9. Data Import and Export. Very often there are cases when you need to transfer data from and to different formats, retrieve it from and provide it to different sources. This chapter deals with management commands for data import and also APIs for data export.
10. Bells and Whistles. There is more to Django. This chapter shows additional snippets and tricks useful in web development with Django. Also you will learn about deployment of Django projects.

Get a copy of this book at Packt Publishing and tell me what you think about it in the comments below. Which recipes did you find the most useful? What would you like to read more in this blog or another edition of the cookbook?

Weather App Tutorial. Part 4 of 5. Template Tag

If you followed the first parts of the tutorial, you should have basic understanding how to create an app with models, set up administration, and retrieve data from third-party services. This part is about displaying collected data in any template using custom template tag.

At first, we need to create a directory templatetags containing an empty __init__.py file in the climate_change directory.

mkdir -p climate_change/templatetags
touch climate_change/templatetags/__init__.py

I will call the template library weather. So I have to create a file weather.py in climate_change/templatetags and define and register the template tag in that file. The template tag get_current_weather should display the current weather for a chosen location. To define what location you choose, you could refer to id, name or location_id, but none of them is appropriate for this reason. id and location_id are not remember-able and not informative enough, whereas the name might be changed to translate the city to another language or to add some more specifics and this change would detach the template tag from the location. For those reasons, it is best to create a new field sysname for the location model which would have a unique non-changeable value as a textual humanized identifier for templates.

But wait! It's such a pain to add new fields and modify database schema... Not, if you are using south! Let's quickly install it and then put it under INSTALLED_APPS.

easy_install south

#...
INSTALLED_APPS = (
    # django core
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.sites",
    "django.contrib.admin",
    # third-party
    "south",
    # project-specific
    "climate_change",
    )
#...

Note that south was installed in the virtual environment not spoiling the global python namespace. Now we will syncronize the database to create south_history table, create the initial migration for climate_change and apply it.

# create the missing database table from south app
python manage.py syncdb
# create initial migration for climate_change app which will be used by new projects
python manage.py startmigration climate_change --initial
# fake this migration for this project
python manage.py migrate climate_change --fake

Now we can finally add the new sysname field, create migration for it and apply it.

#...
class Location(models.Model):
    sysname = models.SlugField(
        _("system name"),
        max_length=200,
        unique=True,
        blank=True,
        default="",
        help_text=_("Do not change this value"),
        )
    name = models.CharField(_("name"), max_length=200)
    location_id = models.CharField(
        _("location ID"),
        max_length=20,
        help_text=_("Location IDs can be retrieved from URLs of weather "
            "at specific cities at Yahoo! Weather, e.g. GMXX0008 from "
            "http://weather.yahoo.com/forecast/GMXX0008.html"),
        )
#...

# create a new migration called "add_sysname"
python manage.py startmigration climate_change add_sysname --auto
# apply it to the database
python manage.py migrate climate_change

I had to set the default value to empty string because otherwise south throws exception when I use sqlite3. Anyway, after running those commands, I started the built in webserver again and added the sysname "berlin" to the record of Berlin's location.

We can get back to weather.py and add the template tag there

# -*- coding: UTF-8 -*-
from django.db import models
from django import template
from django.template import loader

register = template.Library()

### TAGS ###

def do_get_current_weather(parser, token):
    """
    Returns the latest known weather information.

    Usage::

        {% get_current_weather in <location_sysname> [using <template_path>] [as <var_name>] %}
    
    Examples::

        {% get_current_weather in "berlin" using "climate_change/custom_weather.html" %}

        {% get_current_weather in "london" as current_weather %}
        var sCurrentWeather = "{{ current_weather|escapejs }}";

    """    
    bits = token.split_contents()
    tag_name = bits.pop(0)
    template_path = ""
    var_name = ""
    location_sysname = ""
    try:
        while bits:
            first_word = bits.pop(0)
            second_word = bits.pop(0)
            if first_word == "in":
                location_sysname = second_word
            elif first_word == "using":
                template_path = second_word
            elif first_word == "as":
                var_name = second_word
                    
    except ValueError:
        raise template.TemplateSyntaxError, "get_current_weather tag requires a following syntax: {% get_current_weather [using <template_path>] [as <var_name>] %}"
    return CurrentWeatherNode(tag_name, location_sysname, template_path, var_name)

class CurrentWeatherNode(template.Node):
    def __init__(self, tag_name, location_sysname, template_path, var_name):
        self.tag_name = tag_name
        self.location_sysname = location_sysname
        self.template_path = template_path
        self.var_name = var_name
    def render(self, context):
        location_sysname = template.resolve_variable(
            self.location_sysname,
            context,
            )
        template_path = ""
        if self.template_path:
            template_path = template.resolve_variable(
                self.template_path,
                context,
                )
        context.push()
        WeatherLog = models.get_model("climate_change", "WeatherLog")
        logs = WeatherLog.objects.filter(
            location__sysname=location_sysname,
            ).order_by("-timestamp")
        if logs:
            context['weather'] = logs[0]
        output = loader.render_to_string(
            [template_path, "climate_change/current_weather.html"],
            context,
            )
        context.pop()
        if self.var_name:
            context[self.var_name] = output
            return ""
        else:
            return output


register.tag("get_current_weather", do_get_current_weather)

### FILTERS ###

# none at the moment

As you might see from the code, the template tag is using a template which can be redefined by the template designer. We still need the default template itself, so I will create a directory templates/climate_change and a file current_weather.html with this content:

{% load i18n %}
<div class="current-wheather">
    <h3>{{ weather.location.name }}</h3>
    <dl>
        <dt>{% trans "Temperature" %}:</dt>
        <dd>{{ weather.temperature }}° C</dd>
        <dt>{% trans "Humidity" %}:</dt>
        <dd>{{ weather.humidity }} %</dd>
        <dt>{% trans "Wind speed" %}:</dt>
        <dd>{{ weather.wind_speed }} km/h</dd>
        <dt>{% trans "Visibility" %}:</dt>
        <dd>{{ weather.visibility }} km</dd>
    </dl>
</div>

How can I test the template tag? I will need a new page which will include it. So I will add a rule in urls.py to redirect root url to index.html which will extend from base.html.

So the base.html looks like this:

{% block doctype %}>!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN""http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">{% endblock %}

{% load i18n %}

<html>
<head>
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <title>{% block title %}simple document{% endblock %}</title>
    {% block extra_head %}{% endblock %}
</head>
<body>
    <div id="header">{% block header %}{% endblock %}</div>
    <div id="content">{% block content %}{% endblock %}</div>
    <div id="footer">{% block footer %}{% endblock %}</div>
</body>
</html>

The index.html looks like this:

{% extends "base.html" %}
{% load i18n weather %}
{% block content %}
    {% get_current_weather in "berlin" %}
{% endblock %}

And also we'll need an extension in the urls.py:

from django.conf.urls.defaults import *
from django.contrib import admin

admin.autodiscover()

urlpatterns = patterns("",
    (
        r"^$",
        "django.views.generic.simple.direct_to_template",
        {'template': "index.html"},
        ),
    (r"^admin/", include(admin.site.urls)),
    )

When I run the development server and go to http://127.0.0.1:8000/, I see this:



It's time for the graphs! The end of this tutorial will be published here tomorrow.