2017-07-10

Domain Name for Django Development Server

Domain Name for Django Development Server

Isn't it strange that browsing the web you usually access the websites by domain names, however, while developing a Django website, you usually access it through IP address? Wouldn't it be handy to navigate through your local website by domain name too? Let's have a look what possibilities there are to access the local development server by a domain name.

Access via IP Address

You probably know the following line by heart since the first day of developing with Django and can type it with closed eyes?

(myenv)$ python manage.py runserver

When you run a management command runserver, it starts a lightweight Django development server which by default listens to HTTP requests on your local machine's port 8000, whereas by default, HTTP websites are running on the 80 and HTTPS websites are running on 443. Enter http://127.0.0.1:8000 in a browser and you can click through your Django project.

Note that this is a local address and it is not accessible from other devices in the network. Other people accessing the same address from their computers will see what is provided by web servers on their own machines, if any web server is running there at all.

Each device in a local network has its own Internet Protocol (IP) address. There are two versions of IP addresses: IPv4, typically formed from 4 decimal numbers separated by dots (e.g. 197.160.2.1), and IPv6, formed from hexadecimal numbers separated by colons (e.g. [fe80::200:f8ff:fe21:67cf]). The IP address can be set automatically and generated dynamically when you connect to the network, or you can set it manually and make it static. For example, the printer in the network will usually have a static address, whereas a mobile phone or tablet will have a dynamically attached IP addresses.

If you want to access a responsive website on your computer from another device in the network, I recommend you to set the IP address manually in the network settings. It is much more convenient to have an address that doesn't change every time you connect to the same network - you can bookmark it or use in different configuration files. Just don't let it clash with the IP addresses of other devices in the network.

Then run the local development server passing IP address 0.0.0.0 and port 8000:

(myenv)$ python manage.py runserver 0.0.0.0:8000

The 0.0.0.0 is a special case. It allows you to access the website through any IP address that is assigned to your computer: 0.0.0.0 or 127.0.0.1, or the one that is set in your network settings. To access the website through any of those addresses, you will have to list those IP addresses in your Django setting ALLOWED_HOSTS.

Moreover, this allows you to check the website you are building through your computer's IP address, e.g. http://197.160.2.7:8000, not only from your computer, but from any smartphone, tablet, or another computer in the same local network. Also through the same IP address you can access the website from a virtual machine. For example, by installing Windows in Parallels Desktop on a Mac, you can test how Django websites behave in Opera, Microsoft Edge, or Internet Explorer.

Domain Names for Local Host

Sometimes you want to address the website you are developing using a unique host name. This is necessary either when you have subdomains which lead to different parts of the website (e.g. http://aidas.example.com should show my profile), or when you need to test social authentication (e.g. using Python Social Auth).

One of the ways to deal with that is configuring a hosts file, which allows to map host names to IP addresses manually. Unfortunately, the hosts file doesn't support wildcard entries, such as <anything>.example.com, so for any new subdomain, you will need to modify the file as a Super User on Unix-based operating systems or as System Administrator on Windows.

A better way is to use a wildcard domain name that points to the IP of local host: 127.0.0.1. You can either set it up yourself at a domain provider, or use one of the available services.

For example, localtest.me by Scott Forsyth allows you to have unlimited wildcard entries pointing to local host. So all of the following domains would show a website at local host:

http://localtest.me:8000
http://myproject.localtest.me:8000
http://aidas.myproject.localtest.me:8000

Whichever domains you need to make work, don't forget to add them to ALLOWED_HOSTS in the Django project settings.

This enables to use authentication at Facebook or payments by PayPal (except the Instant Payment Notification which we'll cover a little later).

Also you can test subdomain resolution. For example, Django context processor might parse the subdomain and add some context variables, or a middleware might parse the subdomain and rewrite the path or redirect to a specific view.

Unfortunately, you can't test the website from an iPhone or iPad, using such address. And setting up your own domain's Address Record (A record) to the static IP of a computer in a local network is too inconvenient.

Domain Names for Local IP

There is another service - xip.io provided by Basecamp which allows you to use a wild card domain entries pointing to specific IP address.

Supposing that your computer's IP address is 197.160.2.7, all of the following domains would show a website on your computer's local web server:

http://197.160.2.7.xip.io:8000
http://myproject.197.160.2.7.xip.io:8000
http://aidas.myproject.197.160.2.7.xip.io:8000

Add them to ALLOWED_HOSTS in the project settings and you can check the website from any capable device in the local network.

Unless you are using the standard port 80, you will always have to add the port number. Also your website will be shown unsecured under HTTP, not HTTPS, and in some cases you will need to test the Django website under secure conditions, for example, when creating a Facebook canvas app or working with payments.

Tunnelling

Sometimes you want to demonstrate your fresh website to other participants at a hackathon. Or you want to share your website temporarily with the interested colleagues or friends. Or you need to test services that use Webhooks - HTTP callbacks, that post data to your server on specific events, like Instant Payment Notification at PayPal or notifications about sent SMS messages at twilio.

One way to do that is to have a remote staging website and to deploy to it very often to test the development results. For that you need a specific domain and server, and probably some automation for deployment. Also you will need to log all activities and edit log files in Terminal - no ability to make use of handy visual PyCharm debugging with breakpoints.

This is quite inconvenient. Luckily, alternatives to this method exist.

Tunnels are systems making your local host open to the public Internet. Tunnels have a frontend - that's the server by which the website will be accessed, and backend - that's your own development machine. By creating a tunnel, you open access through a firewall from a frontend server to local servers running on specified ports.

The best known open source tunnelling systems are ngrok.com, localtunnel.me, and pagekite.net. Let's have a look at each of them.

ngrok.com

Although it is not under active development now - the last commit was more than a year ago - ngrok is the most popular one. At the time of writing, it has 10573 GitHub stars. The tool was coded in the go programming language.

The ngrok is a freemium service giving you one persistent session and one randomly generated subdomain for free, but if you want to customize the setup or even install it on your own servers, you have to pay an annual fee.

To start a tunnel for a local Django project, you would type the following in the Terminal:

$ ngrok http 8000

Then anybody on the Internet could access your http://127.0.0.1:8000 entering something like https://92832de0.ngrok.io in their browser's address bar.

The default ngrok configuration would also start a special website running at http://localhost:4040 that would show the details of the traffic to and from your Django website.

If you are a paying customer and want to have a custom subdomain for your website, you can start the tunnel typing this in the Terminal:

$ ngrok http -subdomain=myproject 8000

This would create a domain like https://myproject.ngrok.io that would show the content of the Django project on your local host.

Using Canonical Name Records (CNAME records) in DNS configuration, it is also possible to create tunnels within ngrok under custom domain names like https://dev.example.com, and even wildcard entries like https://<anything>.dev.example.com.

To restrict access only to specific users, you can also use the Basic authentication with the following command:

$ ngrok http -auth="username:password" 8000

localtunnel.me

This service was created overnight at a hackathon and then published and maintained as it proved to be a useful tool. Localtunnel.me doesn't require any user account, and it creates a temporary access to your localhost under a randomly generated subdomain like https://nkfmosjsgh.localtunnel.me or a custom subdomain like https://myproject.localtunnel.me if it is available. When you close the tunnel, the address is not saved for you for future usage.

Localtunnel is free and open source. If you want or need, you can install the frontend part on your own server, so called "on premise".

To start a tunnel you would normally type the following in the Terminal:

$ lt --port 8000

If you need a custom domain, you can also type this instead:

$ lt --port 8000 --subdomain myproject

Localtunnel is meant to be relatively simple for quick temporary access. Therefore, CNAME configuration and wildcard subdomains are not possible.

Still this project is under active development. It was programmed in Node JS and by the time of writing it received 4832 GitHub Stars.

pagekite.net

Pagekite is open source, python based, pay-what-you-want solution. Comparing to the previous projects, it has only 368 GitHub Stars, but is also worth giving a try.

You can start a tunnel with Pagekite, by entering a command with your private user's domain name in the Terminal:

$ pagekite.py 8000 myuser.pagekite.me

This will open a secure access to your local Django project from https://myuser.pagekite.me.

For each project you can then have a separate project's address, like https://myproject-myuser.pagekite.me which can be created starting the tunnel like this:

$ pagekite.py 8000 myproject-myuser.pagekite.me

With Pagekite you can have custom domains like https://dev.example.com for your tunnel using CNAME setting in the domain configuration. It's possible to expose non-web services, for example SSH or Minecraft server, too.

The Basic authentication is available using a command like this:

$ pagekite.py 8000 myproject-myuser.pagekite.me +password/username=password

Django Project Configuration

If you want to use tunnelling with your Django project, you will have to do a couple of modifications here and there:

  • Change the URL configuration to show static and media files even in non DEBUG mode:

    # urls.py
    # ...
    import re
    from django.views.static import serve
    
    if settings.STATIC_URL.startswith("/"):
        urlpatterns += [
            url(
                r'^{STATIC_URL}(?P<path>.*)$'.format(STATIC_URL=re.escape(settings.STATIC_URL.lstrip('/'))),
                serve,
                # {'document_root': settings.STATIC_ROOT},
            ),
        ]
    if settings.MEDIA_URL.startswith("/"):
        urlpatterns += [
            url(
                r'^{MEDIA_URL}(?P<path>.*)$'.format(MEDIA_URL=re.escape(settings.MEDIA_URL.lstrip('/'))),
                serve,
                {'document_root': settings.MEDIA_ROOT},
            ),
        ]
    

    If you want the static files to get recognized from various apps automatically, omit the {'document_root': settings.STATIC_ROOT}. Otherwise you will have to run collectstatic management command every time you change a CSS, JavaScript, or styling image file.

  • Have separate settings for the exposed access.

    # settings.local_exposed
    from .local import *
    DEBUG = False
    ALLOWED_HOSTS = [...]  # enter the domains of your tunnel's frontend
    SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
    

    To use those settings run the following in your virtual environment:

    (myenv)$ python manage.py runserver --settings=settings.local_exposed --insecure

    Here the --insecure directive forces automatic static file recognition from different places in your project even in non DEBUG mode. Leave it out, if you are serving the static files collected by collectstatic command.

Security Recommendations

This list of security recommendations is by no means complete. Use tunnelling at your own risk.

  • Don't keep tunnels running all the time. When not in need, close the connection.
  • Share the frontend URL only with trusted people. If you make the URL easy to remember or guess, set the Basic authentication for the tunnel's frontend.
  • Switch off the DEBUG mode in your Django project.
  • Have frequent backups of your project's code, media files, and database.
  • Don't use production data for development.
  • Don't use sensitive data for testing: no real passwords or API tokens of live system, use sandbox credentials for PayPal or Stripe payments, etc.
  • If you don't trust the tunnelling services, you can set up a tunnelling frontend on your own servers.

Do you see any other security issues about using tunnelling with Django development server? Then please share your thoughts in the comments.

Final Thoughts

When you are developing a responsive website with Django and need to check how it works on a mobile device, you can run the development server with 0.0.0.0:8000 and access it on your Wifi network through the IP address of your computer, or you can use xip.io to analogically check it by a domain name.

When you need to check subdomain resolution, you can use the hosts file, configure your private subdomain pointing directly to your local IP, or use localtest.me, xip.io, or one of the tunnelling services.

When you want to debug Webhooks in order to get notified about executed payments, received messages, or completed serverless processes, you can use ngrok.com, localtunnel.me, pagekite.net or some other tunnelling service. Or of course you can set a staging website with logging, but that makes a lot of hassle debugging.

Perhaps you know some other interesting solutions how to deal with domains and local development server. If you do, don't hesitate to share your tips in the comments.

2017-04-17

Recap of DjangoConEurope 2017

"DjangoCon, why is everybody wearing this t-shirt?" wondered the security guys in the airport of Florence, Italy, in the beginning of April. The reason for that was DjangoCon Europe 2017 happening there for a week, full of interesting talks in an exciting location.

What I liked, was that the conference was not only about technical novelties in Django world, but also about human issues that programmers deal with in everyday life.

Interesting Non-tech Topics

According to a manifest, the conference had a goal to strengthen the Django community and to shape responsible attitude towards the works done with Django.

Healthy and Successful Community

We have to build stronger communities including everyone who wants to participate without discrimination. Although, at first, it might be difficult as people have biases, i.e. prejudices for or against one person or group; by being emphatic we can accept and include everyone no matter what is their gender identity or expression, sexual orientation, ethnicity, race, neurodiversity, age, religion, disabilities, geographical location, food diversities, body size, or family status.

Valuing diversity and individual differences is the key for a healthy, positive and successful community, that empowers its members and helps them grow stronger and happier.

Responsibility for How We Use Technology

Information technology companies (Apple, Alphabet, Microsoft, Amazon, and Facebook) are among the most traded companies in the world. IT connects people and their things, automates processes, stores and treats historical data. Usually you don't need many physical resources to start an IT business. Software developers have a power to shape the future, but should use this power responsibly:

With this, great responsibility is upon us: to make the future a better place, to make the future more evenly distributed, across gender gaps and discriminations, breaking economical, political and geographical barriers.

Business

  • When creating an online business, it is important to think about the business value that your product will give to people and the way you will make money with it. Don't make assumptions without talking to your customers.
  • When choosing employees for your company, give them freedom how to prove their knowledge: by a quiz, or whiteboard interview, or a take-home programming task. Different people have different ways how to best represent their skills.
  • Launch as early as possible. Track the usage statistics with Google Analytics or other analytics service. Collect emails for the mailing list. Write about your product in a blog and personalized emails.
  • Django is an open-source project based on the hard work of many professionals, and if you gain any commercial value of it and appreciate the framework, you should donate to the Django Software Foundation.

Interesting Tech Topics

From the technical point of view, I liked several ideas mentioned in the conference:

Automate your processes

  • For starting new projects, you can have boilerplates with the mostly used functionalities already prepared. Django management command startproject has a parameter --template for that where you can pass a URL to a zip file.
  • Developers should have troubleshooting checklists for debugging, just like airplane pilots.
  • There are several types of tests. Unit tests check the functionality of individual functions or methods. Integration tests check how different units work together. The functional tests check how the processes of business requirements work from start to end. Finally, there is manual testing requiring people to click through the website and fill in the forms. Some tests like the ones involving third-party authentication or mobile phones, are hardly possible to automate. Anyway, manual testing is the most expensive in time and resources (besides it being boring for the tester), functional tests go after them, then integration tests, and lastly unit tests. Although automatic testing adds up to the development time, in the long run it makes the systems more stable and error proof.

What about Django

  • You can extend the Django ORM with custom lookups, transactions, and filtered prefetchings, to make your QuerySets more readable and more capable.
  • Once again, PostgreSQL has more capabilities than MySQL and is more stable. Use EXPLAIN ANALYZE ... SQL command to find the bottlenecks of your database queries. You can usually fix them by adding indexes.
  • You can have custom indexes for your database tables, to optimize the performance on PostgreSQL (or some other vendor) database.
  • Django 1.11 is out and it's a long-term support version.

What about Third Party Django Packages

  • After analyzing the 6 most popular model translation packages (parler, hvad, klingon, modeltranslation, nece, and i18nfield) from different angles (database support, integration in django admin and forms, performance, etc.), django-hvad seemed to be the winning approach.
  • You can visually build static websites with very little coded configuration using django-cms and djangocms-cascade. The djangocms-cascade provides an alternative nested-plugins system for Django CMS.

What about Django Projects

  • If you build web apps for developing countries, you have to keep these things in mind: people might be using cell phones instead of computers (you need responsive design with small or no images), Internet connectivity is slow and unstable (websites have to be fast and should preferably have offline versions), the users do not always understand English (the websites should be translated and adapted), and locations where people live do not always have street addresses.
  • Some interesting use cases: tracking the health of the heart with Arduino and Django, providing weather data to the whole Europe using Django, managing a radio station in Birmingham using Django.

Thanks

Finally, thanks to the organizers for making this conference as great as it was. The city was beautiful, the food and coffee was delicious, the location for the talks was impressive. Looking forward to the next DjangoConEurope!

2017-03-01

Tracking the Results of Cron Jobs

Every Django website needs some automatic background tasks to execute regularly. The outdated sessions need to be cleaned up, search index needs to be updated, some data needs to be imported from RSS feeds or APIs, backups need to be created, you name it. Usually, if not all the time, those regular tasks are being set as cron jobs. However, when some task is run in the background, by default, you don't get any feedback whether it was successfully completed, or whether it crashed on the way. In this post I will show you how I handle the results of cron jobs.

In a Django project, all those tasks are usually implemented as management commands. For each such command I write a short bash script, that will call the management command with specific parameters and will print the verbose output to a log file.

Let's say my project structure is like this on a remote server:

/home/myproject
├── bin
├── include
├── lib
├── public_html
├── backups
├── project
│   └── myproject
├── scripts
└── logs

A virtual environment is created in the home directory of myproject linux user. The Django project itself is kept under project directory. The scripts directory is for my bash scripts. And the logs directory is for the verbose output of the bash scripts.

For example, for the default clearsessions command that removes outdated sessions, I would create scripts/cleanup.sh bash script as follows:

#!/usr/bin/env bash
SECONDS=0
PROJECT_PATH=/home/myproject
CRON_LOG_FILE=${PROJECT_PATH}/logs/cleanup.log

echo "Cleaning up the database" > ${CRON_LOG_FILE}
date >> ${CRON_LOG_FILE}

cd ${PROJECT_PATH}
source bin/activate
cd project/myproject    
python manage.py clearsessions --verbosity=2 --traceback >> ${CRON_LOG_FILE}  2>&1

echo "Finished." >> ${CRON_LOG_FILE}
duration=$SECONDS
echo "$(($duration / 60)) minutes and $(($duration % 60)) seconds elapsed." >> ${CRON_LOG_FILE}

To run this command every night at 1 AM, you could create a file myproject_crontab with the following content:

MAILTO=""
00 01 * * * /home/myproject/scripts/cleanup.sh

Then register the cron jobs with:

$ crontab myproject_crontab

By such a bash script, I can track:

  • At what time the script was last executed.
  • What is the verbose output of the management command.
  • If the management command broke, what was in the traceback.
  • Whether the command finished executing or hung up.
  • How long it took to run the command.

In addition, this gives me information whether the crontab was registered and whether the cron service was running at all. As I get the total time of execution in minutes and seconds, I can decide how often I can call the cron job regularly so that it doesn't clash with another cron job.

When you have multiple Django management commands, you can group them thematically into single bash script, or you can wrap them into individual bash scripts. After putting them into the crontab, the only thing left is manually checking the logs from time to time.

If you have any suggestions how I could even improve this setup, I would be glad to hear your opinion in the comments.

Here is the Gist of the scripts in this post. To see some examples of custom Django management commands, you can check Chapter 9, Data Import and Export in my book Web Development with Django Cookbook - Second Edition.