tag:blogger.com,1999:blog-90144570883940599182024-03-13T05:56:28.484+01:00DjangoTricksAidas Bendoraitis on development with Django, Python, and JavaScript.Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.comBlogger72125tag:blogger.com,1999:blog-9014457088394059918.post-24094373235938961002024-02-24T19:27:00.001+01:002024-02-24T20:49:17.475+01:00Django Project on NGINX Unit<img alt="Django Project on NGINX Unit" border="0" data-original-height="512" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhXC9cd1FL7VtiZAiTLMKyvc1MsowQXjkZ1Wfbm7ooE2xoHgUkeThxe8m5Y5dlGj3_aeKJCjawnIQ6fpDmBOzfxTMuiu_8I5KJ2yinVjgxVuL8Ul3QmNaqOgGOGmCb5_LkEPiRQ3f7JRGdftmIAUH2kjnZdHSWYhdf3nqx0PDvjHavsnJThtHFTTMNxeBtf/s1600/django-project-on-nginx-unit.png"/>
<p>Recently, I learned about the NGINX Unit and decided to try it on my DjangoTricks website. Unit is a web server developed by people from NGINX, with pluggable support for Python (WSGI and ASGI), Ruby, Node.js, PHP, and a few other languages. I wanted to see whether it's really easy to set it up, have it locally on my Mac and the remote Ubuntu server, and try out the ASGI features of Django, allowing real-time communication. Also, I wanted to see whether Django is faster with Unit than with NGINX and Gunicorn. This article is about my findings.</p>
<h2>My observations</h2>
<p>Unit service uses HTTP requests to read and update its configuration. The configuration is a single JSON file that you can upload to the Unit service via a command line from the same computer or modify its values by keys in the JSON structure.</p>
<p>Normally, the docs suggest using the <code>curl</code> command to update the configuration. However, as I am using Ansible to deploy my Django websites, I wanted to create a script I could later copy to other projects. I used Google Gemini to convert bash commands from the documentation to Ansible directives and corrected its mistakes.</p>
<p>The trickiest part for me was to figure out how to use Let's Encrypt certificates in the simplest way possible. The docs are extensive and comprehensible, but sometimes, they dig into technical details that are unnecessary for a common Django developer.</p>
<p>Also, it's worth mentioning that the Unit plugin version must match your Python version in the virtual environment. It was unexpected for me when Brew installed Python 3.12 with <code>unit-python3</code> and then required my project to use Python 3.12 instead of Python 3.10 (which I used for the DjangoTricks website). So I had to recreate my virtual environment and probably will have problems later with <code>pip-compile-multi</code> when I prepare packages for the production server, still running Python 3.10.</p>
<p>Below are the instructions I used to set up the NGINX Unit with my existing DjangoTricks website on Ubuntu 22.04. For simplicity, I am writing plain Terminal commands instead of analogous Ansible directives.</p>
<h2>1. Install Unit service to your server</h2>
<p>Follow the installation instructions <a href="https://unit.nginx.org/installation/#ubuntu-2204">from documentation</a> to install <code>unit</code>, <code>unit-dev</code>, <code>unit-python3.10</code>, and whatever other plugins you want. Make sure the service is running.</p>
<h2>2. Prepare Let's Encrypt certificates</h2>
<p>Create a temporary JSON configuration file <code>/var/webapps/djangotricks/unit-config/unit-config-pre.json</code>, which will allow Let's Encrypt certbot to access the <code>.well-known</code> directory for domain confirmation:</p>
<div><pre><code class="language-javascript">{
"listeners": {
"*:80": {
"pass": "routes/acme"
}
},
"routes": {
"acme": [
{
"match": {
"uri": "/.well-known/acme-challenge/*"
},
"action": {
"share": "/var/www/letsencrypt"
}
}
]
}
}</code></pre></div>
<p>Install it to Unit:</p>
<div><pre><code class="language-bash">$ curl -X PUT --data-binary @/var/webapps/djangotricks/unit-config/unit-config-pre.json \
--unix-socket /var/run/control.unit.sock http://localhost/config</code></pre></div>
<p>If you make any mistakes in the configuration, it will be rejected with an error message and not executed.</p>
<p>Create Let's Encrypt certificates:</p>
<div><pre><code class="language-bash">$ certbot certonly -n -w /var/www/letsencrypt -m hello@djangotricks.com \
--agree-tos --no-verify-ssl -d djangotricks.com -d www.djangotricks.com</code></pre></div>
<p>Create a bundle that is required by the NGINX Unit:</p>
<div><pre><code class="language-bash">cat /etc/letsencrypt/live/djangotricks.com/fullchain.pem \
/etc/letsencrypt/live/djangotricks.com/privkey.pem > \
/var/webapps/djangotricks/unit-config/bundle1.pem</code></pre></div>
<p>Install certificate to NGINX Unit as <code>certbot1</code>:</p>
<div><pre><code class="language-bash">curl -X PUT --data-binary @/var/webapps/djangotricks/unit-config/bundle1.pem \
--unix-socket /var/run/control.unit.sock http://localhost/certificates/certbot1</code></pre></div>
<h2>3. Install Django project configuration</h2>
<p>Create a JSON configuration file <code>/var/webapps/djangotricks/unit-config/unit-config.json</code> which will use your SSL certificate and will serve your Django project:</p>
<div><pre><code class="language-javascript">{
"listeners": {
"*:80": {
"pass": "routes/acme"
},
"*:443": {
"pass": "routes/main",
"tls": {
"certificate": "certbot1"
}
}
},
"routes": {
"main": [
{
"match": {
"host": [
"djangotricks.com",
"www.djangotricks.com"
],
"scheme": "https"
},
"action": {
"pass": "applications/django"
}
},
{
"action": {
"return": 444
}
}
],
"acme": [
{
"match": {
"host": [
"djangotricks.com",
"www.djangotricks.com"
],
"uri": "/.well-known/acme-challenge/*"
},
"action": {
"share": "/var/www/letsencrypt"
}
},
{
"action": {
"return": 444
}
}
]
},
"applications": {
"django": {
"type": "python",
"path": "/var/webapps/djangotricks/project/djangotricks",
"home": "/var/webapps/djangotricks/venv/",
"module": "djangotricks.wsgi",
"environment": {
"DJANGO_SETTINGS_MODULE": "djangotricks.settings.production"
},
"user": "djangotricks",
"group": "users"
}
}
}</code></pre></div>
<p>In this configuration, HTTP requests can only be used for certification validation, and HTTPS requests point to the Django project if the domain used is correct. In other cases, the status "444 - No Response" is returned. (It's for preventing access for hackers who point their domains to your
IP address).</p>
<p>In the NGINX Unit, switching between WSGI and ASGI is literally a matter of changing one letter from "w" to "a" in the line about the Django application module, from:</p>
<div><pre><code class="language-javascript">"module": "djangotricks.wsgi",</code></pre></div>
<p>to:</p>
<div><pre><code class="language-javascript">"module": "djangotricks.asgi",</code></pre></div>
<p>I could have easily served the static files in this configuration here, too, but my <code>STATIC_URL</code> contains a dynamic part to force retrieval of new files from the server instead of the browser cache. So, I used <a href="https://whitenoise.readthedocs.io/en/latest/">WhiteNoise</a> to serve the static files.</p>
<p>For redirection from <code>djangotricks.com</code> to <code>www.djangotricks.com</code>, I also chose to use <code>PREPEND_WWW = True</code> setting instead of Unit directives.</p>
<p>And here, finally, installing it to Unit (it will overwrite the previous configuration):</p>
<div><pre><code class="language-bash">$ curl -X PUT --data-binary @/var/webapps/djangotricks/unit-config/unit-config.json \
--unix-socket /var/run/control.unit.sock http://localhost/config</code></pre></div>
<h2>How it performed</h2>
<p>DjangoTricks is a pretty small website; therefore, I couldn't do extensive benchmarks, but I checked two cases: how a filtered list view performs with NGINX and Gunicorn vs. NGINX Unit, and how you can replace NGINX, Gunicorn, and Huey background tasks with ASGI requests using NGINX Unit.</p>
<p>First of all, the <a href="https://www.djangotricks.com/tricks/?categories=development&technologies=django-4-2">https://www.djangotricks.com/tricks/?categories=development&technologies=django-4-2</a> returned the HTML result on average in 139 ms on NGINX with Gunicorn, whereas it was on average 140 ms with NGINX Unit using WSGI and 149 ms with NGINX Unit using ASGI. So, the NGINX Unit with WSGI is 0.72% slower than NGINX with Gunicorn, and the NGINX Unit with ASGI is 7.19% slower than NGINX with Gunicorn.</p>
<p>However, when I checked <a href="https://www.djangotricks.com/detect-django-version/">https://www.djangotricks.com/detect-django-version/</a> how it performs with background tasks and continuous Ajax requests until the result is retrieved vs. asynchronous checking using ASGI, I went on average from 6.62 s to 0.75 s. Of course, it depends on the timeout of the continuous Ajax request, but generally, a real-time ASGI setup can improve the user experience significantly.</p>
<h2>Final words</h2>
<p>Although NGINX Unit with Python is slightly (unnoticeably) slower than NGINX with Gunicorn, it allows Django developers to use asynchronous requests and implement real-time user experience. Also, you could probably have a Django website and Matomo analytics or WordPress blog on the same server. The NGINX Unit configuration is relatively easy to understand, and you can script the process for reusability.</p>
<hr>
<p>Cover Image by <a href="https://www.pexels.com/photo/gebetsfahen-an-mast-16090300/">Volker Meyer</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495424.209772763821142 -21.751296 80.830240436178826 48.561204000000004tag:blogger.com,1999:blog-9014457088394059918.post-24128481173718014172022-10-21T23:50:00.003+02:002022-10-23T23:16:28.104+02:00How to Handle Django Forms within Modal Dialogs<img alt="How to Handle Django Forms within Modal Dialogs" border="0" data-original-height="512" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4cuXnItkq7TjGMbpTE9A_Li3OQqI1bgK8jE7C6Yvr0L6wYgYkMS_iyPezQuJMWAv_iEa5Gq6MRcYDjwsrAbp2MiIlcauLBokesgFprjQNIYtmg4grqsK5b9zcE5LVLtqwtsQSZNb1tA31W_8jHaNOCAtnTESKBLZXKRAT-y2iEwNxs9QSECg2qd5Ayg/s1600/how-to-handle-django-forms-within-modal-dialogs.png"/>
<p>I like django-crispy-forms. You can use it for stylish uniform HTML forms with Bootstrap, TailwindCSS, or even your custom template pack. But when it comes to custom widgets and dynamic form handling, it was always a challenge. Recently I discovered htmx. It's a JavaScript framework that handles Ajax communication based on custom HTML attributes. In this article, I will explore how you can use django-crispy-forms with htmx to provide a form with server-side validation in a modal dialog.</p>
<h2>The setup</h2>
<p>For this experiment, I will be using these PyPI packages:</p>
<ul>
<li>Django - my beloved Python web framework.</li>
<li>django-crispy-forms - library for stylized forms.</li>
<li>crispy-bootstrap5 - Bootstrap 5 template pack for django-crispy-forms.</li>
<li>django-htmx - some handy htmx helpers for Django projects.</li>
</ul>
<p>Also, I will use the CDN versions of Bootstrap5 and htmx.</p>
<h2>The form</h2>
<p>I decided to add some crispy style to the login form by extending Django's authentication form and attaching a crispy helper to it.</p>
<div><pre><code class="language-python">from django.contrib.auth.forms import AuthenticationForm
from crispy_forms.helper import FormHelper
from crispy_forms.layout import Layout
from crispy_bootstrap5 import bootstrap5
class LoginForm(AuthenticationForm):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.helper = FormHelper()
self.helper.form_tag = False
self.helper.include_media = False
self.helper.layout = Layout(
bootstrap5.FloatingField("username", autocomplete="username"),
bootstrap5.FloatingField("password", autocomplete="current-password"),
)</code></pre></div>
<p>Here I set <code>form_tag</code> to <code>False</code> to skip the <code><form></code> tag from the rendered form elements because I want to customize it in the templates. And I set <code>include_media</code> to <code>False</code> because I want to manually handle the media inclusion, as you will see later, instead of automatically including them in the form.</p>
<h2>The views</h2>
<p>I will have two views:</p>
<ul>
<li><strong>The home view</strong> will have a button to open a modal dialog for login.</li>
<li><strong>The login form view</strong> will handle the form and return the HTML markup for the dialog.</li>
</ul>
<div><pre><code class="language-python">from django.shortcuts import render, redirect
from django.contrib.auth import login as auth_login, logout as auth_logout
from django_htmx.http import HttpResponseClientRefresh
from .forms import LoginForm
def home(request):
if request.user.is_authenticated:
return render(request, "dashboard.html")
return render(request, "home.html", context)
def login(request):
if request.method == "POST":
form = LoginForm(request=request, data=request.POST, prefix="login")
template_name = "login_form.html"
if form.is_valid():
user = form.get_user()
auth_login(request, user)
return HttpResponseClientRefresh()
else:
form = LoginForm(request=request, prefix="login")
template_name = "login_dialog.html"
context = {"form": form}
return render(request, template_name, context)
</code></pre></div>
<p>The home view just returns different templates based on whether the user is logged in or not.</p>
<p>The login view handles a login form and renders different templates based on whether the view was accessed by GET or POST method. If the login succeeds, a special <code>HttpResponseClientRefresh</code> response is returned, which tells htmx to refresh the page from where the login form was loaded and submitted. It's an empty response with the <code>HX-Refresh: true</code> <a href="https://htmx.org/reference/#response_headers">header</a>.</p>
<p>Then I plug those two views into my <code>urls.py</code> rules.</p>
<h2>The templates and javascript</h2>
<p>At the end of the base.html template, I include htmx from CDN and my custom <code>dialog.js</code>:</p>
<div><pre><code class="language-markup"><script src="https://unpkg.com/htmx.org@1.8.2" integrity="sha384-+8ISc/waZcRdXCLxVgbsLzay31nCdyZXQxnsUy++HJzJliTzxKWr0m1cIEMyUzQu" crossorigin="anonymous"></script>
<script src="{% static 'js/dialog.js' %}"></script></code></pre></div>
<p>In the <code>home.html</code> template, I add a button which will open the dialog:</p>
<div><pre><code class="language-markup"><button
data-hx-get="{% url 'login' %}"
data-hx-target="main"
data-hx-swap="beforeend"
type="button"
class="btn btn-primary"
>Log in</button></code></pre></div>
<p>Note that htmx allows either <code>hx-*</code> syntax or <code>data-hx-*</code> for its HTML attributes and I prefer the latter because <code>data-*</code> attributes make a valid HTML document.</p>
<p>In the snippet above, I tell htmx to load the login page on the button click and insert it before the end of the <code><main></code> HTML tag.</p>
<p>Let's have a look at my <code>dialog.js</code> file:</p>
<div><pre><code class="language-javascript">function init_widgets_for_htmx_element(target) {
// init other widgets
// init modal dialogs
if (target.tagName === 'DIALOG') {
target.showModal();
htmx.on('.close-dialog', 'click', function(event) {
var dialog = htmx.find('dialog[open]');
dialog.close();
htmx.remove(dialog);
});
}
}
htmx.onLoad(init_widgets_for_htmx_element);</code></pre></div>
<p>Here, when a page loads or a htmx inserts a snippet, the <code>init_widgets_for_htmx_element</code> function will be called with the <code><body></code> or the inserted element as the <code>target</code>. One can use this function to initialize widgets, such as rich text fields, autocompletes, tabs, custom frontend validators, etc. Also, I use this function to open the loaded modal dialogs and add event handlers to close them.</p>
<p>Now the <code>login_dialog.html</code> template looks like this (I just stripped the styling markup):</p>
<div><pre><code class="language-markup"><dialog id="htmx-dialog">
<h1>Login</h1>
<button type="button" class="close-dialog btn-close" aria-label="Close"></button>
{% include "login_form.html" %}
<button type="submit" form="htmx-dialog-form">Log in</button>
</dialog></code></pre></div>
<p>And the <code>login_form.html</code> looks like this:</p>
<div><pre><code class="language-markup"><form
id="htmx-dialog-form"
novalidate
data-hx-post="{% url 'login' %}"
data-hx-swap="outerHTML"
>
{% load crispy_forms_tags %}
{% crispy form %}
</form></code></pre></div>
<p>The htmx attributes tell htmx to submit the form data to the login view by Ajax and replace the <code><form></code> HTML tag with the response received. That is used for form validation. If the response has the <code>HX-Refresh: true</code> header, as mentioned before, then the home page is refreshed.</p>
<p>Here is what the result looks like:</p>
<img alt="Validated form within a modal dialog" border="0" data-original-height="1168" data-original-width="1970" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQ_kpsy6iVM7ubpk-QvPGxmwB7nNSendlM_mDnolBrsTSvvlRQcrC0Ls75GTpnotGRl_X1_5ZiRReOFAkgB3itZx8ovHfADkSUu1t1V9JuHqwjKmtwee5gBlEJ2o1hgloZW0y90zURHLSX6UH1uLIM_52ZEf2zcZ_qSnx03l4PouR_bRzLCS7PujmGDQ/s1600/validated-dialog-form.png"/>
<h2>The form media</h2>
<p>If you noticed before, we excluded the media from the form. Otherwise, the home page would load the media of its own forms (for example, a search form) and the media of the dialog forms. And that would cause double executions of shared scripts, for example, the ones for autocompletes and rich text fields.</p>
<p>You can nicely combine media from different forms by <a href="https://www.djangotricks.com/tricks/AHBBEk53n6cc/">concatenating the media instances</a>. This way, each CSS and Javascript file is included just once.</p>
<p>As the login form isn't part of the home page but instead included on demand, I need to have its media in the home view or any other view where the login button is shown.</p>
<p>Here comes this <a href="https://www.djangotricks.com/tricks/7PnHMbfaHrNw/">custom context processor</a> for help:</p>
<div><pre><code class="language-python">def login_dialog(request):
from .forms import LoginForm
if not request.user.is_authenticated:
form = LoginForm(request=request)
if hasattr(request, "combined_media"):
request.combined_media += form.media
else:
request.combined_media = form.media
return {}</code></pre></div>
<p>It checks for any <code>request.combined_media</code> and attaches the form media from the login form.</p>
<p>Lastly, I attach this context processor to the template settings and render the value of combined media before <code></body></code>:</p>
<div><pre><code class="language-markup">{{ request.combined_media }}</code></pre></div>
<h2>The final words</h2>
<p>Get the <a href="https://github.com/archatas/experiment_with_htmx">code to play with</a> from Github. As you can see, htmx makes Ajax communications pretty straightforward, even when it's about dynamically loading and reloading parts of the content or initializing custom widgets. </p>
<hr>
<p>Cover photo by <a href="https://www.pexels.com/photo/view-of-building-258163/">Pixabay</a></p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495424.209772763821142 -21.751296 80.830240436178826 48.561204000000004tag:blogger.com,1999:blog-9014457088394059918.post-88835668462162860042022-10-04T02:06:00.000+02:002022-10-04T02:06:42.965+02:00How to Rename a Django App<img alt="" border="0" data-original-height="512" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEFWXkvnuEVHukf_OWe970P2s3rRTAzpag-mYjnQSq7XT8SVTnoJzNVdCQw-eu-GIy2Ayt0PhVpSTiU-sUirz-m6hBOuiKeBxYfVQFq4QXjrr7OQiy6-4omrnK8UTimyNVs1MWnTOXNerug7qPJbJkE4KZoP8jO7oY1vchjSrh0ulxSdSDlM_rXyzR3g/s1600/how-to-rename-a-django-app.png"/>
<p>When I initially created my MVP (minimal viable product) for <a href="https://www.1st-things-1st.com">1st things 1st</a>, I considered the whole Django project to be about prioritization. After a few years, I realized that the Django project is about SaaS (software as a service), and prioritization is just a part of all functionalities necessary for a SaaS to function. I ended up needing to rename apps to have clean and better-organized code. Here is how I did that.</p>
<h2>0. Get your code and database up to date</h2>
<p>Ensure you have the latest git pull and execute all database migrations.</p>
<h2>1. Install django-rename-app</h2>
<p>Put <code>django-rename-app</code> into pip requirements and install them or just run:</p>
<div><pre><code class="language-shell">(venv)$ pip install django-rename-app</code></pre></div>
<p>Put the app into <code>INSTALLED_APPS</code> in your settings:</p>
<div><pre><code class="language-python">INSTALLED_APPS = [
# …
"django_rename_app",
]</code></pre></div>
<h2>2. Rename the app directories</h2>
<p>Rename the <code>oldapp</code> as <code>newapp</code> in your apps and templates.</p>
<h2>3. Rename the app name occurrences in the code</h2>
<p>Rename the app in all your imports, relations, migrations, and template paths.</p>
<p>You can do a global search for <code>oldapp</code> and then check case by case where you need to rename that term to <code>newapp</code>, and where not.</p>
<h2>4. Run the management command rename_app</h2>
<p>Run the management command <code>rename_app</code>:</p>
<div><pre><code class="language-bash">(env)$ python manage.py rename_app oldapp newapp</code></pre></div>
<p>This command renames the app prefix the app tables and the records in <code>django_content_type</code> and <code>django_migrations</code> tables.</p>
<p>If you plan to update staging or production servers, add the <code>rename_app</code> command before running migrations in your deployment scripts (Ansible, Docker, etc.)</p>
<h2>5. Update indexes and constraints</h2>
<p>Lastly, create an empty database migration for the app with custom code to update indexes and foreign-key constraints.</p>
<div><pre><code class="language-bash">(env)$ python manage.py makemigrations newapp --empty --name rename_indexes</code></pre></div>
<p>Fill the migration with the following code:</p>
<div><pre><code class="language-python"># newapp/migrations/0002_rename_indexes.py
from django.db import migrations
def named_tuple_fetch_all(cursor):
"Return all rows from a cursor as a namedtuple"
from collections import namedtuple
desc = cursor.description
Result = namedtuple("Result", [col[0] for col in desc])
return [Result(*row) for row in cursor.fetchall()]
def rename_indexes(apps, schema_editor):
from django.db import connection
with connection.cursor() as cursor:
cursor.execute(
"""SELECT indexname FROM pg_indexes
WHERE tablename LIKE 'newapp%'"""
)
for result in named_tuple_fetch_all(cursor):
old_index_name = result.indexname
new_index_name = old_index_name.replace(
"oldapp_", "newapp_", 1
)
cursor.execute(
f"""ALTER INDEX IF EXISTS {old_index_name}
RENAME TO {new_index_name}"""
)
def rename_foreignkeys(apps, schema_editor):
from django.db import connection
with connection.cursor() as cursor:
cursor.execute(
"""SELECT table_name, constraint_name
FROM information_schema.key_column_usage
WHERE constraint_catalog=CURRENT_CATALOG
AND table_name LIKE 'newapp%'
AND position_in_unique_constraint notnull"""
)
for result in named_tuple_fetch_all(cursor):
table_name = result.table_name
old_foreignkey_name = result.constraint_name
new_foreignkey_name = old_foreignkey_name.replace(
"oldapp_", "newapp_", 1
)
cursor.execute(
f"""ALTER TABLE {table_name}
RENAME CONSTRAINT {old_foreignkey_name}
TO {new_foreignkey_name}"""
)
class Migration(migrations.Migration):
dependencies = [
("newapp", "0001_initial"),
]
operations = [
migrations.RunPython(rename_indexes, migrations.RunPython.noop),
migrations.RunPython(rename_foreignkeys, migrations.RunPython.noop),
]</code></pre></div>
<p>Run the migrations:</p>
<div><pre><code class="language-bash">(env)$ python manage.py migrate</code></pre></div>
<p>If something doesn't work as wanted, migrate back, fix the code, and migrate again. You can unmigrate by migrating to one step before the last migration, for example:</p>
<div><pre><code class="language-bash">(env)$ python manage.py migrate 0001</code></pre></div>
<h2>6. Cleanup</h2>
<p>After applying the migration in all necessary environments, you can clean them up by removing <code>django-rename-app</code> from your pip requirements and deployment scripts.</p>
<h2>Final words</h2>
<p>It's rarely possible to build a system that meets all your needs from the beginning. Proper systems always require continuous improvement and refactoring. Using a combination of Django migrations and <code>django-rename-app</code>, you can work on your websites in an Agile, clean, and flexible way.</p>
<p>Happy coding!</p>
<hr>
<p>Cover photo by <a href="https://unsplash.com/photos/I_pOqP6kCOI">freestocks</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495424.209772763821142 -21.751296 80.830240436178826 48.561204000000004tag:blogger.com,1999:blog-9014457088394059918.post-1575203833724173502022-04-18T03:46:00.002+02:002022-04-18T21:02:39.570+02:00How I Integrated Zapier into my Django Project<img alt="" border="0" data-original-height="512" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheXqdYyAbLIPTMZV3VzdpvddQMjnn5E2jyBAGLbHtHWC7CJi3A39r0_4w-jS8Pamma6eqrH7RJpt1GNy41hKRW4lXtLCRERIQJyRHbpERsbcDwEXz7_6MvpFxwx9k18xJ1xRXTkB-Ue0UmuX11ZeBqJnHDLpwiSzJaSMXmLaeGD9CEg06kiaVJfQc7Sw/s1600/how-I-integrated-zapier-into-my-django-project.png"/>
<p>As you might know, I have been developing, providing, and supporting the prioritization tool <a href="https://www.1st-things-1st.com">1st things 1st</a>. One of the essential features to implement was exporting calculated priorities to other productivity tools. Usually, building an export from one app to another takes 1-2 weeks for me. But this time, I decided to go a better route and use Zapier to export priorities to almost all possible apps in a similar amount of time. Whaaat!?? In this article, I will tell you how.</p>
<h2>What is Zapier and how it works?</h2>
<p>The no-code tool Zapier takes <strong>input</strong> from a wide variety of web apps and <strong>outputs</strong> it to many other apps. Optionally you can <strong>filter</strong> the input based on conditions. Or <strong>format</strong> the input differently (for example, convert HTML to Markdown). In addition, you can stack the output actions one after the other. Usually, people use 2-3 steps for their automation, but there are power users who create 50-step workflows.</p>
<img alt="" border="0" data-original-height="1026" data-original-width="2844" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhThN0nvyCPCqsn0M7fHblKceByXOnJLVlXLQg7xu44f0ajjb153yvxgYw52wZD64MufHzpR5sn3yUf7i4_x_2Z0l3VOuo8SBSXoKdeBSByilQ4XuD2Zgm_HsF3EhU3wT9vvKtI2y8HmeZUqiDDeIRtqCScLQv8Ve2tcRzX9A9iJT7ZzeJLD-LdR6jAAQ/s1600/zapier-workflow-step.png"/>
<p>The input is managed by Zapier's <strong>triggers</strong>. The output is controlled by Zapier's <strong>actions</strong>. These can be configured at the website UI or using a command-line tool. I used the UI as this was my first integration. Trigger events accept a JSON feed of objects with unique IDs. Each new item there is treated as a new input item. With a free tier, the triggers are checked every 15 minutes. Multiple triggers are handled in parallel, and the sorting order of execution is not guaranteed. As it is crucial to have the sorting order correct for <a href="https://www.1st-things-1st.com">1st things 1st</a> priorities, people from Zapier support suggested providing each priority with a 1-minute interval to make sure the priorities get listed in the target app sequentially.</p>
<p>The most challenging part of Zapier integration was setting up OAuth 2.0 provider. Even though I used a third-party Django app <a href="https://django-oauth-toolkit.readthedocs.io/en/latest/">django-oauth-toolkit</a> for that. Zapier accepts other authentication options too, but this one is the least demanding for the end-users.</p>
<h2>Authentication</h2>
<p>OAuth 2.0 allows users of one application to use specific data of another application while keeping private information private. You might have used the OAuth 2.0 client directly or via a wrapper for connecting to Twitter apps. For Zapier, one has to set OAuth 2.0 provider.</p>
<p>The <a href="https://django-oauth-toolkit.readthedocs.io/en/latest/getting_started.html">official tutorial for setting up OAuth 2.0 provider with <code>django-oauth-toolkit</code></a> is a good start. However, one problem with it is that by default, any registered user can create OAuth 2.0 applications at your Django website, where in reality, you need just one global application.</p>
<p>First of all, I wanted to allow OAuth 2.0 application creation only for superusers.</p>
<p>For that, I created a new Django app <a href="https://gist.github.com/archatas/e43ff14f116d7b07fc0bfe7ddfcc7bf9"><code>oauth2_provider_adjustments</code></a> with modified views and URLs to use instead of the ones from <code>django-oauth-toolkit</code>.</p>
<p>The views related to OAuth 2.0 app creation extended this <code>SuperUserOnlyMixin</code> instead of <code>LoginRequiredMixin</code>:</p>
<div><pre><code class="language-python">from django.contrib.auth.mixins import AccessMixin
class SuperUserOnlyMixin(AccessMixin):
def dispatch(self, request, *args, **kwargs):
if not request.user.is_superuser:
return self.handle_no_permission()
return super().dispatch(request, *args, **kwargs)</code></pre></div>
<p>Then I replaced the default <code>oauth2_provider</code> URLs:</p>
<div><pre><code class="language-python">urlpatterns = [
# …
path("o/", include("oauth2_provider.urls", namespace="oauth2_provider")),
]</code></pre></div>
<p>with my custom ones:</p>
<div><pre><code class="language-python">urlpatterns = [
# …
path("o/", include("oauth2_provider_adjustments.urls", namespace="oauth2_provider")),
]</code></pre></div>
<p>I set the new OAuth 2.0 application by going to <code>/o/applications/register/</code> and filling in this info:</p>
<p><strong>Name:</strong> Zapier<br>
<strong>Client type:</strong> Confidential<br>
<strong>Authorization grant type:</strong> Authorization code<br>
<strong>Redirect uris:</strong> <code>https://zapier.com/dashboard/auth/oauth/return/1stThings1stCLIAPI/</code> (copied from Zapier)<br>
<strong>Algorithm:</strong> No OIDC support</p>
<p><em>If you have some expertise in the setup choices and see any flaws, let me know.</em></p>
<p>Zapier requires creating a test view that will return anything to check if there are no errors authenticating a user with OAuth 2.0. So I made a simple JSON view like this:</p>
<div><pre><code class="language-python">from django.http.response import JsonResponse
def user_info(request, *args, **kwargs):
if not request.user.is_authenticated:
return JsonResponse(
{
"error": "User not authenticated",
},
status=200,
)
return JsonResponse(
{
"first_name": request.user.first_name,
"last_name": request.user.last_name,
},
status=200,
)</code></pre></div>
<p>Also, I had to have login and registration views for those cases when the user's session was not present.</p>
<p>Lastly, at Zapier, I had to set these values for OAuth 2.0:</p>
<p><strong>Client ID:</strong> The Client ID from registered app<br>
<strong>Client Secret:</strong> The Client Secret from registered app </p>
<p><strong>Authorization URL:</strong> <code>https://apps.1st-things-1st.com/o/authorize/</code><br>
<strong>Scope:</strong> read write<br>
<strong>Access Token Request:</strong> <code>https://apps.1st-things-1st.com/o/token/</code><br>
<strong>Refresh Token Request:</strong> <code>https://apps.1st-things-1st.com/o/token/</code><br>
<strong>I want to automatically refresh on unauthorized error:</strong>
Checked<br>
<strong>Test:</strong> <code>https://apps.1st-things-1st.com/user-info/</code><br>
<strong>Connection Label:</strong> <code>{{first_name}} {{last_name}}</code></p>
<h2>Trigger implementation</h2>
<p>There are two types of triggers in Zapier:</p>
<ul>
<li>(A) <strong>Ones for providing new things to other apps</strong>, for example, sending priorities from <strong>1st things 1st</strong> to other productivity apps.</li>
<li>(B) <strong>Ones for listing things in drop boxes at the former triggers</strong>, for example, letting Zapier users choose the <strong>1st things 1st</strong> project from which to import priorities.</li>
</ul>
<p>The feeds for triggers should (ideally) be paginated. But without meta information for the item count, page number, following page URL, etc., you would usually have with <code>django-rest-framework</code> or other REST frameworks. Provide only an array of objects with unique IDs for each page. The only field name that matters is "id" – others can be anything. Here is an example:</p>
<div><pre><code class="language-javascript">[
{
"id": "39T7NsgQarYf",
"project": "5xPrQbPZNvJv",
"title": "01. Custom landing pages for several project types (83%)",
"plain_title": "Custom landing pages for several project types",
"description": "",
"score": 83,
"priority": 1,
"category": "Choose"
},
{
"id": "4wBSgq3spS49",
"project": "5xPrQbPZNvJv",
"title": "02. Zapier integration (79%)",
"plain_title": "Zapier integration",
"description": "",
"score": 79,
"priority": 2,
"category": "Choose"
},
{
"id": "6WvwwB7QAnVS",
"project": "5xPrQbPZNvJv",
"title": "03. Electron.js desktop app for several project types (42%)",
"plain_title": "Electron.js desktop app for several project types",
"description": "",
"score": 41,
"priority": 3,
"category": "Consider"
}
]</code></pre></div>
<p>The feeds should list items in reverse order for the (A) type of triggers: the newest things go at the beginning. The pagination is only used to cut the number of items: the second and further pages of the paginated list are ignored by Zapier.</p>
<p>In my specific case of priorities, the order matters, and no items should be lost in the void. So I listed the priorities sequentially (not newest first) and set the number of items per page unrealistically high so that you basically get all the things on the first page of the feed.</p>
<p>The feeds for the triggers of (B) type are normally paginated from the first page until the page returns empty results. The order should be alphabetical, chronological, or by sorting order field, whatever makes sense. There you need just two fields, the ID and the title of the item (but more fields are allowed too), for example:</p>
<div><pre><code class="language-javascript">[
{
"id": "5xPrQbPZNvJv",
"title": "1st things 1st",
"owner": "Aidas Bendoraitis"
},
{
"id": "VEXGzThxL6Sr",
"title": "Make Impact",
"owner": "Aidas Bendoraitis"
},
{
"id": "WoqQbuhdUHGF",
"title": "DjangoTricks website",
"owner": "Aidas Bendoraitis"
},
]</code></pre></div>
<p>I used <code>django-rest-framework</code> to implement the API because of the batteries included, such as browsable API, permissions, serialization, pagination, etc.</p>
<p>For the specific Zapier requirements, I had to write a custom pagination class, <code>SimplePagination</code>, to use with my API lists. It did two things: omitted the meta section and showed an empty list instead of a 404 error for pages that didn't have any results:</p>
<div><pre><code class="language-python">from django.core.paginator import InvalidPage
from rest_framework.pagination import PageNumberPagination
from rest_framework.response import Response
class SimplePagination(PageNumberPagination):
page_size = 20
def get_paginated_response(self, data):
return Response(data) # <-- Simple pagination without meta
def get_paginated_response_schema(self, schema):
return schema # <-- Simple pagination without meta
def paginate_queryset(self, queryset, request, view=None):
"""
Paginate a queryset if required, either returning a
page object, or `None` if pagination is not configured for this view.
"""
page_size = self.get_page_size(request)
if not page_size:
return None
paginator = self.django_paginator_class(queryset, page_size)
page_number = self.get_page_number(request, paginator)
try:
self.page = paginator.page(page_number)
except InvalidPage as exc:
msg = self.invalid_page_message.format(
page_number=page_number, message=str(exc)
)
return [] # <-- If no items found, don't raise NotFound error
if paginator.num_pages > 1 and self.template is not None:
# The browsable API should display pagination controls.
self.display_page_controls = True
self.request = request
return list(self.page)</code></pre></div>
<p>To preserve the order of items, I had to make the priorities appear one by one at 1-minute intervals. I did that by having a Boolean field <code>exported_to_zapier</code> at the priorities. The API showed priorities only if that field was set to <code>True</code>, which wasn't the case by default. Then, background tasks were scheduled 1 minute after each other, triggered by a button click at <strong>1st things 1st</strong>, which set the <code>exported_to_zapier</code> to <code>True</code> for each next priority. I was using <code>huey</code>, but the same can be achieved with <code>Celery</code>, cron jobs, or other background task manager:</p>
<div><pre><code class="language-python"># zapier_api/tasks.py
from django.conf import settings
from django.utils.translation import gettext
from huey.contrib.djhuey import db_task
@db_task()
def export_next_initiative_to_zapier(project_id):
from evaluations.models import Initiative
next_initiatives = Initiative.objects.filter(
project__pk=project_id,
exported_to_zapier=False,
).order_by("-total_weight", "order")
count = next_initiatives.count()
if count > 0:
next_initiative = next_initiatives.first()
next_initiative.exported_to_zapier = True
next_initiative.save(update_fields=["exported_to_zapier"])
if count > 1:
result = export_next_initiative_to_zapier.schedule(
kwargs={"project_id": project_id},
delay=settings.ZAPIER_EXPORT_DELAY,
)
result(blocking=False)
</code></pre></div>
<p><strong>One gotcha:</strong> Zapier starts pagination from 0, whereas <code>django-rest-framework</code> starts pagination from 1. To make them work together, I had to modify the API request (written in JavaScript) at Zapier trigger configuration:</p>
<div><pre><code class="language-javascript">const options = {
url: 'https://apps.1st-things-1st.com/api/v1/projects/',
method: 'GET',
headers: {
'Accept': 'application/json',
'Authorization': `Bearer ${bundle.authData.access_token}`
},
params: {
'page': bundle.meta.page + 1 // <-- The custom line for pagination
}
}
return z.request(options)
.then((response) => {
response.throwForStatus();
const results = response.json;
// You can do any parsing you need for results here before returning them
return results;
});</code></pre></div>
<h2>Final Words</h2>
<p>For the v1 of Zapier integration, I didn't need any Zapier actions, so they are yet something to explore, experiment with, and learn about. But the Zapier triggers seem already pretty helpful and a big win compared to individual exports without this tool.</p>
<p>If you want to try the result, do this:</p>
<ul>
<li>Create an account and a project at <a href="https://www.1st-things-1st.com">1st things 1st</a></li>
<li>Prioritize something</li>
<li>Head to <a href="https://zapier.com/apps/1st-things-1st/integrations">Zapier integrations</a> and connect your prioritization project to a project of your favorite to-do list or project management app</li>
<li>Then click on "Export via Zapier" at <strong>1st things 1st</strong>.</li>
</ul>
<hr>
<p>Cover photo by <a href="https://www.pexels.com/photo/persons-hand-doing-thumbs-up-8058540/">Anna Nekrashevich</a></p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.404954-12.571633364979775 -127.22004599999997 90 154.02995399999998tag:blogger.com,1999:blog-9014457088394059918.post-53123844004341764322022-04-09T00:06:00.002+02:002022-10-21T23:52:08.286+02:00Generic Functionality without Generic Relations<img alt="" border="0" data-original-height="512" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHGZADtOZzDWCBjAkkUwrVXgwV_ZHN6lPLsQOxshv2yQpm9N4idHx3xBIsAuqOeQD1ol-b7AZ40_3zojXDhJ70h9Jv7Roh9y6lMnFgha8cYFaiFbPFapL_SmcpQUPdgCByIzXsUSrw7FPWFfebAIbIhUlJK9PGmrPueyBAm0nZ8cYVvrPIlqHXoDcN0w/s1600/generic-functionality-without-generic-relations.png"/>
<p>When you have some generic functionality like anything commentable, likable, or upvotable, it’s common to use <a href="https://simpleisbetterthancomplex.com/tutorial/2016/10/13/how-to-use-generic-relations.html">Generic Relations in Django</a>. The problem with Generic Relations is that they create the relationships at the <strong>application level</strong> instead of the <strong>database level</strong>, and that requires a lot of database queries if you want to aggregate content that shares the generic functionality. There is another way that I will show you in this article. </p>
<p>I learned this technique at my first job in 2002 and then rediscovered it again with Django a few years ago. The trick is to have a generic <code>Item</code> model where every other autonomous model has a one-to-one relation to the <code>Item</code>. Moreover, the <code>Item</code> model has an <code>item_type</code> field, allowing you to recognize the backward one-to-one relationship.</p>
<p>Then whenever you need to have some generic categories, you link them to the <code>Item.</code> Whenever you create generic functionality like media gallery, comments, likes, or upvotes, you attach them to the <code>Item</code>. Whenever you need to work with permissions, publishing status, or workflows, you deal with the <code>Item</code>. Whenever you need to create a global search or trash bin, you work with the <code>Item</code> instances.</p>
<p>Let’s have a look at some code.</p>
<h2>Items</h2>
<p>First, I'll create the <code>items</code> app with two models: the previously mentioned <code>Item</code> and the abstract model <code>ItemBase</code> with the one-to-one relation for various models to inherit:</p>
<div><pre><code class="language-python"># items/models.py
import sys
from django.db import models
from django.apps import apps
if "makemigrations" in sys.argv:
from django.utils.translation import gettext_noop as _
else:
from django.utils.translation import gettext_lazy as _
class Item(models.Model):
"""
A generic model for all autonomous models to link to.
Currently these autonomous models are available:
- content.Post
- companies.Company
- accounts.User
"""
ITEM_TYPE_CHOICES = (
("content.Post", _("Post")),
("companies.Company", _("Company")),
("accounts.User", _("User")),
)
item_type = models.CharField(
max_length=200, choices=ITEM_TYPE_CHOICES, editable=False, db_index=True
)
class Meta:
verbose_name = _("Item")
verbose_name_plural = _("Items")
def __str__(self):
content_object_title = (
str(self.content_object) if self.content_object else "BROKEN REFERENCE"
)
return (
f"{content_object_title} ({self.get_item_type_display()})"
)
@property
def content_object(self):
app_label, model_name = self.item_type.split(".")
model = apps.get_model(app_label, model_name)
return model.objects.filter(item=self).first()
class ItemBase(models.Model):
"""
An abstract model for the autonomous models that will link to the Item.
"""
item = models.OneToOneField(
Item,
verbose_name=_("Item"),
editable=False,
blank=True,
null=True,
on_delete=models.CASCADE,
related_name="%(app_label)s_%(class)s",
)
class Meta:
abstract = True
def save(self, *args, **kwargs):
if not self.item:
model = type(self)
item = Item.objects.create(
item_type=f"{model._meta.app_label}.{model.__name__}"
)
self.item = item
super().save()
def delete(self, *args, **kwargs):
if self.item:
self.item.delete()
super().delete(*args, **kwargs)</code></pre></div>
<p>Then let's create some autonomous models that will have one-to-one relations with the <code>Item</code>. By "autonomous models," I mean those which are enough by themselves, such as posts, companies, or accounts. Models like types, categories, tags, or likes, wouldn't be autonomous.</p>
<h2>Posts</h2>
<p>Second, I create the <code>content</code> app with the <code>Post</code> model. This model extends <code>ItemBase</code> which will create the one-to-one relation on save, and will define the <code>item_type</code> as <code>content.Post</code>: </p>
<div><pre><code class="language-python"># content/models.py
import sys
from django.contrib.auth.base_user import BaseUserManager
from django.db import models
from django.contrib.auth.models import AbstractUser
if "makemigrations" in sys.argv:
from django.utils.translation import gettext_noop as _
else:
from django.utils.translation import gettext_lazy as _
from items.models import ItemBase
class Post(ItemBase):
title = models.CharField(_("Title"), max_length=255)
slug = models.SlugField(_("Slug"), max_length=255)
content = models.TextField(_("Content"))
class Meta:
verbose_name = _("Post")
verbose_name_plural = _("Posts")
</code></pre></div>
<h2>Companies</h2>
<p>Third, I create the <code>companies</code> app with the <code>Company</code> model. This model also extends <code>ItemBase</code> which will create the one-to-one relation on save, and will define the <code>item_type</code> as <code>companies.Company</code>: </p>
<div><pre><code class="language-python"># companies/models.py
import sys
from django.contrib.auth.base_user import BaseUserManager
from django.db import models
from django.contrib.auth.models import AbstractUser
if "makemigrations" in sys.argv:
from django.utils.translation import gettext_noop as _
else:
from django.utils.translation import gettext_lazy as _
from items.models import ItemBase
class Company(ItemBase):
name = models.CharField(_("Name"), max_length=255)
slug = models.SlugField(_("Slug"), max_length=255)
description = models.TextField(_("Description"))
class Meta:
verbose_name = _("Company")
verbose_name_plural = _("Companies")
</code></pre></div>
<h2>Accounts</h2>
<p>Fourth, I'll have a more extensive example with the <code>accounts</code> app containing the <code>User</code> model. This model extends <code>AbstractUser</code> from <code>django.contrib.auth</code> as well as <code>ItemBase</code> for the one-to-one relation. The <code>item_type</code> set at the <code>Item</code> model will be <code>accounts.User</code>: </p>
<div><pre><code class="language-python"># accounts/models.py
import sys
from django.db import models
from django.contrib.auth.base_user import BaseUserManager
from django.contrib.auth.models import AbstractUser
if "makemigrations" in sys.argv:
from django.utils.translation import gettext_noop as _
else:
from django.utils.translation import gettext_lazy as _
from items.models import ItemBase
class UserManager(BaseUserManager):
def create_user(self, username="", email="", password="", **extra_fields):
if not email:
raise ValueError("Enter an email address")
email = self.normalize_email(email)
user = self.model(username=username, email=email, **extra_fields)
user.set_password(password)
user.save(using=self._db)
return user
def create_superuser(self, username="", email="", password=""):
user = self.create_user(email=email, password=password, username=username)
user.is_superuser = True
user.is_staff = True
user.save(using=self._db)
return user
class User(AbstractUser, ItemBase):
# change username to non-editable non-required field
username = models.CharField(
_("Username"), max_length=150, editable=False, blank=True
)
# change email to unique and required field
email = models.EmailField(_("Email address"), unique=True)
bio = models.TextField(_("Bio"))
USERNAME_FIELD = "email"
REQUIRED_FIELDS = []
objects = UserManager()
</code></pre></div>
<h2>Creating new items</h2>
<p>I will use the Django shell to create several autonomous model instances and the related Items too:</p>
<div><pre><code class="language-python">>>> from content.models import Post
>>> from companies.models import Company
>>> from accounts.models import User
>>> from items.models import Item
>>> post = Post.objects.create(
... title="Hello, World!",
... slug="hello-world",
... content="Lorem ipsum…",
... )
>>> company = Company.objects.create(
... name="Aidas & Co",
... slug="aidas-co",
... description="Lorem ipsum…",
... )
>>> user = User.objects.create_user(
... username="aidas",
... email="aidas@example.com",
... password="jdf234oha&6sfhasdfh",
... )
>>> Item.objects.count()
3</code></pre></div>
<h2>Aggregating content from all those relations</h2>
<p>Lastly, here is an example of having posts, companies, and users in a single view. For that, we will use the <code>Item</code> queryset with annotations:</p>
<div><pre><code class="language-python">from django import forms
from django.db import models
from django.shortcuts import render
from django.utils.translation import gettext, gettext_lazy as _
from .models import Item
class SearchForm(forms.Form):
q = forms.CharField(label=_("Search"), required=False)
def all_items(request):
qs = Item.objects.annotate(
title=models.Case(
models.When(
item_type="content.Post",
then="content_post__title",
),
models.When(
item_type="companies.Company",
then="companies_company__name",
),
models.When(
item_type="accounts.User",
then="accounts_user__email",
),
default=models.Value(gettext("<Untitled>")),
),
description=models.Case(
models.When(
item_type="content.Post",
then="content_post__content",
),
models.When(
item_type="companies.Company",
then="companies_company__description",
),
models.When(
item_type="accounts.User",
then="accounts_user__bio",
),
default=models.Value(""),
),
)
form = SearchForm(data=request.GET, prefix="search")
if form.is_valid():
query = form.cleaned_data["q"]
if query:
qs = qs.annotate(
search=SearchVector(
"title",
"description",
)
).filter(search=query)
context = {
"queryset": qs,
"search_form": form,
}
return render(request, "items/all_items.html", context)
</code></pre></div>
<h2>Final words</h2>
<p>You can have generic functionality and still avoid multiple hits to the database by using the <code>Item</code> one-to-one approach instead of generic relations.</p>
<p>The name of the <code>Item</code> model can be different, and you can even have multiple such models for various purposes, for example, <code>TaggedItem</code> for tags only.</p>
<p>Do you use anything similar in your projects?</p>
<p>Do you see how this approach could be improved?</p>
<p>Let me know in the comments!</p>
<hr>
<p>Cover picture by <a href="https://www.pexels.com/photo/assemble-challenge-combine-creativity-269399/">Pixabay</a></p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495424.209772763821142 -21.751296 80.830240436178826 48.561204000000004tag:blogger.com,1999:blog-9014457088394059918.post-59781057651946751562021-11-09T18:14:00.004+01:002021-11-10T03:21:00.520+01:0017 Django Project Ideas that can Make a Positive Impact around You<div class="separator" style="clear: both;"><img alt="17 Django Project Ideas that can Make a Positive Impact around You" border="0" data-original-height="512" data-original-width="1024" src="https://1.bp.blogspot.com/-xYD9njdx5PY/YYqrySvPxZI/AAAAAAAACLM/x6Yq1moMyuAiW32vls2ojUMn6JfnEU8qACLcBGAsYHQ/s0/17-django-project-ideas-that-can-make-a-positive-impact-around-you.png"/></div>
<p>For more than a decade, I was focused only on the technical part of website building with Django. In the process, I have built a bunch of interesting cultural websites. But I always felt that those sleepless nights were not worthy of the impact.</p>
<p>They say, "Don’t work hard, work smart!" I agree with that phrase, and for me it's not about working less hours. For me, it's working as much as necessary, but on things that matter most.</p>
<p>So after years of collecting facts about life, I connected the dots and came up with <a href="https://www.make-impact.org/">make-impact.org</a> – a social donation platform, which became one of the most important long-term projects. All my planning goes around this project.</p>
<p>And I believe I am not the only programmer who sometimes feels that they want to make a positive impact with their skills. So I brainstormed 17 Django project ideas. You can choose one and realize it as a hobby project, open-source platform, startup, or non-profit organization; alone, with a team of developers, or collaborating with some non-technical people.</p>
<h2>Idea #1: Low Qualification Job Search</h2>
<p>The job market is pretty competitive, and not all people can keep up with the train. You could build a job search website for jobs that don't require high education or lots of working experience. It could be helpful for people with language barriers, harsh living conditions, or those who are very young or very old. You could build it for your city, region, or country.</p>
<h2>Idea #2: Discounted Meals and Products</h2>
<p>Get inspired from <a href="https://toogoodtogo.org/">Too Good To Go</a> and build a progressive web app for your city about discounted restaurant meals and shop products whose expiration date is close to the end, but they are still good to eat.</p>
<h2>Idea #3: Personal Health Advisor and Tracker</h2>
<p>Build a website for setting your personal health improvement goals and tracking the progress. For example, maybe one wants to start eating more particular vegetables every week, jogging daily, lose or gain weight, or get rid of unhealthy addictions. Let people choose their health goals and check in with each progressive step. Allow using the website anonymously.</p>
<h2>Idea #4: Online Primary and Elementary School Materials</h2>
<p>Some people don't have access to schools in general or miss some classes because of illnesses. You could build a global and open wiki-based primary and elementary school education website for children and adults. It should be translatable and localizable. It would also be interesting to compare the same subject teachings in different countries side-by-side.</p>
<h2>Idea #5: Psychological Support for Women</h2>
<p>You could build a website with a video chat providing psychological support to discriminated or violently abused women. The help could be given by professionals or emphatic volunteers. The technical part can be implemented using django-channels, WebSockets, and WebRTC.</p>
<h2>Idea #6: Rain-harvesting Companies around the World</h2>
<p>Rain harvesting is one of the available ways to solve the problem of the lack of drinking water. There could be a platform comparing rain-harvesting companies all around the world. What are the installation prices? What are the countries they are working with? How many people have they saved? This website would allow people to find the most optimal company to build a rain harvesting system for them.</p>
<h2>Idea #7: Closest Electric Car Charging Stations</h2>
<p>Use the <a href="https://openchargemap.org/">Open Charge Map</a> API and create a progressive web app that shows the nearest electric car charging station and how to get there.</p>
<h2>Idea #8: Escrow-based Remote Job Search</h2>
<p>As remote jobs are getting more and more popular, there is still a matter of trust between the employees and employers. "Will the job taker complete their job in a good quality?" "Will the company pay the employee on time?" There are Escrow services to fix this issue. These are third parties that take and hold the money until the job is done. You could build a remote job search website promoting the usage of <a href="https://www.escrow.com/">Escrow.com</a> or another escrow service provider.</p>
<h2>Idea #9: Open Work Locations</h2>
<p>You could build a website listing coworking spaces and cafes with free wifi in your city. It should include the map, price ranges, details if registration is required, and other information necessary for remote workers.</p>
<h2>Idea #10: Most Admired Companies</h2>
<p>There could be a social website listing the most admired companies to work for in your country. Companies could be rated by working conditions, salary equality, growth opportunities, work relations, and other criteria. Anyone could suggest such a company, and they would be rated by their current and former employees anonymously.</p>
<h2>Idea #11: Tiny Houses</h2>
<p>The cost of accommodation is a critical problem in many locations of the world. You could develop a website that lists examples of tiny houses and their building schemas and instructions.</p>
<h2>Idea #12: Catalog of Recycled Products</h2>
<p>You could work on a product catalog with links to online shops, selling things produced from collected plastic. For example, <a href="https://products.theoceancleanup.com/">these sunglasses</a> are made of plastic collected from the ocean. Where available, you could use affiliate marketing links.</p>
<h2>Idea #13: Information for Climate-change Migrants</h2>
<p>You could work on a website for climate-change migrants with information about getting registered, housing, education, and jobs in a new city or country with better climate conditions.</p>
<h2>Idea #14: Fishes, Fishing, and Overfishing</h2>
<p>Scrape parts of <a href="https://www.fishbase.se/">FishBase</a> and create a website about fishes, fishing, and overfishing in your region or the world. Engage people about the marine world and inform them about the damage done by overfishing.</p>
<h2>Idea #15: Plant Trees</h2>
<p>Create an E-commerce shop or Software as a Service and integrate <a href="https://digitalhumani.com/">RaaS (Reforestation as a Service)</a>. Let a tree be planted for every sale. </p>
<h2>Idea #16: Positive Parenting</h2>
<p>Create a progressive web app about positive parenting. For inspiration and information check <a href="https://positivepsychology.com/positive-parenting-tips-skills-techniques/">this article</a>.</p>
<h2>Idea #17: Constructive Forum</h2>
<p>Create a forum with topic voting and automatic hate speech detection and flagging. For example, maybe you could use a combination of <a href="http://text-processing.com/docs/sentiment.html">Sentiment analysis from text-processing.com</a> and usage of profanity words to find negativity in forum posts. </p>
<h2>It's your turn</h2>
<p>I hope this post inspired you. If you decided to start a startup with one of those ideas, don't forget to do your research at first. What are the competitors in your area? What would be your unique selling point? Etc.</p>
<p>Also, it would be interesting to hear your thoughts. Which of the projects would seem to you the most crucial? Which of them would you like to work on?</p>
<hr>
<p>Cover photo by <a href="https://unsplash.com/photos/9QF90iLO0q0">Joshua Fuller</a></p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495424.209772763821142 -21.751296 80.830240436178826 48.561204000000004tag:blogger.com,1999:blog-9014457088394059918.post-57802813866380661962021-11-06T07:03:00.005+01:002021-11-06T14:30:02.678+01:00How to Use Semantic Versioning for Shared Django Apps<div class="separator" style="clear: both;"><a href="https://1.bp.blogspot.com/-0amTeE8ELbA/YYYZUZRQz3I/AAAAAAAACLE/vBrO2zOeEy8fYd7K9p7oPFPHoGErBmFRQCLcBGAsYHQ/s0/how-to-use-semantic-versioning-for-shared-django-apps.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="How to Use Semantic Versioning for Shared Django Apps" border="0" data-original-height="512" data-original-width="1024" src="https://1.bp.blogspot.com/-0amTeE8ELbA/YYYZUZRQz3I/AAAAAAAACLE/vBrO2zOeEy8fYd7K9p7oPFPHoGErBmFRQCLcBGAsYHQ/s0/how-to-use-semantic-versioning-for-shared-django-apps.png"/></a></div>
<p>When you are building websites with Django, there is no need to track their software versions. You can just have a stable branch in Git repository and update the production environment whenever it makes sense. However, when you create a Django app or package shared among various websites and maybe with multiple people, it is vital to keep track of the package versions.</p>
<h2>The Benefits</h2>
<p>Versioning allows you to identify a specific state of your package and has these benefits:</p>
<ul>
<li>Developers can be aware of which package version works with their websites together flawlessly.</li>
<li>You can track which versions had which bugs or certain features when communicating with open-source communities or technical support.</li>
<li>In the documentation, you can clearly see which version of the software it is referring to.</li>
<li>When fixing bugs of a particular version, developers of the versioned package have a narrower scope of code to check at version control commits.</li>
<li>Just from the version number, it's clear if the upgrade will only fix the bugs or if it requires more attention to make your software compatible.</li>
<li>When talking to other developers about a particular package, you can clearly state what you are talking about (yes, developers talk about software versions from time to time 😅).</li>
</ul>
<h2>Semantic Versioning</h2>
<p>The most popular versioning scheme is called <a href="https://semver.org/">semantic versioning</a>. The version consists of three numbers separated by dots. Django framework is using it too. For example, the latest stable version at the time of writing is <code>3.2.9</code>. In semantic versioning, the first number is the major version, the second is the minor version, and the third is a patch version: <code>MAJOR.MINOR.PATCH</code>. </p>
<ol>
<li><code>MAJOR</code> version introduces backward incompatible features.</li>
<li><code>MINOR</code> version adds functionality that is backward compatible.</li>
<li><code>PATCH</code> version is for backward-compatible bug fixes.</li>
</ol>
<p>There can also be some versioning outliers for <code>alpha</code>, <code>beta</code>, release candidates <code>rc1</code>, <code>rc2</code>, etc., but we will not cover them in this post.</p>
<p>By convention, the package's version is saved in its <code>myapp/__init__.py</code> file as <code>__version__</code> variable, for example:</p>
<div><pre><code class="language-python">__version__ = "0.2.4"</code></pre></div>
<p>In addition, it is saved in <code>setup.py</code>, <code>README.md</code>, <code>CHANGELOG.md</code>, and probably some more files.</p>
<h2>Changelog</h2>
<p>When you develop a Django app or another Python package that you will share with other developers, it is also recommended to have a <a href="https://keepachangelog.com/en/1.0.0/">changelog</a>. Usually, it's a markdown-based document with the list of features, bugs, and deprecations that were worked on between specific versions of your package.</p>
<p>A good changelog starts with the "Unreleased" section and is followed by sections for each version in reverse order. In each of those sections, there are lists of changes grouped into categories:</p>
<ul>
<li>Added</li>
<li>Changed</li>
<li>Deprecated</li>
<li>Removed</li>
<li>Fixed</li>
<li>Security</li>
</ul>
<p>This could be the starting template for <code>CHANGELOG.md</code>:</p>
<div><pre><code class="language-markdown">Changelog
=========
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
[Unreleased]
------------
<!--
### Added
### Changed
### Deprecated
### Removed
### Fixed
### Security
--></code></pre></div>
<p>As the new version is released, it will replace the <code>Unreleased</code> section while creating a new <code>Unreleased</code> section above it. For example, this:</p>
<div><pre><code class="language-markdown">[Unreleased]
------------
### Added
- Logging the cookie consent choices in the database because of the legal requirements.
### Fixed
- Disabling the buttons while saving the Cookie Choices so that they are not triggered more than once with slow Internet connections.
</code></pre></div>
<p>would become this:</p>
<div><pre><code class="language-markdown">[Unreleased]
------------
[v0.2.0] - 2021-10-27
------------------
### Added
- Logging the cookie consent choices in the database because of the legal requirements.
### Fixed
- Disabling the buttons while saving the Cookie Choices so that they are not triggered more than once with slow Internet connections.
</code></pre></div>
<p>To keep track of the versions manually would be pretty tedious work, with the likelihood of forgetting one or more files or mismatching the version numbers. Gladly, there is a utility tool to do that for you, which is called <code>bump2version</code>.</p>
<h2>Using bump2version</h2>
<h3>Installation</h3>
<p>Install the <strong>bump2version</strong> utility to your virtual environment the standard way with pip:</p>
<div><pre><code class="language-bash">(env)$ pip install bump2version</code></pre></div>
<h3>Preparation</h3>
<p>In your package's <code>__init__.py</code> file, set the version to <code>0.0.0</code>:</p>
<div><pre><code class="language-python">__version__ = "0.0.0"</code></pre></div>
<p>Set the version to <code>0.0.0</code> in all other files, where the version needs to be mentioned, for example, <code>README.md</code> and <code>setup.py</code>.</p>
<p>Then create <code>setup.cfg</code> with the following content:</p>
<div><pre><code class="language-none">[bumpversion]
current_version = 0.0.0
commit = True
tag = True
[bumpversion:file:setup.py]
search = version="{current_version}"
replace = version="{new_version}"
[bumpversion:file:myapp/__init__.py]
search = __version__ = "{current_version}"
replace = __version__ = "{new_version}"
[bumpversion:file:README.md]
search = django_myapp-{current_version}-py2.py3-none-any.whl
replace = django_myapp-{new_version}-py2.py3-none-any.whl
[bumpversion:file:CHANGELOG.md]
search =
[Unreleased]
------------
replace =
[Unreleased]
------------
[v{new_version}] - {utcnow:%%Y-%%m-%%d}
------------------
[bdist_wheel]
universal = 1</code></pre></div>
<p>Replace "django_myapp" and "myapp" with your Django app there. If you have more files, mentioning the package version, make sure to have analogous version replacements in the <code>setup.cfg</code>.</p>
<p>Commit and push your changes to the Git repository.</p>
<h3>Usage</h3>
<p>Every time you want to create a new version, type this in the shell:</p>
<div><pre><code class="language-bash">$ bump2version patch</code></pre></div>
<p>or </p>
<div><pre><code class="language-bash">$ bump2version minor</code></pre></div>
<p>or </p>
<div><pre><code class="language-bash">$ bump2version major</code></pre></div>
<p>followed by the command to build the package:</p>
<div><pre><code class="language-bash">$ python3 setup.py sdist bdist_wheel</code></pre></div>
<p>As mentioned before, <code>patch</code> is for the bug fixes, <code>minor</code> is for backward-compatible changes, and <code>major</code> is for backward-incompatible changes.</p>
<p>The <code>bump2version</code> command will use the configuration at <code>setup.cfg</code> and will do these things:</p>
<ul>
<li>It will increment the current version number depending on the parameter passed to it.</li>
<li>It will replace the old version with the new one in the <code>setup.py</code>, <code>myapp/__init__.py</code>, and <code>README.md</code>.</li>
<li>It will take care of correct versioning in the <code>CHANGELOG.md</code>.</li>
<li>Then, it will commit the changes and create a tag according to the pattern <code>vMAJOR.MINOR.PATCH</code> there.</li>
</ul>
<h3>Some further details</h3>
<p>There are two things to note there.</p>
<p>First, in Markdown, there are two ways of setting the headings:</p>
<div><pre><code class="language-markdown">Changelog
=========
[Unreleased]
------------</code></pre></div>
<p>is identical to</p>
<div><pre><code class="language-markdown"># Changelog
## [Unreleased]</code></pre></div>
<p>Usually, I would use the second format for Markdown, but the replacement function in <code>setup.cfg</code> treats lines with the <code>#</code> symbol as comments, and escaping doesn't work either.</p>
<p>For example, <strong>this wouldn't work</strong>:</p>
<div><pre><code class="language-none">[bumpversion:file:CHANGELOG.md]
search =
## [Unreleased]
replace =
## [Unreleased]
## [v{new_version}] - {utcnow:%%Y-%%m-%%d}</code></pre></div>
<p><em>If anybody knows or finds a simple solution with the shorter Markdown variation, let me know.</em></p>
<p>Second, instead of using <code>setup.cfg</code>, you can use <code>.bumpversion.cfg</code>, but that file is hidden, so I recommend sticking with <code>setup.cfg</code>.</p>
<h2>Final words</h2>
<p>You can see this type of semantic versioning configuration in the paid <a href="https://websightful.gumroad.com/l/django-gdpr-cookie-consent">Django GDPR Cookie Consent</a> app I recently published on Gumroad.</p>
<p>Happy programming!</p>
<hr>
<p>Cover photo by <a href="https://unsplash.com/photos/ScZSBiQRhMA">Jacob Campbell</a></p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495439.284674424154318 -4.173171 65.755338775845658 30.983079tag:blogger.com,1999:blog-9014457088394059918.post-18372677823796916722021-04-05T13:41:00.000+02:002022-10-21T23:51:27.780+02:00Guest Post: Django Crispy Forms Advanced Usage Example<div class="separator" style="clear: both;"><img alt="Guest Post: Django Crispy Forms Advanced Usage Example" border="0" data-original-height="512" data-original-width="1024" src="https://1.bp.blogspot.com/-WNxQP5J1hyA/YGr2KBs0SXI/AAAAAAAACHs/K_pxX0owjTok5uzQrcRJCzmInJM5ydXOwCLcBGAsYHQ/s0/guest-post-django-crispy-forms-advanced-usage-example.png"/></div>
<p><em>This is a guest post by <a href="https://www.linkedin.com/in/kushchenko/">Serhii Kushchenko</a>, Python/Django developer and data analyst. He is skillful in SQL, Python, Django, RESTful APIs, statistics, and machine learning.</em></p>
<p>This post aims to demonstrate the creation and processing of the large and complex form in Django using <code>django-crispy-forms</code>. The form contains several buttons that require executing different actions during the processing phase. Also, in the form, the number of rows and columns is not static but varies depending on some conditions. In our case, the set of rows and columns changes depending on the number of instances (columns) associated with the main object (data schema) in a many-to-one relationship.</p>
<p>The <code>django-crispy-forms</code> documentation includes a page <a href="https://django-crispy-forms.readthedocs.io/en/d-0/dynamic_layouts.html">Updating layouts on the go</a>. That page is quite helpful. However, it does not contain any detailed example of a working project. IMHO such an example is much needed. I hope my project serves as a helpful addition to the official Django Crispy Forms documentation. Feel free to copy-paste the pieces of code that you find applicable.</p>
<p>Please see the full codebase <a href="https://github.com/s-kust/django-advanced-forms">here</a>. It is ready for Heroku deployment.</p>
<h2>Task</h2>
<p>Suppose we have a database with information about people: name, surname, phone numbers, place of work, companies owned, etc. The task is not to process the data but to work with meta-information about that data. </p>
<p>Different users may need to extract varying information from the database. These users do not want to write and run SQL queries. They demand some simple and more visual solutions. The assignment is to make it possible for users to create the data schemas visually. In other words, to develop such a form and make it fully functioning.</p>
<div class="separator" style="clear: both;"><a href="https://1.bp.blogspot.com/-UWdE9IX0NQc/YGr20gL0fhI/AAAAAAAACH0/LJPSM0z5UEQxjfdt8FhKvtaX7LC_YFu9QCLcBGAsYHQ/s0/descr_1.PNG" style="display: block; padding: 1em 0; text-align: center; "><img alt="Schema editing form" border="0" data-original-height="816" data-original-width="783" src="https://1.bp.blogspot.com/-UWdE9IX0NQc/YGr20gL0fhI/AAAAAAAACH0/LJPSM0z5UEQxjfdt8FhKvtaX7LC_YFu9QCLcBGAsYHQ/s0/descr_1.PNG"/></a></div>
<p>Using their schemas, users will be able to CRUD data in the database. However, these operations are beyond the scope of the current project.</p>
<p>Different columns can have their specific parameters. For example, integer columns have lower and upper bounds. It is necessary to develop functionality for editing those parameters for all types of columns. For that editing, forms are used that arise after clicking the "Edit details" button on the main form.</p>
<p>Moreover, we have to develop a form "Create new schema" and a page with a list of all available schemas.</p>
<h2>Solution</h2>
<p>Described below:</p>
<ol>
<li>Data models used.</li>
<li>How the required forms are generated.</li>
<li>Recognizing and handling the user-initiated state changes.</li>
</ol>
<p>The task described above can be better solved using JavaScript together with Django forms. It would reduce the number of requests to the server and increase the speed of the application. So the user experience would improve. However, the project aimed to create an advanced example of working with Django Crispy Forms.</p>
<p>Here you can learn the following tricks:</p>
<ol>
<li>Compose a complex Django Crispy Form using the Layout, Row, Column, Fieldset, Field, and other available classes.</li>
<li>During form generation, iterate over the Foreign Key related objects and create a form row for each of them.</li>
<li>When it makes sense to use an abstract base class and when it doesn't.</li>
<li>How to encode the required action and other data in the button name during its creation.</li>
<li>Determine which button the user clicked and implement the analog of switch-case statement to perform the required action.</li>
<li>Automatically populate the newly generated form with the request.POST data if you want and if that data is available.</li>
<li>Validation of user-entered data (phone number) using a regular expression.</li>
<li>If you have many similar models, use metaprogramming to generate ModelForm classes for those models without violating the DRY principle.</li>
</ol>
<h3>Models</h3>
<p>According to the task, the number of columns in the schemas can be different. The users add new columns and delete existing columns. Also, they can change the type and order of columns. The columns and schemas have a many-to-one relationship that is described using the Foreign Key in Django models.</p>
<p>The picture shows that every schema has its name, 'Column separator' field, and 'String character' field. Also, it would be nice to save the date of the last schema modification. The following code from <code>schemas\models.py</code> file is pretty simple.</p>
<div><pre><code class="language-python">INTEGER_CH = "IntegerColumn"
FULLNAME_CH = "FullNameColumn"
JOB_CH = "JobColumn"
PHONE_CH = "PhoneColumn"
COMPANY_CH = "CompanyColumn"
COLUMN_TYPE_CHOICES = [
(INTEGER_CH, "Integer"),
(FULLNAME_CH, "Full Name"),
(JOB_CH, "Job"),
(PHONE_CH, "Phone"),
(COMPANY_CH, "Company"),
]
DOUBLE_QUOTE = '"'
SINGLE_QUOTE = "'"
STRING_CHARACTER_CHOICES = [
(DOUBLE_QUOTE, 'Double-quote(")'),
(SINGLE_QUOTE, "Single-quote(')"),
]
COMMA = ","
SEMICOLON = ";"
COLUMN_SEPARATOR_CHOICES = [(COMMA, "Comma(,)"), (SEMICOLON, "Semicolon(;)")]
class DataSchemas(models.Model):
name = models.CharField(max_length=100)
column_separator = models.CharField(
max_length=1,
choices=COLUMN_SEPARATOR_CHOICES,
default=COMMA,
)
string_character = models.CharField(
max_length=1,
choices=STRING_CHARACTER_CHOICES,
default=DOUBLE_QUOTE,
)
modif_date = models.DateField(auto_now=True)
def get_absolute_url(self):
return reverse("schema_add_update", args=[str(self.id)])</code></pre></div>
<p>Each column has a name, type, and order. All of these fields are in the base <code>SchemaColumn(models.Model)</code> class. This class cannot be abstract because in such a case, the code <code>schema.schemacolumn_set.all()</code> would not work.</p>
<p>Columns of type integer, first and last name, job, company, and phone number are implemented as classes derived from the base class SchemaColumn.</p>
<div><pre><code class="language-python">class SchemaColumn(models.Model):
name = models.CharField(max_length=100)
schema = models.ForeignKey(DataSchemas, on_delete=models.CASCADE)
order = models.PositiveIntegerField()
class Meta:
unique_together = [["schema", "name"], ["schema", "order"]]
def save(self, *args, **kwargs):
self.validate_unique()
super(SchemaColumn, self).save(*args, **kwargs)
class IntegerColumn(SchemaColumn):
range_low = models.IntegerField(blank=True, null=True, default=-20)
range_high = models.IntegerField(blank=True, null=True, default=40)
class FullNameColumn(SchemaColumn):
first_name = models.CharField(max_length=10, blank=True, null=True)
last_name = models.CharField(max_length=15, blank=True, null=True)
class JobColumn(SchemaColumn):
job_name = models.CharField(max_length=100, blank=True, null=True)
class CompanyColumn(SchemaColumn):
company_name = models.CharField(max_length=100, blank=True, null=True)
class PhoneColumn(SchemaColumn):
phone_regex = RegexValidator(
regex=r"^\+?1?\d{9,15}$",
message="Phone number must be entered in the format: '+999999999'. Up to 15 digits allowed.",
)
phone_number = models.CharField(
validators=[phone_regex], max_length=17, blank=True, null=True
) # validators should be a list</code></pre></div>
<h3>Forms</h3>
<p>The schema editing form is quite complex. We do not use the Django built-in <code>ModelForm</code> class here because it is not flexible enough. Our class DataSchemaForm is a derivative of the <code>forms.Form</code> class. Of course, <code>django-crispy-forms</code> was very helpful and even essential. </p>
<div><pre><code class="language-python">from crispy_forms.layout import (
Layout,
Submit,
Row,
Column,
Fieldset,
Field,
Hidden,
ButtonHolder,
HTML,
)</code></pre></div>
<p>The type of column in the form depends on the class of the column. How to determine that class? The problems can arise if we use the built-in <code>isinstance()</code> function for derived classes such as our various column types. The following code demonstrates how the subclass check was implemented in the <code>forms.py</code> file when generating the form.</p>
<div><pre><code class="language-python">INTEGER_CH = "IntegerColumn"
FULLNAME_CH = "FullNameColumn"
JOB_CH = "JobColumn"
PHONE_CH = "PhoneColumn"
COMPANY_CH = "CompanyColumn"
COLUMN_TYPE_CHOICES = [
(INTEGER_CH, "Integer"),
(FULLNAME_CH, "Full Name"),
(JOB_CH, "Job"),
(PHONE_CH, "Phone"),
(COMPANY_CH, "Company"),
]
subclasses = [
str(subclass).split(".")[-1][:-2].lower()
for subclass in SchemaColumn.__subclasses__()
]
# yes, somewhat redundant
column_type_switcher = {
"integercolumn": INTEGER_CH,
"fullnamecolumn": FULLNAME_CH,
"jobcolumn": JOB_CH,
"companycolumn": COMPANY_CH,
"phonecolumn": PHONE_CH,
}
column_type_field_name = "col_type_%s" % (column.pk,)
self.fields[column_type_field_name] = forms.ChoiceField(
label="Column type", choices=COLUMN_TYPE_CHOICES
)
for subclass in subclasses:
if hasattr(column, subclass):
self.fields[column_type_field_name].initial = [
column_type_switcher.get(subclass)
]
break
</code></pre></div>
<h4>How new schemas are created</h4>
<p>The function that generates the schema editing form may get the primary key of the existing schema. If such a key is not available, then the function creates a new schema and its first column. After that, the user can change the parameters of the schema, as well as add new columns.</p>
<div><pre><code class="language-python">if schema_pk:
schema = DataSchemas.objects.get(pk=schema_pk)
else:
# no existing schema primary key passed from the caller,
# so create new schema and its first column
# with default parameters
schema = DataSchemas.objects.create(name="New Schema")
int1 = IntegerColumn.objects.create(
name="First Column",
schema=schema,
order=1,
range_low=-20,
range_high=40,
)
self.fields["name"].initial = schema.name
self.fields["column_separator"].initial = schema.column_separator
self.fields["string_character"].initial = schema.string_character</code></pre></div>
<p>In addition to the schema editing form, the application also contains a list of all created schemas. </p>
<div class="separator" style="clear: both;"><a href="https://1.bp.blogspot.com/-49_ubj9FwuY/YGr3CcNgZ2I/AAAAAAAACH4/YSJfl7guxH8OOy_98f52JsDAhvEIPNBMwCLcBGAsYHQ/s0/descr_2.PNG" style="display: block; padding: 1em 0; text-align: center; "><img alt="All schemas list" border="0" data-original-height="389" data-original-width="576" src="https://1.bp.blogspot.com/-49_ubj9FwuY/YGr3CcNgZ2I/AAAAAAAACH4/YSJfl7guxH8OOy_98f52JsDAhvEIPNBMwCLcBGAsYHQ/s0/descr_2.PNG"/></a></div>
<p>There is nothing special about that page, so I will not describe it in detail here. Pleas see the full code and templates at <a href="https://github.com/s-kust/django-advanced-forms">https://github.com/s-kust/django-advanced-forms</a>. </p>
<h3>Requests processing</h3>
<p>The picture shows that the schema editing form contains several types of buttons:</p>
<ul>
<li>Submit form</li>
<li>Add new column</li>
<li>Delete column</li>
<li>Edit column details</li>
</ul>
<p>We need to determine which button the user pressed and perform the required action. </p>
<p>During the form creation, the required action is encoded in the names of all the buttons. Also, the column primary key or schema primary key is encoded there. For example, <code>delete_btn = 'delete_col_%s' % (column.pk,)</code> or <code>submit_form_btn = 'submit_form_%s' % (schema.pk,)</code>. </p>
<p>The name of the button can be found in the request.POST data only if the user has pressed that button. The following code from the views.py file searches for the button name in the request.POST data and calls the required function. A well-known method is used to implement the Python analog of <code>switch-case</code> statement.</p>
<div><pre><code class="language-python">btn_functions = {
"add_new_col": process_btn_add_column,
"delete_col": process_btn_delete_column,
"edit_col": process_btn_edit_column_details,
"submit_form": process_btn_submit_form,
"save_column_chng": process_btn_save_chng_column,
}
btn_pressed = None
# source of key.startswith idea - https://stackoverflow.com/questions/13101853/select-post-get-parameters-with-regular-expression
for key in request.POST:
if key.startswith("delete_col_"):
btn_pressed = "delete_col"
if key.startswith("edit_col_"):
btn_pressed = "edit_col"
if key.startswith("add_column_btn_"):
btn_pressed = "add_new_col"
if key.startswith("submit_form_"):
btn_pressed = "submit_form"
if key.startswith("save_column_chng_btn_"):
btn_pressed = "save_column_chng"
if btn_pressed is not None:
func_to_call = self.btn_functions.get(btn_pressed)
self.pk, form = func_to_call(self, key, form_data=request.POST)
break</code></pre></div>
<h4>Editing parameters of different types of columns</h4>
<p>Different types of columns have their specific parameters. For example, integer columns have lower and upper bounds. Phone columns have the phone number field that requires validation before saving. The number of different types of columns can increase over time. </p>
<p>How to handle the Edit Details button click? The straightforward solution is to make individual ModelForm classes for each type of column. However, it would violate the DRY principle. Perhaps, in this case, the use of metaprogramming is justified.</p>
<div><pre><code class="language-python">def get_general_column_form(self, model_class, column_pk):
class ColumnFormGeneral(ModelForm):
def __init__(self, *args, **kwargs):
super(ColumnFormGeneral, self).__init__(*args, **kwargs)
self.helper = FormHelper(self)
save_chng_btn = "save_column_chng_btn_%s" % (column_pk,)
self.helper.layout.append(Submit(save_chng_btn, "Save changes"))
class Meta:
model = model_class
exclude = ["schema", "order"]
return ColumnFormGeneral </code></pre></div>
<p>First, the type of column is determined using its primary key. After that, the <code>get_general_column_form</code> function is called. It returns the customized <code>ModelForm</code> class. Next, an instance of that class is created and used.</p>
<div><pre><code class="language-python">column = get_object_or_404(SchemaColumn, pk=column_pk)
for subclass in self.subclasses:
if hasattr(column, subclass):
column_model = apps.get_model("schemas", subclass)
column = get_object_or_404(column_model, pk=column_pk)
form_class = self.get_general_column_form(column_model, column_pk)
form = form_class(
initial=model_to_dict(
column, fields=[field.name for field in column._meta.fields]
)
)
break</code></pre></div>
<h4>Handling of the column type change</h4>
<p>The user may change the type of one or several columns. If it happens, it means that the class of these columns has changed. Here, it is not enough to change the value of some attribute of the object. We have to delete the old object and create a new object belonging to the new class instead. How do we handle it:</p>
<ol>
<li>First, a form is generated using the schema's primary key. </li>
<li>Then, in the newly created form, the data from the database is replaced with the request.POST data, if that data is available. <em>It happens automatically</em>.</li>
<li>In the next step, the form is validated. For that, we have to call the <code>form.is_valid()</code> method explicitly.</li>
<li>If the validation is successful, then we process every column of the schema. For each column, its type from the database is compared with its type from the form. It means that its database type is compared with its <code>request.POST</code> type. If these types differ, the old column is deleted, and a new one is created instead.</li>
</ol>
<div><pre><code class="language-python"># elem is a 'Submit Form' button that the user pressed.
# schema primary key is encoded in its name
# first, let's decode it
self.pk = [int(s) for s in elem.split("_") if s.isdigit()][0]
# form_data is request.POST
form = DataSchemaForm(form_data, schema_pk=self.pk)
if form.is_valid():
schema = get_object_or_404(DataSchemas, pk=self.pk)
schema.name = form.cleaned_data["name"]
schema.column_separator = form.cleaned_data["column_separator"]
schema.string_character = form.cleaned_data["string_character"]
schema.save()
# the following code is in the save_schema_columns(self, schema, form) function
schema_columns = schema.schemacolumn_set.all()
for column in schema_columns:
column_name_field_name = "col_name_%s" % (column.pk,)
column_order_field_name = "col_order_%s" % (column.pk,)
column_type_field_name = "col_type_%s" % (column.pk,)
type_form = form.cleaned_data[column_type_field_name]
type_changed = False
for subclass in self.subclasses:
if hasattr(column, subclass):
type_db = self.column_type_switcher.get(subclass)
if type_db != type_form:
new_class = globals()[type_form]
new_column = new_class()
new_column.name = form.cleaned_data[column_name_field_name]
new_column.order = form.cleaned_data[column_order_field_name]
new_column.schema = schema
column.delete()
new_column.save()
type_changed = True
break
if not type_changed:
column.name = form.cleaned_data[column_name_field_name]
column.order = form.cleaned_data[column_order_field_name]
column.save()</code></pre></div>
<h2>Final Words</h2>
<p>I hope this post serves as a helpful addition to the <a href="https://django-crispy-forms.readthedocs.io/en/latest/">official Django Crispy Forms documentation</a>. Feel free to copy-paste the pieces of code that you find applicable. Also, write comments and share your experience of using Django Crispy Forms.</p>
<hr>
<p>Cover photo by <a href="https://unsplash.com/photos/z9QkBBwzWTg">Juan Pablo Malo</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0tag:blogger.com,1999:blog-9014457088394059918.post-1758275458674515412020-04-10T20:32:00.001+02:002020-04-25T03:47:11.480+02:00How I Tested ReactJS-based Webapp with Selenium<p><img border="0" src="https://3.bp.blogspot.com/-k3gynx3y_ZM/XpC2-t1ek0I/AAAAAAAACCU/1tZuk-wFpV8NZEEL9GtR3Lp8FyAUluwkQCLcBGAsYHQ/s1600/how-i-tested-reactjs-based-webapp-with-selenium.png" data-original-width="1024" data-original-height="512" alt="How I Tested ReactJS-based Webapp with Selenium" /></p>
<p>For quite some time, I have been building a SaaS product - <a href="https://www.1st-things-1st.com">strategic prioritizer 1st things 1st</a>. It's using Django in the backend and ReactJS in the frontend and communicating between those ends by REST API. Every week I try to make progress with this project, be it a more prominent feature, some content changes, or small styling tweaks. In the past week, I implemented frontend testing with <a href="https://selenium-python.readthedocs.io/">Selenium</a>, and I want to share my journey with you.</p>
<h2>What can you do with 1st things 1st</h2>
<p>1st things 1st allows you to evaluate a list of items by multiple criteria and calculates priorities for you to follow and take action. The service has 4 main steps:</p>
<ol>
<li>Defining criteria.</li>
<li>Listing out things.</li>
<li>Evaluating things by each criterion.</li>
<li>Exploring the priorities.</li>
</ol>
<p>Selenium is a testing tool that mimics user interaction in the browser: you can fill in fields, trigger events, or read out information from the HTML tags. To test the frontend of 1st things 1st with Selenium, I had to</p>
<ol>
<li>enter the user credentials and login,</li>
<li>create a project from a blank project template,</li>
<li>add some criteria,</li>
<li>add some things to do,</li>
<li>evaluate each thing by each criterion, and</li>
<li>see if the generated list of priorities was correct.</li>
</ol>
<p>Let's see how I did it.</p>
<h2>Preparation</h2>
<p>In 2020, Chrome is the most popular browser, and it's my default browser, so I decided to develop tests using it.</p>
<p>I had to install Selenium with pip into my virtual environment:</p>
<div><pre><code class="language-none">(venv)$ pip install selenium</code></pre></div>
<p>Also, I needed a <a href="https://chromedriver.chromium.org/downloads">binary chromedriver</a>, which makes Selenium talk to your Chrome browser. I downloaded it and placed it under <code>myproject/drivers/chromedriver</code>.</p>
<p>In the Django project configuration, I needed a couple of settings. I usually have separate settings-file for each of the environments, such as:</p>
<ul>
<li><code>myproject.settings.local</code> for the local development,</li>
<li><code>myproject.settings.staging</code> for the staging server,</li>
<li><code>myproject.settings.test</code> for testing, and</li>
<li><code>myproject.settings.production</code> for production.</li>
</ul>
<p>All of them import defaults from a common base, and I have to set only the differences for each environment. </p>
<p>In the <code>myproject.settings.test</code> I added these settings:</p>
<div><pre><code class="language-python">WEBSITE_URL = 'http://my.1st-things-1st.127.0.0.1.xip.io:8080' # no trailing slash
TESTS_SHOW_BROWSER = True</code></pre></div>
<p>Here for the <code>WEBSITE_URL</code>, I was using the <a href="http://xip.io">xip.io</a> service. It allows you to create domains dynamically pointing to the localhost or any other IP. The Selenium tests will use this URL.</p>
<p>The <code>TEST_SHOW_BROWSER</code> was my custom setting, telling whether to show a browser while testing the frontend or just to run the tests in the background.</p>
<h2>The test case</h2>
<p>In one of my apps, <code>myproject.apps.evaluations</code>, I created a <code>tests</code> package, and there I placed a test case <code>test_evaluations_frontend.py</code> with the following content:</p>
<div><pre><code class="language-python">import os
from time import sleep
from datetime import timedelta
from django.conf import settings
from django.test import LiveServerTestCase
from django.test import override_settings
from django.contrib.auth import get_user_model
from django.utils import timezone
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
User = get_user_model()
SHOW_BROWSER = getattr(settings, "TESTS_SHOW_BROWSER", False)
@override_settings(DEBUG=True)
class EvaluationTest(LiveServerTestCase):
host = settings.WEBSITE_URL.rsplit(":", 1)[0].replace(
"http://", ""
) # domain before port
port = int(settings.WEBSITE_URL.rsplit(":", 1)[1]) # port
USER1_USERNAME = "user1"
USER1_FIRST_NAME = "user1"
USER1_LAST_NAME = "user1"
USER1_EMAIL = "user1@example.com"
USER1_PASSWORD = "change-me"
@classmethod
def setUpClass(cls):
# …
@classmethod
def tearDownClass(cls):
# …
def wait_until_element_found(self, xpath):
# …
def wait_a_little(self, seconds=2):
# …
def test_evaluations(self):
# …</code></pre></div>
<p>It's a live-server test case, which runs a Django development server under the specified IP and port and then runs the Chrome browser via Selenium and navigates through the DOM and fills in forms.</p>
<p>By default, the <code>LiveServerTestCase</code> runs in non-debug mode, but I want to have the debug mode on so that I could see any causes of server errors. With the <code>@override_settings</code> decorator, I could change the <code>DEBUG</code> setting to <code>True</code>.</p>
<p>The <code>host</code> and <code>port</code> attributes define on which host and port the test server will be running (instead of a 127.0.0.1 and a random port). I extracted those values from the <code>WEBSITE_URL</code> setting.</p>
<p>The test case also had some attributes for the user who will be navigating through the web app.</p>
<p>Let's dig deeper into the code for each method.</p>
<h2>Test-case setup and teardown</h2>
<p>Django test cases can have class-level setup and teardown, which run before and after all methods whose names start with <code>test_</code>:</p>
<div><pre><code class="language-python"> @classmethod
def setUpClass(cls):
super().setUpClass()
cls.user1 = User.objects.create_user(
cls.USER1_USERNAME, cls.USER1_EMAIL, cls.USER1_PASSWORD
)
# … add subscription for this new user …
driver_path = os.path.join(settings.BASE_DIR, "drivers", "chromedriver")
chrome_options = Options()
if not SHOW_BROWSER:
chrome_options.add_argument("--headless")
chrome_options.add_argument("--window-size=1200,800")
cls.browser = webdriver.Chrome(
executable_path=driver_path, options=chrome_options
)
cls.browser.delete_all_cookies()
@classmethod
def tearDownClass(cls):
super().tearDownClass()
cls.browser.quit()
# … delete subscription for the user …
cls.user1.delete()</code></pre></div>
<p>In the setup, I created a new user, added a subscription to them, and prepared the Chrome browser to use.</p>
<p>If the <code>TEST_SHOW_BROWSER</code> setting was <code>False</code>, Chrome was running headless, that is, in the background without displaying a browser window.</p>
<p>When the tests were over, the browser closed, and the subscription, as well as the user, were deleted. </p>
<h2>Utility methods</h2>
<p>I created two utility methods for my Selenium test: <code>wait_until_element_found()</code> and <code>wait_a_little()</code>:</p>
<div><pre><code class="language-python"> def wait_until_element_found(self, xpath):
WebDriverWait(self.browser, timeout=10).until(
lambda x: self.browser.find_element_by_xpath(xpath)
)
def wait_a_little(self, seconds=2):
if SHOW_BROWSER:
sleep(seconds)</code></pre></div>
<p>I used the <code>wait_until_element_found(xpath)</code> method to keep the test running while pages switched.</p>
<p>I used the <code>wait_a_little(seconds)</code> method to stop the execution for 2 or more seconds so that I could follow what's on the screen, make some screenshots, or even inspect the DOM in the Web Developer Inspector.</p>
<h2>XPath</h2>
<p>Selenium allows to select DOM elements <a href="https://selenium-python.readthedocs.io/locating-elements.html">by ID, name, CSS class, tag name, and other ways</a>, but the most flexible approach, in my opinion, is selecting elements by <a href="https://en.wikipedia.org/wiki/XPath">XPath (XML Path Language)</a>.</p>
<p>Contrary to jQuery, ReactJS doesn't use IDs or CSS classes in the markup to update the contents of specific widgets. So the straightforward Selenium's methods for finding elements by IDs or classes won't always work.</p>
<p>XPath is a very flexible and powerful tool. For example, you can:</p>
<ul>
<li>Select elements by ID: <code>"//input[@id='id_title']"</code></li>
<li>Select elements by any other attribute: <code>"//div[@aria-label='Blank']"</code></li>
<li>Select elements by innerText: <code>"//button[.='Save']"</code></li>
<li>Select elements by CSS class and innerText: <code>"//button[contains(@class,'btn-primary')][.='Save']"</code></li>
<li>Select the first element by innerText: <code>"(//button[.='yes'])[1]"</code></li>
</ul>
<p>You can try out XPath syntax and capabilities in Web Developer Console in Chrome and Firefox, using the <code>$x()</code> function, for example:</p>
<div><pre><code class="language-none">» $x("//h1[.='Projects']")
← Array [ h1.display-4.mb-4 ]</code></pre></div>
<h2>Login and adding a project</h2>
<p>I started with opening a login page, dismissing cookie consent notification, filling in user credentials into the login form, creating a new project from a blank template, setting title and description, etc.</p>
<div><pre><code class="language-python"> def test_evaluations(self):
self.browser.get(f"{self.live_server_url}/")
self.wait_until_element_found("//h1[.='Log in or Sign up']")
# Accept Cookie Consent
self.wait_until_element_found("//a[.='Got it!']")
self.browser.find_element_by_xpath("//a[.='Got it!']").click()
# Log in
self.browser.find_element_by_id("id_email").send_keys(self.USER1_EMAIL)
self.browser.find_element_by_id("id_password").send_keys(self.USER1_PASSWORD)
self.browser.find_element_by_xpath('//button[text()="Log in"]').send_keys(
"\n"
) # submit the form
self.wait_until_element_found("//h1[.='Projects']")
# Click on "Add new project"
self.wait_until_element_found("//a[.='Add new project']")
self.wait_a_little()
self.browser.find_element_by_xpath("//a[.='Add new project']").send_keys("\n")
self.wait_until_element_found("//div[@aria-label='Blank']")
# Create a project from the project template "Blank"
self.wait_a_little()
self.browser.find_element_by_xpath("//div[@aria-label='Blank']").send_keys("\n")
# Enter project title and description
self.wait_until_element_found("//input[@id='id_title']")
self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
"Urgent and Important Activities"
)
self.browser.find_element_by_xpath(
"//textarea[@id='id_description']"
).send_keys("I want to find which things to do and which to skip.")
self.browser.find_element_by_xpath("//button[.='Next']").send_keys("\n")
# Keep the default verbose names for the criteria and initiatives
self.wait_until_element_found("//input[@id='id_initiative_verbose_name_plural']")
self.wait_a_little()
self.browser.find_element_by_xpath("//button[.='Next']").send_keys("\n")</code></pre></div>
<p>If <code>TESTS_SHOW_BROWSER</code> was set to <code>True</code>, we would see all this workflow in an opened browser window.</p>
<p>I was creating the test by carefully inspecting the markup in Web Developer Inspector and creating appropriate DOM navigation with XPath. For most of the navigation, I was using <code>send_keys()</code> method, which triggers keyboard events. During the testing, I also noticed that my cookie consent only worked with a mouse click, and I couldn't approve it by the keyboard. That's some room for improving accessibility.</p>
<p>I ran the test with the following command each time I added some more lines:</p>
<div><pre><code class="language-none">(venv)$ python manage.py test myproject.apps.evaluations --settings=myproject.settings.test</code></pre></div>
<img border="0" src="https://4.bp.blogspot.com/-xb0q4UlsJCs/XpC24d9V_UI/AAAAAAAACCA/1nboSL0Szkkb3k9cQpDbp0g_Wd40_HsBwCLcBGAsYHQ/s1600/00-creating-a-project.png" data-original-width="1600" data-original-height="1113" />
<p>The test case failed if any command in the test failed. I didn't even need asserts.</p>
<h2>Adding criteria</h2>
<p>Now it was time to add some criteria:</p>
<div><pre><code class="language-python"> self.wait_until_element_found("//h2[.='Criteria']")
# Add new criterion "Urgent" with the evaluation type Yes/No/Maybe
self.wait_until_element_found("//a[.='Add new criterion']")
self.browser.find_element_by_xpath("//a[.='Add new criterion']").send_keys("\n")
self.wait_until_element_found("//input[@id='id_title']")
self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
"Urgent"
)
self.browser.find_element_by_xpath("//input[@id='widget_y']").send_keys(" ")
self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")
# Add new criterion "Important" with the evaluation type Yes/No/Maybe
self.wait_until_element_found("//a[.='Add new criterion']")
self.browser.find_element_by_xpath("//a[.='Add new criterion']").send_keys("\n")
self.wait_until_element_found("//input[@id='id_title']")
self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
"Important"
)
self.browser.find_element_by_xpath("//input[@id='widget_y']").send_keys(" ")
self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")
# Click on the button "Done"
self.wait_until_element_found("//a[.='Done']")
self.browser.find_element_by_xpath("//a[.='Done']").send_keys("\n")</code></pre></div>
<p>I added two criteria, "Urgent" and "Important", with evaluation type "Yes/No/Maybe".</p>
<p><img border="0" src="https://2.bp.blogspot.com/-WeWRz4PHeW0/XpC24807KTI/AAAAAAAACCE/IPY6f0BQGzE7vtb2wvf_ibwoYaEIXuuaACLcBGAsYHQ/s1600/01-define-criteria.png" data-original-width="1600" data-original-height="1113" alt="Defining criteria" /></p>
<h2>Adding things</h2>
<p>Then I created some activities to evaluate:</p>
<div><pre><code class="language-python"> self.wait_until_element_found("//h2[.='Things']")
# Add new thing "Write a blog post"
self.wait_until_element_found("//a[.='Add new thing']")
self.browser.find_element_by_xpath("//a[.='Add new thing']").send_keys("\n")
self.wait_until_element_found("//input[@id='id_title']")
self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
"Write a blog post"
)
self.browser.find_element_by_xpath("//textarea[@id='id_description']").send_keys(
"I have an idea of a blog post that I want to write."
)
self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")
# Add new thing "Fix a bug"
self.wait_until_element_found("//a[.='Add new thing']")
self.browser.find_element_by_xpath("//a[.='Add new thing']").send_keys("\n")
self.wait_until_element_found("//input[@id='id_title']")
self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
"Fix a bug"
)
self.browser.find_element_by_xpath("//textarea[@id='id_description']").send_keys(
"There is a critical bug that bothers our clients."
)
self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")
# Add new thing "Binge-watch a series"
self.wait_until_element_found("//a[.='Add new thing']")
self.browser.find_element_by_xpath("//a[.='Add new thing']").send_keys("\n")
self.wait_until_element_found("//input[@id='id_title']")
self.browser.find_element_by_xpath("//input[@id='id_title']").send_keys(
"Binge-watch a series"
)
self.browser.find_element_by_xpath("//textarea[@id='id_description']").send_keys(
"There is an exciting series that I would like to watch."
)
self.browser.find_element_by_xpath("//button[.='Save']").send_keys("\n")
# Click on the button "Done"
self.wait_until_element_found("//a[.='Done']")
self.browser.find_element_by_xpath("//a[.='Done']").send_keys("\n")
</code></pre></div>
<p>These were three activities: "Write a blog post", "Fix a bug", and "Binge-watch a series" with their descriptions:</p>
<p><img border="0" src="https://4.bp.blogspot.com/-fmGf7-rchBA/XpC245D_9bI/AAAAAAAACCI/NfQtrfmEbUMDrTVGTNJ0WG6XsXOgny9FQCLcBGAsYHQ/s1600/02-list-out-things.png" data-original-width="1600" data-original-height="1113" alt="Listing out things" /></p>
<h2>Evaluating things</h2>
<p>In this step, there was a list of widgets to evaluate each thing by each criterion with answers "No", "Maybe", or "Yes". The buttons for those answers had no specific id or CSS class, but I could target them by the text on the button using XPath like <code>"//button[.='maybe']"</code>:</p>
<div><pre><code class="language-python"> self.wait_until_element_found("//h2[.='Evaluations']")
self.wait_until_element_found("//button[.='maybe']")
# Evaluate all things by Urgency
self.browser.find_element_by_xpath("(//button[.='no'])[1]").send_keys("\n")
self.wait_until_element_found("//footer[.='Evaluation saved.']")
self.browser.find_element_by_xpath("(//button[.='yes'])[2]").send_keys("\n")
self.wait_until_element_found("//footer[.='Evaluation saved.']")
self.browser.find_element_by_xpath("(//button[.='no'])[3]").send_keys("\n")
self.wait_until_element_found("//footer[.='Evaluation saved.']")
# Evaluate all things by Importance
self.browser.find_element_by_xpath("(//button[.='yes'])[4]").send_keys("\n")
self.wait_until_element_found("//footer[.='Evaluation saved.']")
self.browser.find_element_by_xpath("(//button[.='yes'])[5]").send_keys("\n")
self.wait_until_element_found("//footer[.='Evaluation saved.']")
self.browser.find_element_by_xpath("(//button[.='maybe'])[6]").send_keys("\n")
self.wait_until_element_found("//footer[.='Evaluation saved.']")
# Click on the button "Done"
self.browser.find_element_by_xpath("//a[.='Done']").send_keys("\n")
</code></pre></div>
<p><img border="0" src="https://4.bp.blogspot.com/-xD-6Co_n_EQ/XpC28SOF8JI/AAAAAAAACCM/mU1aAAW16g4dobe4pCTHa_SIs5O1JbdZACLcBGAsYHQ/s1600/03-evaluate-things.png" data-original-width="1600" data-original-height="1113" alt="Evaluating things" /></p>
<p>These were my evaluations:</p>
<ul>
<li>"Write a blog post" was not urgent, but important.</li>
<li>"Fix a bug" was urgent and important.</li>
<li>"Binge-watch a series" was not urgent and maybe important (because one has to have rest and feed imagination too).</li>
</ul>
<h2>Checking priorities</h2>
<p>So in the last step, I got the calculated priorities:</p>
<div><pre><code class="language-python"> self.wait_until_element_found("//h2[.='Priorities']")
self.wait_until_element_found("//h5[.='1. Fix a bug (100%)']")
self.wait_until_element_found("//h5[.='2. Write a blog post (50%)']")
self.wait_until_element_found("//h5[.='3. Binge-watch a series (25%)']")
self.wait_a_little()</code></pre></div>
<p><img border="0" src="https://1.bp.blogspot.com/-lwrMVGlrX2M/XpC2-FFcOdI/AAAAAAAACCQ/kJ-K5M4FbnAnirAkn2uh5uCUzbVPtgkEgCLcBGAsYHQ/s1600/04-explore-priorities.png" data-original-width="1600" data-original-height="1113" alt="Exploring priorities" /></p>
<p>The results looked correct:</p>
<ul>
<li>"Fix a bug" was of the 100% priority.</li>
<li>"Write a blog post" was of the 50% priority.</li>
<li>"Binge-watch a series was of the 25% priority.</li>
</ul>
<h2>Final words</h2>
<ul>
<li>Selenium needs a binary browser driver that lets you manipulate DOM in the browser from Python. </li>
<li>You can set a specific host and port for a <code>LiveServerTestCase</code>.</li>
<li>The Chrome browser can be displayed or executed in the background, depending on your settings.</li>
<li>XPath is a flexible and powerful tool to address DOM elements by any attributes or even inner text.</li>
<li>Selenium can trigger keyboard or mouse events that are handled by JavaScript functions.</li>
</ul>
<p>I hope that my journey was useful to you too.</p>
<p>Happy coding!</p>
<hr>
<p>Thanks a lot to <a href="https://adamj.eu/">Adam Johnson</a> for the review.<br>
Cover photo by <a href="https://unsplash.com/photos/E1eCQdiO_E4">Science in HD</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495452.21073109999999 12.759507 52.829282099999986 14.050401tag:blogger.com,1999:blog-9014457088394059918.post-19470472039872191692020-03-22T08:58:00.000+01:002020-03-22T08:58:50.263+01:00How to Upload a File Using Django REST Framework<img border="0" src="https://1.bp.blogspot.com/-7vcHzmAKokk/XncZSdjACdI/AAAAAAAACBs/WIsr7P3U2bc3A6jf8lanZZvqf0pJNPQ9gCLcBGAsYHQ/s1600/how-to-upload-a-file-using-django-rest-framework.png" data-original-width="1024" data-original-height="512" alt="How to Upload a File Using Django REST Framework" />
<p>When you develop a web app or a mobile app with Django, it is common to use the Django REST Framework for communication with the server-side. The client-side makes GET, POST, PUT, and DELETE requests to the REST API to read, create, update, or delete data there. The communication by Ajax is pretty uncomplicated, but how would you upload an image or another file to the server? I will show you that in this article by creating user avatar upload via REST API. <a href="https://github.com/archatas/experiment-with-drf-and-image-uploads">Find the full code for this feature on Github</a>.</p>
<h2 id="toc_1">Extend Django User model</h2>
<p>We will start by installing Pillow for image handling to the virtual environment using the standard pip command:</p>
<div><pre><code class="language-bash">(venv)$ pip install Pillow</code></pre></div>
<p>Create <code>accounts</code> app with a custom <code>User</code> model:</p>
<div><pre><code class="language-python"># myproject/apps/accounts/models.py
import os
import sys
from django.db import models
from django.contrib.auth.models import AbstractUser
from django.utils import timezone
from django.utils.translation import gettext_lazy as _
def upload_to(instance, filename):
now = timezone.now()
base, extension = os.path.splitext(filename.lower())
milliseconds = now.microsecond // 1000
return f"users/{instance.pk}/{now:%Y%m%d%H%M%S}{milliseconds}{extension}"
class User(AbstractUser):
# …
avatar = models.ImageField(_("Avatar"), upload_to=upload_to, blank=True)</code></pre></div>
<p>You can add there as many fields as you need, but the noteworthy part there is the <code>avatar</code> field.</p>
<p>Update the settings and add the <code>accounts</code> app to <code>INSTALLED_APPS</code>, set the <code>AUTH_USER_MODEL</code>, and the configuration for the static and media directories:</p>
<div><pre><code class="language-python"># myproject/settings.py
INSTALLED_APPS = [
# …
"myproject.apps.accounts",
]
AUTH_USER_MODEL = "accounts.User"
STATICFILES_DIRS = [os.path.join(BASE_DIR, "myproject", "site_static")]
STATIC_ROOT = os.path.join(BASE_DIR, "myproject", "static")
STATIC_URL = "/static/"
MEDIA_ROOT = os.path.join(BASE_DIR, "myproject", "media")
MEDIA_URL = "/media/"</code></pre></div>
<p>Next small steps:</p>
<ul>
<li>Create and run migrations with the <code>makemigrations</code> and <code>migrate</code> management commands.</li>
<li>Set up the custom model administration for the new User model.</li>
<li>Create the superuser with the <code>createsuperuser</code> management command.</li>
</ul>
<h2 id="toc_2">Install and configure Django REST Framework</h2>
<p>Install Django REST Framework for the REST APIs to your virtual environment, as always, using pip:</p>
<div><pre><code class="language-bash">(venv)$ pip install djangorestframework</code></pre></div>
<p>We'll be using authentication by tokens in this example. So add Django REST Framework to <code>INSTALLED_APPS</code> in the settings and set <code>TokenAuthentication</code> as the default authentication in the <code>REST_FRAMEWORK</code> configuration:</p>
<div><pre><code class="language-python"># myproject/settings.py
INSTALLED_APPS = [
# …
"rest_framework",
"rest_framework.authtoken",
# …
]
REST_FRAMEWORK = {
'DEFAULT_AUTHENTICATION_CLASSES': [
'rest_framework.authentication.TokenAuthentication',
]
}</code></pre></div>
<h2 id="toc_3">Prepare the serializer and the view</h2>
<p>In Django REST Framework, serializers are used for data validation, rendering, and saving. They are similar to Django forms. Prepare <code>UserAvatarSerializer</code> for avatar uploads:</p>
<div><pre><code class="language-python"># myproject/apps/accounts/serializers.py
from django.contrib.auth import get_user_model
from rest_framework.serializers import ModelSerializer
User = get_user_model()
class UserAvatarSerializer(ModelSerializer):
class Meta:
model = User
fields = ["avatar"]
def save(self, *args, **kwargs):
if self.instance.avatar:
self.instance.avatar.delete()
return super().save(*args, **kwargs)</code></pre></div>
<p>Now create an API view <code>UserAvatarUpload</code> for avatar uploads. </p>
<div><pre><code class="language-python"># myproject/apps/accounts/views.py
from rest_framework import status
from rest_framework.parsers import MultiPartParser, FormParser
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
from rest_framework.views import APIView
from .serializers import UserAvatarSerializer
class UserAvatarUpload(APIView):
parser_classes = [MultiPartParser, FormParser]
permission_classes = [IsAuthenticated]
def post(self, request, format=None):
serializer = UserAvatarSerializer(data=request.data, instance=request.user)
if serializer.is_valid():
serializer.save()
return Response(serializer.data, status=status.HTTP_200_OK)
else:
return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)</code></pre></div>
<p>Make sure that the view uses <code>MultiPartParser</code> as one of the parser classes. That's necessary for the file transfers.</p>
<h2 id="toc_4">Prepare the URL configuration</h2>
<p>In the URL configuration, we will need those URL rules:</p>
<ul>
<li>The path for the index page. Let's make it a direct <code>TemplateView</code>.</li>
<li>The path for logging in by user credentials and obtaining the authentication token.</li>
<li>The path for user avatar upload.</li>
<li>The path for model administration.</li>
<li>The path for static URLs.</li>
<li>And finally, the path for media URLs.</li>
</ul>
<div><pre><code class="language-python"># myroject/urls.py
from django.conf.urls.static import static
from django.contrib import admin
from django.urls import path
from django.views.generic import TemplateView
from django.conf import settings
from myproject.accounts.views import UserAvatarUpload
from rest_framework.authtoken.views import obtain_auth_token
urlpatterns = [
path("", TemplateView.as_view(template_name="index.html")),
path("api/auth-token/", obtain_auth_token, name="rest_auth_token"),
path("api/user-avatar/", UserAvatarUpload.as_view(), name="rest_user_avatar_upload"),
path("admin/", admin.site.urls),
]
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)</code></pre></div>
<h2 id="toc_5">Prepare the frontend HTML and JavaScript</h2>
<p>I will illustrate the frontend using Bootstrap HTML and Vanilla JavaScript. Of course, you can implement the same using ReactJS, Vue, Angular, or other JavaScript framework and any other CSS framework.</p>
<p>The template for the index page has one login form with username and password or email and password fields (depending on your implementation), and one avatar upload form with a file selection field. Also, it includes a JavaScript file <code>avatar.js</code> for Ajax communication.</p>
<div><pre><code class="language-django">{# myproject/templates/index.html #}
<!doctype html>
{% load static %}
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css"
integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous">
<title>Hello, World!</title>
</head>
<body>
<div class="container">
<div class="row">
<div class="col-md-8">
<p class="text-muted my-3"><small>Open Developer Console for information about responses.</small></p>
<h1 class="my-3">1. Log in</h1>
<form id="login_form">
<div class="form-group">
<label for="id_email">Email address</label>
<input type="email" class="form-control" id="id_email" aria-describedby="emailHelp"
placeholder="Enter email"/>
</div>
<div class="form-group">
<label for="id_password">Password</label>
<input type="password" class="form-control" id="id_password" placeholder="Password"/>
</div>
<button type="submit" class="btn btn-primary">Log in</button>
</form>
<h1 class="my-3">2. Upload an avatar</h1>
<form id="avatar_form">
<div class="form-group">
<label for="id_avatar">Choose an image for your avatar</label>
<input type="file" class="form-control-file" id="id_avatar"/>
</div>
<button type="submit" class="btn btn-primary">Upload</button>
</form>
</div>
</div>
</div>
<script src="{% static 'site/js/avatar.js' %}"></script>
</body>
</html></code></pre></div>
<p>Last but not least, create the JavaScript file <code>avatar.js</code>. It contains these things:</p>
<ul>
<li>a global variable to store the user token. In the real-world application, you would probably save the token in a cookie or local storage.</li>
<li>a login-form submit handler which posts user credentials to the server and retrieves the authentication token.</li>
<li>an avatar-form submit handler which posts the selected file and the token to the server and retrieves the path of the saved file on the server.</li>
</ul>
<div><pre><code class="language-javascript">// myproject/site_static/site/js/avatar.js
let userToken;
document.getElementById('login_form').addEventListener('submit', function(event) {
event.preventDefault();
let email = document.getElementById('id_email').value;
let password = document.getElementById('id_password').value;
fetch('http://127.0.0.1:8000/api/auth-token/', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
"username": email,
"password": password,
})
}).then( response => {
return response.json();
}).then(data => {
console.log(data);
userToken = data.token;
console.log('Logged in. Got the token.');
}).catch((error) => {
console.error('Error:', error);
});
});
document.getElementById('avatar_form').addEventListener('submit', function(event) {
event.preventDefault();
let input = document.getElementById('id_avatar');
let data = new FormData();
data.append('avatar', input.files[0]);
fetch('http://127.0.0.1:8000/api/user-avatar/', {
method: 'POST',
headers: {
'Authorization': `Token ${userToken}`
},
body: data
}).then(response => {
return response.json();
}).then(data => {
console.log(data);
}).catch((error) => {
console.error('Error:', error);
});
});</code></pre></div>
<p>In the JavaScript file, we are using <a href="https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch">fetch API</a> for the REST API requests. The noteworthy part there is the <code>FormData</code> class that we use to send the file to the server.</p>
<p>Now run the local development server and go to the <code>http://127.0.0.1:8000</code>. There you will have something like this:</p>
<p><img border="0" src="https://2.bp.blogspot.com/-TrgQBwOxL9A/XncZidON4RI/AAAAAAAACBw/vP_AU8v104Mjscr2MWVpt8dZP0Xkp-fSACLcBGAsYHQ/s1600/how-to-upload-a-file-using-django-rest-framework-frontend.png" data-original-width="1600" data-original-height="1217" alt="The frontend of the experiment" /></p>
<h2 id="toc_6">Final Thoughts</h2>
<p>As <a href="https://www.broadbandsearch.net/blog/mobile-desktop-internet-usage-statistics">more than a half Internet usage happens on mobile devices</a>, there is a demand to switch from usual HTML websites and platforms to mobile apps. Whether you create a native mobile app, a hybrid app, or Progressive Web App, you will likely have to communicate with the server via REST API or GraphQL. It is pretty clear how to transfer textual data from and to a remote server. But after this exercise, we can also transfer binary files like images, PDF or Word documents, music, and videos.</p>
<p>Happy coding!</p>
<hr>
<p>Cover Photo by <a href="https://unsplash.com/photos/PxX9ssopXtI">Dan Silva</a></p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0tag:blogger.com,1999:blog-9014457088394059918.post-47700064621297346742020-01-24T19:45:00.001+01:002020-01-24T19:45:57.549+01:00Guest Post: Sending Emails with Django<img border="0" src="https://2.bp.blogspot.com/-fRtWx7zu4VU/XisxLKIxrxI/AAAAAAAACBA/MyuwZLBzXQgCngYOS4uCtVKoJLbNTyUpACLcBGAsYHQ/s1600/guest-post-sending-emails-with-django.png" data-original-width="1024" data-original-height="512" />
<p><em>This is a guest post by Mailtrap.io team. The original post on <a href="https://blog.mailtrap.io/django-send-email/">Sending emails with Django</a> was published at Mailtrap's blog.</em></p>
<p>Some time ago, we discovered <a href="https://blog.mailtrap.io/sending-emails-in-python-tutorial-with-code-examples/">how to send an email with Python</a> using smtplib, a built-in email module. Back then, the focus was made on the delivery of different types of messages via SMTP server. Today, we prepared a similar tutorial but for Django. This popular Python web framework allows you to accelerate email delivery and make it much easier. And these code samples of sending emails with Django are going to prove that.</p>
<h2>A simple code example of how to send an email</h2>
<p>Let's start our tutorial with a few lines of code that show you how simple it is to send an email in Django. </p>
<p>Import <code>send_mail</code> at the beginning of the file</p>
<div><pre><code class="language-python">from django.core.mail import send_mail</code></pre></div>
<p>And call the code below in the necessary place.</p>
<div><pre><code class="language-python">send_mail(
"That's your subject",
"That's your message body",
"from@yourdjangoapp.com",
["to@yourbestuser.com"],
fail_silently=False,
)</code></pre></div>
<p>These lines are enclosed in the <code>django.core.mail</code> module that is based on <code>smtplib</code>. The message delivery is carried out via SMTP host, and all the settings are set by default:</p>
<ul>
<li><code>EMAIL_HOST</code>: "localhost"</li>
<li><code>EMAIL_PORT</code>: 25</li>
<li><code>EMAIL_HOST_USER</code>: (Empty string)</li>
<li><code>EMAIL_HOST_PASSWORD</code>: (Empty string)</li>
<li><code>EMAIL_USE_TLS</code>: False</li>
<li><code>EMAIL_USE_SSL</code>: False</li>
</ul>
<p>You can learn other default values <a href="https://docs.djangoproject.com/en/dev/ref/settings/">here</a>. Most likely you will need to adjust them. Therefore, let's tweak the <code>settings.py</code> file.</p>
<h3>Setting up</h3>
<p>Before actually sending your email, you need to set up for it. So, let's add some lines to the <code>settings.py</code> file of your Django app.</p>
<div><pre><code class="language-python">EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
EMAIL_HOST = "smtp.yourserver.com"
EMAIL_PORT = "<your-server-port>"
EMAIL_HOST_USER = "your@djangoapp.com"
EMAIL_HOST_PASSWORD = "your-email account-password"
EMAIL_USE_TLS = True
EMAIL_USE_SSL = False</code></pre></div>
<p><code>EMAIL_HOST</code> is different for each email provider you use. For example, if you use Gmail SMTP server, you'll have <code>EMAIL_HOST = "smtp.gmail.com"</code>. Also, validate other values that are relevant to your email server. Eventually, you need to choose the way to encrypt the mail and protect your user account by setting the variable <code>EMAIL_USE_TLS</code> or <code>EMAIL_USE_SSL</code>. If you have an email provider that explicitly tells you which option to use, then it is clear. Otherwise, you may try different combinations using True and False operators. Mind that only one of these options can be set to True.</p>
<p><code>EMAIL_BACKEND</code> tells Django which custom or predefined email backend will work with <code>EMAIL_HOST</code>. You can set up this parameter as well. </p>
<h3>SMTP email backend </h3>
<p>In the example above, <code>EMAIL_BACKEND</code> is specified as <code>django.core.mail.backends.smtp.EmailBackend</code>. It is the default configuration that uses SMTP server for email delivery. Defined email settings will be passed as matching arguments to EmailBackend. </p>
<ul>
<li><code>host</code>: <a href="https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-EMAIL_HOST"><code>EMAIL_HOST</code></a></li>
<li><code>port</code>: <a href="https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-EMAIL_PORT"><code>EMAIL_PORT</code></a></li>
<li><code>username</code>: <a href="https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-EMAIL_HOST_USER"><code>EMAIL_HOST_USER</code></a></li>
<li><code>password</code>: <a href="https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-EMAIL_HOST_PASSWORD"><code>EMAIL_HOST_PASSWORD</code></a></li>
<li><code>use_tls</code>: <a href="https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-EMAIL_USE_TLS"><code>EMAIL_USE_TLS</code></a></li>
<li><code>use_ssl</code>: <a href="https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-EMAIL_USE_SSL"><code>EMAIL_USE_SSL</code></a></li>
</ul>
<p>Unspecified arguments default to <code>None</code>. </p>
<p>Besides <code>django.core.mail.backends.smtp.EmailBackend</code>, you can use:</p>
<ul>
<li><code>django.core.mail.backends.console.EmailBackend</code> - the console backend that composes the emails that will be sent to the standard output. Not intended for production use.</li>
<li><code>django.core.mail.backends.filebased.EmailBackend</code> - the file backend that creates emails in the form of a new file per each new session opened on the backend. Not intended for production use.</li>
<li><code>django.core.mail.backends.locmem.EmailBackend</code> - the in-memory backend that stores messages in the local memory cache of django.core.mail.outbox. Not intended for production use.</li>
<li><code>django.core.mail.backends.dummy.EmailBackend</code> - the dummy cache backend that implements the cache interface and does nothing with your emails. Not intended for production use.</li>
<li>Any out-of-the-box backend for Amazon SES, Mailgun, SendGrid, and other services. </li>
</ul>
<h2>How to send emails via SMTP </h2>
<p>Once you have that configured, all you need to do to send an email is to import the <code>send_mail</code> or <code>send_mass_mail</code> function from django.core.mail. These functions differ in the connection they use for messages. The <code>send_mail</code> uses a separate connection for each message. The <code>send_mass_mail</code> opens a single connection to the mail server and is mostly intended to handle mass emailing. </p>
<h3>Sending email with send_mail</h3>
<p>This is the most basic function for email delivery in Django. It comprises four obligatory parameters to be specified: <code>subject</code>, <code>message</code>, <code>from_email</code>, and <code>recipient_list</code>. </p>
<p>In addition to them, you can adjust the following:</p>
<ul>
<li> <code>auth_user</code>: If <code>EMAIL_HOST_USER</code> has not been specified, or you want to override it, this username will be used to authenticate to the SMTP server. </li>
<li> <code>auth_password</code>: If <code>EMAIL_HOST_PASSWORD</code> has not been specified, this password will be used to authenticate to the SMTP server.</li>
<li> <code>connection</code>: The optional email backend you can use without tweaking <code>EMAIL_BACKEND</code>.</li>
<li> <code>html_message</code>: Lets you send multipart emails.</li>
<li> <code>fail_silently</code>: A boolean that controls how the backend should handle errors. If <code>True</code> - exceptions will be silently ignored. If <code>False</code> - <code>smtplib.SMTPException</code> will be raised. </li>
</ul>
<p>For example, it may look like this:</p>
<div><pre><code class="language-python">from django.core.mail import send_mail
send_mail(
subject="That's your subject",
message="That's your message body",
from_email="from@yourdjangoapp.com",
recipient_list=["to@yourbestuser.com"],
auth_user="Login",
auth_password="Password",
fail_silently=False,
)</code></pre></div>
<p>Other functions for email delivery include mail<em>admins and mail</em>managers. Both are shortcuts to send emails to the recipients predefined in <code>ADMINS</code> and <code>MANAGERS</code> settings, respectively. For them, you can specify such arguments as <code>subject</code>, <code>message</code>, <code>fail_silently</code>, <code>connection</code>, and <code>html_message</code>. The <code>from_email</code> argument is defined by the <code>SERVER_EMAIL</code> setting.</p>
<h3>What is EmailMessage for? </h3>
<p>If the email backend handles the email sending, the <code>EmailMessage</code> class answers for the message creation. You'll need it when some advanced features like BCC or an attachment are desirable. That's how an initialized <code>EmailMessage</code> may look:</p>
<div><pre><code class="language-python">from django.core.mail import EmailMessage
email = EmailMessage(
subject="That's your subject",
body="That's your message body",
from_email="from@yourdjangoapp.com",
to=["to@yourbestuser.com"],
bcc=["bcc@anotherbestuser.com"],
reply_to=["whoever@itmaybe.com"],
)</code></pre></div>
<p>In addition to the <code>EmailMessage</code> objects you can see in the example, there are also other optional parameters:</p>
<ul>
<li><code>connection</code>: defines an email backend instance for multiple messages. </li>
<li><code>attachments</code>: specifies the attachment for the message.</li>
<li><code>headers</code>: specifies extra headers like <code>Message-ID</code> or <code>CC</code> for the message. </li>
<li><code>cc</code>: specifies email addresses used in the "CC" header.</li>
</ul>
<p>The methods you can use with the <code>EmailMessage</code> class are the following:</p>
<ul>
<li><code>send</code>: get the message sent.</li>
<li><code>message</code>: composes a MIME object (<code>django.core.mail.SafeMIMEText</code> or <code>django.core.mail.SafeMIMEMultipart</code>).</li>
<li><code>recipients</code>: returns a list of the recipients specified in all the attributes including <code>to</code>, <code>cc</code>, and <code>bcc</code>.</li>
<li><code>attach</code>: creates and adds a file attachment. It can be called with a <a href="https://docs.python.org/3/library/email.mime.html#email.mime.base.MIMEBase"><code>MIMEBase</code></a> instance or a triple of arguments consisting of filename, content, and mime type.</li>
<li><code>attach_file</code>: creates an attachment using a file from a filesystem. We'll talk about adding attachments a bit later.</li>
</ul>
<h3>How to send multiple emails</h3>
<p>To deliver a message via SMTP, you need to open a connection and close it afterward. This approach is quite awkward when you need to send multiple transactional emails. Instead, it is better to create one connection and reuse it for all messages. This can be done with the send_messages method. Check out the following example:</p>
<div><pre><code class="language-python">from django.core import mail
connection = mail.get_connection()
connection.open()
email1 = mail.EmailMessage(
"That's your subject",
"That's your message body",
"from@yourdjangoapp.com",
["to@yourbestuser1.com"],
connection=connection,
)
email1.send()
email2 = mail.EmailMessage(
"That's your subject #2",
"That's your message body #2",
"from@yourdjangoapp.com",
["to@yourbestuser2.com"],
)
email3 = mail.EmailMessage(
"That's your subject #3",
"That's your message body #3",
"from@yourdjangoapp.com",
["to@yourbestuser3.com"],
)
connection.send_messages([email2, email3])
connection.close()</code></pre></div>
<p>What you can see here is that the connection was opened for <code>email1</code>, and <code>send_messages</code> uses it to send emails #2 and #3. After that, you close the connection manually.</p>
<h3>How to send multiple emails with send<em>mass</em>mail</h3>
<p><code>send_mass_mail</code> is another option to use only one connection for sending different messages.</p>
<div><pre><code class="language-python">message1 = (
"That's your subject #1",
"That's your message body #1",
"from@yourdjangoapp.com",
["to@yourbestuser1.com", "to@yourbestuser2.com"]
)
message2 = (
"That's your subject #2",
"That's your message body #2",
"from@yourdjangoapp.com",
["to@yourbestuser2.com"],
)
message3 = (
"That's your subject #3",
"That's your message body #3",
"from@yourdjangoapp.com",
["to@yourbestuser3.com"],
)
send_mass_mail((message1, message2, message3), fail_silently=False)</code></pre></div>
<p>Each email message contains a datatuple made of <code>subject</code>, <code>message</code>, <code>from_email</code>, and <code>recipient_list</code>. Optionally, you can add other arguments that are the same as for <code>send_mail</code>.</p>
<h3>How to send an HTML email</h3>
<p>When the article was published, the latest Django official version was 2.2.4. All versions starting from 1.7 let you send an email with HTML content using <code>send_mail</code> like this:</p>
<div><pre><code class="language-python">from django.core.mail import send_mail
subject = "That's your subject"
html_message = render_to_string("mail_template.html", {"context": "values"})
plain_message = strip_tags(html_message)
from_email = "from@yourdjangoapp.com>"
to = "to@yourbestuser.com"
mail.send_mail(subject, plain_message, from_email, [to], html_message=html_message)</code></pre></div>
<p>Older versions users will have to mess about with <code>EmailMessage</code> and its subclass <code>EmailMultiAlternatives</code>. It lets you include different versions of the message body using the <code>attach_alternative</code> method. For example:</p>
<div><pre><code class="language-python">from django.core.mail import EmailMultiAlternatives
subject = "That's your subject"
from_email = "from@yourdjangoapp.com>"
to = "to@yourbestuser.com"
text_content = "That's your plain text."
html_content = """<p>That's <strong>the HTML part</strong></p>"""
message = EmailMultiAlternatives(subject, text_content, from_email, [to])
message.attach_alternative(html_content, "text/html")
message.send()</code></pre></div>
<h3>How to send an email with attachments </h3>
<p>In the <code>EmailMessage</code> section, we've already mentioned sending emails with attachments. This can be implemented using attach or <code>attach_file</code> methods. The first one creates and adds a file attachment through a triple of arguments - filename, content, and mime type. The second method uses a file from a filesystem as an attachment. That's how each method would look like in practice:</p>
<div><pre><code class="language-python">message.attach("Attachment.pdf", file_to_be_sent, "file/pdf")</code></pre></div>
<p>or</p>
<div><pre><code class="language-python">message.attach_file("/documents/Attachment.pdf")</code></pre></div>
<h2>Custom email backend</h2>
<p>Obviously, you're not limited to the abovementioned email backend options and are able to tailor your own. For this, you can use standard backends as a reference. Let's say you need to create a custom email backend with the <code>SMTP_SSL</code> connection support required to interact with Amazon SES. The default SMTP backend will be the reference. First, add a new email option to <code>settings.py</code>.</p>
<div><pre><code class="language-python">AWS_ACCESS_KEY_ID = "your-aws-access-key-id"
AWS_SECRET_ACCESS_KEY = "your-aws-secret-access-key"
AWS_REGION = "your-aws-region"
EMAIL_BACKEND = "your_project_name.email_backend.SesEmailBackend"</code></pre></div>
<p>Make sure that you are allowed to send emails with Amazon SES using these <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code> (or an error message will tell you about it :D)</p>
<p>Then create a file <code>your_project_name/email_backend.py</code> with the following content:</p>
<div><pre><code class="language-python">import boto3
from django.core.mail.backends.smtp import EmailBackend
from django.conf import settings
class SesEmailBackend(EmailBackend):
def __init__(
self,
fail_silently=False,
**kwargs
):
super().__init__(fail_silently=fail_silently)
self.connection = boto3.client(
"ses",
aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
region_name=settings.AWS_REGION,
)
def send_messages(self, email_messages):
for email_message in email_messages:
self.connection.send_raw_email(
Source=email_message.from_email,
Destinations=email_message.recipients(),
RawMessage={"Data": email_message.message().as_bytes(linesep="\r\n")}
)</code></pre></div>
<p>This is the minimum needed to send an email using SES. Surely you will need to add some error handling, input sanitization, retries, etc. but this is out of our topic. </p>
<p>You might see that we have imported <code>boto3</code> at the beginning of the file. Don't forget to install it using a command</p>
<div><pre><code class="language-bash">pip install boto3</code></pre></div>
<p>It's not necessary to reinvent the wheel every time you need a custom email backend. You can find already existing libraries, or just receive SMTP credentials in your Amazon console and use default email backend. </p>
<h2>Sending emails using SES from Amazon</h2>
<p>So far, you can benefit from several services that allow you to send transactional emails at ease. If you can't choose one, check out our blogpost about <a href="https://blog.mailtrap.io/sendgrid-vs-mandrill-vs-mailgun/">Sendgrid vs. Mandrill vs. Mailgun</a>. It will help a lot. But today, we'll discover how to make your Django app send emails via Amazon SES. It is one of the most popular services so far. Besides, you can take advantage of a ready-to-use Django email backend for this service - <a href="https://github.com/django-ses/django-ses">django-ses</a>.</p>
<h3>Set up the library</h3>
<p>You need to execute pip install django-ses to install django-ses. Once it's done, tweak your <code>settings.py</code> with the following line:</p>
<div><pre><code class="language-python">EMAIL_BACKEND = "django_ses.SESBackend"</code></pre></div>
<h3>AWS credentials</h3>
<p>Don't forget to set up your AWS account to get the required credentials - AWS access keys that consist of access key ID and secret access key. For this, add a user in Identity and Access Management (IAM) service. Then, choose a user name and Programmatic access type. Attach AmazonSESFullAccess permission and create a user. Once you've done this, you should see AWS access keys. Update your <code>settings.py</code>:</p>
<div><pre><code class="language-python">AWS_ACCESS_KEY_ID = "********"
AWS_SECRET_ACCESS_KEY = "********"</code></pre></div>
<h3>Email sending</h3>
<p>Now, you can send your emails using <code>django.core.mail.send_mail</code>:</p>
<div><pre><code class="language-python">from django.core.mail import send_mail
send_mail(
"That's your subject",
"That's your message body",
"from@yourdjangoapp.com",
["to@yourbestuser.com"]
)</code></pre></div>
<p><code>django-ses</code> is not the only preset email backend you can leverage. At the end of our article, you'll find more useful libraries to optimize email delivery of your Django app. But first, a step you should never send emails without.</p>
<h2>Testing email sending in Django </h2>
<p>Once you've got everything prepared for sending email messages, it is necessary to do some initial testing of your mail server. In Python, this can be done with one command:</p>
<div><pre><code class="language-bash">python -m smtpd -n -c DebuggingServer localhost:1025</code></pre></div>
<p>It allows you to send emails to your local SMTP server. The DebuggingServer feature won't actually send the email but will let you see the content of your message in the shell window. That's an option you can use off-hand.</p>
<h3>Django's TestCase</h3>
<p>TestCase is a solution to test a few aspects of your email delivery. It uses <code>django.core.mail.backends.locmem.EmailBackend</code>, which, as you remember, stores messages in the local memory cache - <code>django.core.mail.outbox</code>. So, this test runner does not actually send emails. Once you've selected this email backend</p>
<div><pre><code class="language-python">EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"</code></pre></div>
<p>you can use the following unit test sample to test your email sending capability.</p>
<div><pre><code class="language-python">from django.core import mail
from django.test import TestCase
class EmailTest(TestCase):
def test_send_email(self):
mail.send_mail(
"That's your subject", "That's your message body",
"from@yourdjangoapp.com", ["to@yourbestuser.com"],
fail_silently=False,
)
self.assertEqual(len(mail.outbox), 1)
self.assertEqual(mail.outbox[0].subject, "That's your subject")
self.assertEqual(mail.outbox[0].body, "That's your message body")</code></pre></div>
<p>This code will test not only your email sending but also the correctness of the subject and message body. </p>
<h3>Testing with Mailtrap</h3>
<p><a href="https://mailtrap.io/">Mailtrap</a> can be a rich solution for testing. First, it lets you test not only the SMTP server but also the email content and do other essential checks from the <a href="https://blog.mailtrap.io/email-testing-checklist/">email testing checklist</a>. Second, it is a rather easy-to-use tool.</p>
<p>All you need to do is to copy the SMTP credentials from your demo inbox and tweak your <code>settings.py</code>. Or you can just copy/paste these four lines from the <strong>Integrations</strong> section by choosing Django in the pop-up menu.</p>
<div><pre><code class="language-python">EMAIL_HOST = "smtp.mailtrap.io"
EMAIL_HOST_USER = "********"
EMAIL_HOST_PASSWORD = "*******"
EMAIL_PORT = "2525"</code></pre></div>
<p>After that, feel free to send your HTML email with an attachment to check how it goes.</p>
<div><pre><code class="language-python">from django.core.mail import EmailMultiAlternatives
from django.template.loader import render_to_string
subject = "That's your subject"
html_message = render_to_string("mail_template.html", {"context": "values"})
plain_message = strip_tags(html_message)
from_email = "from@yourdjangoapp.com"
to_email = "to@yourbestuser.com"
message = EmailMultiAlternatives(subject, plain_message, from_email, [to_email])
message.attach_alternative(html_message, "text/html")
message.attach_file("/documents/Attachment.pdf")
message.send()</code></pre></div>
<p>If there is no message in the Mailtrap Demo inbox or there are some issues with HTML content, you need to polish your code. </p>
<h2>Django email libraries to simplify your life</h2>
<p>As a conclusion to this blog post about sending emails with Django, we've included a brief introduction of a few libraries that will facilitate your email workflow. </p>
<h3>django-anymail </h3>
<p>This is a collection of email backends and webhooks for numerous famous email services, including SendGrid, Mailgun, and others. <a href="https://github.com/anymail/django-anymail">django-anymail</a> works with <code>django.core.mail</code> module and normalizes the functionality of transactional email service providers.</p>
<h3>django-mailer</h3>
<p><a href="https://github.com/pinax/django-mailer">django-mailer</a> is a Django app you can use to queue the email sending. With it, scheduling your emails is much easier.</p>
<h3>django-post_office</h3>
<p>With this app, you can send and manage your emails. <a href="https://github.com/ui/django-post_office">django-post_office</a> offers many cool features like asynchronous email sending, built-in scheduling, multiprocessing, etc.</p>
<h3>django-templated-email</h3>
<p>This app is about sending templated emails. In addition to its own functionalities, <a href="https://github.com/vintasoftware/django-templated-email">django-templated-email</a> can be used in tow with django-anymail to integrate transactional email service providers.</p>
<h3>django-mailbox</h3>
<p>You might use <a href="https://github.com/coddingtonbear/django-mailbox">django-mailbox</a> if you need to import messages from local mailboxes, POP3, IMAP, or directly receive messages from Postfix or Exim4.</p>
<p>We hope that this small list of packages will facilitate your email workflow. You can always find more apps at <a href="https://djangopackages.org/grids/g/email/">Django Packages</a>.</p>
<hr>
<p>Cover Photo by <a href="https://unsplash.com/photos/ieic5Tq8YMk">Chris Ried</a></p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com1tag:blogger.com,1999:blog-9014457088394059918.post-54913596050410114082019-10-24T05:09:00.000+02:002019-10-24T05:09:41.243+02:00Things I want to remember about SSH<img border="0" src="https://4.bp.blogspot.com/-XwGdz4CBlvU/XbEVi1PVdyI/AAAAAAAACAY/E6dzjlXwcqgvkSED82EtChrPYeWww-NxACLcBGAsYHQ/s1600/things-i-want-to-remember-about-ssh.png" data-original-width="1024" data-original-height="512" />
<p>SSH, short for Secure Shell, is a protocol for secure network communications. It is widely used for executing commands on remote servers, and for file uploads or downloads. If you are working with Django, use Git version control, or administrate servers, you surely are using SSH. In this post, I want to share some technical details about it.</p>
<p>Secure Shell is using private and public key pairs. You can either use automatically generated private and public keys combined with a password, or manually generated private and public keys. In the latter case, you need to keep your private key on your computer and upload the public key to the remote server. </p>
<h2>Creating a pair of SSH keys manually</h2>
<p>If you are using GitHub, Bitbucket, DigitalOcean, or some other service, you might have seen the possibility to upload public SSH keys for direct access to remote servers.</p>
<p>Here is how you usually create the SSH keys on the computer from which you want to establish a secure connection (your local machine or one of your servers that has access to other servers or services). In the Terminal you would execute these commands:</p>
<div><pre><code class="language-bash">$ ssh-keygen
$ ssh-agent /usr/local/bin/bash
$ ssh-add ~/.ssh/id_rsa</code></pre></div>
<p>The <code>id_rsa</code> is the name of the default SSH private key. The public key would be <code>id_rsa.pub</code>. And by default they both will be located under <code>~/.ssh/</code>.</p>
<p>When running <code>ssh-keygen</code> you can choose different key names and even add a passphrase. For instance, you could have <code>github_id_rsa</code> and <code>github_id_rsa.pub</code> keys for communication with GitHub. My recommendation would be for each new service to create a new private-public key pair so that in case you need to transfer your computer's data to a different machine, you could selectively transfer the access to the remote servers.</p>
<p>Also, if you are not using the passphrase for the SSH key pair, I would recommend having your disk encrypted and a secure user password for your computer. If your laptop gets stolen, the thief wouldn't be able to get to your remote servers without knowing your computer's password.</p>
<h2>Creating an access to a remote server by SSH key</h2>
<p>In the case of GitHub, Bitbucket, and other online services with SSH communication, you usually have to copy the contents of the public key into a text field in a web form.</p>
<p>If you want to create a secure communication by manually generated private-public keys with a server where your Django project is deployed, you should append the contents of the public key to the <code>~/.ssh/authorized_keys</code> file on the remote server. </p>
<p>To get the content of the public key in the Terminal, you can use:</p>
<div><pre><code class="language-bash">$ cat ~/.ssh/id_rsa.pub</code></pre></div>
<p>Then copy the output to the clipboard.</p>
<p>Or on macOS you can run <code>pbcopy</code> as follows:</p>
<div><pre><code class="language-bash">$ pbcopy < ~/.ssh/id_rsa.pub </code></pre></div>
<p>To append the contents of the public key to the remote server, you can do this:</p>
<div><pre><code class="language-bash">$ echo "...pasted public key...">>~/.ssh/authorized_keys</code></pre></div>
<h2>Creating authorization at a remote server by password</h2>
<p>If you want to establish an SSH connection with a password and automatically generated private-public keys, you would need to edit <code>/etc/ssh/sshd_config</code> and ensure these two settings:</p>
<div><pre><code class="language-none">PasswordAuthentication yes
PermitEmptyPasswords no</code></pre></div>
<p>After the change, you would restart the ssh server with the following command:</p>
<div><pre><code class="language-none">$ sudo service ssh restart</code></pre></div>
<p>Also, make sure that the user you are connecting with has a password:</p>
<div><pre><code class="language-none">$ sudo passwd the_user</code></pre></div>
<h2>Connecting to a remote server</h2>
<p>The default way to connect via SSH to a remote server with a password is executing the following in the Terminal:</p>
<div><pre><code class="language-bash">$ ssh the_user@example.com</code></pre></div>
<p>To connect with a private key, you would execute this:</p>
<div><pre><code class="language-bash">$ ssh -i ~/.ssh/examplecom_id_rsa the_user@example.com</code></pre></div>
<p>Next, let's see how we can simplify this using some local SSH configuration.</p>
<h2>Configuring local SSH client</h2>
<p>Edit <code>~/.ssh/config</code> and add the following lines for each SSH connection that you want to define:</p>
<div><pre><code class="language-none">Host examplecom
HostName example.com
User the_user
IdentityFile ~/.ssh/examplecom_id_rsa</code></pre></div>
<p>If the domain of the website is not yet pointing to the IP address of the server, you can also connect by IP address:</p>
<div><pre><code class="language-none">Host examplecom
HostName 1.2.3.4
User the_user
IdentityFile ~/.ssh/examplecom_id_rsa</code></pre></div>
<p>The following allows you to login to your remote servers by manually generated private-public key with just these lines:</p>
<div><pre><code class="language-bash">$ ssh examplecom</code></pre></div>
<p>To request for password instead of using the manually generated keys, you would need to modify the snippet as follows:</p>
<div><pre><code class="language-none">Host examplecom
HostName example.com
User the_user
PubkeyAuthentication=no</code></pre></div>
<p>When you connect via SSH and wait don't type anything for 30 minutes or so, the connection gets lost. But you can require your client to connect to the server every 4 minutes or so by adding the following lines to the beginning of the <code>~/.ssh/config</code> on your local computer:</p>
<div><pre><code class="language-none">Host *
ServerAliveInterval 240</code></pre></div>
<h2>Uploading and downloading files using SSH connection</h2>
<p>Typically, Secure Shell allows you to execute terminal commands on the remote server using bash, zsh, sh, or another shell. But very often, you also need to transfer files securely to and from the server. For that, you have these options: <code>scp</code> command, <code>rsync</code> command, or FTP client with <code>SFTP</code> support.</p>
<h3>scp</h3>
<p>The <code>scp</code> stands for Secure Copy.</p>
<p>This is how you would copy the <code>secrets.json</code> file from the remote server to your local development environment:</p>
<div><pre><code class="language-none">$ scp the_user@example.com:~/src/myproject/myproject/settings/secrets.json ./myproject/settings/secrets.json</code></pre></div>
<p>Here is an example of the same, but with custom <code>~/.ssh/config</code> configuration:</p>
<div><pre><code class="language-none">$ scp examplecom:~/src/myproject/myproject/settings/secrets.json ./myproject/settings/secrets.json</code></pre></div>
<p>To copy the file from the local computer to the remote server, you would switch the places of source and target:</p>
<div><pre><code class="language-none">$ scp ./myproject/settings/secrets.json examplecom:~/src/myproject/myproject/settings/secrets.json</code></pre></div>
<h3>rsync</h3>
<p>To synchronize directories on the server and locally, you can use the <code>rsync</code> command. This is how to do it for downloading the <code>media/</code> directory (note that the trailing slashes matter):</p>
<div><pre><code class="language-none">$ rsync --archive --compress --partial --progress the_user@example.com:~/src/myproject/myproject/media/ ./myproject/media/</code></pre></div>
<p>Here is an example of the same with a custom <code>~/.ssh/config</code> configuration:</p>
<div><pre><code class="language-none">$ rsync --archive --compress --partial --progress examplecom:~/src/myproject/myproject/media/ ./myproject/media/</code></pre></div>
<p>To upload the <code>media/</code> directory to the remote server, you would again switch places for the source and target:</p>
<div><pre><code class="language-none">$ rsync --archive --compress --partial --progress ./myproject/media/ examplecom:~/src/myproject/myproject/media/</code></pre></div>
<h3>sftp</h3>
<p>FTP clients like Transmit allow you to have SFTP connections either by username and password or by username and private key. You can even generate the private-public keys directly in the app there.</p>
<p>SFTP works like FTP, but all communication is encrypted there.</p>
<h2>The final words</h2>
<p>Use only encrypted connections for your network communications, encrypt your hard disk if you use manually generated private-public keys, and use strong passwords.</p>
<p>Be safe!</p>
<hr>
<p>Cover photo by <a href="https://unsplash.com/photos/7BhTfoKsheQ">Jason D</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495399999997552.21073109999999 12.759506999999974 52.829282099999986 14.050400999999976tag:blogger.com,1999:blog-9014457088394059918.post-15605233973607034642019-10-08T22:53:00.000+02:002019-10-08T22:53:22.419+02:00Working with Dates and Times in the Forms<img border="0" src="https://1.bp.blogspot.com/-zpoafwSVHi4/XZz2XRYcNDI/AAAAAAAAB_8/X-YdgAN74IQ_hJ0YoOHA7uwRkY2G56MTwCLcBGAsYHQ/s1600/working-with-dates-and-times-in-the-forms.png" data-original-width="1024" data-original-height="512" />
<p>HTML5 comes with a bunch of new types for the input fields that are rendered as rich native widgets. Browsers even restrict invalid values and validate the input immediately. Let's explore how we could make use of them in Django forms.</p>
<p>We will be using an <code>Exhibition</code> model with <code>models.DateField</code>, <code>models.TimeField</code>, and <code>models.DateTimeField</code>:</p>
<div><pre><code class="language-python"># exhibitions/models.py
from django.db import models
from django.utils.translation import gettext_lazy as _
class Exhibition(models.Model):
title = models.CharField(_("Title"), max_length=200)
start = models.DateField(_("Start"))
end = models.DateField(_("End"), blank=True, null=True)
opening = models.TimeField(_("Opening every day"))
closing = models.TimeField(_("Closing every day"))
vernissage = models.DateTimeField(_("Vernissage"), blank=True, null=True)
finissage = models.DateTimeField(_("Finissage"), blank=True, null=True)
class Meta:
verbose_name = _("Exhibition")
verbose_name_plural = _("Exhibitions")
def __str__(self):
return self.title</code></pre></div>
<p>Here is a quick model form for the <code>Exhibition</code> model:</p>
<div><pre><code class="language-python"># exhibitions/forms.py
from django import forms
from .models import Exhibition
class ExhibitionForm(forms.ModelForm):
class Meta:
model = Exhibition
fields = "__all__"</code></pre></div>
<p>If we now open a Django shell and create an instance of the model form with some initial values, then print the form as HTML to the console, we will notice, that all date and time fields are rendered as <code><input type="text" /></code> and the values for the dates are in a local format, not the ISO standard YYYY-MM-DD:</p>
<div><pre><code class="language-python">(venv)$ python manage.py shell
>>> from exhibitions.forms import ExhibitionForm
>>> from datetime import datetime, date, time
>>> form = ExhibitionForm(initial={
... "start": date(2020, 1, 1),
... "end": date(2020, 3, 31),
... "opening": time(11, 0),
... "closing": time(20, 0),
... "vernissage": datetime(2019, 12, 27, 19, 0),
... "finissage": datetime(2020, 4, 1, 19, 0),
>>> })
>>> print(form.as_p())
<p><label for="id_title">Title:</label> <input type="text" name="title" maxlength="200" required id="id_title"></p>
<p><label for="id_start">Start:</label> <input type="text" name="start" value="01.01.2020" required id="id_start"></p>
<p><label for="id_end">End:</label> <input type="text" name="end" value="31.03.2020" id="id_end"></p>
<p><label for="id_opening">Opening every day:</label> <input type="text" name="opening" value="11:00:00" required id="id_opening"></p>
<p><label for="id_closing">Closing every day:</label> <input type="text" name="closing" value="20:00:00" required id="id_closing"></p>
<p><label for="id_vernissage">Vernissage:</label> <input type="text" name="vernissage" value="27.12.2019 19:00:00" id="id_vernissage"></p>
<p><label for="id_finissage">Finissage:</label> <input type="text" name="finissage" value="01.04.2020 19:00:00" id="id_finissage"></p>
</code></pre></div>
<p>Let's modify the model form and customize the date and time inputs. We will extend and use <code>forms.DateInput</code>, <code>forms.TimeInput</code>, and <code>forms.DateTimeInput</code> widgets. We want to show date inputs as <code><input type="date" /></code>, time inputs as <code><input type="time" /></code>, and date-time inputs as <code><input type="datetime-local" /></code>. In addition, the format for the dates should be based on ISO standard.</p>
<div><pre><code class="language-python"># exhibitions/forms.py
from django import forms
from .models import Exhibition
class DateInput(forms.DateInput):
input_type = "date"
def __init__(self, **kwargs):
kwargs["format"] = "%Y-%m-%d"
super().__init__(**kwargs)
class TimeInput(forms.TimeInput):
input_type = "time"
class DateTimeInput(forms.DateTimeInput):
input_type = "datetime-local"
def __init__(self, **kwargs):
kwargs["format"] = "%Y-%m-%dT%H:%M"
super().__init__(**kwargs)
class ExhibitionForm(forms.ModelForm):
class Meta:
model = Exhibition
fields = "__all__"
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.fields["start"].widget = DateInput()
self.fields["end"].widget = DateInput()
self.fields["opening"].widget = TimeInput()
self.fields["closing"].widget = TimeInput()
self.fields["vernissage"].widget = DateTimeInput()
self.fields["vernissage"].input_formats = ["%Y-%m-%dT%H:%M", "%Y-%m-%d %H:%M"]
self.fields["finissage"].widget = DateTimeInput()
self.fields["finissage"].input_formats = ["%Y-%m-%dT%H:%M", "%Y-%m-%d %H:%M"]</code></pre></div>
<p>Let's see now in the Django shell if that worked as expected:</p>
<div><pre><code class="language-python">(venv)$ python manage.py shell
>>> from exhibitions.forms import ExhibitionForm
>>> from datetime import datetime, date, time
>>> form = ExhibitionForm(initial={
... "start": date(2020, 1, 1),
... "end": date(2020, 3, 31),
... "opening": time(11, 0),
... "closing": time(20, 0),
... "vernissage": datetime(2019, 12, 27, 19, 0),
... "finissage": datetime(2020, 4, 1, 19, 0),
>>> })
>>> print(form.as_p())
<p><label for="id_title">Title:</label> <input type="text" name="title" maxlength="200" required id="id_title"></p>
<p><label for="id_start">Start:</label> <input type="date" name="start" value="2020-01-01" required id="id_start"></p>
<p><label for="id_end">End:</label> <input type="date" name="end" value="2020-03-31" id="id_end"></p>
<p><label for="id_opening">Opening every day:</label> <input type="time" name="opening" value="11:00:00" required id="id_opening"></p>
<p><label for="id_closing">Closing every day:</label> <input type="time" name="closing" value="20:00:00" required id="id_closing"></p>
<p><label for="id_vernissage">Vernissage:</label> <input type="datetime-local" name="vernissage" value="2019-12-27T19:00" id="id_vernissage"></p>
<p><label for="id_finissage">Finissage:</label> <input type="datetime-local" name="finissage" value="2020-04-01T19:00" id="id_finissage"></p>
</code></pre></div>
<p>The same way you can also create widgets for other HTML5 input types: <code>color</code>, <code>email</code>, <code>month</code>, <code>number</code>, <code>range</code>, <code>tel</code>, <code>url</code>, <code>week</code>, and alike.</p>
<p>Happy coding!</p>
<hr>
<p>Cover photo by <a href="https://unsplash.com/photos/FoKO4DpXamQ">Eric Rothermel</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495399999997552.21073109999999 12.759506999999974 52.829282099999986 14.050400999999976tag:blogger.com,1999:blog-9014457088394059918.post-8641898419522265082019-05-04T09:00:00.000+02:002019-05-04T09:00:03.007+02:00My 5 Favorite Talks at DjangoCon Europe 2019<img border="0" src="https://1.bp.blogspot.com/-BzP7Pb1KYnc/XMy8IpYcmyI/AAAAAAAAB-c/7kaW3l-YHmI1iDar2sLjRgM-G82qIseIQCLcBGAs/s1600/my-5-favorite-talks-at-djangocon-europe-2019.png" data-original-width="1024" data-original-height="512" alt="Django people at the DjangoCon Europe 2019" />
<p>This year DjangoCon Europe happened in Copenhagen, Denmark, at a very creative and special place <a href="http://www.afuk.dk/">AFUK - Academy for Untamed Creativity</a>. Surrounded by artistic souls, we learned more about web technologies, got to know each other at ecologic reusable disposable cups of coffee, enjoyed delicious authentic food, and socialized with Django-branded beverages. As always, there was also a party with people drinking IPA (not to be confused with API). And at the weekend Django developers were solving bugs for Django and related open-source software at the coding sprints.</p>
<p>Here I would like to present you with the top 5 talks that I liked most of all.</p>
<h2>Django and Web Security Headers</h2>
<p>Adam Johnson (<a href="https://twitter.com/AdamChainz">@AdamChainz</a>) was talking about special response headers that tell browsers to treat data of the website more securely and which Django settings are responsible for those headers.</p>
<div class="videoWrapper"><iframe width="560" height="315" src="https://www.youtube.com/embed/gvQW1vVNohg" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>
<p><a href="https://rixx.de/blog/djangocon-europe-2019-django-and-web-security-headers/">Summary by rixx</a></p>
<h2>Docs or it didn't Happen!</h2>
<p>Mikey Ariel (<a href="https://twitter.com/ThatDocsLady">@ThatDocsLady</a>) was talking about the necessity of documentation and what to write there.</p>
<div class="videoWrapper"><iframe width="560" height="315" src="https://www.youtube.com/embed/gwVxNHO9Lh0" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>
<p><a href="https://rixx.de/blog/djangocon-europe-2019-docs-or-it-didn-t-happen-with-q-a/">Summary by rixx</a></p>
<h2>Pushing the ORM to its Limits</h2>
<p>Sigurd Ljødal (<a href="https://twitter.com/sigurdlj">@sigurdlj</a>) was talking about advanced Django ORM use cases.</p>
<div class="videoWrapper"><iframe width="560" height="315" src="https://www.youtube.com/embed/MPpPu6c8wsM" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>
<p><a href="https://rixx.de/blog/djangocon-europe-2019-pushing-the-orm-to-its-limits/">Summary by rixx</a></p>
<h2>Logging Rethought 2: The Actions of Frank Taylor Jr.</h2>
<p>Markus Holtermann (<a href="https://twitter.com/m_holtermann">@m_holtermann</a>) was talking about logging structured data to log files instead of the traditional plain text messages.</p>
<div class="videoWrapper"><iframe width="560" height="315" src="https://www.youtube.com/embed/Y5eyEgyHLLo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>
<p><a href="https://rixx.de/blog/djangocon-europe-2019-logging-rethought-2-the-actions-of-frank-taylor-jr/">Summary by rixx</a></p>
<h2>Maintaining a Django Codebase after 10k Commits</h2>
<p>Joachim Jablon (<a href="https://twitter.com/Ewjoachim">@Ewjoachim</a>) and Stéphane Angel (<a href="https://twitter.com/twidi">@twidi</a>) were talking about the best practices developing large-scale Django projects.</p>
<div class="videoWrapper"><iframe width="560" height="315" src="https://www.youtube.com/embed/_DIlE-yc9ZQ" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>
<p><a href="https://rixx.de/blog/djangocon-europe-2019-maintaning-a-django-codebase-after-10k-commits/">Summary by rixx</a></p>
<h2>Honorable Mentions</h2>
<ul>
<li><a href="https://2019.djangocon.eu/talks/feeding-the-pony-contributing-back-to-django-how-t/">"Feeding the Pony: Contributing back to Django & How to make that work for you"</a> by Carlton Gibson.</li>
<li><a href="https://2019.djangocon.eu/talks/sketching-out-a-django-redesign/">"Sketching out a Django redesign"</a> by Tom Christie.</li>
<li><a href="https://2019.djangocon.eu/talks/reduce-reuse-recycle-persisting-websocket-connecti/">"Reduce, Reuse, Recycle - Persisting WebSocket connections with SharedWorkers"</a> by Aaronn Bassett.</li>
<li><a href="https://2019.djangocon.eu/talks/maps-with-geodjango-postgis-and-leaflet/">"Maps with GeoDjango, PostGIS and Leaflet"</a> by Paolo Melchiorre.</li>
<li><a href="https://2019.djangocon.eu/talks/here-come-the-robots-django-and-machine-learning/">"Here Come The Robots - Django and Machine Learning"</a> by Tom Dyson.</li>
</ul>
<h2>More Information</h2>
<p>You can see all talk descriptions, video records and some slides at the <a href="https://2019.djangocon.eu/talks/">official conference website</a>.</p>
<p><a href="https://www.flickr.com/photos/djangocon/sets/72157704663920022">Amazing conference photos</a> were taken by Bartek Pawlik.</p>
<p>I was also really astonished how effective were <a href="https://duckduckgo.com/?sites=https%3A%2F%2Frixx.de%2F&k8=%23444444&k9=%23D51920&kt=h&q=DjangoCon+Europe+2019&ia=web">the talk summaries written by rixx</a> almost in real time.</p>
<hr>
<p>Cover photo by <a href="https://www.flickr.com/photos/djangocon/sets/72157704663920022">Bartek Pawlik</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0tag:blogger.com,1999:blog-9014457088394059918.post-91344529202892121232019-04-27T09:00:00.000+02:002019-04-30T00:34:33.493+02:00Improving Page Speed with Incremental Loading<img border="0" src="https://2.bp.blogspot.com/-89TTMDEAZuY/XMN2pCTO6bI/AAAAAAAAB94/hWaxKVaurSYJuco4hfkINjVCv0SbX2r0wCLcBGAs/s1600/improving-page-speed-with-incremental-loading.png" data-original-width="1024" data-original-height="512" alt="Improving Page Speed with Incremental Loading" />
<p>Summary: you can use <a href="https://github.com/archatas/django-include-by-ajax">django-include-by-ajax</a> to improve the performance and usability of your website by forcing some parts of the Django website page to be loaded and shown before other parts of the page.</p>
<hr>
<p>Web browsers load and render traditional HTML pages from top to down, from left to right and as a developer you have little control over what will be shown first, second, and last. However, sometimes you need a different loading sequence to improve user experience and usability. Let's examine a couple of cases when it is advantageous to have primary content showing up immediately and secondary content loading in a moment.</p>
<h2>Case 1. Above the Fold vs. Below the Fold</h2>
<p>People want speed. 47% of visitors expect the website to be loaded in less than 2 seconds. If the website takes more than 3 seconds to show up, it's a big chance, that you will lose 40% of visitors. If you sell something on your website, every one-second delay causes 7% fewer visitors becoming buyers.</p>
<p>One technique to improve the perception of the speed of the website is to display the visible part of the screen as soon as possible, and then load the rest of the website in another go. Usually, the website pages are long vertically scrollable areas. The part of it that fits in the screen is called "above the fold" and the part underneath is called "below the fold".</p>
<img border="0" src="https://1.bp.blogspot.com/-UiqYyJGUREY/XMN2tCmqnfI/AAAAAAAAB98/Wm_dMsBwpKEfhBuqwnRQf2JPdm_rB4ZgwCPcBGAYYCw/s1600/above-the-fold-vs-below-the-fold.png" data-original-width="1024" data-original-height="1263" alt="Primary content above the fold and secondary content below the fold" />
<p>It is recommended to load the part above the fold in 6 requests, including all your HTML, CSS, JavaScript, images and fonts. It's only 6 requests for a reason - that's the maximal number of requests that <a href="https://stackoverflow.com/a/14768266/322818">most browsers keep to the same HTTP/1.1 server at the same time</a>. With HTTP/2 there is no such limitation.</p>
<p>You can only achieve this minimal load if you bundle and minimize your CSS and JavaScript to single files, and use only a couple of images and fonts. Going one step further you can split your CSS and JavaScript into parts that are used above the fold, and the ones that are used below the fold.</p>
<h2>Case 2. Main Content vs. Navigation</h2>
<p>For the users to have best user experience and smooth loading, you could display the content of articles or blog post first, and then load and display the website navigation in the header, sidebars, or footer.</p>
<img border="0" src="https://2.bp.blogspot.com/-iItjGK_qgtE/XMN2tZknzQI/AAAAAAAAB-Q/-KEyx8ug-FgElBeW34wkyPV-2mOo2SUmgCPcBGAYYCw/s1600/content-vs-navigation.png" data-original-width="1024" data-original-height="1030" alt="Content is primary and the navigation is secondary" />
<p>If the visitor navigated to a specific page of your website, they most likely want to see the content of that page rather than navigate out to other pages.</p>
<p>If you have extensive nested navigation, you can also save some milliseconds of its loading at the first request, by skipping it there, but loading it by Ajax at the next go. </p>
<p>Additionally, if visitor disables JavaScript in their browser, they will still be able to read the content.</p>
<h2>Case 3. Own Content vs. Third-party Content</h2>
<p>Wouldn't you agree, websites that show ads before their own content are pretty annoying? One way to improve the user experience is to show the main content at first and show the ads or third-party widgets after several seconds. </p>
<img border="0" src="https://1.bp.blogspot.com/-PYN_NknLPhE/XMN2tWOQNOI/AAAAAAAAB-U/WPOdtPZc05MmOGZ6axMo286cu9wg1OM2wCPcBGAYYCw/s1600/content-vs-ads.png" data-original-width="1024" data-original-height="1009" alt="Own content is primary and third-party widgets are secondary" />
<p>The primary content will be correctly indexed by search engines, whereas the included widgets might be skipped, depending on implementation, which we'll examine next.</p>
<h2>Solution 1. Iframes</h2>
<p>One way to load the delayed secondary content is to use iframes.</p>
<p>Pros:</p>
<ul>
<li>Works without JavaScript.</li>
</ul>
<p>Cons:</p>
<ul>
<li>For each iframed section, you need a separate HTML with custom CSS.</li>
<li>You have to predefine and hardcode all heights of each secondary section, so it wouldn't work well with increased or decreased font size or different amounts of content.</li>
<li>You cannot have interactive elements like tooltips that would go outside the boundaries of the iframe.</li>
</ul>
<h2>Solution 2. API Calls by Ajax</h2>
<p>The page would load with empty placeholders for the secondary content and then some JavaScript function would load content for the missing sections in HTML, JSON, or XML format by Ajax, parse them, and include into the placeholders. This approach <a href="https://www.facebook.com/notes/facebook-engineering/bigpipe-pipelining-web-pages-for-high-performance/389414033919">has been used by Facebook</a>.</p>
<p>Pros:</p>
<ul>
<li>You can use the same global CSS for everything.</li>
<li>The amount of content is flexible, so the designs would look good with different variations.</li>
</ul>
<p>Cons:</p>
<ul>
<li>For each secondary section, you need to define a separate API endpoint.</li>
<li>There are many extra requests (unless you use GraphQL for that).</li>
</ul>
<h2>Solution 3. A Second Request to the Same Page with Specific Query Parameters</h2>
<p>The page loads with empty placeholders for the secondary content. A JavaScript function uses Ajax to load the HTML of the same page this time containing all rendered primary and secondary content. Then another JavaScript function goes through all placeholders and fills the content from the second load.</p>
<p>Pros:</p>
<ul>
<li>You can use the same global CSS for everything.</li>
<li>The amount of content is flexible, so the designs could look good with different variations.</li>
<li>Each page uses a single data endpoint.</li>
<li>Only one extra request is necessary for the full HTML.</li>
</ul>
<p>Cons:</p>
<ul>
<li>If there is a lot of primary content and not so much of secondary content, it might take too long to load and parse the secondary content.</li>
</ul>
<h2>Implementation for a Django Website using django-include-by-ajax</h2>
<p>You can implement the third solution in a Django website using my open-source Django app <strong>django-include-by-ajax</strong>. It is meant to be understandable and simple to use for frontend Django developers, who don't touch Python code but need to work on the layouts and styling.</p>
<p>The idea is that instead of including different sections of a template with the <code>{% include template_name %}</code> template tag, you do the same using <code>{% include_by_ajax template_name %}</code> template tag. This template tag renders as an empty placeholder unless you access the page from a search crawler or if you access the page with a specific query parameter. Otherwise, it works more-or-less the same as the <code>{% include %}</code> template tag.</p>
<p>By adding jQuery and one jQuery-based JavaScript file to your page template, you enable the magic that does all the loading and parsing. Since version 1.0, CSS and JavaScript files can also be included in those delayed sections.</p>
<p>You can see <strong>django-include-by-ajax</strong> in action at the start page of my personal project <a href="https://www.1st-things-1st.com">1st things 1st</a>. There I use the above-the-fold case with the visible content coming to the screen almost immediately and the offscreen content loading in several more seconds.</p>
<h2>Installation</h2>
<p>It should be trivial to convert any standard heavy website page to a page loading the secondary content dynamically. There are mainly these steps to follow:</p>
<ol>
<li><p>Install the app with your Python package manager:</p>
<div><pre><code class="language-shell">(venv)$ pip install django-include-by-ajax==1.0.0</code></pre></div></li>
<li><p>Put the app into the <code>INSTALLED_APPS</code> in your Django project settings:</p>
<div><pre><code class="language-python"># settings.py
INSTALLED_APPS = [
# ...
# Third-party apps
'include_by_ajax',
# ...
]</code></pre></div></li>
<li><p>Put these in your <code>base.html</code>:</p>
<div><pre><code class="language-markup">{% load staticfiles %}
<script src="https://code.jquery.com/jquery-3.4.0.min.js" crossorigin="anonymous"></script>
<script src="{% static 'include_by_ajax/js/include_by_ajax.min.js' %}" defer></script></code></pre></div>
<p>It should also work with older or newer jQuery versions.</p></li>
<li><p>Use the new template tag in any template where you need it:</p>
<div><pre><code class="language-django">{% load include_by_ajax_tags %}
{% include_by_ajax "blog/includes/latest_blog_posts.html" %}</code></pre></div>
<p>You can even define the content for the placeholder that will be shown while the main content is loading:</p>
<div><pre><code class="language-django">{% load include_by_ajax_tags %}
{% include_by_ajax "blog/includes/latest_blog_posts.html" placeholder_template_name="utils/loading.html" %}</code></pre></div></li>
<li><p>If you need some JavaScript action to be called after all content is loaded, you can use the custom <code>include_by_ajax_all_loaded</code> event on the <code>document</code> like this:</p>
<div><pre><code class="language-javascript">$(document).on('include_by_ajax_all_loaded', function() {
console.log('Now all placeholders are loaded and replaced with content. Hurray!');
});</code></pre></div></li>
</ol>
<p>I would be glad if you tried it on some of your projects and see how it improved user experience and loading times. If you use it in production, please mention it in the comments of this blog post.</p>
<h2>More Information</h2>
<p>The app on Github: <a href="https://github.com/archatas/django-include-by-ajax">A Django App Providing the <code>{% include_by_ajax %}</code> Template Tag</a></p>
<p>The practical usage example: <a href="https://www.1st-things-1st.com">Strategic planner - Prioritizer - 1st things 1st</a>.</p>
<p>An article on performance and usability: <a href="https://www.websitemagazine.com/blog/5-reasons-visitors-leave-your-website">5 Reasons Visitors Leave Your Website</a></p>
<hr>
<p>Cover photo by <a href="https://unsplash.com/photos/KraXdvWzKNw">Thomas Tucker</a>.</p>
<p>Thanks to <a href="https://adamj.eu/">Adam Johnson</a> for reviewing this post.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com1Berlin, Germany52.520006599999988 13.40495399999997552.21073109999999 12.759506999999974 52.829282099999986 14.050400999999976tag:blogger.com,1999:blog-9014457088394059918.post-69246616101010655132019-02-15T01:43:00.000+01:002019-03-11T04:47:55.944+01:00How to Export Data to XLSX Files<img border="0" src="https://1.bp.blogspot.com/-isH4SKIR1No/XGYHR4fL36I/AAAAAAAAB8c/21tlFQt5RJYeNuKWZKReVRyRY0CULZcUwCLcBGAs/s1600/how-to-export-data-to-xlsx-files.png" data-original-width="1024" data-original-height="512" />
<p>A while ago I wrote <a href="https://djangotricks.blogspot.com/2013/12/how-to-export-data-as-excel.html">an article about exporting data to different spreadsheet formats</a>. As recently I was reimplementing export to Excel for the <a href="https://www.1st-things-1st.com">1st things 1st</a> project, I noticed that the API changed a little, so it's time to blog about that again.</p>
<p>For Excel export I am using the XLSX file format which is a zipped XML-based format for spreadsheets with formatting support. XLSX files can be opened with Microsoft Excel, Apache OpenOffice, Apple Numbers, LibreOffice, Google Drive, and a handful of other applications. For building the XLSX file I am using <strong>openpyxl</strong> library.</p>
<h2>Installing openpyxl</h2>
<p>You can install openpyxl to your virtual environment the usual way with pip:</p>
<div><pre><code class="language-bash">(venv) pip install openpyxl==2.6.0</code></pre></div>
<h2>Simplest Export View</h2>
<p>To create a function exporting data from a QuerySet to XLSX file, you would need to create a view that returns a response with a special content type and file content as an attachment. Plug that view to URL rules and then link it from an export button in a template.</p>
<p>Probably the simplest view that generates XLSX file out of Django QuerySet would be this:</p>
<div><pre><code class="language-python"># movies/views.py
from datetime import datetime
from datetime import timedelta
from openpyxl import Workbook
from django.http import HttpResponse
from .models import MovieCategory, Movie
def export_movies_to_xlsx(request):
"""
Downloads all movies as Excel file with a single worksheet
"""
movie_queryset = Movie.objects.all()
response = HttpResponse(
content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
)
response['Content-Disposition'] = 'attachment; filename={date}-movies.xlsx'.format(
date=datetime.now().strftime('%Y-%m-%d'),
)
workbook = Workbook()
# Get active worksheet/tab
worksheet = workbook.active
worksheet.title = 'Movies'
# Define the titles for columns
columns = [
'ID',
'Title',
'Description',
'Length',
'Rating',
'Price',
]
row_num = 1
# Assign the titles for each cell of the header
for col_num, column_title in enumerate(columns, 1):
cell = worksheet.cell(row=row_num, column=col_num)
cell.value = column_title
# Iterate through all movies
for movie in movie_queryset:
row_num += 1
# Define the data for each cell in the row
row = [
movie.pk,
movie.title,
movie.description,
movie.length_in_minutes,
movie.rating,
movie.price,
]
# Assign the data for each cell of the row
for col_num, cell_value in enumerate(row, 1):
cell = worksheet.cell(row=row_num, column=col_num)
cell.value = cell_value
workbook.save(response)
return response</code></pre></div>
<p>If you try this, you will notice, that there is no special formatting in it, all columns are of the same width, the value types are barely recognized, the header is displayed the same as the content. This is enough for further data export to CSV or manipulation with <strong>pandas</strong>. But if you want to present the data for the user in a friendly way, you need to add some magic.</p>
<h2>Creating More Worksheets</h2>
<p>By default, each Excel file has one worksheet represented as a tab. You can access it with:</p>
<div><pre><code class="language-python">worksheet = workbook.active
worksheet.title = 'The New Tab Title'</code></pre></div>
<p>If you want to create tabs dynamically with data from the database of Python structures, you can at first delete the current tab and add the others with:</p>
<div><pre><code class="language-python">workbook.remove(workbook.active)
for index, category in enumerate(category_queryset):
worksheet = workbook.create_sheet(
title=category.title,
index=index,
)</code></pre></div>
<p>Although not all spreadsheet applications support this, you can set the background color of the worksheet tab with:</p>
<div><pre><code class="language-python">worksheet.sheet_properties.tabColor = 'f7f7f9'</code></pre></div>
<h2>Working with Cells</h2>
<p>Each cell can be accessed by its 1-based indexes for the rows and for the columns:</p>
<div><pre><code class="language-python">top_left_cell = worksheet.cell(row=1, column=1)
top_left_cell.value = "This is good!"</code></pre></div>
<p>Styles and formatting are applied to individual cells instead of rows or columns. There are several styling categories with multiple configurations for each of them. You can find some available options from the <a href="https://openpyxl.readthedocs.io/en/latest/styles.html">documentation</a>, but even more by exploring the <a href="https://bitbucket.org/openpyxl/openpyxl/">source code</a>.</p>
<div><pre><code class="language-python">from openpyxl.styles import Font, Alignment, Border, Side, PatternFill
top_left_cell.font = Font(name='Calibri', bold=True)
top_left_cell.alignment = Alignment(horizontal='center')
top_left_cell.border = Border(
bottom=Side(border_style='medium', color='FF000000'),
)
top_left_cell.fill = PatternFill(
start_color='f7f7f9',
end_color='f7f7f9',
fill_type='solid',
)</code></pre></div>
<p>If you are planning to have multiple styled elements, instantiate the font, alignment, border, fill options upfront and then assign the instances to the cell attributes. Otherwise, you can get into memory issues when you have a lot of data entries. </p>
<h2>Setting Column Widths</h2>
<p>If you want to have some wider or narrower width for some of your columns, you can do this by modifying column dimensions. They are accessed by column letter which can be retrieved using a utility function:</p>
<div><pre><code class="language-python">from openpyxl.utils import get_column_letter
column_letter = get_column_letter(col_num)
column_dimensions = worksheet.column_dimensions[column_letter]
column_dimensions.width = 40</code></pre></div>
<p>The units here are some relative points depending on the width of the letters in the specified font. I would suggest playing around with the width value until you find what works for you.</p>
<p>When defining column width is not enough, you might want to wrap text into multiple lines so that everything can be read by people without problems. This can be done with the alignment setting for the cell as follows:</p>
<div><pre><code class="language-python">from openpyxl.styles import Alignment
wrapped_alignment = Alignment(vertical='top', wrap_text=True)
cell.alignment = wrapped_alignment</code></pre></div>
<h2>Data Formatting</h2>
<p>Excel automatically detects text or number types and aligns text to the left and numbers to the right. If necessary that can be overwritten.</p>
<p>There are some gotchas on how to format cells when you need a percentage, prices, or time durations.</p>
<h3>Percentage</h3>
<p>For percentage, you have to pass the number in float format from 0.0 till 1.0 and style should be 'Percent' as follows:</p>
<div><pre><code class="language-python">cell.value = 0.75
cell.style = 'Percent'</code></pre></div>
<h3>Currency</h3>
<p>For currency, you need values of <code>Decimal</code> format, the style should be 'Currency', and you will need a special number format for currency other than American dollars, for example:</p>
<div><pre><code class="language-python">from decimal import Decimal
cell.value = Decimal('14.99')
cell.style = 'Currency'
cell.number_format = '#,##0.00 €'</code></pre></div>
<h3>Durations</h3>
<p>For time duration, you have to pass timedelta as the value and define special number format:</p>
<div><pre><code class="language-python">from datetime import timedelta
cell.value = timedelta(minutes=90)
cell.number_format = '[h]:mm;@'</code></pre></div>
<p>This number format ensures that your duration can be greater than '23:59', for example, '140:00'.</p>
<h2>Freezing Rows and Columns</h2>
<p>In Excel, you can freeze rows and columns so that they stay fixed when you scroll the content vertically or horizontally. That's similar to <code>position: fixed</code> in CSS.</p>
<p>To freeze the rows and columns, locate the top-left cell that is below the row that you want to freeze and is on the right from the column that you want to freeze. For example, if you want to freeze one row and one column, the cell would be 'B2'. Then run this:</p>
<div><pre><code class="language-python">worksheet.freeze_panes = worksheet['B2']</code></pre></div>
<h2>Fully Customized Export View</h2>
<p>So having the knowledge of this article now we can build a view that creates separate sheets for each movie category. Each sheet would list movies of the category with titles, descriptions, length in hours and minutes, rating in percent, and price in Euros. The tabs, as well as the headers, can have different background colors for each movie category. Cells would be well formatted. Titles and descriptions would use multiple lines to fully fit into the cells.</p>
<div><pre><code class="language-python"># movies/views.py
from datetime import datetime
from datetime import timedelta
from openpyxl import Workbook
from openpyxl.styles import Font, Alignment, Border, Side, PatternFill
from openpyxl.utils import get_column_letter
from django.http import HttpResponse
from .models import MovieCategory, Movie
def export_movies_to_xlsx(request):
"""
Downloads all movies as Excel file with a worksheet for each movie category
"""
category_queryset = MovieCategory.objects.all()
response = HttpResponse(
content_type='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
)
response['Content-Disposition'] = 'attachment; filename={date}-movies.xlsx'.format(
date=datetime.now().strftime('%Y-%m-%d'),
)
workbook = Workbook()
# Delete the default worksheet
workbook.remove(workbook.active)
# Define some styles and formatting that will be later used for cells
header_font = Font(name='Calibri', bold=True)
centered_alignment = Alignment(horizontal='center')
border_bottom = Border(
bottom=Side(border_style='medium', color='FF000000'),
)
wrapped_alignment = Alignment(
vertical='top',
wrap_text=True
)
# Define the column titles and widths
columns = [
('ID', 8),
('Title', 40),
('Description', 80),
('Length', 15),
('Rating', 15),
('Price', 15),
]
# Iterate through movie categories
for category_index, category in enumerate(category_queryset):
# Create a worksheet/tab with the title of the category
worksheet = workbook.create_sheet(
title=category.title,
index=category_index,
)
# Define the background color of the header cells
fill = PatternFill(
start_color=category.html_color,
end_color=category.html_color,
fill_type='solid',
)
row_num = 1
# Assign values, styles, and formatting for each cell in the header
for col_num, (column_title, column_width) in enumerate(columns, 1):
cell = worksheet.cell(row=row_num, column=col_num)
cell.value = column_title
cell.font = header_font
cell.border = border_bottom
cell.alignment = centered_alignment
cell.fill = fill
# set column width
column_letter = get_column_letter(col_num)
column_dimensions = worksheet.column_dimensions[column_letter]
column_dimensions.width = column_width
# Iterate through all movies of a category
for movie in category.movie_set.all():
row_num += 1
# Define data and formats for each cell in the row
row = [
(movie.pk, 'Normal'),
(movie.title, 'Normal'),
(movie.description, 'Normal'),
(timedelta(minutes=movie.length_in_minutes), 'Normal'),
(movie.rating / 100, 'Percent'),
(movie.price, 'Currency'),
]
# Assign values, styles, and formatting for each cell in the row
for col_num, (cell_value, cell_format) in enumerate(row, 1):
cell = worksheet.cell(row=row_num, column=col_num)
cell.value = cell_value
cell.style = cell_format
if cell_format == 'Currency':
cell.number_format = '#,##0.00 €'
if col_num == 4:
cell.number_format = '[h]:mm;@'
cell.alignment = wrapped_alignment
# freeze the first row
worksheet.freeze_panes = worksheet['A2']
# set tab color
worksheet.sheet_properties.tabColor = category.html_color
workbook.save(response)
return response</code></pre></div>
<h2>The Takeaways</h2>
<ul>
<li>Spreadsheet data can be used for further mathematical processing with <strong>pandas</strong>.</li>
<li>XLSX file format allows quite a bunch of formatting options that can make your spreadsheet data more presentable and user-friendly.</li>
<li>To see Excel export in action, go to <a href="https://www.1st-things-1st.com">1st things 1st</a>, log in as a demo user, and navigate to project results where you can export them as XLSX. Feedback is always welcome.</li>
</ul>
<hr>
<p>Cover photo by <a href="https://unsplash.com/photos/Uf-c4u1usFQ">Tim Evans</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495399999997552.21073109999999 12.759506999999974 52.829282099999986 14.050400999999976tag:blogger.com,1999:blog-9014457088394059918.post-42290621932392763862019-02-02T23:48:00.000+01:002019-11-04T06:51:14.624+01:00Equivalents in Python and JavaScript. Bonus<img border="0" src="https://3.bp.blogspot.com/-AbBqknEOT04/XFYb-oh0udI/AAAAAAAAB7w/d9zY2pG0O8IG2-D75GUkyT8RjckJU5FKACLcBGAs/s1600/equivalents-in-python-and-javascript-bonus.png" data-original-width="1024" data-original-height="512" />
<p>From time to time I google for the right syntax how to process lists and dictionaries in Python or arrays and objects in JavaScript. So I decided to extend my series of equivalents with those functions. After all, it's me too, who will be using the information I provide here.</p>
<h2>All truthful elements</h2>
<p>Sometimes we need to check from a list of conditions if all of them are true, or from a list of elements if all of them are not empty.</p>
<p>This can be checked with the following in <strong>Python</strong>:</p>
<div><pre><code class="language-python">items = [1, 2, 3]
all_truthy = all(items)
# True</code></pre></div>
<p>And here is an equivalent in <strong>JavaScript</strong>:</p>
<div><pre><code class="language-javascript">items = [1, 2, 3];
all_truthy = items.every(Boolean);
// true</code></pre></div>
<h2>Any truthful elements</h2>
<p>Similarly, we can check if at least one of the conditions is true, or there is at least one non-empty element in a list.</p>
<p>It <strong>Python</strong> we would do that with:</p>
<div><pre><code class="language-python">items = [0, 1, 2, 3]
some_truthy = any(items)
# True</code></pre></div>
<p>And in <strong>JavaScript</strong> we would check it like this:</p>
<div><pre><code class="language-javascript">items = [0, 1, 2, 3];
some_truthy = items.some(Boolean);
// true</code></pre></div>
<h2>Iterate through each element and its index</h2>
<p>Here is an example of how to iterate through a list of items and also check their indices in <strong>Python</strong>. It is useful for verbose console output when creating different command line tools that process data:</p>
<div><pre><code class="language-python">items = ['a', 'b', 'c', 'd']
for index, element in enumerate(items):
print(f'{index}: {element};')</code></pre></div>
<p>In <strong>JavaScript</strong> an analogous way to do the same would be using the <code>forEach()</code> method. The usual <code>for</code> loop is also an option, but I find the <code>forEach()</code> more elegant and clear.</p>
<div><pre><code class="language-javascript">items = ['a', 'b', 'c', 'd'];
items.forEach(function(element, index) {
console.log(`${index}: ${element};`);
});</code></pre></div>
<h2>Map elements to the results of a function</h2>
<p>To process all elements of a list, you can either iterate through them with the <code>for</code> loop and create a new list with modifications, or you can do that in one step by mapping the list items to a modification function. In <strong>Python</strong> this can be done with the <code>map()</code> function:</p>
<div><pre><code class="language-python">items = [0, 1, 2, 3]
all_doubled = list(map(lambda x: 2 * x, items))
# [0, 2, 4, 6]</code></pre></div>
<p>In <strong>JavaScript</strong> the <code>map()</code> is a method of an array:</p>
<div><pre><code class="language-javascript">items = [0, 1, 2, 3];
all_doubled = items.map(x => 2 * x);
// [0, 2, 4, 6]</code></pre></div>
<h2>Filter elements by a function</h2>
<p>When you need to search for some elements in a list or array and want to avoid <code>for</code> loops, you can use the filtering functionality. In <strong>Python</strong> that is doable with the <code>filter()</code> function that accepts the filtering function and the list and returns a new filtered list.</p>
<div><pre><code class="language-python">items = [0, 1, 2, 3]
only_even = list(filter(lambda x: x % 2 == 0, items))
# [0, 2]</code></pre></div>
<p>In <strong>JavaScript</strong> there is a <code>filter()</code> method of the array for that.</p>
<div><pre><code class="language-javascript">items = [0, 1, 2, 3];
only_even = items.filter(x => x % 2 === 0);
// [0, 2]</code></pre></div>
<p>In both cases, the filtering function checks each item if it is matching the filter criteria and returns true in that case.</p>
<h2>Reduce elements by a function to a single value</h2>
<p>When you want to apply some function to a list of items to get a single result in one go, you can use the reduce function. It works for summing, multiplying, ORing, ANDing, or checking maximums and minimums.</p>
<p>In <strong>Python</strong> there is a <code>reduce()</code> function for that.</p>
<div><pre><code class="language-python">from functools import reduce
items = [1, 2, 3, 4]
total = reduce(lambda total, current: total + current, items)
# 10</code></pre></div>
<p>In <strong>JavaScript</strong> there is a <code>reduce()</code> method of the array.</p>
<div><pre><code class="language-javascript">items = [1, 2, 3, 4];
total = items.reduce((total, current) => total + current);
// 10</code></pre></div>
<h2>Merge dictionaries</h2>
<p>There are multiple ways to merge dictionaries in Python or objects in JavaScript. But these are probably the simplest ones.</p>
<p>In <strong>Python</strong> it's decomposing dictionaries to tuples of keys and arrays, joining them, and creating a new dictionary.</p>
<div><pre><code class="language-python">d1 = {'a': 'A', 'b': 'B'}
d2 = {'a': 'AAA', 'c': 'CCC'}
merged = dict(list(d1.items()) + list(d2.items()))
# {'a': 'AAA', 'b': 'B', 'c': 'CCC'}</code></pre></div>
<p>Analogously, in <strong>JavaScript</strong> it's spreading two objects into a new object:</p>
<div><pre><code class="language-javascript">d1 = {a: 'A', b: 'B'}
d2 = {a: 'AAA', c: 'CCC'}
merged = {...d1, ...d2};
// {a: 'AAA', b: 'B', c: 'CCC'}</code></pre></div>
<h2>The Takeaways</h2>
<ul>
<li>In both languages, you can traverse through lists of items without explicitly incrementing and referencing an index.</li>
<li>For processing list items, you don't necessarily need a loop. The dedicated methods or functions <code>all() / every()</code>, <code>any() / some()</code>, <code>map()</code>, <code>filter()</code>, and <code>reduce()</code> are there to help you.</li>
<li>In both languages, you can merge multiple dictionaries into one. If the same key appears in several dictionaries, the latest one will be used in the merged dictionary.</li>
</ul>
<p>Of course, I also updated the cheat sheet with the full list of equivalents in <strong>Python</strong> and <strong>JavaScript</strong> that you saw here described. This cheat sheet helps me with a good overview next to my laptop, so I believe that it would be helpful to you too. The new revision 10 is with syntax highlighting, so it makes it even better to explore and understand.</p>
<p><a href="https://www.djangotricks.com/goodies/YbnpiLKBmAZi/" target="_blank">Get the Ultimate Cheat Sheet of Equivalents in Python and JavaScript</a></p>
<p>Use it for good!</p>
<hr>
<p>Cover photo by <a href="https://unsplash.com/photos/0nHOfjH0emE">Darren Chan</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com1tag:blogger.com,1999:blog-9014457088394059918.post-21932833604013467482019-01-11T16:10:00.000+01:002019-01-12T23:15:39.643+01:00How to Create PDF Documents with Django in 2019<img border="0" src="https://1.bp.blogspot.com/-7ryrK96vtEc/XDgT8U3TaSI/AAAAAAAAB7k/IfjBWM8qyk8Q-hpRRTgW9Upq4EvT3DJcgCLcBGAs/s1600/how-to-create-pdf--documents-with-django-in-2019.png" data-original-width="1024" data-original-height="512" />
<p>If you've read my <strong>Web Development with Django Cookbook</strong>, you might remember a recipe for creating PDF documents using <strong>Pisa xhtml2pdf</strong>. Well, this library does its job, but it supports only a subset of HTML and CSS features. For example, for multi-column layouts, you have to use tables, like it's 1994.</p>
<p>I needed some fresh and flexible option to generate donation receipts for the <a href="http://www.make-impact.org/">donation platform <strong>www.make-impact.org</strong></a> and reports for <a href="https://www.1st-things-1st.com">the strategic planner <strong>1st things 1st</strong></a> I have been building. After a quick research I found another much more suitable library. It's called <strong><a href="http://weasyprint.org/">WeasyPrint</a></strong>. In this article, I will tell you how to use it with Django and what's valuable in it.</p>
<h2>Features</h2>
<p><strong>WeasyPrint</strong> uses HTML and CSS 2.1 to create pixel-perfect, or let's rather say point-perfect, PDF documents. <strong>WeasyPrint</strong> doesn't use WebKit or Gecko but has its own rendering engine. As a proof that it works correctly, it passes the famous among web developers Acid2 test which was created back in the days before HTML5 to check how compatible browsers are with CSS 2 standards.</p>
<p>All supported features (and unsupported exceptions) are listed in <a href="http://weasyprint.readthedocs.io/en/latest/features.html">the documentation</a>. But my absolute favorites are these:</p>
<ul>
<li><strong>Layouts with floated elements.</strong> You don't have to use tables anymore if you want to have the recipient address on the left side and the sender information on the right side in a letter, or if you want to have the main content and the side notes in an exercise book. Just use floated elements.</li>
<li><strong>Working links.</strong> The generated document can have clickable links to external URLs and internal anchors. You can straightforwardly create a clickable table of contents or a banner that leads back to your website.</li>
<li><strong>Support for web fonts.</strong> With the wide variety of embeddable web fonts, your documents don't need to look boring anymore. Why not write titles in elegant cursive or in bold western letters?</li>
<li><strong>Background images.</strong> By default, when you print an HTML page, all foreground images get printed, but the backgrounds are skipped. When you generate a PDF document for printing, you can show background images anywhere, even in the margins of the printed page.</li>
<li><strong>SVG kept as vector images.</strong> When you have diagrams and graphics in a PDF document, you usually want to preserve the quality of the lines. Even if they look good on the screen, raster images might be not what you want, because on a printed page the resolution will differ and the quality can be lost. WeasyPrint keeps SVG images as vector images, so you have the highest possible quality in the prints.</li>
</ul>
<h2>Important Notes</h2>
<p>WeasyPrint needs <strong>Python 3.4</strong> or newer. That's great for new Django projects, but might be an obstacle if you want to integrate it into an existing website running on Python 2.7. Can it be the main argumentation for you to upgrade your old Django projects to the new Python version?</p>
<p>WeasyPrint is <strong>dependent on several OS libraries</strong>: Pango, GdkPixbuf, Cairo, and Libffi. In the documentation, there are <a href="http://weasyprint.readthedocs.io/en/latest/install.html">understandable one-line instructions</a> how to install them on different operating systems. You can have a problem only if you don't have full control of the server where you are going to deploy your project.</p>
<p>If you need some <strong>basic headers and footers</strong> for all pages, you can use <code>@page</code> CSS selector for that. If you need <strong>extended headers and footers</strong> for each page, it's best to combine the PDF document out of separate HTML documents for each page. Examples follow below.</p>
<p>The fun fact, <strong>Emojis</strong> are drawn using some weird raster single-color font. I don't recommend using them in your PDFs unless you replace them with SVG images.</p>
<h2>Show Me the Code</h2>
<p>A technical article is always more valuable when it has some quick code snippets to copy and paste. Here you go!</p>
<h3>Simple PDF View</h3>
<p>This snippet generates a donation receipt and shows it directly in the browser. Should the PDF be downloadable immediately, change content disposition from <code>inline</code> to <code>attachment</code>.</p>
<div><pre><code class="language-python"># -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from django.http import HttpResponse
from django.template.loader import render_to_string
from django.utils.text import slugify
from django.contrib.auth.decorators import login_required
from weasyprint import HTML
from weasyprint.fonts import FontConfiguration
from .models import Donation
@login_required
def donation_receipt(request, donation_id):
donation = get_object_or_404(Donation, pk=donation_id, user=request.user)
response = HttpResponse(content_type="application/pdf")
response['Content-Disposition'] = "inline; filename={date}-{name}-donation-receipt.pdf".format(
date=donation.created.strftime('%Y-%m-%d'),
name=slugify(donation.donor_name),
)
html = render_to_string("donations/receipt_pdf.html", {
'donation': donation,
})
font_config = FontConfiguration()
HTML(string=html).write_pdf(response, font_config=font_config)
return response</code></pre></div>
<h3>Page Configuration Using CSS</h3>
<p>Your PDF document can have a footer with an image and text on every page, using <code>background-image</code> and <code>content</code> properties:</p>
<div><pre><code class="language-markup">{% load staticfiles i18n %}
<link href="https://fonts.googleapis.com/css?family=Playfair+Display:400,400i,700,700i,900" rel="stylesheet" />
<style>
@page {
size: "A4";
margin: 2.5cm 1.5cm 3.5cm 1.5cm;
@bottom-center {
background: url({% static 'site/img/logo-pdf.svg' %}) no-repeat center top;
background-size: auto 1.5cm;
padding-top: 1.8cm;
content: "{% trans "Donation made via www.make-impact.org" %}";
font: 10pt "Playfair Display";
text-align: center;
vertical-align: top;
}
}
</style></code></pre></div>
<h3>Pagination</h3>
<p>You can show page numbers in the footer using CSS as follows.</p>
<div><pre><code class="language-css">@page {
margin: 3cm 2cm;
@top-center {
content: "Documentation";
}
@bottom-right {
content: "Page " counter(page) " of " counter(pages);
}
}</code></pre></div>
<h3>Horizontal Page Layout</h3>
<p>You can rotate the page to horizontal layout with <code>size: landscape</code>.</p>
<div><pre><code class="language-css">@page {
size: landscape;
}</code></pre></div>
<h3>HTML-based Footer</h3>
<p>Another option to show an image and text in the header or footer on every page is to use an HTML element with <code>position: fixed</code>. This way you have more flexibility about formatting, but the element on all your pages will have the same content.</p>
<div><pre><code class="language-markup"><style>
footer {
position: fixed;
bottom: -2.5cm;
width: 100%;
text-align: center;
font-size: 10pt;
}
footer img {
height: 1.5cm;
}
</style></code></pre></div>
<div><pre><code class="language-markup"><footer>
{% with website_url="https://www.make-impact.org" %}
<a href="{{ website_url }}">
<img alt="" src="{% static 'site/img/logo-contoured.svg' %}" />
</a><br />
{% blocktrans %}Donation made via <a href="{{ website_url }}">www.make-impact.org</a>{% endblocktrans %}
{% endwith %}
</footer></code></pre></div>
<h3>Document Rendering from Page to Page</h3>
<p>When you need to have a document with complex unique headers and footers, it is best to render each page as a separate HTML document and then to combine them into one. This is how to do that:</p>
<div><pre><code class="language-python">def letter_pdf(request, letter_id):
letter = get_object_or_404(Letter, pk=letter_id)
response = HttpResponse(content_type='application/pdf')
response['Content-Disposition'] = (
'inline; '
f'filename={letter.created:%Y-%m-%d}-letter.pdf'
)
COMPONENTS = [
'letters/pdf/cover.html',
'letters/pdf/page01.html',
'letters/pdf/page02.html',
'letters/pdf/page03.html',
]
documents = []
font_config = FontConfiguration()
for template_name in COMPONENTS:
html = render_to_string(template_name, {
'letter': letter,
})
document = HTML(string=html).render(font_config=font_config)
documents.append(document)
all_pages = [page for document in documents for page in document.pages]
documents[0].copy(all_pages).write_pdf(response)
return response</code></pre></div>
<h2>Final Thoughts</h2>
<p>I believe that <strong>WeasyPrint</strong> could be used not only for invoices, tickets, or booking confirmations but also for online magazines and small booklets. If you want to see PDF rendering with WeasyPrint in action, make a donation to your chosen organization at <a href="http://www.make-impact.org">www.make-impact.org</a> (when it's ready) and download the donation receipt. Or check the demo account at <a href="https://my.1st-things-1st.com">my.1st-things-1st.com</a> and find the button to download the results of a prioritization project as PDF document.</p>
<hr>
<p>Cover photo by <a href="https://unsplash.com/photos/FgK0yuynevY">Daniel Korpai</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com1tag:blogger.com,1999:blog-9014457088394059918.post-45920182023933547422018-12-19T03:53:00.000+01:002018-12-19T03:53:14.787+01:00What's New in the Third Edition of Web Development with Django Cookbook?<img border="0" src="https://1.bp.blogspot.com/-1k5hzAEhLP8/XBmwNxD6hhI/AAAAAAAAB7I/Y5MGx3XPX9ksf1lEPTntMeuesixgoskewCLcBGAs/s1600/whats-new-in-the-third-edition-of-web-development-with-django-cookbook.png" data-original-width="1024" data-original-height="512" />
<p>A couple of months ago the third release of Django Cookbook was published under the title <a href="https://www.packtpub.com/web-development/django-2-web-development-cookbook-third-edition">Django 2 Web Development Cookbook - Third Edition</a>. This edition was thoroughly and progressively re-written by Jake Kronika, the guy who had reviewed my second edition and had added a lot of value to it. I was sure that he wouldn't disappoint the readers, so I invited him to write the update. In this article, I will guide you through the main new highlights of over 500 pages of this new book.</p>
<h2>Up to Date</h2>
<p>Just like <a href="https://wsvincent.com/books/">William S. Vincent's books</a>, Django 2 Web Development Cookbook - Third Edition is adapted to Django 2.1 and Python 3.6. So you will be dealing with the state of the art technologies building your Django projects. Unicode strings, f-strings, <code>super()</code> without parameters, HTML5 tags, and object-oriented JavaScript to mention a few are used all over the book. The code is carefully generalized and even more adapted to the Don't-Repeat-Yourself (DRY) principle.</p>
<h2>Working with Docker</h2>
<p>Docker is one of the most popular deployment technologies and Jake gives a good compact introduction how to use it with Django.</p>
<h2>Using Environment Variables for Configuration</h2>
<p><a href="https://12factor.net/config">12-factor app guidelines</a> suggest saving app configuration in environment variables. In the book, there is a practical example of how to use it.</p>
<h2>Multilingual Fields even with Region-specific Language Codes</h2>
<p>I introduced multilingual fields in previous editions of the book, but there they had a limitation, that region-specific languages like Australian English or Swiss German were not supported. Now they are!</p>
<h2>Using Precisely Semantic Markup with schema.org Microdata</h2>
<p>Schema.org Microdata allows you to define the context of the content more specifically so that the content is more machine-readable. This was new to me and I still don't know the exact practical value of it, but I guess it is related to accessibility, new ways of presenting data via plugins, and Artificial Intelligence.</p>
<h2>Defining Custom Templates for the Default Django Form Fields</h2>
<p>Since Django 1.11 form fields are rendered using templates instead of Python code and those templates can be customized. There is a recipe that shows you how to do that.</p>
<h2>Providing Responsive Images</h2>
<p>HTML5 has the <code><picture></code> tag with <code><source></code> children that can be used in combination with the <code>sorl-thumbnail</code> Python package to generate different versions of the image based on your viewport size: load small image on the mobile, middle image on the tablet, and big image on the desktop or smart TV.</p>
<h2>Uploading Images and Deleting them by Ajax</h2>
<p>In my previous editions, I only showed how to upload a file by Ajax and attach it to a Django model. In Jake's update, it is shown how you can also delete the image.</p>
<h2>Validating Passwords with Special Requirements</h2>
<p>Since Django 1.11 you can define special requirements for the passwords of your users, for example, have a mix of small and big letters or include at least 1 number and 3 special characters, etc. There is a practical recipe how to do that.</p>
<h2>Adding Watermarks to Images</h2>
<p>When it comes to branding or copyright protection, it is common to add special watermarks, semitransparent images on top of your normal pictures. Jake added an example, how to do that and it was very interesting to me.</p>
<h2>Authenticating with Auth0</h2>
<p>In one recipe it is shown how to login to a Django website using Auth0, which seems to be a passwordless authentication system with integrations of popular connection services like OpenID Connect, Facebook, Google, Github, LinkedIn, PayPal, Yahoo!, and others. I haven't tried that myself yet, but it can be an interesting option for a social website. </p>
<h2>Using Redis for Caching</h2>
<p>It is common to cache websites using Memcached service, but a good alternative is caching with Redis and django-redis. Moreover, you can easily save user sessions there.</p>
<h2>Creating Hierarchies with django-treebeard</h2>
<p>In the previous editions, I introduced django-mptt for creating hierarchical structures. However, recently many projects are moving towards its alternative - django-treebeard which has more stability and writing speed. In the book, Jake shows you how to work with it.</p>
<h2>Conclusion</h2>
<p>There was a lot of new things to learn. For example, for me personally Docker usage was new, and I haven't heard of schema.org microdata and Auth0 which were introduced in this book. All in all, I think, Jake Kronika did an enormous job with this update and it's really worth purchasing this book, especially as there is a winter-holidays sale where you can <a href="https://www.packtpub.com/web-development/django-2-web-development-cookbook-third-edition">get the EPUB, MOBI, and PDF with the code examples</a> just for ~ 5 €.</p>
<p>Have a nice Christmas time and come back to this blog next year!</p>
<hr>
<p>Cover photo by <a href="https://unsplash.com/search/photos/book-shelf">chuttersnap</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0tag:blogger.com,1999:blog-9014457088394059918.post-50414386887777475222018-07-09T09:00:00.000+02:002019-11-04T06:50:54.343+01:00Equivalents in Python and JavaScript. Part 4<img border="0" src="https://3.bp.blogspot.com/-IUXEO64Y6q8/WzrC947Je8I/AAAAAAAAB58/KyzGfiel_6AxNOHLu4lHsZhTPyznTvmfQCLcBGAs/s1600/equivalents-in-python-and-javascript-part-4.png" data-original-width="1024" data-original-height="512" />
<p>In the last three parts of the series of articles about analogies in <strong>Python</strong> and <strong>JavaScript</strong>, we explored lots of interesting concepts like serializing to JSON, error handling, using regular expressions, string interpolation, generators, lambdas, and many more. This time we will delve into function arguments, creating classes, using class inheritance, and defining getters and setters of class properties.</p>
<h2>Function arguments</h2>
<p><strong>Python</strong> is very flexible with argument handling for functions: you can set default values there, allow a flexible amount of positional or keyword arguments (<code>*args</code> and <code>**kwargs</code>). When you pass values to a function, you can define by name to which argument that value should be assigned. All that in a way is now possible in <strong>JavaScript</strong> too.</p>
<p>Default values for function arguments in <strong>Python</strong> can be defined like this:</p>
<div><pre><code class="language-python">from pprint import pprint
def report(post_id, reason='not-relevant'):
pprint({'post_id': post_id, 'reason': reason})
report(42)
report(post_id=24, reason='spam')</code></pre></div>
<p>In <strong>JavaScript</strong> that can be achieved similarly:</p>
<div><pre><code class="language-javascript">function report(post_id, reason='not-relevant') {
console.log({post_id: post_id, reason: reason});
}
report(42);
report(post_id=24, reason='spam');</code></pre></div>
<hr />
<p>Positional arguments in <strong>Python</strong> can be accepted using the <code>*</code> operator like this:</p>
<div><pre><code class="language-python">from pprint import pprint
def add_tags(post_id, *tags):
pprint({'post_id': post_id, 'tags': tags})
add_tags(42, 'python', 'javascript', 'django')</code></pre></div>
<p>In <strong>JavaScript</strong> positional arguments can be accepted using the <code>...</code> operator:</p>
<div><pre><code class="language-javascript">function add_tags(post_id, ...tags) {
console.log({post_id: post_id, tags: tags});
}
add_tags(42, 'python', 'javascript', 'django'); </code></pre></div>
<hr />
<p>Keyword arguments are often used in <strong>Python</strong> when you want to allow a flexible amount of options:</p>
<div><pre><code class="language-python">from pprint import pprint
def create_post(**options):
pprint(options)
create_post(
title='Hello, World!',
content='This is our first post.',
is_published=True,
)
create_post(
title='Hello again!',
content='This is our second post.',
)</code></pre></div>
<p>A common practice to pass multiple optional arguments to a <strong>JavaScript</strong> function is through a dictionary object, for example, <code>options</code>.</p>
<div><pre><code class="language-javascript">function create_post(options) {
console.log(options);
}
create_post({
'title': 'Hello, World!',
'content': 'This is our first post.',
'is_published': true
});
create_post({
'title': 'Hello again!',
'content': 'This is our second post.'
});
</code></pre></div>
<h2>Classes and inheritance</h2>
<p>Python is an object-oriented language. Since ECMAScript 6 standard support, it's also possible to write object-oriented code in <strong>JavaScript</strong> without hacks and weird prototype syntax.</p>
<p>In <strong>Python</strong> you would create a class with the constructor and a method to represent its instances textually like this:</p>
<div><pre><code class="language-python">class Post(object):
def __init__(self, id, title):
self.id = id
self.title = title
def __str__(self):
return self.title
post = Post(42, 'Hello, World!')
isinstance(post, Post) == True
print(post) # Hello, World!</code></pre></div>
<p>In <strong>JavaScript</strong> to create a class with the constructor and a method to represent its instances textually, you would write:</p>
<div><pre><code class="language-javascript">class Post {
constructor (id, title) {
this.id = id;
this.title = title;
}
toString() {
return this.title;
}
}
post = new Post(42, 'Hello, World!');
post instanceof Post === true;
console.log(post.toString()); // Hello, World!</code></pre></div>
<hr />
<p>Now we can create two classes <code>Article</code> and <code>Link</code> in <strong>Python</strong> that will extend the <code>Post</code> class. Here you can also see how we are using <code>super</code> to call methods from the base <code>Post</code> class.</p>
<div><pre><code class="language-python">class Article(Post):
def __init__(self, id, title, content):
super(Article, self).__init__(id, title)
self.content = content
class Link(Post):
def __init__(self, id, title, url):
super(Link, self).__init__(id, title)
self.url = url
def __str__(self):
return '{} ({})'.format(
super(Link, self).__str__(),
self.url,
)
article = Article(1, 'Hello, World!', 'This is my first article.')
link = Link(2, 'DjangoTricks', 'https://djangotricks.blogspot.com')
isinstance(article, Post) == True
isinstance(link, Post) == True
print(link)
# DjangoTricks (https://djangotricks.blogspot.com)</code></pre></div>
<p>In <strong>JavaScript</strong> the same is also doable by the following code:</p>
<div><pre><code class="language-javascript">class Article extends Post {
constructor (id, title, content) {
super(id, title);
this.content = content;
}
}
class Link extends Post {
constructor (id, title, url) {
super(id, title);
this.url = url;
}
toString() {
return super.toString() + ' (' + this.url + ')';
}
}
article = new Article(1, 'Hello, World!', 'This is my first article.');
link = new Link(2, 'DjangoTricks', 'https://djangotricks.blogspot.com');
article instanceof Post === true;
link instanceof Post === true;
console.log(link.toString());
// DjangoTricks (https://djangotricks.blogspot.com)</code></pre></div>
<h2>Class properties: getters and setters</h2>
<p>In object oriented programming, classes can have attributes, methods, and properties. Properties are a mixture of attributes and methods. You deal with them as attributes, but in the background they call special getter and setter methods to process data somehow before setting or returning to the caller.</p>
<p>The basic wireframe for getters and setters of the <code>slug</code> property in <strong>Python</strong> would be like this:</p>
<div><pre><code class="language-python">class Post(object):
def __init__(self, id, title):
self.id = id
self.title = title
self._slug = ''
@property
def slug(self):
return self._slug
@slug.setter
def slug(self, value):
self._slug = value
post = new Post(1, 'Hello, World!')
post.slug = 'hello-world'
print(post.slug)</code></pre></div>
<p>In <strong>JavaScript</strong> getters and setters for the <code>slug</code> property can be defined as:</p>
<div><pre><code class="language-javascript">class Post {
constructor (id, title) {
this.id = id;
this.title = title;
this._slug = '';
}
set slug(value) {
this._slug = value;
}
get slug() {
return this._slug;
}
}
post = new Post(1, 'Hello, World!');
post.slug = 'hello-world';
console.log(post.slug);</code></pre></div>
<h2>The Takeaways</h2>
<ul>
<li>In both languages, you can define default argument values for functions.</li>
<li>In both languages, you can pass a flexible amount of positional or keyword arguments for functions.</li>
<li>In both languages, object-oriented programming is possible.</li>
</ul>
<p>As you might have noticed, I am offering a cheat sheet with the full list of equivalents in <strong>Python</strong> and <strong>JavaScript</strong> that you saw here described. At least for me, it is much more convenient to have some printed sheet of paper with valuable information next to my laptop, rather than switching among windows or tabs and scrolling to get the right piece of snippet. So I encourage you to get this cheat sheet and improve your programming!</p>
<p><center><a href="https://www.djangotricks.com/goodies/YbnpiLKBmAZi/" target="_blank">Get the Ultimate Cheat Sheet of Equivalents in Python and JavaScript</a></center></p>
<p>Use it for good!</p>
<hr />
<p>Cover photo by <a href="https://unsplash.com/photos/Mn9Fa_wQH-M">Andre Benz</a></p>Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com2Berlin, Germany52.520006599999988 13.40495399999997552.21073109999999 12.759506999999974 52.829282099999986 14.050400999999976tag:blogger.com,1999:blog-9014457088394059918.post-77984277114883900372018-07-06T09:00:00.000+02:002019-11-04T06:50:28.674+01:00Equivalents in Python and JavaScript. Part 3<img border="0" src="https://1.bp.blogspot.com/-QOkE0Sse2NU/WzrCwrbe4qI/AAAAAAAAB54/Y0P1FNJqDc4gnR1JiAjv4MLGllcrZDFvACLcBGAs/s1600/equivalents-in-python-and-javascript-part-3.png" data-original-width="1024" data-original-height="512" />
<p>This is the 3rd part of 4-article series about analogies in <strong>Python</strong> and <strong>JavaScript</strong>. In <a href="https://djangotricks.blogspot.com/2018/07/equivalents-in-python-and-javascript-part-2.html">the previous parts</a> we covered a large part of the traditional <strong>Python</strong> 2.7 and <strong>JavaScript</strong> based on the ECMAScript 5 standard. This time we will start looking into <strong>Python</strong> 3.6 and <strong>JavaScript</strong> based on the ECMAScript 6 standard. ECMAScript 6 standard is pretty new and supported only the newest versions of browsers. For older browsers you will need <a href="https://babeljs.io/">Babel</a> to compile your next-generation <strong>JavaScript</strong> code to the cross-browser-compatible equivalents. It opens the door to so many interesting things to explore. We will start from string interpolation, unpacking lists, lambda functions, iterations without indexes, generators, and sets!</p>
<h2>Variables in strings</h2>
<p>The old and inefficient way to build strings with values from variables is this concatenation:</p>
<div><pre><code class="language-python">name = 'World'
value = 'Hello, ' + name + '!\nWelcome!'</code></pre></div>
<p>This can get very sparse and difficult to read. Also it is very easy to miss whitespaces in the sentence around variables.</p>
<p>Since <strong>Python</strong> version 3.6 and <strong>JavaScript</strong> based on the ECMAScript 6 standard, you can use so called string interpolation. These are string templates which are filled in with values from variables.</p>
<p>In <strong>Python</strong> they are also called f-string, because their notation starts with letter "f": </p>
<div><pre><code class="language-python">name = 'World'
value = f"""Hello, {name}!
Welcome!"""
price = 14.9
value = f'Price: {price:.2f} €' # 'Price: 14.90 €'</code></pre></div>
<p>In <strong>JavaScript</strong> string templates start and end with backticks:</p>
<div><pre><code class="language-javascript">name = 'World';
value = `Hello, ${name}!
Welcome!`;
price = 14.9;
value = `Price ${price.toFixed(2)} €`; // 'Price: 14.90 €'</code></pre></div>
<p>Note that string templates can be of a single line as well as of multiple lines. For f-strings in <strong>Python</strong> you can pass the format for variables, but you can't call methods of a variable unless they are properties and call getter methods.</p>
<h2>Unpacking lists</h2>
<p>Python and now <strong>JavaScript</strong> has an interesting feature to assign items of sequences into separate variables. For example, we can read the three values of a list into variables a, b, and c with the following syntax:</p>
<div><pre><code class="language-python">[a, b, c] = [1, 2, 3]</code></pre></div>
<hr />
<p>For tuples the parenthesis can be omitted. The following is a very popular way to swap values of two variables in <strong>Python</strong>:</p>
<div><pre><code class="language-python">a = 1
b = 2
a, b = b, a # swap values</code></pre></div>
<p>With the next generation <strong>JavaScript</strong> this can also be achieved:</p>
<div><pre><code class="language-javascript">a = 1;
b = 2;
[a, b] = [b, a]; // swap values</code></pre></div>
<hr />
<p>In <strong>Python</strong> 3.6 if we have an unknown number of items in a list or tuple, we can assign them to a tuple of several values while also unpacking the rest to a list:</p>
<div><pre><code class="language-python">first, second, *the_rest = [1, 2, 3, 4]
# first == 1
# second == 2
# the_rest == [3, 4]</code></pre></div>
<p>This can also be done with <strong>JavaScript</strong> (ECMAScript 6):</p>
<div><pre><code class="language-javascript">[first, second, ...the_rest] = [1, 2, 3, 4];
// first === 1
// last === 2
// the_rest === [3, 4]</code></pre></div>
<h2>Lambda functions</h2>
<p><strong>Python</strong> and <strong>JavaScript</strong> have a very neat functionality to create functions in a single line. These functions are called lambdas. Lambdas are very simple functions that take one or more arguments and return some calculated value. Usually lambdas are used when you need to pass a function to another function as a callback or as a function to manipulate every separate elements in a sequence. </p>
<p>In <strong>Python</strong>, you would define a lambda using the <code>lambda</code> keyword, like this:</p>
<div><pre><code class="language-python">sum = lambda x, y: x + y
square = lambda x: x ** 2</code></pre></div>
<p>In <strong>JavaScript</strong> lambdas use the <code>=></code> notation. If there are more than one arguments, they have to be in parenthesis:</p>
<div><pre><code class="language-javascript">sum = (x, y) => x + y;
square = x => Math.pow(x, 2);</code></pre></div>
<h2>Iteration without indexes</h2>
<p>Many programming languages allow iterating through a sequence only by using indexes and incrementing their values. Then to get an item at some position, you would read it from an array, for example:</p>
<div><pre><code class="language-javascript">for (i=0; i<items.length; i++) {
console.log(items[i]);
}</code></pre></div>
<p>This is not a nice syntax and is very technical - it doesn't read naturally. What we really want is just to grab each value from the list. And <strong>Python</strong> has a very neat possibility just to iterate through the elements:</p>
<div><pre><code class="language-python">for item in ['A', 'B', 'C']:
print(item)</code></pre></div>
<p>In the modern <strong>JavaScript</strong> this is also possible with the <code>for..of</code> operator:</p>
<div><pre><code class="language-javascript">for (let item of ['A', 'B', 'C']) {
console.log(item);
}</code></pre></div>
<hr />
<p>You can also iterate through a string characters in <strong>Python</strong>:</p>
<div><pre><code class="language-python">for character in 'ABC':
print(character)</code></pre></div>
<p>And in the modern <strong>JavaScript</strong>:</p>
<div><pre><code class="language-javascript">for (let character of 'ABC') {
console.log(character);
}</code></pre></div>
<h2>Generators</h2>
<p>Python and modern <strong>JavaScript</strong> has a possibility to define special functions through which you can iterate. With each iteration they return the next generated value in a sequence. These functions are called generators. With generators you can get numbers in a range, lines from a file, data from different paginated API calls, <a href="https://www.youtube.com/watch?v=T8xgfVzef_E">fibonacci numbers</a>, and any other dynamicly generated sequences.</p>
<p>Technically generators are like normal functions, but instead of returning a value, they yield a value. This value will be returned for one iteration. And this generation happens as long as the end of the function is reached.</p>
<p>To illustrate that, the following <strong>Python</strong> code will create a generator <code>countdown()</code> which will return numbers from the given one back to 1, (like 10, 9, 8, ..., 1):</p>
<div><pre><code class="language-python">def countdown(counter):
while counter > 0:
yield counter
counter -= 1
for counter in countdown(10):
print(counter)</code></pre></div>
<p>Exactly the same can be achieved in modern <strong>JavaScript</strong>, but notice the asterisk at the <code>function</code> statement. It defines that it's a generator:</p>
<div><pre><code class="language-javascript">function* countdown(counter) {
while (counter > 0) {
yield counter;
counter--;
}
}
for (let counter of countdown(10)) {
console.log(counter);
}</code></pre></div>
<h2>Sets</h2>
<p>We already had a look at lists, tuples and arrays. But here is another type of data - sets. Sets are groups of elements that ensure that each element there exists only once. Set theory also specifies set operations like <a href="http://2ality.com/2015/01/es6-set-operations.html">union, intersection, and difference</a>, but we won't cover them here today.</p>
<p>This is how to create a set, add elements to it, check if a value exists, check the total amount of elements in a set, and iterate through its values, and remove a value using <strong>Python</strong>:</p>
<div><pre><code class="language-python">s = set(['A'])
s.add('B'); s.add('C')
'A' in s
len(s) == 3
for elem in s:
print(elem)
s.remove('C')</code></pre></div>
<p>Here is how to achieve the same in modern <strong>JavaScript</strong>:</p>
<div><pre><code class="language-javascript">s = new Set(['A']);
s.add('B').add('C');
s.has('A') === true;
s.size === 3;
for (let elem of s.values()) {
console.log(elem);
}
s.delete('C')</code></pre></div>
<h2>The Takeaways</h2>
<ul>
<li>String interpolation or literal templates allows you to have much cleaner and nicer code even with a possibility to have multiple lines of text.</li>
<li>You can iterate through elements in a sequence or group without using indexes.</li>
<li>Use generators when you have a sequence of nearly unlimited elements.</li>
<li>Use sets when you want to ensure fast check if data exists in a collection.</li>
<li>Use lambdas when you need short and clear single-line functions. </li>
</ul>
<p>As you know from the previous parts, I am offering a cheat sheet with the whole list of equivalents in <strong>Python</strong> and <strong>JavaScript</strong>, both, time honored and future proof. To have something printed in front of your eyes is much more convenient than switching among windows or scrolling up and down until you find what you exactly were searching for. So I suggest you to get the cheat sheet and use it for good!</p>
<p><center><a href="https://www.djangotricks.com/goodies/YbnpiLKBmAZi/" target="_blank">Get the Ultimate Cheat Sheet of Equivalents in Python and JavaScript</a></center></p>
<p>In the <a href="https://djangotricks.blogspot.com/2018/07/equivalents-in-python-and-javascript-part-4.html">next and last part</a> of the series, we will have a look at function arguments, classes, inheritance, and properties. Stay tuned!</p>
<hr />
<p>Cover photo by <a href="https://unsplash.com/photos/Akz00I_GGjU">Alex Knight</a></p>Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495399999997552.21073109999999 12.759506999999974 52.829282099999986 14.050400999999976tag:blogger.com,1999:blog-9014457088394059918.post-53491092164408542692018-07-03T09:00:00.000+02:002019-11-04T06:50:05.182+01:00Equivalents in Python and JavaScript. Part 2<img border="0" src="https://4.bp.blogspot.com/-ZEZgva-20Og/WzrCcYKB-NI/AAAAAAAAB5w/8Pv9p7uP_kYXxEL-Y_wkFxvTtU_ZXJK6gCLcBGAs/s1600/equivalents-in-python-and-javascript-part-2.png" data-original-width="1024" data-original-height="512" />
<p><a href="https://djangotricks.blogspot.com/2018/06/equivalents-in-python-and-javascript-part-1.html">Last time</a> we started a new series of articles about analogies in <strong>Python</strong> and <strong>JavaScript</strong>. We had a look at lists, arrays, dictionaries, objects, and strings, conditional assignments, and parsing integers. This time we will go through more interesting and more complex things like serializing dictionaries and lists to JSON, operations with regular expressions, as well as raising and catching errors.</p>
<h2>JSON</h2>
<p>When working with APIs it is very usual to serialize objects to JSON format and be able to parse JSON strings.</p>
<p>In <strong>Python</strong> it is done with the <code>json</code> module like this:</p>
<div><pre><code class="language-python">import json
json_data = json.dumps(dictionary, indent=4)
dictionary = json.loads(json_data)</code></pre></div>
<p>Here we'll indent the nested elements in the JSON string by 4 spaces.</p>
<p>In <strong>JavaScript</strong> there is a <code>JSON</code> object that has methods to create and parse JSON strings:</p>
<div><pre><code class="language-javascript">json_data = JSON.stringify(dictionary, null, 4);
dictionary = JSON.parse(json_data);</code></pre></div>
<h2>Splitting strings by regular expressions</h2>
<p>Regular expressions are multi-tool that once you master, you can accomplish lots of things.</p>
<p>In the last article, we saw how one can join lists of strings into a single string. But how can you split a long string into lists of strings? What if the delimiter can be not a single character as the comma, but a range of possible variations? This can be done with regular expressions and the <code>split()</code> method.</p>
<p>In <strong>Python</strong>, the <code>split()</code> method belongs to the regular expression pattern object. This is how you could split a text string into sentences by punctuation marks:</p>
<div><pre><code class="language-python">import re
# One or more characters of "!?." followed by whitespace
delimiter = re.compile(r'[!?\.]+\s*')
text = "Hello!!! What's new? Follow me."
sentences = delimiter.split(text)
# sentences == ['Hello', "What's new", 'Follow me', '']</code></pre></div>
<p>In <strong>JavaScript</strong> the <code>split()</code> method belongs to the string:</p>
<div><pre><code class="language-javascript">// One or more characters of "!?." followed by whitespace
delimiter = /[!?\.]+\s*/;
text = "Hello!!! What's new? Follow me.";
sentences = text.split(delimiter)
// sentences === ["Hello", "What's new", "Follow me", ""]</code></pre></div>
<h2>Matching regular expression patterns in strings</h2>
<p>Regular expressions are often used to validate data from the forms.</p>
<p>For example, to validate if the entered email address is correct, you would need to match it against a regular expression pattern. In <strong>Python</strong> that would look like this:</p>
<div><pre><code class="language-python">import re
# name, "@", and domain
pattern = re.compile(r'([\w.+\-]+)@([\w\-]+\.[\w\-.]+)')
match = pattern.match('hi@example.com')
# match.group(0) == 'hi@example.com'
# match.group(1) == 'hi'
# match.group(2) == 'example.com'</code></pre></div>
<p>If the text matches the pattern, it returns a match object with the <code>group()</code> method to read the whole matched string, or separate captures of the pattern that were defined with the parenthesis. 0 means getting the whole string, 1 means getting the match in the first group, 2 means getting the match in the second group, and so on. If the text doesn't match the pattern, the <code>None</code> value will be returned.</p>
<p>In <strong>JavaScript</strong> the <code>match()</code> method belongs to the string and it returns either a match object, or <code>null</code>. Pretty similar:</p>
<div><pre><code class="language-javascript">// name, "@", and domain
pattern = /([\w.+\-]+)@([\w\-]+\.[\w\-.]+)/;
match = 'hi@example.com'.match(pattern);
// match[0] === 'hi@example.com'
// match[1] === 'hi'
// match[2] === 'example.com'</code></pre></div>
<p>The match object in <strong>JavaScript</strong> acts as an array. Its value at the zeroth position is the whole matched string. The other indexes correspond to the captures of the pattern defined with the parenthesis. </p>
<hr />
<p>Moreover, sometimes you need to search if a specific value exists in a string and at which letter position it will be found. That can be done with the <code>search()</code> method.</p>
<p>In <strong>Python</strong> this method belongs to the regular expression pattern and it returns the match object. The match object has the <code>start()</code> method telling at which letter position the match starts:</p>
<div><pre><code class="language-python">text = 'Say hi at hi@example.com'
first_match = pattern.search(text)
if first_match:
start = first_match.start() # start == 10</code></pre></div>
<p>In <strong>JavaScript</strong> the <code>search()</code> method belongs to the string and it returns just an integer telling at which letter position the match starts. If nothing is found, <code>-1</code> is returned:</p>
<div><pre><code class="language-javascript">text = 'Say hi at hi@example.com';
first_match = text.search(pattern);
if (first_match > -1) {
start = first_match; // start === 10
}
</code></pre></div>
<h2>Replacing patterns in strings using regular expressions</h2>
<p>Replacing with regular expressions usually happen when cleaning up data, or adding additional features. For example, we could take some text and make all email addresses clickable.</p>
<p><strong>Python</strong> developers would use the <code>sub()</code> method of the regular expression pattern:</p>
<div><pre><code class="language-python">html = pattern.sub(
r'<a href="mailto:\g<0>">\g<0></a>',
'Say hi at hi@example.com',
)
# html == 'Say hi at <a href="mailto:hi@example.com">hi@example.com</a>'</code></pre></div>
<p><strong>JavaScript</strong> developers would use the <code>replace()</code> method of the string:</p>
<div><pre><code class="language-javascript">html = 'Say hi at hi@example.com'.replace(
pattern,
'<a href="mailto:$&">$&</a>',
);
// html === 'Say hi at <a href="mailto:hi@example.com">hi@example.com</a>'</code></pre></div>
<p>In <strong>Python</strong> the captures, also called as "backreferences", are accessible in the replacement string as <code>\g<0></code>, <code>\g<1></code>, <code>\g<2></code>, etc. In <strong>JavaScript</strong> the same is accessible as <code>$&</code>, <code>$1</code>, <code>$2</code>, etc. Backreferences are usually used to wrap some strings or to switch places of different pieces of text.</p>
<hr />
<p>It is also possible to replace a match with a function call. This can be used to do replacements within replacements or to count or collect some features of a text. For example, using replacements with function calls in <strong>JavaScript</strong>, I once wrote a fully functional HTML syntax highlighter.</p>
<p>Here let's change all email addresses in a text to UPPERCASE.</p>
<p>In <strong>Python</strong>, the replacement function receives the match object. We can use its <code>group()</code> method to do something with the matched text and return a text as a replacement:</p>
<div><pre><code class="language-python">text = pattern.sub(
lambda match: match.group(0).upper(),
'Say hi at hi@example.com',
)
# text == 'Say hi at HI@EXAMPLE.COM'</code></pre></div>
<p>In <strong>JavaScript</strong> the replacement function receives the whole match string, the first capture, the second capture, etc. We can do what we need with those values and then return some string as a replacement:</p>
<div><pre><code class="language-javascript">text = 'Say hi at hi@example.com'.replace(
pattern,
function(match, p1, p2) {
return match.toUpperCase();
}
);
// text === 'Say hi at HI@EXAMPLE.COM'</code></pre></div>
<h2>Error handling</h2>
<p>Contrary to <strong>Python</strong>, client-side <strong>JavaScript</strong> normally isn't used for saving or reading files or connecting to remote databases. So <code>try..catch</code> blocks are quite rare in <strong>JavaScript</strong> compared to <code>try..except</code> analogy in <strong>Python</strong>.</p>
<p>Anyway, error handling can be used with custom user errors implemented and raised in <strong>JavaScript</strong> libraries and caught in the main code.</p>
<p>The following example in <strong>Python</strong> shows how to define a custom exception class <code>MyException</code>, how to raise it in a function, and how to catch it and handle in a <code>try..except..finally</code> block:</p>
<div><pre><code class="language-python">class MyException(Exception):
def __init__(self, message):
self.message = message
def __str__(self):
return self.message
def proceed():
raise MyException('Error happened!')
try:
proceed()
except MyException as err:
print('Sorry! {}'.format(err))
finally:
print('Finishing') </code></pre></div>
<p>The following example in <strong>JavaScript</strong> does exactly the same: here we define a <code>MyException</code> class, throw it in a function, and catch it and handle in the <code>try..catch..finally</code> block.</p>
<div><pre><code class="language-javascript">function MyException(message) {
this.message = message;
this.toString = function() {
return this.message;
}
}
function proceed() {
throw new MyException('Error happened!');
}
try {
proceed();
} catch (err) {
if (err instanceof MyException) {
console.log('Sorry! ' + err);
}
} finally {
console.log('Finishing');
}
</code></pre></div>
<p>The <code>MyException</code> class in both languages has a parameter <code>message</code> and a method to represent itself as a string using the value of the <code>message</code>.</p>
<p>Of course, exceptions should be raised/thrown just in the case of errors. And you define what is an error in your module design.</p>
<h2>The Takeaways</h2>
<ul>
<li>Serialization to JSON is quite straightforward in both, <strong>Python</strong> and <strong>JavaScript</strong>.</li>
<li>Regular expressions can be used as multi-tools when working with textual data.</li>
<li>You can do replacements with function calls in both languages.</li>
<li>For more sophisticated software design you can use custom error classes.</li>
</ul>
<p>As I mentioned last time, you can grab a side-by-side comparison of <strong>Python</strong> and <strong>JavaScript</strong> that I compiled for you (and my future self). Side by side you will see features from <strong>traditional</strong> list, array, dictionary, object, and string handling to <strong>modern</strong> string interpolation, lambdas, generators, sets, classes, and everything else. Use it for good.</p>
<p><center><a href="https://www.djangotricks.com/goodies/YbnpiLKBmAZi/" target="_blank">Get the Ultimate Cheat Sheet of Equivalents in Python and JavaScript</a></center></p>
<p>In the <a href="https://djangotricks.blogspot.com/2018/07/equivalents-in-python-and-javascript-part-3.html">next part</a> of the series, we will have a look at textual templates, list unpacking, lambda functions, iteration without indexes, generators, and sets. Stay tuned!</p>
<hr />
<p>Cover photo by <a href="https://unsplash.com/photos/fs9hGJUevXY">Benjamin Hung</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495399999997552.21073109999999 12.759506999999974 52.829282099999986 14.050400999999976tag:blogger.com,1999:blog-9014457088394059918.post-77921359358968528242018-06-30T05:33:00.000+02:002019-11-04T06:49:39.575+01:00Equivalents in Python and JavaScript. Part 1<img border="0" src="https://2.bp.blogspot.com/-vPKYTFzcQIE/WzrCDyN3S_I/AAAAAAAAB5k/7bCSikeIKrgP27OEDrcJp4NsjDZnSt2YACLcBGAs/s1600/equivalents-in-python-and-javascript-part-1.png" data-original-width="1024" data-original-height="512" />
<p>Although <strong>Python</strong> and <strong>JavaScript</strong> are quite different languages, there are some analogies which full stack <strong>Python</strong> developers should know when developing web projects. In this series of 4 parts, I will explore what is similar in each of those languages and what are the common ways to solve common problems. This is not meant to be a reference and I will skip the basics like primitive variable types, conditions, and loops. But I will dig into more complex structures and data operations using both, <strong>Python</strong> and <strong>JavaScript</strong>. Also, I will try to focus on the practical use cases. This series should be interesting for the developers of Django, Flask, or another <strong>Python</strong> framework who want to get a grasp of traditional and modern vanilla <strong>JavaScript</strong>. On the other hand, it will be useful for the front-enders who want to better understand how the backend is working and maybe even start their own Django website.</p>
<h2>Parsing integers</h2>
<p>We'll begin with integer parsing.</p>
<p>In <strong>Python</strong> that's straightforward:</p>
<div><pre><code class="language-python">number = int(text)</code></pre></div>
<p>But in <strong>JavaScript</strong> you have to explain what number system you expect: decimal, octal, hexadecimal, or binary:</p>
<div><pre><code class="language-javascript">number = parseInt(text, 10);</code></pre></div>
<p>To use the "normal" decimal number system we are passing number 10 as the second parameter of the <code>parseInt()</code> function. 8 goes for octal, 16 for hexadecimal, or 2 – for binary. If the <strong>second parameter is missing</strong>, the number in text starts with zero, and you are using a slightly older browser, the number in the text will be interpreted as octal. For example,</p>
<div><pre><code class="language-javascript">parseInt('012') == 10 // in some older browsers
parseInt('012', 10) == 12</code></pre></div>
<p>And that can really mess up your calculations.</p>
<h2>Conditional assignment</h2>
<p>For conditional assignment, <strong>Python</strong> and <strong>JavaScript</strong> have different syntaxes, but conditional assignments are quite popular in both languages. That's popular, because it's just a single statement to have a condition checking, the true-case value, and the false-case value.</p>
<p>Since <strong>Python</strong> 2.7 you can write conditional assignments like this:</p>
<div><pre><code class="language-python">value = 'ADULT' if age >= 18 else 'CHILD'</code></pre></div>
<p>In <strong>JavaScript</strong> conditional assignments are done using ternary operator <code>?:</code>, similar to the ones in C, C++, C#, Java, Ruby, PHP, Perl, Swift, and ActionScript:</p>
<div><pre><code class="language-javascript">value = age >= 18? 'ADULT': 'CHILD';</code></pre></div>
<h2>Object attribute value by attribute name</h2>
<p>The normal way to access an object's attribute is by the dot notation in both, <strong>Python</strong> and <strong>JavaScript</strong>:</p>
<div><pre><code class="language-python">obj.color = 'YELLOW'</code></pre></div>
<p>But what if you want to refer to an attribute by its name saved as a string? For example, the attribute name could be coming from a list of attributes or the attribute name is combined from two strings like <code>'title_' + lang_code</code>.</p>
<p>For that reason, in <strong>Python</strong>, there are functions <code>getattr()</code> and <code>setattr()</code>. I use them a lot.</p>
<div><pre><code class="language-python">attribute = 'color'
value = getattr(obj, attribute, 'GREEN')
setattr(obj, attribute, value)</code></pre></div>
<p>In <strong>JavaScript</strong> you can treat an object like a dictionary and pass the attribute name in square brackets:</p>
<div><pre><code class="language-javascript">attribute = 'color';
value = obj[attribute] || 'GREEN';
obj[attribute] = value;</code></pre></div>
<p>To retrieve a default value when an object has no such attribute, in <strong>Python</strong>, <code>getattr()</code> has the third parameter. In <strong>JavaScript</strong>, if <code>obj</code> attribute doesn't exist, it will return the <code>undefined</code> value. Then it can be OR-ed with the default value that you want to assign. That's a common practice in <strong>JavaScript</strong> that you can find in many <strong>JavaScript</strong> libraries and frameworks.</p>
<h2>Dictionary value by key</h2>
<p>This is similar to the previous one. The normal way to assign a dictionary's value by key in both languages is using the square brackets:</p>
<div><pre><code class="language-python">dictionary = {}
dictionary['color'] = 'YELLOW'</code></pre></div>
<p>To read a value in <strong>Python</strong> you can use the square-bracket notation, but it will fail on non-existing keys with <code>KeyError</code>. The more flexible way is to use the <code>get()</code> method which returns <code>None</code> for non-existing keys. Also you can pass an optional default value as the second parameter:</p>
<div><pre><code class="language-python">key = 'color'
value = dictionary.get(key, 'GREEN')</code></pre></div>
<p>In <strong>JavaScript</strong> you would use the same trick as with object attributes, because dictionaries and objects are the same there:</p>
<div><pre><code class="language-javascript">key = 'color';
value = dictionary[key] || 'GREEN';</code></pre></div>
<h2>Slicing lists and strings</h2>
<p><strong>Python</strong> has the slice <code>[:]</code> operator to get parts of lists, tuples, and similar more complex structures, for example Django QuerySets:</p>
<div><pre><code class="language-python">items = [1, 2, 3, 4, 5]
first_two = items[:2] # [1, 2]
last_two = items[-2:] # [4, 5]
middle_three = items[1:4] # [2, 3, 4]</code></pre></div>
<p>In <strong>JavaScript</strong> arrays have the <code>slice()</code> method with the same effect and similar usage:</p>
<div><pre><code class="language-javascript">items = [1, 2, 3, 4, 5];
first_two = items.slice(0, 2); // [1, 2]
last_two = items.slice(-2); // [4, 5]
middle_three = items.slice(1, 4); // [2, 3, 4]</code></pre></div>
<p>But don't mix it up with the <code>splice()</code> method which modifies the original array!</p>
<hr />
<p>The <code>[:]</code> slice operator in <strong>Python</strong> also works for strings:</p>
<div><pre><code class="language-python">text = 'ABCDE'
first_two = text[:2] # 'AB'
last_two = text[-2:] # 'DE'
middle_three = text[1:4] # 'BCD'</code></pre></div>
<p>In <strong>JavaScript</strong> strings just like arrays have the <code>slice()</code> method:</p>
<div><pre><code class="language-javascript">text = 'ABCDE';
first_two = text.slice(0, 2); // 'AB'
last_two = text.slice(-2); // 'DE'
middle_three = text.slice(1, 4); // 'BCD'</code></pre></div>
<h2>Operations with list items</h2>
<p>In programming it is very common to collect and analyze sequences of elements. In <strong>Python</strong> that is usually done with lists and in <strong>JavaScript</strong> with arrays. They have similar syntax and operations, but different method names to add and remove values.</p>
<p>This is how to concatenate two lists, add one value to the end, add one value to the beginning, get and remove a value from the beginning, get and remove a value from the end, and delete a certain value by index in <strong>Python</strong>:</p>
<div><pre><code class="language-python">items1 = ['A']
items2 = ['B']
items = items1 + items2 # items == ['A', 'B']
items.append('C') # ['A', 'B', 'C']
items.insert(0, 'D') # ['D', 'A', 'B', 'C']
first = items.pop(0) # ['A', 'B', 'C']
last = items.pop() # ['A', 'B']
items.delete(0) # ['B']</code></pre></div>
<p>This is how to do exactly the same with arrays in <strong>JavaScript</strong>:</p>
<div><pre><code class="language-javascript">items1 = ['A'];
items2 = ['B'];
items = items1.concat(items2); // items === ['A', 'B']
items.push('C'); // ['A', 'B', 'C']
items.unshift('D'); // ['D', 'A', 'B', 'C']
first = items.shift(); // ['A', 'B', 'C']
last = items.pop(); // ['A', 'B']
items.splice(0, 1); // ['B']</code></pre></div>
<h2>Joining lists of strings</h2>
<p>It is very common after having a list or array of strings, to combine them into one string by a separator like comma or new line.</p>
<p>In <strong>Python</strong> that is done by the <code>join()</code> method of a string where you pass the list or tuple. Although it might feel unnatural, you start with the separator there. But I can assure that you get used to it after several times of usage.</p>
<div><pre><code class="language-python">items = ['A', 'B', 'C']
text = ', '.join(items) # 'A, B, C'</code></pre></div>
<p>In <strong>JavaScript</strong> the array has the <code>join()</code> method where you pass the separator:</p>
<div><pre><code class="language-javascript">items = ['A', 'B', 'C'];
text = items.join(', '); // 'A, B, C'</code></pre></div>
<h2>The Takeaways</h2>
<ul>
<li>List and tuples in <strong>Python</strong> are similar to arrays in <strong>JavaScript</strong>.</li>
<li>Dictionaries in <strong>Python</strong> are similar to objects in <strong>JavaScript</strong>.</li>
<li>Strings in <strong>Python</strong> are similar to strings in <strong>JavaScript</strong>.</li>
<li>Numbers in <strong>JavaScript</strong> should be parsed with care.</li>
<li>Single-line conditional assignments exist in both languages.</li>
<li>Joining sequences of strings in <strong>Python</strong> is confusing, but you can quickly get used to it.</li>
</ul>
<p>I compiled the whole list of equivalents of <strong>Python</strong> and <strong>JavaScript</strong> to a cheat sheet that you can print out and use for good. Side by side it compares traditional <strong>Python</strong> 2.7 and <strong>JavaScript</strong> based on ECMAScript 5 standard, as well as newer <strong>Python</strong> 3.6 and <strong>JavaScript</strong> based on ECMAScript 6 standard with such goodies as string interpolation, lambdas, generators, classes, etc.</p>
<p><center><a href="https://www.djangotricks.com/goodies/YbnpiLKBmAZi/" target="_blank">Get the Ultimate Cheat Sheet of Equivalents in Python and JavaScript</a></center></p>
<p>In <a href="https://djangotricks.blogspot.com/2018/07/equivalents-in-python-and-javascript-part-2.html">the next part</a> of the series, we will have a look at JSON creation and parsing, operations with regular expressions, and error handling. Stay tuned! </p>
<hr />
<p>Cover photo by <a href="https://unsplash.com/photos/GAFZcKrWnO8">Benjamin Hung</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0Berlin, Germany52.520006599999988 13.40495399999997552.21073109999999 12.759506999999974 52.829282099999986 14.050400999999976tag:blogger.com,1999:blog-9014457088394059918.post-88353070856131475372018-06-17T15:44:00.000+02:002019-11-04T06:34:15.204+01:00Data Filtering in a Django Website using Elasticsearch<img border="0" src="https://2.bp.blogspot.com/-ctkyxVMewZA/WzrB2CSjdeI/AAAAAAAAB5c/AnVJ1pzMyzUbErv1Lj5VTzrka564EW5ggCLcBGAs/s1600/data-filtering-in-a-django-website-using-elasticsearch.png" data-original-width="1024" data-original-height="512" />
<p>In my <a href="https://www.packtpub.com/web-development/web-development-django-cookbook-second-edition">Web Development with Django Cookbook</a> section
<em>Forms and Views</em> there is a recipe <em>Filtering object lists</em>. It shows you how to filter a Django QuerySet dynamically by different filter parameters selected in a form. From practice, the approach is working well, but with lots of data and complex nested filters, the performance might get slow. You know - because of all those INNER JOINS in SQL, the page might take even 12 seconds to load. And that is not preferable behavior. I know that I could denormalize the database or play with indices to optimize SQL. But I found a better way to increase the loading speed. Recently we started using Elasticsearch for one of the projects and its data filtering performance seems to be enormously faster: in our case, it increased from 2 to 16 times depending on which query parameters you choose.</p>
<h2>What is Elasticsearch?</h2>
<p>Elasticsearch is java-based search engine which stores data in JSON format and allows you to query it using special JSON-based query language. Using <a href="https://elasticsearch-dsl.readthedocs.io/en/latest/">elasticsearch-dsl</a> and <a href="https://github.com/sabricot/django-elasticsearch-dsl">django-elasticsearch-dsl</a>, I can bind my Django models to Elasticsearch indexes and rewrite my object list views to use Elasticsearch queries instead of Django ORM. The API of Elasticsearch DSL is chainable like with Django QuerySets or jQuery functions, and we'll have a look at it soon.</p>
<h2>The Setup</h2>
<p>At first, let's install Elasticsearch server. Elasticsearch is quite a complex system, but it comes with convenient configuration defaults.</p>
<p>On macOS you can install and start the server with Homebrew:</p>
<div><pre><code class="language-none">$ brew install elasticsearch
$ brew services start elasticsearch</code></pre></div>
<p>For other platforms, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html">the installation instructions</a> are also quite clear.</p>
<p>Then in your Django project's virtual environment install <strong>django-elasticsearch-dsl</strong>. I guess, "DSL" stands for "domain specific language".</p>
<p>With pipenv it would be the following from the project's directory:</p>
<div><pre><code class="language-none">$ pipenv install django-elasticsearch-dsl</code></pre></div>
<p>If you are using just <strong>pip</strong> and <strong>virtual environment</strong>, then you would do this with your project's environment activated.</p>
<div><pre><code class="language-python">(venv)$ pip install django-elasticsearch-dsl</code></pre></div>
<p>This, in turn, will install related lower level client libraries: <strong>elasticsearch-dsl</strong> and <strong>elasticsearch-py</strong>.</p>
<p>In the Django project settings, add <code>'django_elasticsearch_dsl'</code> to <code>INSTALLED_APPS</code>.</p>
<p>Finally, add the lines defining default connection configuration there:</p>
<div><pre><code class="language-python">ELASTICSEARCH_DSL={
'default': {
'hosts': 'localhost:9200'
},
}</code></pre></div>
<h2>Elasticsearch Documents for Django Models</h2>
<p>For the illustration how to use Elasticsearch with Django, I'll create <code>Author</code> and <code>Book</code> models, and then I will create Elasticsearch index document for the books. </p>
<h3>models.py</h3>
<div><pre><code class="language-python"># -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from django.db import models
from django.utils.translation import ugettext_lazy as _
from django.utils.encoding import python_2_unicode_compatible
@python_2_unicode_compatible
class Author(models.Model):
first_name = models.CharField(_("First name"), max_length=200)
last_name = models.CharField(_("Last name"), max_length=200)
author_name = models.CharField(_("Author name"), max_length=200)
class Meta:
verbose_name = _("Author")
verbose_name_plural = _("Authors")
ordering = ("author_name",)
def __str__(self):
return self.author_name
@python_2_unicode_compatible
class Book(models.Model):
title = models.CharField(_("Title"), max_length=200)
authors = models.ManyToManyField(Author, verbose_name=_("Authors"))
publishing_date = models.DateField(_("Publishing date"), blank=True, null=True)
isbn = models.CharField(_("ISBN"), blank=True, max_length=20)
class Meta:
verbose_name = _("Book")
verbose_name_plural = _("Books")
ordering = ("title",)
def __str__(self):
return self.title
</code></pre></div>
<p>Nothing fancy here. Just an <code>Author</code> model with fields <code>id</code>, <code>first_name</code>, <code>last_name</code>, <code>author_name</code>, and a <code>Book</code> model with fields <code>id</code>, <code>title</code>, <code>authors</code>, <code>publishing_date</code>, and <code>isbn</code>. Let's go to the documents.</p>
<h3>documents.py</h3>
<p>In the same directory of your app, create <code>documents.py</code> with the following content:</p>
<div><pre><code class="language-python"># -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from django_elasticsearch_dsl import DocType, Index, fields
from .models import Author, Book
# Name of the Elasticsearch index
search_index = Index('library')
# See Elasticsearch Indices API reference for available settings
search_index.settings(
number_of_shards=1,
number_of_replicas=0
)
@search_index.doc_type
class BookDocument(DocType):
authors = fields.NestedField(properties={
'first_name': fields.TextField(),
'last_name': fields.TextField(),
'author_name': fields.TextField(),
'pk': fields.IntegerField(),
}, include_in_root=True)
isbn = fields.KeywordField(
index='not_analyzed',
)
class Meta:
model = Book # The model associated with this DocType
# The fields of the model you want to be indexed in Elasticsearch
fields = [
'title',
'publishing_date',
]
related_models = [Author]
def get_instances_from_related(self, related_instance):
"""If related_models is set, define how to retrieve the Book instance(s) from the related model."""
if isinstance(related_instance, Author):
return related_instance.book_set.all()
</code></pre></div>
<p>Here we defined a <code>BookDocument</code> which will have fields: <code>title</code>, <code>publishing_date</code>, <code>authors</code>, and <code>isbn</code>. </p>
<p>The <code>authors</code> will be a list of nested dictionaries at the <code>BookDocument</code>. The <code>isbn</code> will be a <code>KeywordField</code> which means that it will be not tokenized, lowercased, nor otherwise processed and handled the whole as is.</p>
<p>The values for those document fields will be read from the <code>Book</code> model. </p>
<p>Using signals, the document will be automatically updated either when a <code>Book</code> instance or <code>Author</code> instance is added, changed, or deleted. In the method <code>get_instances_from_related()</code>, we tell the search engine which books to update when an author is updated.</p>
<h3>Building the Index</h3>
<p>When the index document is ready, let's build the index at the server:</p>
<div><pre><code class="language-python">(venv)$ python manage.py search_index --rebuild</code></pre></div>
<h2>Django QuerySets vs. Elasticsearch Queries</h2>
<p>The concepts of SQL and Elasticsearch queries are quite different. One is working with relational tables and the other works with dictionaries. One is using queries that are kind of human-readable logical sentences and another is using nested JSON structures. One is using the content verbosely and another does string processing in the background and gives search relevance for each result.</p>
<p>Even when there are lots of differences, I will try to draw analogies between Django ORM and <strong>elasticsearch-dsl</strong> API as close as possible.</p>
<h3>1. Query definition</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">queryset = MyModel.objects.all()</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">search = MyModelDocument.search()</code></pre></div>
<h3>2. Count</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">queryset = queryset.count()</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">search = search.count()</code></pre></div>
<h3>3. Iteration</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">for item in queryset:
print(item.title)</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">for item in search:
print(item.title)</code></pre></div>
<h3>4. To see the generated query:</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">>>> queryset.query</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">>>> search.to_dict()</code></pre></div>
<h3>5. Filter by single field containing a value</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">queryset = queryset.filter(my_field__icontains=value)</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">search = search.filter('match_phrase', my_field=value)</code></pre></div>
<h3>6. Filter by single field equal to a value</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">queryset = queryset.filter(my_field__exact=value)</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">search = search.filter('match', my_field=value)</code></pre></div>
<p>If a field type is a string, not a number, it has to be defined as <code>KeywordField</code> in the index document:</p>
<div><pre><code class="language-python">my_field = fields.KeywordField()</code></pre></div>
<h3>7. Filter with either of the conditions (OR)</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">from django.db import models
queryset = queryset.filter(
models.Q(my_field=value) |
models.Q(my_field2=value2)
)</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">from elasticsearch_dsl.query import Q
search = search.query(
Q('match', my_field=value) |
Q('match', my_field2=value2)
)</code></pre></div>
<h3>8. Filter with all of the conditions (AND)</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">from django.db import models
queryset = queryset.filter(
models.Q(my_field=value) &
models.Q(my_field2=value2)
)</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">from elasticsearch_dsl.query import Q
search = search.query(
Q('match', my_field=value) &
Q('match', my_field2=value2)
)</code></pre></div>
<h3>9. Filter by values less than or equal to certain value</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">from datetime import datetime
queryset = queryset.filter(
published_at__lte=datetime.now(),
)</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">from datetime import datetime
search = search.filter(
'range',
published_at={'lte': datetime.now()}
)</code></pre></div>
<h3>10. Filter by a value in a nested field</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">queryset = queryset.filter(
category__pk=category_id,
)</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">from elasticsearch_dsl.query import Q
search = search.filter(
'nested',
path='category',
query=Q('match', category__pk=category_id)
)</code></pre></div>
<h3>11. Filter by one of many values in a related model</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">queryset = queryset.filter(
category__pk__in=category_ids,
)</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">from django.utils.six.moves import reduce
from elasticsearch_dsl.query import Q
search = search.query(
reduce(operator.ior, [
Q(
'nested',
path='category',
query=Q('match', category__pk=category_id),
)
for category_id in category_ids
])
)</code></pre></div>
<p>Here the <code>reduce()</code> function combines a list of <code>Q()</code> conditions using the bitwise OR operator (|).</p>
<h3>12. Ordering</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">queryset = queryset.order_by('-my_field', 'my_field2')</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">search = search.sort('-my_field', 'my_field2')</code></pre></div>
<h3>13. Creating query dynamically</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">import operator
from django.utils.six.moves import reduce
filters = []
if value1:
filters.append(models.Q(
my_field1=value1,
))
if value2:
filters.append(models.Q(
my_field2=value2,
))
queryset = queryset.filter(
reduce(operator.iand, filters)
)</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">import operator
from django.utils.six.moves import reduce
from elasticsearch_dsl.query import Q
queries = []
if value1:
queries.append(Q(
'match',
my_field1=value1,
))
if value2:
queries.append(Q(
'match',
my_field2=value2,
))
search = search.query(
reduce(operator.iand, queries)
)</code></pre></div>
<h3>14. Pagination</h3>
<p>Django QuerySet:</p>
<div><pre><code class="language-python">from django.core.paginator import (
Paginator, Page, EmptyPage, PageNotAnInteger
)
paginator = Paginator(queryset, paginate_by)
page_number = request.GET.get('page')
try:
page = paginator.page(page_number)
except PageNotAnInteger:
page = paginator.page(1)
except EmptyPage:
page = paginator.page(paginator.num_pages)
</code></pre></div>
<p>Elasticsearch query:</p>
<div><pre><code class="language-python">from django.core.paginator import (
Paginator, Page, EmptyPage, PageNotAnInteger
)
from django.utils.functional import LazyObject
class SearchResults(LazyObject):
def __init__(self, search_object):
self._wrapped = search_object
def __len__(self):
return self._wrapped.count()
def __getitem__(self, index):
search_results = self._wrapped[index]
if isinstance(index, slice):
search_results = list(search_results)
return search_results
search_results = SearchResults(search)
paginator = Paginator(search_results, paginate_by)
page_number = request.GET.get('page')
try:
page = paginator.page(page_number)
except PageNotAnInteger:
page = paginator.page(1)
except EmptyPage:
page = paginator.page(paginator.num_pages)
</code></pre></div>
<p>ElasticSearch doesn't work with Django's pagination by default. Therefore, we have to wrap the search query with lazy <code>SearchResults</code> class to provide the necessary functionality.</p>
<h2>Example</h2>
<p>I built an example with books written about Django. You can <a href="https://github.com/archatas/experiment-with-django-and-elasticsearch">download it from Github</a> and test it. </p>
<h2>Takeaways</h2>
<ul>
<li>Filtering with Elasticsearch is much faster than with SQL databases.</li>
<li>But it comes at the cost of additional deployment and support time.</li>
<li>If you have multiple websites using Elasticsearch on the same server, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/zip-targz.html#zip-targz-configuring">configure a new cluster and node</a> for each of those websites.</li>
<li>Django ORM can be in a way mapped to Elasticsearch DSL.</li>
<li>I summarized the comparison of Django ORM and Elasticsearch DSL, mentioned in this article, into <a href="https://www.djangotricks.com/goodies/tq4NRGdnXkiM/">a cheat sheet</a>. Print it on a single sheet of paper and use it as a reference for your developments.</li>
</ul>
<p align="center"><a href="https://www.djangotricks.com/goodies/tq4NRGdnXkiM/" target="_blank">Get Django ORM vs. Elasticsearch DSL Cheat Sheet</a></p>
<hr>
<p>Cover photo by <a href="https://unsplash.com/photos/TYIzeCiZ_60">Karl Fredrickson</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0tag:blogger.com,1999:blog-9014457088394059918.post-2754166402398501432018-05-27T23:45:00.000+02:002018-07-03T02:22:21.865+02:00QuerySet Filters on Many-to-many Relations<img border="0" src="https://3.bp.blogspot.com/-mpWcOBwZVIA/WzrBrDK1mJI/AAAAAAAAB5Y/zd9_YGzKi9gMRs0TqJfeq0YQRHQl5nrgACLcBGAs/s1600/queryset-filters-on-many-to-many-relations.png" data-original-width="1024" data-original-height="512" />
<p>Django ORM (Object-relational mapping) makes querying the database so intuitive, that at some point you might forget that SQL is being used in the background.</p>
<p>This year at the DjangoCon Europe <a href="https://www.youtube.com/watch?v=AIke7IZdVJI&list=PLY_che_OEsX3aZo5RttI6Fj2XZ7nTjhBu&index=15">Katie McLaughlin was giving a talk</a> and mentioned one thing that affects the SQL query generated by Django ORM, depending on how you call the QuerySet or manager methods. This particularity is especially relevant when you are creating your QuerySets dynamically. Here it is. When you have a many-to-many relationship, and you try to filter objects by the fields of the related model, every new <code>filter()</code> method of a QuerySet creates a new <code>INNER JOIN</code> clause. I won't discuss whether that's a Django bug or a feature, but these are my observations about it.</p>
<h2>The Books and Authors Example</h2>
<p>Let's create an app with books and authors, where each book can be written by multiple authors.</p>
<div><pre><code class="language-python"># -*- coding: UTF-8 -*-
from __future__ import unicode_literals
from django.db import models
from django.utils.translation import ugettext_lazy as _
from django.utils.encoding import python_2_unicode_compatible
@python_2_unicode_compatible
class Author(models.Model):
first_name = models.CharField(_("First name"), max_length=200)
last_name = models.CharField(_("Last name"), max_length=200)
author_name = models.CharField(_("Author name"), max_length=200)
class Meta:
verbose_name = _("Author")
verbose_name_plural = _("Authors")
ordering = ("author_name",)
def __str__(self):
return self.author_name
@python_2_unicode_compatible
class Book(models.Model):
title = models.CharField(_("Title"), max_length=200)
authors = models.ManyToManyField(Author, verbose_name=_("Authors"))
publishing_date = models.DateField(_("Publishing date"), blank=True, null=True)
class Meta:
verbose_name = _("Book")
verbose_name_plural = _("Books")
ordering = ("title",)
def __str__(self):
return self.title
</code></pre></div>
<p>The similar app with sample data can be found in <a href="https://github.com/archatas/experiment-with-django-and-elasticsearch">this repository</a>.</p>
<h2>Inefficient Filter</h2>
<p>With the above models, you could define the following QuerySet to select <a href="https://www.packtpub.com/books/info/authors/aidas-bendoraitis">books which author is me, Aidas Bendoraitis</a>.</p>
<div><pre><code class="language-python">queryset = Book.objects.filter(
authors__first_name='Aidas',
).filter(
authors__last_name='Bendoraitis',
)</code></pre></div>
<p>We can check what SQL query it would generate with <code>str(queryset.query)</code> (or <code>queryset.query.__str__()</code>).</p>
<p>The output would be something like this:</p>
<div><pre><code class="language-sql">SELECT `libraryapp_book`.`id`, `libraryapp_book`.`title`, `libraryapp_book`.`publishing_date`
FROM `libraryapp_book`
INNER JOIN `libraryapp_book_authors` ON ( `libraryapp_book`.`id` = `libraryapp_book_authors`.`book_id` )
INNER JOIN `libraryapp_author` ON ( `libraryapp_book_authors`.`author_id` = `libraryapp_author`.`id` )
INNER JOIN `libraryapp_book_authors` T4 ON ( `libraryapp_book`.`id` = T4.`book_id` )
INNER JOIN `libraryapp_author` T5 ON ( T4.`author_id` = T5.`id` )
WHERE (`libraryapp_author`.`first_name` = 'Aidas' AND T5.`last_name` = 'Bendoraitis')
ORDER BY `libraryapp_book`.`title` ASC;</code></pre></div>
<p>Did you notice, that the database table <code>libraryapp_author</code> was attached through the <code>libraryapp_book_authors</code> table to the <code>libraryapp_book</code> table TWICE where just ONCE would be enough?</p>
<h2>Efficient Filter</h2>
<p>On the other hand, if you are defining query expressions in the same <code>filter()</code> method like this:</p>
<div><pre><code class="language-python">queryset = Book.objects.filter(
authors__first_name='Aidas',
authors__last_name='Bendoraitis',
)</code></pre></div>
<p>The generated SQL query will be much shorter and (theoretically) would perform faster:</p>
<div><pre><code class="language-sql">SELECT `libraryapp_book`.`id`, `libraryapp_book`.`title`, `libraryapp_book`.`publishing_date`
FROM `libraryapp_book`
INNER JOIN `libraryapp_book_authors` ON ( `libraryapp_book`.`id` = `libraryapp_book_authors`.`book_id` )
INNER JOIN `libraryapp_author` ON ( `libraryapp_book_authors`.`author_id` = `libraryapp_author`.`id` )
WHERE (`libraryapp_author`.`first_name` = 'Aidas' AND `libraryapp_author`.`last_name` = 'Bendoraitis')
ORDER BY `libraryapp_book`.`title` ASC;</code></pre></div>
<p>The same SQL query can be achieved using the <code>Q()</code> objects:</p>
<div><pre><code class="language-python">queryset = Book.objects.filter(
models.Q(authors__first_name='Aidas') &
models.Q(authors__last_name='Bendoraitis')
)</code></pre></div>
<p>The <code>Q()</code> objects add a lot of flexibility to filters allowing to <a href="https://docs.djangoproject.com/en/2.0/topics/db/queries/#complex-lookups-with-q-objects">OR, AND, and negate query expressions</a>.</p>
<h2>Dynamic Filtering</h2>
<p>So to have faster performance, when creating QuerySets dynamically, DON'T use <code>filter()</code> multiple times:</p>
<div><pre><code class="language-python">queryset = Book.objects.all()
if first_name:
queryset = queryset.filter(
authors__first_name=first_name,
)
if last_name:
queryset = queryset.filter(
authors__last_name=last_name,
)</code></pre></div>
<p>DO this instead:</p>
<div><pre><code class="language-python">filters = models.Q()
if first_name:
filters &= models.Q(
authors__first_name=first_name,
)
if last_name:
filters &= models.Q(
authors__last_name=last_name,
)
queryset = Book.objects.filter(filters)</code></pre></div>
<p>Here the empty <code>Q()</code> doesn't have any impact for the generated SQL query, so you don't need the complexity of creating a list of filters and then joining all of them with the bitwise AND operator, like this:</p>
<div><pre><code class="language-python">import operator
from django.utils.six.moves import reduce
filters = []
if first_name:
filters.append(models.Q(
authors__first_name=first_name,
))
if last_name:
filters.append(models.Q(
authors__last_name=last_name,
))
queryset = Book.objects.filter(reduce(operator.iand, filters))</code></pre></div>
<h2>Profiling</h2>
<p>In DEBUG mode, you can check how long the previously executed SQL queries took by checking <code>django.db.connection.queries</code>:</p>
<div><pre><code class="language-python">>>> from django.db import connection
>>> connection.queries
[{'sql': 'SELECT …', 'time': '0.001'}, {'sql': 'SELECT …', 'time': '0.004'}]</code></pre></div>
<h2>The Takeaways</h2>
<ul>
<li>When querying many-to-many relationships, avoid using multiple <code>filter()</code> methods, make use of <code>Q()</code> objects instead.</li>
<li>You can check the SQL query of a QuerySet with <code>str(queryset.query)</code>.</li>
<li>Check the performance of recently executed SQL queries with <code>django.db.connection.queries</code>.</li>
<li>With small datasets, the performance difference is not so obvious. For your specific cases you should do the benchmarks yourself.</li>
</ul>
<hr />
<p>Cover photo by <a href="https://unsplash.com/photos/ljp-ewA23lc?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Tobias Fischer</a>.</p>
Aidas Bendoraitis a.k.a. archatashttp://www.blogger.com/profile/18444891139539061590noreply@blogger.com0