Software localization

Babel’s I18n Advantages for Multilingual Apps

Python Babel is a well-known library for Python i18n and l10n. Learn how to use it and integrate it with Jinja2 templates in this tutorial.
Software localization blog category featured image | Phrase

We have given detailed tutorials on gettext tools and integrating gettext with Python. We are going to extend our knowledge by using Babel and see some practical examples of its usage in python i18n. We are also going to see how to integrate it with Jinja2 templates and how to integrate Phrase's in-context editor into a Flask application to help with the translation process by simply browsing the website and editing texts along the way, making your Python localization process a lot simpler. You can also find the code described in this tutorial on GitHub.

About Babel

Babel provides internationalization (python i18n) and localization (python l10n) helpers and tools that work in two areas. The first is the gettext module that uses gettext to update, extract, and compile message catalogs and manipulate PO files. The second one is the usage of CLDR (Common Locale Data Repository) to provide formatting methods for currencies, dates, numbers, etc. based on a locale parameter. Both aspects aim to help automate the process of internationalizing Python applications as well as provide convenient methods for accessing and using this data.

Babel, in essence, works as an abstraction mechanism for a larger message extraction framework, as you can extend it with your own extractors and strategies that are not tied to a particular platform.

Installing and using Babel

Installing Babel is simple using pip

$ pip install Babel

If you don't have pip installed, you can get it with easy_install

$ sudo easy_install pip

Working with Locale Data

(CLDR) Unicode Common Locale Data Repository is a standardized repository of locale data used for formatting, parsing, and displaying locale-specific information. Instead of translating, for example, day names or month names for a particular language or script, you can make use of the translations provided by the locale data included with Babel based on CLDR data.

Let's see some examples...

Create a file name loc.py and add the following code:

from babel import Locale

# Parsing

l = Locale.parse('de-DE', sep='-')

print("Locale name: {0}".format(l.display_name))

l = Locale.parse('und_GR', sep='_')

print("Locale name: {0}".format(l.display_name))

# Detecting

l = Locale.negotiate(['de_DE', 'en_AU'], ['de_DE', 'de_AT'])

print("Locale negociated: {0}".format(l.display_name))

print(Locale('it').english_name)

print(Locale('it').get_display_name('fr_FR'))

print(Locale('it').get_language_name('de_DE'))

print(Locale('de', 'DE').languages['zh'])

print(Locale('el', 'GR').scripts['Copt'])

# Calendar

locale = Locale('it')

month_names = locale.days['format']['wide'].items()

print(list(month_names))

We are showing some examples of the Locale class. It is used to print, negotiate or identify language tags. If you run this example, you will see the following output:

$ python loc.py

Locale name: Deutsch (Deutschland)

Locale name: Ελληνικά (Ελλάδα)

Locale negociated: Deutsch (Deutschland)

Italian

italien

Italienisch

Chinesisch

Κοπτικό

[(6, 'domenica'), (0, 'lunedì'), (1, 'martedì'), (2, 'mercoledì'), (3, 'giovedì'), (4, 'venerdì'), (5, 'sabato')]

These are useful as they are provided by the (CLDR) dataset and do not need translation.

Apart from that, there are several functions that format dates, times, currencies, units etc. Let's see some examples:

from babel.dates import format_date, format_datetime, format_time

from babel.numbers import format_number, format_decimal, format_percent, parse_decimal

from babel.units import format_unit

from datetime import date, datetime, time

# Date, time

d = date(2010, 3, 10)

print(format_date(d, format='short', locale='it'))

print(format_date(d, format='full', locale='it'))

print(format_date(d, "EEEE, d.M.yyyy", locale='de'))

dt = datetime.now()

print(format_datetime(dt, "yyyy.MMMM.dd GGG hh:mm a", locale='en'))

# Numbers/ Units

print(format_decimal(123.45123, locale='en_US'))

print(format_decimal(123.45123, locale='de'))

print(format_unit(12, 'length-meter', locale='en_GB'))

print(format_unit(12, 'length-meter', locale='en_US'))

parse_decimal('2.029,98', locale='de')

The output is:

$ python loc.py

10/03/10

mercoledì 10 marzo 2010

Mittwoch, 10.3.2010

2018.August.28 AD 10:24 AM

123.451

123,451

12 metres

12 meters

2029.98

Message Extraction

Babel has an extraction mechanism similar to gettext. It works by walking through the specified directories and based on the configuration rules it applies extractor functions to those files matched. This way there is more flexibility than gettext as you can leverage the expression power of Python to extend the tool.

Babel comes with a few built-in extractors such as python, javascript, and ignore (which extracts nothing) and you can create your own extractors. There are two different front-ends to access this functionality:

  • A Command-Line Interface
  • Distutils/Setuptools Integration

In this tutorial, we are going to use the Command-Line Interface.

To use it just invoke the pybabel tool for example to print all known locales

$ pybabel --list-locales:

To actually use the tooling, let's walk through the process of extracting messages using pybabel :

Create a file named main.py and add the following code:

import gettext

_ = gettext.gettext

print(_('This is a translatable string.'))

Note that the usage of gettext is only convenient because the default extractor uses gettext behind the scenes, but Babel, in general, is not tied to that.

Use the pybabel extract command to create the initial message catalog:

$ mkdir locale

$ pybabel extract . -o locale/base.pot

That will create a base pot file that will contain the following messages:

#: main.py:4

msgid "This is a translatable string."

msgstr ""

You don't have to edit this file now.

Using the init command, we can create a new translation catalog based on that POT template file:

$ pybabel init -l el_GR de_DE en_US -i locale/base.pot -d locale

creating catalog locale/el_GR/LC_MESSAGES/messages.po based on locale/base.pot

$ pybabel init -l de_DE en_US -i locale/base.pot -d locale

creating catalog locale/de_DE/LC_MESSAGES/messages.po based on locale/base.pot

$ pybabel init -l en_US -i locale/base.pot -d locale

creating catalog locale/en_US/LC_MESSAGES/messages.po based on locale/base.pot

Those files are ready to be translated. When the translations are done, you can use the compile command to turn them into MO files:

$ pybabel compile -d locale

compiling catalog locale/en_US/LC_MESSAGES/messages.po to locale/en_US/LC_MESSAGES/messages.mo

compiling catalog locale/el_GR/LC_MESSAGES/messages.po to locale/el_GR/LC_MESSAGES/messages.mo

compiling catalog locale/de_DE/LC_MESSAGES/messages.po to locale/de_DE/LC_MESSAGES/messages.mo

Now, if you make changes to the base.pot file you can update the rest using the update command. For example, add the following line to base.pot:

msgid "Hello world."

msgstr ""

Then run the tool to update the rest of the PO files:

$ pybabel update -i locale/base.pot -d locale

updating catalog locale/en_US/LC_MESSAGES/messages.po based on locale/base.pot

updating catalog locale/el_GR/LC_MESSAGES/messages.po based on locale/base.pot

updating catalog locale/de_DE/LC_MESSAGES/messages.po based on locale/base.pot

Integration with Jinja2 templates

Babel can integrate with Jinja templates using the jinja2.ext.i18n extension.

It can then be used to mark and translate messages from templates and it is useful for internationalizing HTML pages.

In order to integrate both of those tools together, you need to provide some config.

First, install jinja using pip:

$ pip install Jinja2

You need to instruct babel to parse jinja templates when extracting the messages and for that, you need to add a configuration file.

Create a file named babel-mapping.ini and add the following text:

[python: **.py]

[jinja2: **/templates/**.html]

extensions=jinja2.ext.i18n,jinja2.ext.autoescape,jinja2.ext.with_

So now when you invoke pybabel commands referencing that file, it will also extract messages from Jinja templates.

Let's see how we can load Babel and Jinja templates together:

Create a template called index.html that will be used to extract our messages:

<title>{% trans %}{{title}}{% endtrans %}</title>

<p>{% trans count=mailservers|length %}

There is {{ count }} {{ name }} server.

{% pluralize %}

There are {{ count }} {{ name }} servers.

{% endtrans %}

</p>

Invoke the following command to extract the base messages:

¢ pybabel extract -F babel-mapping.ini -o locale/messages.pot ./

That will generate the following catalog:

# Translations template for PROJECT.

# Copyright (C) 2018 ORGANIZATION

# This file is distributed under the same license as the PROJECT project.

# FIRST AUTHOR <EMAIL@ADDRESS>, 2018.

#

#, fuzzy

msgid ""

msgstr ""

"Project-Id-Version: PROJECT VERSION\n"

"Report-Msgid-Bugs-To: EMAIL@ADDRESS\n"

"POT-Creation-Date: 2018-08-29 10:24+0100\n"

"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"

"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"

"Language-Team: LANGUAGE <LL@li.org>\n"

"MIME-Version: 1.0\n"

"Content-Type: text/plain; charset=utf-8\n"

"Content-Transfer-Encoding: 8bit\n"

"Generated-By: Babel 2.6.0\n"

#: templates/index.html:1

#, python-format

msgid "%(title)s"

msgstr ""

#: templates/index.html:2

#, python-format

msgid ""

"\n"

" There is %(count)s %(name)s server.\n"

" "

msgid_plural ""

"\n"

" There are %(count)s %(name)s servers.\n"

" "

msgstr[0] ""

msgstr[1] ""

Now initialize the Italian translations using the init command and provide the translations:

$ pybabel init -d locale -l it -i locale/messages.pot

creating catalog locale/it/LC_MESSAGES/messages.po based on locale/messages.pot

Run the compile command to generate the MO files:

$ pybabel compile -d locale -l it

compiling catalog locale/it/LC_MESSAGES/messages.po to locale/it/LC_MESSAGES/messages.mo

Create a file named app.py to hook everything together:

from jinja2 import Environment, FileSystemLoader, select_autoescape

from babel.support import Translations

templateLoader = FileSystemLoader( searchpath="templates" )

env = Environment(

  loader=templateLoader,

  extensions=['jinja2.ext.i18n', 'jinja2.ext.autoescape'],

  autoescape=select_autoescape(['html', 'xml'])

)

translations = Translations.load('locale', ['it'])

env.install_gettext_translations(translations)

template = env.get_template('index.html')

print(template.render(mailservers=range(10), name='mail'))

We are using the Translations component to load the message catalog, which we compiled earlier. Then we load them to the Jinja environment using the install_gettext_translations method. Then we render the template.

If you run this program, you will see the following output:

$ python app.py

<title>Titolo</title>

<p>Esistono 10 mail servers.

</p>

If we want to change the locale, we need to do the same procedure. For example:

translations = Translations.load('locale', ['en_US'])

env.install_gettext_translations(translations)

template = env.get_template('index.html')

print(template.render())

Adding Phrase in-context editor with Flask and Babel

We can also introduce Flask into the picture and integrate Phrase in-context editor by using a technique to replace the gettext callables for the jinja environment.

First, we need to install the required packages:

$ pip install flask, flask-babel

Create a file named web.py and add the following code:

from flask import Flask, render_template, request

from flask_babel import Babel

from phrase import Phrase, gettext, ngettext

class Config(object):

    LANGUAGES = {

        'it': 'Italian',

        'de_DE': 'Deutsch'

    },

    BABEL_DEFAULT_LOCALE= 'it'

    PHRASEAPP_ENABLED = True

    PHRASEAPP_PREFIX = '{{__'

    PHRASEAPP_SUFFIX = '__}}'

app = Flask(__name__)

app.config.from_object(Config)

babel = Babel(app)

phrase = Phrase(app)

@babel.localeselector

def get_locale():

    return request.accept_languages.best_match(app.config['LANGUAGES'][0].keys())

@app.route('/')

def index():

    return render_template('index.html', locale=get_locale() or babel.default_locale)

We set up some configuration first to define the list of supported languages and the PHRASEAPP_* specific keys. What the in-context editor needs is wrapping the translatable strings with specific tags '{{__'  and '__}}'.

Now let's see the contents of the phrase.py file

from __future__ import print_function

try:

    from flask_babel import gettext as gettext_original, ngettext as ngettext_original

    from flask import current_app

except ImportError:

    print("Flask-Babel is required.")

class Phrase(object):

    def __init__(self, app=None):

        self.app = app

        app.jinja_env.install_gettext_callables(

            gettext,

            ngettext,

            newstyle=True

        )

def phrase_enabled():

    return current_app.config['PHRASEAPP_ENABLED']

def phrase_key(msgid):

    return current_app.config['PHRASEAPP_PREFIX'] + 'phrase_' + msgid + current_app.config['PHRASEAPP_SUFFIX']

def gettext(msgid):

    if phrase_enabled():

        return phrase_key(msgid)

    else:

        return gettext_original(msgid)

def ngettext(msgid1, msgid2, n, **kwargs):

    if phrase_enabled():

        return phrase_key(msgid1)

    else:

        return ngettext_original(msgid1, msgid2, n, **kwargs)

We need to make sure we forward the original parameters to the original gettext functions in case we disabled the editor.

Update the index.html file to include the script to load the editor:

<!DOCTYPE html>

<html lang="{{locale}}">

<head>

  <meta charset="UTF-8">

  <meta name="viewport" content="width=device-width, initial-scale=1.0">

  <meta http-equiv="X-UA-Compatible" content="ie=edge">

  <title>{% trans %}title{% endtrans %}</title>

</head>

<body>

    <p>{% trans %}title{% endtrans %}</p>

        window.PHRASEAPP_CONFIG = {

            projectId: "YOUR-PROJECT-ID"

        };

        (function() {

            var phraseapp = document.createElement('script'); phraseapp.type = 'text/javascript'; phraseapp.async = true;

            phraseapp.src = ['https://', 'phraseapp.com/assets/in-context-editor/2.0/app.js?', new Date().getTime()].join('');

            var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(phraseapp, s);

        })();

</body>

</html>

If you haven't done that already, go to phrase.com and signup to try it for free.

Once you set up your account, you can create a project and navigate to Project Settings to find your projectId key.

Phrase sign up | Phrase

New project screen in Phrase | Phrase

Use that to assign the PHRASE_APP_TOKEN environment variable before you start the server.

Project Settings in Phrase | PhraseWhen you navigate to the page you will see a login modal and once you are authenticated you will see the translated strings change to include edit buttons next to them. The in-context editor panel will show also.

$ export FLASK_APP=web.py

$ flask run

In-context translator | Phrase

Conclusion

In this article, we have seen how to do python localization using the Python babel library. We’ve also seen how we can integrate it with Jinja templates and Phrase's in-context editor in our workflow. If you have any other questions left, do not hesitate to get in touch. Thank you for reading and see you again next time!