android2po

Convert Android string resources to gettext .po files, and import them right back.

The goal is to remove as many syntax elements required by the Android string resource files when exporting, to present the text to translators in as easy a format as possible; and correctly writing everything back to Android on import while keeping the generated XML files easily readable as well (i.e. no unnecessary escaping).

Requirements

The following Python modules are required, but will mostly be auto-installed. See Installation below.

babel >= 1.0dev (only the dev version has support for contexts)

http://babel.edgewall.org/

lxml

http://codespeak.net/lxml/

argparse

http://argparse.googlecode.com/

Since the .po files this script generates use contexts (msgctx), that's what you're gettext software will have to support as well.

Installation

To install the current release, you can simply do:

$ easy_install android2po

That's it!

If you want to install the current development version of android2po instead, get the source code, then run:

$ python setup.py install

setup.py should automatically install all the dependencies. Alternatively, you can also use pip if you prefer:

$ pip install -r requirements.pip

Note: The development version of Babel is required, which isn't straighforward to install, so both the setup.py file and the requirements.pip file link to a custom distribution I made for android2po. If you prefer to install the babel SVN version yourself, you need to follow the instructions on this page:

http://babel.edgewall.org/wiki/SubversionCheckout

Usage

The basic idea is that:

• values/strings.xml holds the reference strings in their original language (the gettext msgid fields).
• The gettext .po files generated and updated by this script contain the reference version of the translations.
• The values-XX/strings.xml files are fully generated based on your .po files, and should not be modified manually.

In addition to your authoritative strings.xml file, you will usually also want to keep your .po files in source control; The generated language-specific strings.xml files then contain no additional information, and do not need to be source controlled, though you are free to if you like.

The environment

To be able to run, the script necessarily needs to know about two filesystem locations: The directory where your Android resources are located, and the directory where the gettext .po files should be stored:

$ a2po COMMAND --android myproject/res --gettext myproject/locale

However, to simplify usage, the program will automatically try to detect the location of these folders, as follows:

• It will search the directory hierarchy, starting with the your working directory, for a AndroidManifest.xml or .android2po file.
• As soon as it finds either of those files, it will stop, and consider the it's location the project directory.
• Unless explicitly overriden by you, it will place the .po files in a subdirectory ./locale of that project directory.
• Only if a AndroidManifest.xml file is in the project directory will it assume that the Android resources are located in a subfolder named ./res.
• If a .android2po file is in the project directory, it automatically will be loaded as a configuration file. See the section below on the format of the configuration file, and possible values.
• The script automatically processes all the languages it can find. It will normally look at the existing .po files to determine the list of languages, with the exception of the init command, in which case the list of languages inside the Android resource directory will be used.

Initial setup

When switching to android2po, you will first want to create an initial export of your current translations.

$ a2po init

This will ignore any languages for which a .po file already exists.

For testing purposes, you may want to immediately import the generated files back in, to compare with what you originally had, and make sure the script was able to process your files correctly. At this point, make sure you have a backup, since your language-specific strings.xml files are going to be replaced (you are using source control, right?!):

$ a2po import
$ git diff --ignore-all-space res/values-XX/strings.xml

In the example above, git is used for source control. git provides a nice option to show a diff while ignoring whitespace changes, which will make it much easier to spot problems with the import. If you use a different tool, see if there is a comparable feature.

Hopefully, your translated XML files at this point hold the same information as before. The whitespace will probably have changed, comments will have been removed, and some strings may have changed visually (i.e. use different escaping). However, their meaning should not have changed. If it has, please report a bug.

Updating

After hacking on your code for a while, you have changed some strings (in your authoritative values/strings.xml file), and now you need to pass those on to your translators through your .po files.

Simply do:

$ a2po export

This will update your .po files with your changes.

Importing

Your translators have come back to you with their changes, and you want to include them in the next build. Simply do:

$ a2po import

This will fully regenerate your language-specific strings.xml based on the gettext .po files.

You can do this step manually, or add it to your build process.

Adding a new language

As noted above, android2po will automatically process all the languages it can find, based on the .po files that exist. To add a new language, simply run

$ a2po init {LANGUAGE CODES}

For example:

$ a2po init de fr

This will create both new .po and strings.xml files for German and French.

You are also free to simply create the appropriate strings.xml files yourself, and let

$ a2po init

initialize their proper .po counterparts (in case of the init command, the languages found in the Android resource directory will be processed).

Configuration file

A configuration file can be used to avoid manually specifying all the required options. The format of the file is simply a list of command line option, each specified on a line of it's own. For example:

--no-template
# Paths - don't specify --android, default location is used.
--gettext ../locale

As you can see, comments are supported by using #, and the mechanism to automatically try to detect the directories for .po files and Android strings.xml files is still in place if you don't specify locations explicitly.

The configuration file may be specified by using the --config option. Alternatively, if a .android2po file is found in the project directory, it will be used.

See --help for a list of possible configuration options. There's also an example configuration file in example.config that you can have a look at, or use as a template for your own.

Plurals support

<plurals> are supported, but merit some additional explanation.

Android's plural support is based on CLDR keywords like "one" and "many". The rules specifying which quantity n maps to which keyword are built into Android itself, by way of the CLDR database. It is important to understand that a keyword like "one" may be used for quantities other then 1.

In the gettext system, on the other hand, each catalog has the ability to define the plural rules it wants to use itself, via an expression like nplurals=2; plural=((n == 1) ? 0 : 1). The expression returns the index of the string to use for the quantity n.

android2po converts between those two system in the following way:

• When writing .po files, it will generate a plural rule expression like above based on the CLDR data, custom-fit for the language in question. The result is a .po file that defines as many plural forms as required for the language, and your translation tool will ask for a different string for each plural form.
• During import, it will generate a <plurals> tag with the correct quantity keywords based on it's knowledge (CLDR) about which such keywords the language supports.
• The init command, having to convert existing <plurals> tags to gettext, will pick those quantity keywords the language supports, and ignore others (and display a warning in those cases).
• The export command will ensure that the catalog uses the correct plural definition, but it otherwise does not have to deal with individual plural forms / quantities.

If this is confusing, consider the issue: Android lets you define a number of different quantity keywords for each <plurals> element, but ignores all keywords that are not supported by the language (see this erroneous bug report). gettext only allows you to define a fixed number of plural rules, as many as the language purports to require via the catalog's plural rule expression.

To cleanly convert between the two systems, we are forced to ignore keywords in an Android XML resource that are really not supported - but only if Android itself would also ignore them. So view this as essentially a validation feature.

A final note: plurals can be complex (and there are many languages) and the CLDR database is regularly updated. In French, whether 0 is treated as plural or singular possibly even depends on the dialect. As such, you may find that different plural rules for the same languages are in use in the wild. android2po uses the CLDR rules, but not necessarily the same version as Android does, and Android presumably will upgrade their CLDR version over time as well. I think the goal here would be to always make android2po use a reasonably recent version of the CLDR data, and accept that old Android versions with outdated plural information might not be able to correctly internationalize some plural strings into into those languages where the data is incorrect.

Further reading:

The CLDR plural system and rules

http://unicode.org/repos/cldr-tmp/trunk/diff/supplemental/language_plural_rules.html http://cldr.unicode.org/index/cldr-spec/plural-rules

Plural information about various languages:

http://translate.sourceforge.net/wiki/l10n/pluralforms https://translations.launchpad.net/+languages https://developer.mozilla.org/en/Localization_and_Plurals

Understanding / Debugging the android2po

If something doesn't work as expected, it may be helpful to understand which files are processed how and when:

On init, android2po will take your language-neutral (English) values/strings.xml file and convert it to a .pot template.

Further on init, if there are existing values-{lang}/strings.xml files, it will take the strings from there, match them with the strings in the language-neutral values/strings.xml file, and generate .po files for these languages which already contain translations, in addition to the template. This is the only time that the values-{lang}/strings.xml files will be looked at and considered.

On export, android2po will take the language-neutral values/strings.xml file, generate a new .pot template, and then merge the new template into any existing .po catalogs, i.e. update the .po catalogs for each language with the changes. This is how gettext normally works (msgmerge). The values-{lang}/strings.xml files do not play a role here.

On 'import', android2po will only look at the .po catalogs for each language and generate values-{lang}/strings.xml files, without looking at anything else.

Notes

Initially based on:

http://code.google.com/p/openintents/source/browse/tools/Androidxml2po/androidxml2po.bash

Links of interest:

http://www.gnu.org/software/hello/manual/gettext/PO-Files.html

GNU PO file format docs.

http://docs.oasis-open.org/xliff/v1.2/xliff-profile-po/xliff-profile-po-1.2.html

Explains the gettext format according to how xliff interprets it.

http://www.artfulbits.com/Android/aiLocalizer.aspx

App to localize Android xml string files directly. They seems to be involved with the Ukrainian translation of Android itself.

http://groups.google.com/group/android-platform/browse_thread/thread/a2626195205e8543

Notes that Google internally manages Android translations in their own system.

There is a converter from and to XLIFF in frameworks/base/tools/localize, which might be what they are using. It looks pretty decent too. Why isn't this promoted more?

https://launchpad.net/intltool

Converts to and from .po und "can be extended to support other types of XML" - sounds like something we could've used? It's Perl though, ugh.