Proof of concept / Research: Add django-modeltranslation package and translation fields to ThemeConfig and FooterConfig

[drikusroor]

This pull request adds the django-modeltranslation package to the project's requirements and includes translation fields in the ThemeConfig and FooterConfig models. The translation fields allow for multilingual content in the application.

Configuration

The list of supported languages are to be added in base_settings.py to the LANGUAGES array or the MODELTRANSLATION_LANGUAGES array.

In the translation.py file (to be created by you) in an app (theme in my POC) you can register models and fields to be translated. In this POC I opted for translating description for ThemeConfig and translating disclaimer and privacy for FooterConfig.

Administration dashboard

Django ModelTranslation generates extra migrations in the form of newly added fields which are suffixed with the language. For example: I added the ThemeConfig's field description to the list of fields to be translated and the library adds description_en, description_nl, description_pt and description_zh_hans.

With some extra configuration, you can show these inputs in the administration dashboard. Initially, they will show up like this:

Which is fairly annoying and gets more annoying with every language you add. Luckily, they've added some auxiliary JavaScript and CSS that allow for "tabbed" inputs per language, which then look like this:

Screen.Recording.2024-07-08.at.14.51.05.mov

Unfortunately, as we can see, the markdown preview widget does not work with this so it would need to be modified too. I don't think this will be too much work, but it is something we have to keep in the back of our mind.

Retrieval of internationalised content

Below you can see a screenshot that shows the description field of the ThemeConfig of an ExperimentCollection in two browsers: Firefox set to Dutch language (left) and Safari set to English (inherited from OS system setting). We can therefore conclude it works out of the box based on the Accept-Language header that gets sent from the browser.

I'm not sure yet how we can make this work with #1177 - in which we might want to restrict or force certain languages, depending on the context. Perhaps we can do something with configuring a fallback per language or by just not providing any other language than Dutch in the case of TH Kids.

[BeritJanssen]

Thanks for digging into this! This looks pretty promising. In an ideal world, it would be good to add a language "on demand" from the admin interface per block / experiment object, without having to add fields for each new language we want to translate to: It is conceivable that there is one researcher who wants to run their experiment in, say, Japan and the Netherlands, to do some cross-linguistic / cross-cultural research, and another might want to compare Spain and the US instead. Then it becomes very cumbersome at some point that we will just have to keep expanding the number of translated fields. Unfortunately, that's just how SQL works. We'd have to either build up a custom setup with ForeignKeys, which would slow down lookups, or even go no-SQL in order to get the flexibility of my "ideal world" scenario. With Django being built around SQL, I don't think we'll find much by way of support in that direction. (I was briefly wondering whether we could integrate Elasticsearch for this, which a lot of text heavy projects I'm working on lean on, but to set this up and build the interface around it would be too time consuming to make this a realistic option at this stage.)

As for the plugin alternative of django-parler which you mentioned in another channel: this package doesn't seem to be actively supported anymore. So I think django-modeltranslations are a good way to go. I do agree that we'll have to still invest some work in tweaking the admin interface though: perhaps instead of the tabs, we have some "plus" button to "add" a language. Behind the scenes, the fields will be all there, but in the admin interface, we only show a translation field once the admin user decides to work on a translation for a given language. We should especially look into the markdown fields: if we have headings, tables, or even tags in the markdown field, it might be a nice convenience to pre-populate a translated field with the contents of the original entry, so that users don't need to re-enter markup.

In the long run, I don't think we'll replace the translation infrastructure of MUSCLE altogether by translations stored in the database. The capacity to translate I scored {n} points is something for which we'll keep using Django's localization system. But there are a lot of objects - questions, explainers, debriefs - which would benefit from database based translations. Consent forms may actually be something which we consciously decide not to add translation capacities for: they have to be approved separately for every language. In my cross-cultural research example from above, perhaps it would be cleaner to make two experiment objects, each with their own consent form, and to give separate links to participants from different languages. That being said, we might actually choose to handle translations for experiment/block related content (i.e., fields directly on these models) through object copies (which will automatically provide prepopulated fields which can be edited). So I'd say, don't merge this POC as-is, but let's perhaps focus on the Question related models first (see #1138), as these are the moving parts of the application that will need separate fields for each language in the database for sure. @albertas-jn , what do you think of the capacities of django-modeltranslation, as shown in this POC, for translating questions texts and choices? Would this be a good way to go? For questions, I would also consider showing all the language choices (preferrably with the tabbed view) as an advantage, as admin users can then feel encouraged to add translations for a given question, even if they are not actively working on supporting a language.

[drikusroor]

Thanks for digging into this! This looks pretty promising. In an ideal world, it would be good to add a language "on demand" from the admin interface per block / experiment object, without having to add fields for each new language we want to translate to: It is conceivable that there is one researcher who wants to run their experiment in, say, Japan and the Netherlands, to do some cross-linguistic / cross-cultural research, and another might want to compare Spain and the US instead. Then it becomes very cumbersome at some point that we will just have to keep expanding the number of translated fields. Unfortunately, that's just how SQL works. We'd have to either build up a custom setup with ForeignKeys, which would slow down lookups, or even go no-SQL in order to get the flexibility of my "ideal world" scenario. With Django being built around SQL, I don't think we'll find much by way of support in that direction. (I was briefly wondering whether we could integrate Elasticsearch for this, which a lot of text heavy projects I'm working on lean on, but to set this up and build the interface around it would be too time consuming to make this a realistic option at this stage.)

Technically, there could be a third solution, in which you don't have to add extra fields to a model like in django-modeltranslate and making a custom setup with key-value pairs (maybe this is what you meant with custom setup already):

List of languages to be able to use: Hard-coded list, a Language model.

ExperimentLanguage - model that relates an Experiment to a Language

• language key or FK to Language (see above)
• FK to Experiment

ExperimentTranslation

• language key or FK to Language ('nl' or 1)
• FK to Experiment (1)
• Property name / key ('description')
• Value ('Lorem ipsum')

This would make it very focused to Experiments and not suitable for Blocks, ThemeConfigs, and so on.

One could make it more generic ,like:

Translation

• language key or FK to Language ('nl' or 1)
• FK to model (1), nullable
• modeltype (Experiment), nullable - these two properties are supported by Django, for making generic relationships
• Property name / key ('description')
• Value ('Lorem ipsum')

You could either FK to a specific model, or leave it open as null and allow for generic use of translations in non-model specific translations.

This solution looks very much like solutions I have worked with in the past at previous companies, but that was often for bigger organisations with more resources to make and maintain this kind of architecture. It will probably be more work to create nice admin interfaces for this as Django won't have a built-in solution for this.

I wouldn't worry about slowing down SQL though. Our DB is still very small and SQL is designed to work with millions of records. AFAIK we haven't even optimised many things using indexes and caching or have we?

[BeritJanssen]

Thanks for the possible design for a home-baked setup of translations. Though my hunch is that this would indeed be overkill both for maintainers and users of MUSCLE.

[albertas-jn]

I think this would be nice for custom questions that could be created from admin. But for the built-in questions, the django native translation seems more fitting. For built-in questions, the translated fields would have to be filled-in automatically and they would be read-only.