Cannot get TemplatedConfigLoader + jinja to work

[mjmare]

I'm a Kedro newb and need some help getting TemplatedConfigLoader with a catalog.yml that contains a ninja snippet to work. The relevant parts (mostly lifted from the docs):

hooks.py:

class ProjectHooks:
    @hook_impl
    def register_config_loader(self, conf_paths: Iterable[str]) -> ConfigLoader:
        return TemplatedConfigLoader(
            conf_paths,
            globals_pattern="*globals.yml",  # read the globals dictionary from project config
            globals_dict={},  # extra keys to add to the globals dictionary, take precedence over globals_pattern
        )

conf/base/globals.yml

my_tables:
  - AA
  - BB
  - CC

part of catalog.yml

:
:

{% for table in my_tables %}
{{ table }}:
  layer: primary
  type: pandas.ParquetDataSet
  filepath: data/03_primary/{{table}}.parquet
{% endfor %}

I don't see any errors (but my pipeline does not acces the new tables yet). In some debugging sessions I see globals.yml being accessed.
However when using Kedro ipython to use the catalog.list(), the newly created data tables are not there. As if the loop is skipped.

Sure enough, when I change the first jinja line to:

{% for table in my_tables|default(['ZZ', 'XX']) %}

I see two tables 'ZZ', and 'XX' appear in the catalog. So it seems my_tables is empty.

Conclusion: maybe my understanding of globals_pattern and globals_dict is wrong. I thought they were used to populate a dict that was passed to the jinja template to use as global variables.

Please elucidate me. TIA

[datajoely]

Unfortunately you can't mix Kedro Templated globals and Jinja scoped variables out of the box. An open source user on our discord documented how to do this here:
https://discord.com/channels/778216384475693066/846330075535769601/933658953773101078

[datajoely]

Sorry I should make myself clearer - there is a big piece of research going on on issue #891

We may end up keeping Jinja support. Fundamentally iti is a solution to Kedro requiring too much config at a certain scale. This then begs the question, is this the right problem being solved? Can we find a way to reduce the config overhead on users? The Google SRE team did a really great piece of research on this topic and they highlight similar concerns.

In any case - we will likely find a way of introducing our new approach without breaking the old one, but hopefully this gives you a view of our current thinking.

[mjmare]

I'm glad to see that the Kedro team is spending considerable effort to make our config life better. ;-)
Curious what the outcome will be.

[gianniwgm]

I can't see the content of the discord link and I have the exact same issue. Could you share the solution in another way? Thanks

[mjmare]

I had the same problem initially with Discord (totally confusing UI): you have to press some small green checkmark to accept the terms.
Anyway here is the code for future reference:

class TemplatedConfigLoaderGlobals(TemplatedConfigLoader):
    def _load_config_file(self, config_file: Path) -> Dict[str, Any]:
        """Load an individual config file using `anyconfig` as a backend.
        Args:
            config_file: Path to a config file to process.
        Returns:
            Parsed configuration.
        """
        # for performance reasons
        import anyconfig  # pylint: disable=import-outside-toplevel

        if "globals" in str(config_file):
            globals = {}
        else:
            globals_pattern = "*globals.yml"
            globals = self.get(globals_pattern) if globals_pattern else {}

        return {
            k: v
            for k, v in anyconfig.load(
                config_file, ac_template=True, ac_context=globals
            ).items()
            if not k.startswith("_")
        }

class ProjectHooks:
    @hook_impl
    def register_config_loader(self, conf_paths: Iterable[str]) -> ConfigLoader:
        return TemplatedConfigLoaderGlobals(
            conf_paths,
            globals_pattern="*globals.yml",  # read the globals dictionary from project config
            globals_dict={},  # extra keys to add to the globals dictionary, take precedence over globals_pattern
        )

[datajoely]

Thanks @mjmare ! the trick to this is the ac_context=globals part which isn't in our current class

[mjmare]

How does this work in v0.18.0? My understanding is that the register_config_loader hook has gone away. Does TemplatedConfigLoader support this out of the box?

[datajoely]

This should be fixed shortly in 0.18.1

[datajoely]

#1437

[mjmare]

Thanks. What's the ETA for 0.18.1?

[merelcht]

We're aiming to release at the end of this sprint, so in approximately 2 weeks.

[mjmare]

OK, thx