Thoughts on monorepos (dependency isolation)

[astrojuanlu]

With the deprecation of micropackaging #3854, it would be good to explore monorepo approaches. What "monorepo" actually means depends on the use case.

[astrojuanlu]

One request that comes up from time to time is "1 Kedro project, different groups of dependencies". At the moment, 1 Kedro project = 1 Python package, therefore having completely disjoint dependencies is not possible. There are several possible solutions:

• 1 Python package
  • one could have a very minimal set of common dependencies (say, just kedro>=0.19.6) and then groups of dependencies per use case using PEP 621 [project.optional-dependencies].
  • There's also the draft PEP 735 about dependency groups (to be submitted in 2 weeks?)
• 1 "root" Python package, other connected packages
  • There's uv workspaces (akin to Cargo monorepos) https://docs.astral.sh/uv/concepts/workspaces/

[astrojuanlu]

Requested clarification on "non-workspaces" astral-sh/uv#5605 (comment)

[astrojuanlu]

Beyond the "different groups of dependencies" use case, today a user asked about having different packages under src:

my_project
├── src
|   ├── use_case_1
|       ├── __init__.py
|       ├── __main__.py
|       ├── pipelines
|       |   ├── use_case_1_data_engineering
|       |   ├── use_case_1_modeling
|   ├── use_case_2
|       ├── __init__.py
|       ├── __main__.py
|       ├── pipelines
|       |   ├── use_case_2_data_ingestion
|       |   ├── use_case_2_data_science

I tried this with both setuptools and PDM build backends and it worked like a charm (hatchling on the other hand chocked). Sample pyproject.toml:

[project]
name = "multi"
version = "0.1.0"
readme = "README.md"
requires-python = ">=3.11"
dependencies = []

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

However, I'm sure Kedro has some hardcoded logic that looks into a specific path inside src/. That logic also enables kedro pipeline create and other commands that manipulate the source code. So supporting this fully in Kedro might require some thought.

[astrojuanlu]

To note, PEP 735 (Dependency Groups) is now accepted. https://peps.python.org/pep-0735/

[astrojuanlu]

uv and tox already shipped support for it

...but pip hasn't, and might never do it pypa/pip#12963 (comment)

[astrojuanlu]

Useful ecosystem overview of how other languages do this https://gist.github.com/konstin/6d04f111563641beb10facb617fe0eb3

[astrojuanlu]

Yet another user concerned about the 1 Kedro pipeline = 1 set of requirements https://kedro.hall.community/evaluating-kedro-for-data-engineering-processes-qFYCwWk5VKQh

[astrojuanlu]

@marrrcin @Galileo-Galilei @sbrugman would be interested if you also agree that dependency isolation is something we need to also consider within this problem space?

I think it's a valid consideration. At the very least, there needs to be an answer as to how you handle pipelines that have multiple sets of dependencies (e.g. should data_engineering and data_science be separate pipelines, or should they be separate Kedro projects, if they have a separate set of, potentially-incompatible, dependencies?). In my past work on large Kedro projects (ages ago), I heavily leveraged monorepo approaches, as in #4147, because there is also a tradeoff to maintaining multiple Kedro projects for a single use case.

I can't find my previous comments to this extent (perhaps they were verbal), but my opinions are:

• Micro-packaging was always a bad idea (for Kedro framework to support); they were introduced to solve a QB-internal use case that never even leveraged them/built their own solution.
• However, being able to define separate requirements for top-level pipelines IS useful, especially for deployment. The story for running a pipeline locally with multiple sets of dependencies was never really there, but it seems useful.
• Supporting complex node grouping is unnecessary, at least to start with; that's akin to the micro-packaging over-engineering problem. The vast majority of users would benefit from being able to deploy each modular pipeline (with it's own namespace) separately. This also creates pretty clear boundaries where node persistence, etc. may be required.

TL;DR I agree with the idea of namespaces and supporting deployment based on namespaces, but I further posit that namespaces just being per modular pipeline is sufficient, even for deployment, and that will also make them easier to adopt.

Originally posted by @deepyaman in #4319 (comment)

[astrojuanlu]

I mostly agree with all of deepyaman's comments. I actually crave for pipelines requirements isolation, but I've never thought about requirements isolation at namespace level. This would make sense from an orchestrator point of view with "big nodes" filtered out from namespaces pipelines, but I am not sure this is what we should focus on in the short term because it's likely a hard problem.

Originally posted by @Galileo-Galilei in #4319 (comment)

[astrojuanlu]

I agree with the need for dependency isolation for large projects 👍🏻

Originally posted by @marrrcin in #4319 (comment)