west.yml: lazy module loading

[cfriedt]

Introduction

This topic should really be of interest for anyone who uses west to pull in Zephyr components, which is likely a large majority at this time.

Generally though, the topic should be of interest to end users, TSC members, maintainers, product designers, module authors, etc.

Problem description

Modules are the primary vehicle used to scale Zephyr.

At the beginning, it was fairly inconsequential that there were 5-10 modules pulling in maybe ~100 MB of additional repositories, HALs, libraries, etc. However, as Zephyr and the Zephyr Community have grown, it's become a more frequent concern that the default behaviour (west update) pulls in several GB of repository data. Moreover, for the a common goal like running hello_world on a particular board, the vast majority of modules, and the GB of git repository data downloaded, are largely unused.

There was previously an attempt to push the problem onto users by grouping west repositories into categories (core, hal, etc). However, that does not seem to have worked as well we had hoped.

Proposed change

In an ideal world, everything would be optional. Our default should be to minimize the required packages to install and the required modules to download.

We propose something along the lines of "lazy module loading". I.e. automatically updating only the the git repositories necessary for a particular app using a particular configuration running on a particular board.

Automatically downloading dependencies during a build phase are generally not a great idea, so this should be an optional feature that is disabled by default.

Some potentially "nice" features to have would be

• maybe prompt users to download repos (if in an interactive shell)?
• maybe something like a "--fetch-only" option that does the fetching in a separate step for a given app / board / config
• maybe something like a "--dry-run" print out the list of repos that would need to be downloaded, so that people can manually do that in advance for their particular CI systems
• maybe something like a "deps --list" option

Additional notes

• It's not just west modules, but the tools (e.g. ubuntu / python packages) needed to deal with those modules
  • augment module yaml schema with necessary packages
    • ubuntu: (array of deb package-names..)
    • macos: (array of package-names..)
    • windows: (array of package-names..)
    • pip: (array of package-names..)
    • redhat: (array of package-names..)
  • each of the above would of course be optional, but every module would be required to list its own dependencies outside of the most basic requirements for building Zephyr.

• West update tooling dependency management #55056

Detailed RFC

TODO

Proposed change (Detailed)

TODO

Dependencies

TODO

Concerns and Unresolved Questions

TODO

Alternatives

Deja-vu - we have had discussions and solutions about this several times.

[cfriedt]

Some PRs with associated conversations about this topic

• thrift: module integration, samples, and tests #54013
• Introduce Pigweed Module #53760

[galak]

I think one question is what kinda of means do we convey how a given module is needed. For example, is that via some Kconfig symbols?

[cfriedt]

Also, just throwing this out there, but we could consider this as a problem for a GSoC contributor 🤷

[mgielda]

Thanks @cfriedt for opening up the issue. I think it's probably more important to us than a GSoC proposal, which after all 1) will not necessarily be picked and 2) can fail (not that we can't have a GSoC about it, but then probably we should wait until the GSoC completes with undertaking other attempts; I assume the ideal GSoC project should not be on a critical path, just to avoid unnecessary pressure on the student).

I am for an apt-like behaviour w/r to asking the user - i.e. by default it will stop on recomending packages to download and wait for approval, you have to explicitly override that behavior but then everyone assumes you know what you are doing. A --download-all flag would probably make sense too.

For determining modules, Kconfig makes sense to me too.

[cfriedt]

Yea, I just won't have time to do this myself. So cannot be the assignee unfortunately. A GSoC contributor would be great in terms of someone to offload work to. Even if a GSoC fails, the work could still be used, etc. But yea, it might not necessarily be picked.

Getting this right will not be an overnight problem though, by any means. Given that we have 55 modules right now, I would expect that part of the task would be to itemize the packages that each module depends on (pip, .deb, etc). That alone could take a considerable amount of time.

[mbolivar-nordic]

@MaureenHelm MaureenHelm assigned mbolivar-nordic 5 days ago

I welcome someone developing a package manager based on west, but I have disclaimed this assignment already elsewhere.

@cfriedt feel free to reopen and assign yourself or someone else if you're aware of a volunteer.

[cfriedt]

@mbolivar-nordic - I don't know why this was assigned to you. It was originally an RFC

[marc-hb]

I welcome someone developing a package manager based on west,

Great article
https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527

[cfriedt]

Great article https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae9c17d9527

Lol - I definitely do not want to write a package manager. YAML FTW! Maybe an additional point, for that matter, should be a flag to completely silence package recommendations, because, as the article states, there are many different scenarios, and the YAML would just be a list of recommendations.

E.g. if someone wants to build Zephyr on a Linux From Scratch system, and they have built all of the required packages, then they should be able to silence warnings.

I think we just want to list recommended packages for the somewhat fixed OS versions that Zephyr supports.

[mbolivar-ampere]

one proposal based on some recent work and discussions we had: #61505

Pardon my ignorance on West, but would it be possible to take that PR then add a west install that would allow you to install any module listed in the West manifests that's not currently installed? This way you won't have to opt into all the optionals, but would be able to pick just the ones you want. Optionals would become a bit like an a-la-cart menu. It would probably be easy to add tab complete too or at least a way to lost all the manifest items that aren't yet installed.

It wouldn't be easy in the general case. The import feature makes this entire thing nontrivial because it necessarily involves a recursive process that has network and file system impact at every step in the recursion tree. I'd like to ask that people who want to get involved in this please take the time to understand how imports work before proposing solutions.

[marc-hb]

I'd like to ask that people who want to get involved in this please take the time to understand how imports work before proposing solutions.

There's an app Github label for that:
https://github.com/zephyrproject-rtos/west/issues?q=+label%3A%22Partial+imports%22+

Unfortunately not for the faint of heart... :-(

However, as Zephyr and the Zephyr Community have grown, it's become a more frequent concern that the default behaviour (west update) pulls in several GB of repository data.

BTW west init && west update --fetch-opt=--filter=tree:0 has been much faster and rock solid in our CI:

• Consider adding config option for treeless clones (--fetch-opt=--filter=...) west#638

There is of course another Github label for this sort of optimizations too:
https://github.com/zephyrproject-rtos/west/issues?q=label%3Aperformance

[uqtsherr]

Hi,

I just made a start with migrating our project to zephyr 3.5, and found that zscilib was not available. I see that this change has been integrated into v3.5 and I assume I need to add zscilib as an optional include into our west manifest/cmakelists.

Any guide as to how I go about this?
I checked the release notes and migration guide, and west modules documentation, and couldnt see any instructions.

[fabiobaltieri]

I just made a start with migrating our project to zephyr 3.5, and found that zscilib was not available. I see that this change has been integrated into v3.5 and I assume I need to add zscilib as an optional include into our west manifest/cmakelists.

Any guide as to how I go about this? I checked the release notes and migration guide, and west modules documentation, and couldnt see any instructions.

You can enable the module again by running west config manifest.group-filter -- +optional. The change was actually #61505, not this one.

@kartben should we update the 3.5 migration guide and backport it?

[kartben]

@kartben should we update the 3.5 migration guide and backport it?

Ya I was thinking just this when reading this earlier. Frankly I am not sure how we missed it but this is very much a "required change" thing.

+1 for adding it to the migration guide and making sure this also gets backported. Luckily my guess is that many people will access the migration guide from "/latest" so the only place it will still be missing is the actual 3.5.0.

[jhedberg]

+1 for adding it to the migration guide and making sure this also gets backported. Luckily my guess is that many people will access the migration guide from "/latest" so the only place it will still be missing is the actual 3.5.0.

Instead of having the full version in the URL of version-specific documentation, would it perhaps make sense to drop the last version component and just use <major>.<minor>, i.e. instead of:
https://docs.zephyrproject.org/3.5.0/
we would have:
https://docs.zephyrproject.org/3.5/

And then the version-specific documentation for a release would always be generated based on stable branch of that release (e.g. v3.5-branch in the case of Zephyr 3.5). That way we could backport important documentation fixes, and have them published on docs.zephyrproject.org. Thoughts?

[uqtsherr]

Thanks for the quick answer.

Luckily my guess is that many people will access the migration guide from "/latest" so the only place it will still be missing is the actual 3.5.0.

just checked as a layman and I hit the latest release notes and end up on the latest migration guide. Also hit latest when I go look at the v3.4.0 guide. So in my experience this is true

[fabiobaltieri]

Opened #64619

[mrx23dot]

Noob here,
in device tree we can define what HAL to use at what given version, which I guess it fetches anyway.

So all that needs to be done is not to download everything at installation, isn't it?

[glenn-andrews]

This breaks module samples like those in samples\modules\tflite-micro

[mrx23dot]

But if I didn't select a target environments then there shouldn't be any example in the first place, otherwise it's a waste of space/bandwidth.
Ideally we would start from empty env and bring in what's needed (like in platformio).