[feature] Default profile/settings/options/binary configuration in .conanrc

[aander80]

What is your suggestion?

Hello!

I am trying to figure out a good way to manage profiles over time. Let's assume you have multiple projects that is continuously developed over 2, 3, 5 years or more. During that time the binary configuration such as compiler version might change. To trace these changes and to be able to easily go back to old releases and make bugfixes, it would greatly simplify if the intended profile/settings+options to build with were committed to those projects' repositories and be automatically picked up by conan.

The most simple example I can think of right now is to have clang configured as compiler, but the clang version is incremented over time. If I check out a commit for a year ago, I will probably not know which version of clang was used for this release, but I want to use the same version. In my opinion, this should be committed to the repositories together with the source code since they are heavily tied together; at least it should be possible to source control this configuration.

My suggestion so far is to put this in .conanrc, but there could potentially be more suitable alternatives? I realize this could be used with .conanrc and using a conan_home configuration where you have defined default profiles, but I imagine it would be very tedious to have an isolated conan cache per repository so that you cannot share packages between repositories. Moreover, we might have some library which is used in multiple applications which are using different compiler versions, so if we were to use a "master" .conanrc file it would need to contain multiple profiles and configurations anyway and be synchronized, and at that point I guess we are back to my original problem that I most likely want this configured for each repository.

I have looked a bit at config.toml in Cargo. I am thinking that this could be a useful file to shamelessly steal from into Conan. In my specific question regarding profiles, you could define profiles here (similar to targets in config.toml) and simply build your library for -pr appa or pr appb.

I expect that this could also be used for workspaces (#15992). However, I expect that in my case we are likely to have one conan package per repository, so a workspace configuration/conan.rc will need to encompass multiple projectsindividual files. Perhaps this could be aggregated in a workspace, and allow the workspace to select which configuration to use (e.g. onlyappb`)?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @aander80

Thanks for the feedback.

The most simple example I can think of right now is to have clang configured as compiler, but the clang version is incremented over time. If I check out a commit for a year ago, I will probably not know which version of clang was used for this release, but I want to use the same version. In my opinion, this should be committed to the repositories together with the source code since they are heavily tied together; at least it should be possible to source control this configuration.

There are other ways for this. If you manage your configuration with conan config install-pkg, then the configuration itself is also versioned, and not only that, it can be updated, and it can be part of the lockfiles. So it is easy to reproduce exactly the same profiles any point later in time. The problem when having different packages in different repositories, it is not practical to keep profiles copied in all repositories, as there are sometimes projects with dozens or hundreds of them. Have you had a look to this feature?

For things like defining a default profile for a workspace, we have already started to consider this, yes, but not as part of the .conanrc, but part of the workspace, as you point out. The new workspaces predates the .conanrc, most likely it will be deprecated/superseded by it.

[aander80]

There are other ways for this. If you manage your configuration with conan config install-pkg, then the configuration itself is also versioned, and not only that, it can be updated, and it can be part of the lockfiles. So it is easy to reproduce exactly the same profiles any point later in time.

This is not something I knew, thank you for pointing it out, I will have a look at this and come back to you when I have managed to test it out!

The problem when having different packages in different repositories, it is not practical to keep profiles copied in all repositories, as there are sometimes projects with dozens or hundreds of them.

This is a very real problem, but hopefully conan config install-pkg will work just fine for us and allow us to centralize the configuration and only keep a reference to them in lockfiles!

[memsharded]

Sounds good. It is a pity, we wanted to do a blog post for the conan config install-pkg feature, but we didn't have time so far, I think it is a great feature, looking forward your feedback.

[aander80]

I have had some time to test conan config install-pkg briefly. It seems the feature itself is very useful! I have two comments regarding it so far, though.

1. It would be useful if you could run conan config install-pkg --lockfile conan.lock and install all the config_requires from the lockfile directly, rather than having to specify each config_require in the lockfile individually. I imagine this would greatly simplify CI scripts that would only have to run conan config install-pkg once, instead of parsing conan.lock, getting the config_requires and running conan config install-pkg for each config package.

2. When running conan build or conan create with a lockfile that contains config_requires, there is no verification that the locked configuration is what is actually installed in the conan cache. For instance, I tried creating two config packages: config/0.1.0 and config/0.2.0. When building with a lockfile that contained config/0.1.0 as config_requires, I could build with config/0.2.0 installed without getting any error that the locked config version did not match what is in config_version.json in the cache.

   I am thinking that if there is a config requirement in the lockfile, it should be honored and not ignored. Hence, I am thinking that perhaps an error should be raised if the config_requires in the lockfile does not match config_version.json. However, I am not sure how this should work if there is configuration that is not present in config_version.json or if e.g. only one package is locked in conan.lock but there are several with different names in config_version.json.

   This could of course also cause issues if you have multiple projects you are working on at the same time that are locking different configurations. It could be very tedious to switch to different projects and having to switch configuration all the time, although for correctness you should probably do that since the locked config_requires are in fact different.

   One solution that I am thinking of is to allow the static configuration to be loaded in at the start of conan create (and other commands). Then, when loading the lockfile, config_requires is downloaded (if needed) and loaded into the configuration in memory in the current process, rather than updating the static installation through conan config install-pkg. This means that I can run conan create with different lockfiles containing config/0.1.0 and config/0.2.0 respectively, without having to run conan config install-pkg all the time.

   A large downside to this dynamic config loading is of course that the static configuration that you are loading in initially is not locked in the lockfile, so this is flawed for reproducability and correctness. An example of where this might fail is if config/0.2.0 is statically installed. Then we try to load config/0.1.0 in memory when finding it in the lockfile, but it fails to unload/override all the static configuration from conan/0.2.0 so it is not the same config as installing config/0.1.0 in a clean cache (although this is a problem with running conan config install-pkg manually too, I suppose).

   This is a very difficult problem to solve in my opinion, but I am leaning towards perhaps erroring when running conan create if the locked config_requires does not match config_version.json.

What are your opinions on this?

[memsharded]

I have had some time to test conan config install-pkg briefly. It seems the feature itself is very useful!

Happy to hear that!

It would be useful if you could run conan config install-pkg --lockfile conan.lock and install all the config_requires from the lockfile directly, rather than having to specify each config_require in the lockfile individually. I imagine this would greatly simplify CI scripts that would only have to run conan config install-pkg once, instead of parsing conan.lock, getting the config_requires and running conan config install-pkg for each config package.

Indeed, this could be a good addition, lets give it a try

When running conan build or conan create with a lockfile that contains config_requires, there is no verification that the locked configuration is what is actually installed in the conan cache. For instance, I tried creating two config packages: config/0.1.0 and config/0.2.0. When building with a lockfile that contained config/0.1.0 as config_requires, I could build with config/0.2.0 installed without getting any error that the locked config version did not match what is in config_version.json in the cache.

This can also be interesting, yes.

We have been holding these kind of improvements so far, we would like to get first a larger user base of this feature and more extended feedback in production, to be able to get right those new functionalities.

And you introduce these in your next paragraphs, like However, I am not sure how this should work if there is configuration that is not present in config_version.json

This is exactly the kind of thing that often is about user expectations and UX, not easy to get this right without enough feedback.

I am going to assign this to next-next (2.11) release, to try to consider some of these ideas. Thanks for the feedback!