Mix manager should set git versioning git dependencies

[1oglop1]

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

Gituhb 38.101.1

Please tell us more about your question or problem

This is a follow-up on Bad Tag name for git dependencies in Elixir project.

Problem

It looks like the mix manager handles git dependencies incorrectly and when "datasource": "git-tags", "versioning": "hex" renovate runs the hex versioning and reaches this line

renovate/lib/modules/versioning/hex/index.ts

Line 104 in 8b2d1fa

newSemver = `== ${newSemver}`;

As far as I understand mix does not support version ranges for git dependencies. And it looks like the extract.ts implementation is completely wrong.
Eg. ref: can be a tag, branch or digest. But the current implementation treats it as digest and replaces tag with SHA which is probably not the desired solution.

It produces an invalid newVersion from v.0.1 -> == v0.2 which fails to download the tag (because it does not exist) and mix deps.update fails.

Fix proposal

I think the fix can follow the logic of the existing Poetry manager, where it sets versioning to semver in case the dependency is a type of semver

renovate/lib/modules/manager/poetry/schema.ts

Lines 133 to 135 in 8b2d1fa

	if (dep.datasource === GitRefsDatasource.id && dep.currentDigest) {
	dep.versioning = gitVersioning.id;
	return dep;

Here are the use cases the mix manager should cover

    {:basic_dep, git: "https://github.com/user/repo.git"},  // skip not enough data
    {:branch_dep, git: "https://github.com/user/repo.git", branch: "main"}, // if semver support, else skip
    {:commit_dep, git: "https://github.com/user/repo.git", ref: "abc123"}, // if semver support, else skip
    {:make_dep, git: "https://github.com/user/repo.git", ref: "abc123", manager: :make}, // skip mix deps.update cannot run custom managers and rebar is not implemented
    {:tag_dep, git: "https://github.com/user/repo.git", tag: "v1.0.0"}, // if semver support, else skip

After doing more tests, it turns out the mix.exs file changes in an unexpected way, and for some reason mix.lock is not changed at all
https://github.com/jans-forks/renovate-reporoduction-31778/pull/3/files

I've made some modifications to the code and achieved this.
https://github.com/jans-forks/renovate-playground/compare/main..e5f0dce01dd3b321dd3d2e46c7bf09605cead7ed
I am still not sure about the desired behaviour for branch where renovate has changed ref: branch-name -> ref: SHA1, but I think this should only happen in a lock file not the mix.exs.

Besides the logic fix, I propose to update the docs page https://docs.renovatebot.com/modules/manager/mix/
To mention that git dependencies may require manager process may require extra git credentials. Which I achieved with customenvvariables.

Reproduction repo

• https://github.com/jans-forks/renovate-reporoduction-31778

Logs (if relevant)

Logs

DEBUG: branches info extended (repository=jans-forks/renovate-reporoduction-31778)
       "branchesInformation": [
         {
           "branchName": "renovate/mix",
           "prNo": 3,
           "prTitle": "Update mix to v0.2.0",
           "result": "done",
           "upgrades": [
             {
               "datasource": "git-tags",
               "depName": "git_lib_pkg",
               "displayPending": "",
               "fixedVersion": "v0.1.0",
               "currentVersion": "v0.1.0",
               "currentValue": "v0.1.0",
               "newValue": "v0.2.0",
               "newVersion": "v0.2.0",
               "packageFile": "el_app/mix.exs",
               "updateType": "minor",
               "packageName": "https://github.com/jans-forks/elixir_lib"
             },
             {
               "datasource": "git-tags",
               "depName": "lib_pkg",
               "displayPending": "",
               "fixedVersion": "v0.1.0",
               "currentVersion": "v0.1.0",
               "currentValue": "v0.1.0",
               "newValue": "== v0.2.0",
               "newVersion": "v0.2.0",
               "packageFile": "el_app/mix.exs",
               "updateType": "minor",
               "packageName": "https://github.com/jans-forks/elixir_lib"
             }
           ]
         }
       ]

[github-actions[bot]]

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. Discussions without reproductions are less likely to be converted to Issues.

Please follow these steps:

1. Read our guide on creating a minimal reproduction.
2. Go to our minimal reproduction template repository.
3. Select the Use this template button to create a new repository based on the template.
4. Work on the minimal reproduction in your own repository.
5. Fill out the information in your repository's README.md.
6. Add the link to your reproduction to the first post of your Discussion. If you are not the original author, you can post a new comment with the link.

If you need help with running renovate on your minimal reproduction repository, please refer to our Running Renovate guide.
Good luck,

The Renovate team

[1oglop1]

@rarkins please lemme know if there is something else I am missing.

[1oglop1]

It seems to reproduce, but now needs dependencies reduced to minimal. The other dependencies don't appear to be necessary

I am not sure what you mean, there is only a single dependency on my dummy elixir package, but it is mentioned multiple times to reproduce all/most possible cases.

Would you suggest a different way of doing so?

[rarkins]

Remove all the other dependencies

[1oglop1]

I'm sorry but I am confused by your answers. What do you mean by "other dependencies", all lines in the link are relevant to the problem. Could you clearly point out what would you like to remove?
I am trying to help but I do not think that this process makes it any easier.
https://github.com/jans-forks/renovate-reporoduction-31778/blob/main/el_app/mix.exs#L22-L28

[rarkins]

Your reproduction has 6 PRs. Why are so many needed to demonstrate one problem?

[1oglop1]

It was one PR (with a group) reproducing a complex problem, but then you complained about too much config. The behaviour of mix manager is quite complex and the current buggy solution is simplistic and naive while not considering possible combinations. So fixing one combination can break the other.

It is possible to use different notation for git entries in mix.exs:
tag: "v0.2.0", ref: "v0.1.0" but ref can also be digest, tag or even a branch name.

They are then represented in the mix.lock like this
SHA1, [reference_type]

"284def38b9c7909f9eb99d5c19b7bb7eb80f90ef", [tag: "v0.2.0"]
"4888c25b27e47369296ebd66502a3d3ec6e955db", [ref: "v0.1.0"]
"83b72d4940b13cc68e70a32aa14eb624b29b880a", [ref: "main"]
"4888c25b27e47369296ebd66502a3d3ec6e955db", [ref: "4888c25b27e47369296ebd66502a3d3ec6e955db"]

in the end the the mapping between mix.exs -> mix.lock is

tag: "v0.2.0" -> "284def38b9c7909f9eb99d5c19b7bb7eb80f90ef", [tag: "v0.2.0"]
ref: "v0.1.0" -> "4888c25b27e47369296ebd66502a3d3ec6e955db", [ref: "v0.1.0"]
ref: "main" -> "83b72d4940b13cc68e70a32aa14eb624b29b880a", [ref: "main"]
ref: "4888c25b27e47369296ebd66502a3d3ec6e955db" -> "4888c25b27e47369296ebd66502a3d3ec6e955db", [ref: "4888c25b27e47369296ebd66502a3d3ec6e955db"]

The current behaviour incorrectly evaluates tags that look like a version and changes ref from anything to digest inmix.exs this is all covered in the PR example.
I believe this is incorrect, and this behaviour should have more logic.
Ideally, the change should be propagated in both mix.exs and mix.lock.
So I'd like to understand more about the strategies or behaviour patterns of managers.
One strategy I do not fully understand is update-lockfile introduced in #27924

I'd like to discuss possible approaches for implementation

[kivra-pauoli]

Is there a renovate pull request in this repository, that I could look at? I searched for one before opening #32539, but maybe I didn't use the right keywords (?)

[kivra-pauoli]

You're right. In this case, should the manager be the one deciding on the versioning? Because there's a couple of types of versioning Mix supports and it can't just decide "later", when the manager should know what type of versioning is more appropriate.

Something like hex-git or hex-github depending on the use cases (?)

[rarkins]

Can mix ever use constraints/ranges for git dependencies?

[kivra-pauoli]

Can mix ever use constraints/ranges for git dependencies?

Not from my experience/tests, but if you prefer I can ask in other Elixir channels to get a more informed reply (lest I go to the code to check it out). It can use a tag, a branch, or a SHA. If it can do something else the documentation there probably also needs an update 😄

[1oglop1]

The mix shouldn't use ranges for git dependencies, therefore opted for "semver like tags/branch names" with pin strategy.

It seems to me that the cause of this problem is that hex versioning is being used for git dependencies when it shouldn't be
This is something I wasn't able to understand properly from the code. Where exactly is the versioning decided.
I know there is

renovate/lib/modules/manager/mix/index.ts

Line 10 in a413b75

export const defaultConfig = {

So here is a sketch of what I did for git versions in mix/extract.ts
tl; dr; set git datasource for git dependency, if git dependency is "semver like" use semver versioning.
As I already mentioned I had one edge case, but it is better to see the code as a whole.

function handleGitDependecy(args: {
  ref?: string;
  branchOrTag?: string;
  git?: string;
  github?: string;
}): PackageDependency | undefined {
  const { ref, branchOrTag, git, github } = args;

  // not a git dep
  if (!(git ?? github)) return undefined;

  // one of branch, ref or tag
  const isSemverVersion = semver.isVersion((ref ?? branchOrTag)!);

  const dep: PackageDependency = {

    
    currentValue: branchOrTag,
    datasource: git ? GitTagsDatasource.id : GithubTagsDatasource.id,
    packageName: git ?? github,
  };

  if (isSemverVersion && ref) { // ref is semver tag
    dep.currentValue = ref;
    dep.versioning = semver.id;
  } else if (isSemverVersion) { // ref is semver branch
    dep.currentValue = branchOrTag;
    dep.versioning = semver.id;
  } else if(gitVersioning.isVersion(ref)) { // ref is SHA
    dep.currentDigest = ref
  } else {
    // dep.skipReason = 'unsupported';
    dep.warnings = [{message:"hahsa",topic:"topic"}]
    dep.currentValue = branchOrTag
    dep.currentDigest = ref
  }

  return dep;
}

@rarkins Is there any place in the code where I could unit test the flow of manager->datasource->versioning? I could only find separate tests.

[kivra-pauoli]

@1oglop1, do you have a pull request I could look at?

And also, could we change this discussion's title to "Manager mix not setting proper Versioning module git for Git dependencies"?