dub fetch --annotate doesn't generate complete dub.selections.lock file

[TomaSajt]

System information

• dub version: dub >=1.37.0
• OS Platform and distribution: Linux (NixOS)
• compiler version N/A

Bug Description

Note: all of the examples below also work with dub upgrade instead of dub fetch

Running dub fetch --annotate used to generate a complete dub.selections.json file.
After dub 1.37.0 it only fetches some but misses some deeper subprojects deeper in the dependency tree.

Just running dub fetch without --annotate will still work correctly, but it actually downloads the deps, which I do not want.

How to reproduce?

• clean your DUB_HOME: rm -r ~/.dub
• remove dub.selections.json if it exists: rm dub.selections.json
• depend on a subproject
  • e.g. add dependency "dcv:core" version="~>0.3.0" to dub.sdl
• run dub fetch --annotate --verbose
• see logs:

Using fixed selection dcv 0.3.0
Subpackage dcv:core@0.3.0 not found.

• get only this as the selection:

{
  "fileVersion": 1,
  "versions": {
    "dcv": "0.3.0"
  }
}

Expected Behavior

• run dub fetch --annotate --verbose
  • note: here I just copied the result from dub fetch --verbose without --annotate
• see logs similar to this, but without the actual fetching:

Using fixed selection dcv 0.3.0
Subpackage dcv:core@0.3.0 not found.
Dependency resolution result:
  dcv: 0.3.0
    Fetching dcv 0.3.0 (getting selected version)
Placing to User...
  Found dependency dcv:core 0.3.0
    Version selection for dependency bcaa (bcaa) of dcv:core is missing.
    Missing dependency bcaa 0.0.8 of dcv:core
    Version selection for dependency mir-algorithm (mir-algorithm) of dcv:core is missing.
    Missing dependency mir-algorithm >=3.15.3 of dcv:core
    Version selection for dependency mir-random (mir-random) of dcv:core is missing.
    Missing dependency mir-random 2.2.19 of dcv:core
Using fixed selection dcv 0.3.0

• get selections

{
  "fileVersion": 1,
  "versions": {
    "bcaa": "0.0.8",
    "dcv": "0.3.0",
    "mir-algorithm": "3.22.1",
    "mir-core": "1.7.1",
    "mir-linux-kernel": "1.2.1",
    "mir-random": "2.2.19",
    "silly": "1.1.1"
  }
}

---

Note: after having the dependencies downloaded into your DUB_HOME, the running the command with --annotate works as expected

[Geod24]

I didn't consider --annotate when making changes to dub fetch, so there's definitely an issue here.

However, there's a few questions here:

1. If there's a selections file, it can easily print what it would fetch at the top level;
2. Because we don't know a dependency's dependency until we fetch the recipe file for the dependency, dub fetch --annotate will not be able to generate a selections file without doing network requests, but I think this is going against the spirit of --annotate;
3. dub upgrade --annotate should IMO be the preferred option here because you're asking for dependency resolution;

[TomaSajt]

dub upgrade --annotate should IMO be the preferred option here because you're asking for dependency resolution;

I discovered this issue while using dub upgrade, but it seems like somewhere in the middle of testing this for other versions, I started using dub fetch accidentally (it seemed to do the same thing when there's no selection file).

To clarify, I am using dub upgrade --annotate as a suggestion command to run to generate the missing selections file that will be parsed by a helper CLI tool. I can just edit it to recommend running dub upgrade instead, so it's no big deal. I just didn't want people to download files twice (since the CLI tool downloads them in a special way).

[Geod24]

I just didn't want people to download files twice (since the CLI tool downloads them in a special way).

Then dub fetch is really not what you are looking for! For the record dub fetch used to not work, only the dub fetch <package_name> variant was working before the feature was overhauled in v1.37.0.

Regarding the CLI tool, my understanding is that it will become unnecessary once #1723 is implemented ?

[TomaSajt]

Regarding the CLI tool, my understanding is that it will become unnecessary once #1723 is implemented ?

Yes, if there's a proper hash based lockfile we will be able to mostly deprecate our custom lockfile format.

[TomaSajt]

Then dub fetch is really not what you are looking for! For the record dub fetch used to not work, only the dub fetch <package_name> variant was working before the feature was overhauled in v1.37.0.

Yes, when trying older dub versions, was when I realized I mixed up the two commands