cpm not checking version number in snapshot

[barefootcoder]

Perhaps I just have a fundamental misunderstanding of how the snapshot file is being used.

Let's say I have version 0.01 of module Foo::Bar installed. Using a cpanfile that just lists requires 'Foo::Bar';, but a snapshot file that lists Foo-Bar-0.02, and giving cpm the --cpanfile switch, the --snapshot switch, and the --resolver snapshot switch, should cpm install Foo::Bar 0.02? What I think is happening is that cpm sees Foo::Bar in the cpanfile, says, "oh, that's installed already: we're all good," and never even checks the version number listed in the snapshot.

So my question is:

• Is that what's happening, and it's supposed to happen that way?
• Is that what's happening, but it's wrong?
• Is that not really what's happening, and I've mistaken one bug for another? :-)

Hopefully you have some insight for me to get me back on track to address this problem. Thx for all you do: cpm has become a major part of our Perl module management, so we really appreciate all your hard work on this module.

[barefootcoder]

The only workaround I've been able to think of is to put the version numbers for all the modules into the cpanfile. But i was hoping to avoid that, because the versions are in the snapshot file already, and then I would have to worry about them getting out-of-sync. But so far I haven't thought of a viable alternative ...

[barefootcoder]

Any thoughts on this? I suppose I can generate a temporary cpanfile which has exact version numbers for everything, based on the ones in the snapshot. That would guarantee they'd always be in sync. But I hate to go to all that trouble if you have a quicker way, or if I'm just misinterpreting how something is supposed to be working.

Thx again for your hard work on cpm.

[skaji]

Sorry for the late reply.

This is an expected behavior.
cpm does NOT claim that "cpm install --resolver snapshot" reproduces modules with the exact versions in snapshot;
it just uses snapshot files as a resolver.

So, if you have requires 'Foo::Bar'; in cpanfile,
cpm install --resolver snapshot first checks whether you already have Foo::Bar or not.

• If so, it's fine and there is nothing to do.
• If not, then
  • cpm looks at snapshot,
  • And Foo::Bar will be resolved to https://www.cpan.org/.../Foo-Bar-0.02.tar.gz
  • And cpm will install https://www.cpan.org/.../Foo-Bar-0.02.tar.gz.

If you want to reproduce modules with the exact versions in snapshot,
you may remove ./local directory first, and run cpm install --resolver snapshot.

[skaji]

In the future, I may add such "reproducible" feature to cpm.
Even then, cpm will create the dependency tree in a clean directory first (not reusing ./local),
and replace ./local by the directory.

[barefootcoder]

Thx for the reply!

If you want to reproduce modules with the exact versions in snapshot, you may remove ./local directory first, and run cpm install --resolver snapshot.

Sadly, I don't think that's going to be practical for us ... we've got 850+ modules to install, so having to redo them all every time we want to see if anything needs to upgrade (even as fast as cpm is for such things) is not really feasible. :-(

Right now, I keep my distributions in a .yaml file and simply generate the cpanfile.snapshot from that. This is because a) the snapshot format contains info I don't want to have to maintain manually (e.g. the list of modules in each dist), and also doesn't contain some info I need (e.g. whether a version is pinned). Generating the snapshot file is fairly trivial.

So, I think my solution will be that, instead of maintaining cpanfile directly, I'll maintain a set of master cpanfiles and just generate the cpanfile I need, adding the exact version numbers from the dists.yaml file as I generate it (and, other than that, leaving it untouched). This should also be very simple, and it also allows me to fix a problem more elegantly that I had previously hacked using feature. (Mainly I explain this for anyone who gets here via a search and wonders how I "fixed" the problem.)

So I think that will work for now. I'll look forward to any future improvements for cpm that might make this process easier.

Thx again for all your hard work!

[miyagawa]

@barefootcoder:
This is the premise of Carmel - if your entire goal is about preserving what's in the snapshot, Carmel might be an alternative option for you.

@skaji:
Having said that, I was thinking of writing some guide to mix and match Carmel and cpm, i.e. to manage dependencies in development and pin versions with Carmel, then package up vendor/cache to install on CI/production systems using cpm install.

cpm does NOT claim that "cpm install --resolver snapshot" reproduces modules with the exact versions in snapshot;

I didn't realize this, and that might be a surprising behavior for the use case above. I wonder if there could be an option (off by default is fine) to stop falling back to CPAN and give an error? Carton has the option for this (carton install --deployment) and I'm thinking of adding a new option in Carmel to do the same: miyagawa/Carmel#57 cpanm does this with --mirror-only or --from option, given a specific mirror or a mirror index file from the snapshot.

[barefootcoder]

@miyagawa:

This is the premise of Carmel - if your entire goal is about preserving what's in the snapshot, Carmel might be an alternative option for you.

I had actually made a note to myself to look into Carmel, so thx for the reminder. Looking now, I see that Carmel does seem to address most of the issues that caused me to originally reject Carton as part of my solution. I'll have dig into the details to see if this is a viable alternative for my current approach (which I outline in the 4/28/21 comment above). It's turned out to have a few complications—almost entirely due to mismatches between distro versions and module versions—but seems to be workable so far. But I definitely think that simplifying the code on my side and relying on well-maintained CPAN code as much as possible will always be a win.

Happy to discuss what features I would want to see in Carmel to make it fit better, but this is probably not the right venue. :-) Although I will say that, whatever the end solution ends up being, I definitely want to keep cpm as a part of it: while it has some minor annoyances (such as this one), it has overall been the best option for module installation that I've found, and I would consider any solution that didn't include it a step backwards. ;->