downloads folder size/contents is just crazy

[jdx-john]

We have about a dozen vcpkg dependencies, which of course have their own. Copying vcpkg/downloads between machines is in theory a way to speed everything up but mine is 46,000 files. It takes multiple minutes for Windows Explorer just to discover how many there are, let alone copy them :) It's 2Gb but the issue is the incredible number of tiny files.

I thought downloads was literally just the raw files it got from the internet, but is it unpacking them there too? Is this normal... and if so is there a way to streamline it? It's going to take me about half as long to copy the files as just doing the build from scratch :)

[Neumann-A]

Only copy vcpkg/downloads without subfolders and you should be fine.

[jdx-john]

It's a waste of time anyway. I spend all this time making sure my machines have the same VS version, building cached binaries, copying everything across... then it just decide all my cache versions are incompatible and now I've spent an hour and still have to spend 2-3 hours building from scratch anyway.

[autoantwort]

If you want to cache your download folder use Asset Caching instead of copying the download folder

[jdx-john]

I never knew about that option, how useful. IIRC it might also factor something into the hash but I've lost where I read this - maybe CMake version or something? But I might try that anyway it certainly makes things easier :)

[autoantwort]

maybe CMake version or something

It hashes everything into the binary hash. All used tool versions, scripts, dependencies. You can see it in installed/<triplet>/share/<port>/vcpkg_abi_info.txt

[jdx-john]

And then that entire file is hashed, right? I think what might be causing it is versions of cmake, git or powershell. I've noticed that vcpkg doesn't seem to look for your installed VS for these tools - if CMake isn't on your %PATH% it downloads a copy, whereas if you run from a VS command prompt it doesn't. So I could easily have a situation this is different.

I did see that even though my VS was up to date, it insisted on using portable Git. I have 2.31.1.windows.1, vcpkg installed 2.31.2 (IIRC). Similarly even though I have Powershell it insists on getting Powershell Core.
It seems like if developers have already installed these tools, there is more chance of version mismatches. This might be where asset-caching would help me as it would include these things. Will vcpkg prefer its own tools over those in %PATH%?

[autoantwort]

And then that entire file is hashed, right?

I think so.

Afaik on windows it always uses a own copy of every tool and completely ignores what is on your PATH. But I have found microsoft/vcpkg-tool#138, so maybe this is only true for mac and linux

[jdx-john]

Yeah I think on Windows it must use installed tools in some cases. I started building and thought "it's crazy to be downloading tools I already have" so I tried again in a VS command prompt and then it didn't want to get CMake.

Thanks for that link, it's amazing how complicated 'simple' things can get! Forcing everyone to use consistent tools seems good especially if we can cache them all on a network drive.