dev/ci: deploy npm pull-through cache

[bobheadxi]

Flakes as a result of npm failures have become a quite common source of failures:

sum(count by (build) (count_over_time({app="buildkite",branch="main",state="failed"}
|= "An unexpected error occurred: \"https://registry.npmjs.org/"[1d])))

https://sourcegraph.grafana.net/goto/GEAhSC6nk?orgId=1 (I think this is a correct query)

We should investigate deploying a pull-through cache for npm to mitigate these issues, such as Verdacio:

Verdaccio is a simple, zero-config-required local private NPM registry. No need for an entire database just to get started. Verdaccio comes out of the box with its own tiny database, and the ability to proxy other registries (eg. npmjs.org), also introduces caching the downloaded modules along the way. For those who are looking to extend their storage capabilities, Verdaccio supports various community-made plugins to hook into services such as Amazon's S3, Google Cloud Storage or create your own plugin.

[bobheadxi]

Alright @kalanchan and I looked at this for a few hours, and I think Verdaccio is a non-starter - the deployment we set up is at https://github.com/sourcegraph/infrastructure/pull/3655 . How it works:

1. Install via a self-hosted registry, e.g. yarn install --registry ${verdaccio} ${package} (see below for why ${package})
2. Verdaccio acts as a pull-through cache - repeated requests successfully goes through Verdaccio and uses the cached hit

Now the problems arise: yarn.lock tracks the exact registry that was resolved. So right now, yarn.lock looks like this:

lodash@^4.14.0, lodash@^4.15.0, lodash@^4.17.10, lodash@^4.17.12, lodash@^4.17.14, lodash@^4.17.15, lodash@^4.17.19, lodash@^4.17.20, lodash@^4.17.4, lodash@^4.17.5, lodash@^4.7.0, lodash@~4.17.0, lodash@~4.17.2, lodash@~4.17.4:
  version "4.17.21"
  resolved "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz#679591c564c3bffaae8454cf0b3df370c3d6911c"
  integrity sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg==

This means that no matter how hard you try with --registry or similar configs, yarn will completely ignore you and use resolved, i.e. https://registry.npmjs.org

Now, if you use a registry, e.g. yarn install --registry ${verdaccio} ${package}:

lodash@^4.17.21:
  version "4.17.21"
  resolved "http://127.0.0.1:8080/lodash/-/lodash-4.17.21.tgz#679591c564c3bffaae8454cf0b3df370c3d6911c"
  integrity sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg==

It gets committed to yarn.lock. This means to use the registry proxy approach, yarn.lock must be completely re-resolved through the proxy, and every engineer at Sourcegraph must set up the exact same proxy locally whenever they want to dev, or we set up an ingress for the proxy that everyone must auth to in order to do anything at all.

IMO the above makes the registry proxy a complete non-starter 😬 I had expected this to work similarly to GOPROXY but it appears significantly more invasive than that

[bobheadxi]

The alternative is to look at https://yarnpkg.com/configuration/yarnrc#httpsProxy , with something like http://www.squid-cache.org/

[bobheadxi]

One idea from @jhchabran is to hot-patch yarn.lock when in CI specifically to use a specific registry, which could work but I'm not sure if it's best practice - sending this over to FPT to see if they have any ideas.

[valerybugakov]

Now the problems arise: yarn.lock tracks the exact registry that was resolved.

It is a known issue with the yarn version we use. It's solved in modern package versions. I experimented with migrating us to pnpm which also doesn't have this problem.

If it's the only blocker for using verdaccio, then it's another signal that we should upgrade our package manager. @bobheadxi, could you confirm that removing explicit registry URLs from the lock file would unblock this?

One idea from @jhchabran is to hot-patch yarn.lock when in CI specifically

Not a best practice but a workaround that people came up with because of the limitations of yarn v1.

[valerybugakov]

The alternative is to look at https://yarnpkg.com/configuration/yarnrc#httpsProxy

Note that this is the documentation for modern versions of yarn, and we need to upgrade to use it.

[bobheadxi]

It looks like this was specifically accounted for in Yarn 2: yarnpkg/yarn#5892 (comment)

I'd rather not do the hack for now if a migration is on the horizon, so I'd say this (using Yarn 2 or pnpm) is a blocker to us setting up a registry cache

[jhchabran]

I'd rather not do the hack for now if a migration is on the horizon, so I'd say this (using Yarn 2 or pnpm) is a blocker to us setting up a registry cache

@jasongornall is there a migration on the horizon btw? Because the dirty fix could still save a lot of time is it's not coming in the next 3 weeks :D

[valerybugakov]

@jhchabran, I will create a tracking issue for GitStart to pick up. They should be able to deliver it in three weeks. I will let you know if we hit any blockers along the way.

[jhchabran]

I did some tests over https://buildkite.com/sourcegraph/sourcegraph/builds/162466#_ and sadly, changing the registry in yarn.lock seems to tell yarn to re-download everything, even if stuff in /node_modules is present. This effectively add a flat 3m on all yarn related tasks.

Given the crudeness of the code to hotpatch yarn.lock, I think it's better to call it a day and wait for GitStart to update yarn to the next major version.

[valerybugakov]

@jhchabran, @bobheadxi, the Verdacio integration is unblocked by the yarn upgrade.