Support count-based requiredStatusChecks

[jlsjonas]

What would you like Renovate to be able to do?
As we have a plethora of tests, but some (due to agent restrictions) only trigger after the first set is completed, we get a race-condition when using not-pending on it's own, combined with branching-based automerge.

Describe the solution you'd like
Allow passing a number to requiredStatusChecks (or alternative introduce a similar parameter accepting numbers) that will allow automerging when the defined number of tests are succeeded.

Describe alternatives you've considered
Listing all test ids as requiredStatusChecks is eventually intended is not implemented yet.
This would also be a good approuch, but I believe several projects would benefit from the simpler count-based approach.

Additional context
Bitbucket server (/stash) provides a "Minimum successful builds" merge check, which already does the job for PRs, however, due to other requirements for PRs (think reviewers, etc) we opted to do patch & pin automerges using branching; which avoids the merge checks.
Using the number-based approach also ensures more consistency between how PRs are checked & how automerges are checked

[rarkins]

Todo:

• an option name
• research into whether each platform returns a list/count and if they would be suitable
• determine the best way to pass check count from platform layer to workers layer

[jlsjonas]

Naming-wise I'd personally consider additionally allowing numeric values for requiredStatusChecks so it can either accept a number of tests to pass, or their actual names

In the case of bitbucket-server, the call required is actually already in place

And, similar to the current approach, getBranchStatus looks like the ideal spot to handle this; where the green check is expanded to check for X succesful builds, instead of more than 0.

[viceice]

@jlsjonas I'm against reusing the same config option because of another use-case.

Maybe i have a lot of status checks for a commit / pr, but only some of them are required. The required check are very slow and completing eg after next renovate run. If i only configure the number renovate would automerge and miss the required check.

So i would recommend to have both options and combining them with and, but ignore if not required (eg null or 0 ).

[rarkins]

I'd prefer not to overload an option to accept different types of values - e.g. a count or a list of names. It's already a bit overloaded because we use the value null to mean "ignore status checks". On the other hand, good interfaces should make it as hard as possible to configure contradicting values, and any new count option could contract requiredStatusChecks if both are configured.

In the case of bitbucket-server, the call required is actually already in place

(a) we won't implement this for one platform only - it needs to be consistent
(b) I'm aware we already have low level API calls already but what is missing is the ability to pass the count back from getBranchStatus e.g.

renovate/lib/platform/bitbucket-server/index.ts

Line 521 in d144013

): Promise<BranchStatus> {

Option 1: the minimum count is passed to getBranchStatus() and it doesn't return success unless the count is met, or
Option 2: each platform's getBranchStatus() returns both the status as well as count

Option 1 will result in duplicated logic in each platform. I think the logic better belongs in the application layer so option 2 is preferred.

[viceice]

@rarkins I think we should refactor the status check handling, so the platform code only return the needed information in a single type shared by all platforms and do check checks in worker code as you suggested.

This way we can remove a lot of duplicated code currently shared on most platforms

[jlsjonas]

If i only configure the number renovate would automerge and miss the required check.

Ah I see, in a case where you'd want let's say 10/20 minimum completed & successful already, but only 3 of them required by name? Didn't think of that use-case :)

Do note named-checks are (as far as I can tell) not actually implemented yet

(a) we won't implement this for one platform only - it needs to be consistent

Of course, I mainly gave it as an indication as I recognized the call; I suspect some (or most?) other systems will also have this info available already, or with a minor change

[jlsjonas]

todos:
an option name

requiredStatusCount perhaps?

research into whether each platform returns a list/count and if they would be suitable

The information can be gathered from at least Github, Bitbucket & Bitbucket server

determine the best way to pass check count from platform layer to workers layer

I'd propose an object containing a list of failed, in progress and successful buildnames, or (similar to this map on GH's status check) just a list of objects.
The latter could require more transforms before dealing with the actual data, but could be easier to expand upon in the future.