Sometimes GPG check fails, even though all commits are verified

[rallytime]

I am not sure why, but occasionally I see the GPG check fail, even though all of the individual commits are verified.

Here are some examples:

• Add multiple spm integration tests saltstack/salt#44253
• begin switching in kitchen-salt for running the test suite saltstack/salt#44259
• Fix file.managed on Windows with test=True saltstack/salt#43379

At first I thought it might have something to do with the shear number of commits in a PR because I thought i was seeing them fail only on PRs with many commits. However, that doesn't appear to be the case since I found a PR today that only had 2 commits in it.

[jarrodldavis]

Hmm... interesting. It doesn't look like the cause is a rebase since the 2-commit PR doesn't look like it was force pushed so I'm not sure what's going on. I'll need to add logging (#45) to get more information about what the app sees so I can track it down. Thanks for reporting the issue.

[rallytime]

@jarrodldavis Sounds good. Thank you for looking into it and let me know if you need anything else from me.

[jarrodldavis]

Hey @rallytime, I just merged #61 into develop and I have a development instance of the app running. If you want to help me track down the issue, you can install https://github.com/apps/gpg-development on one of your repositories so I can see if the issue happens again. Be sure to have the production instance of the app disabled on any repo you install this dev instance on to prevent any conflicts.

[jarrodldavis]

This gets even weirder… saltstack/salt#44253 and saltstack/salt#44259 have been updates with success statuses where it previous marked them as failure. Looking through the GitHub API, something triggered the app to update the statuses again on the 25th, a day after the commits in question were made…

saltstack/salt#43379 remains failed, though.

[jarrodldavis]

Hmm, so I saw a couple more pull requests with the issue today. It appears the initial open was marked as success but subsequent pushes were marked with error. After manually going through the verification steps locally and also redelivering web hooks, I can't reproduce the issue. There has to be some timing issue with the API or something.

@rallytime do you think you could switch the saltstack/salt repo over to the development deployment so I can catch the logs when this happens?

[rallytime]

@jarrodldavis - yes, I will look into getting the development deployment installed today.

[rallytime]

Ok, I have switched those over now on the https://github.com/saltstack/salt repo. Please let me know if there is anything else I can help with. :)

[jarrodldavis]

After more investigation, there appears to be an issue with the GitHub API after a pull request is updated with additional commits. My assumption is that when a pull request (or probably a branch in general) is pushed to, GitHub's signature verification service re-verifies all of the commits, and there's a delay between when the webhook is sent in response to the push and when the verification service completes.

[rallytime]

Is there a way to handle the delayed response/verification service when a pull request receives a push?

[jarrodldavis]

Other than hard coding a delay when receiving a pull_request.synchronized event or implementing a retry mechanism when encountering any unsigned commit, not really. I get the same response from the API as if the commit was actually unsigned, even though the reason field is documented with some values for when the verification service has issues.

I've reached out to GitHub Support about the issue and they've followed up with some requests for more info, so hopefully we can track down the cause of the problem. I'll keep you updated.

[rallytime]

@jarrodldavis Thanks for looking into it! I look forward to any updates.