Pushes to gitlab fail with SSL error

[nlopezgi]

from bazelbuild/rules_docker#820:

When attempting to run a docker push through a container_push rule to a registry.gitlab.com target, I am recieving the following error:
registry.gitlab.com/gauntletwizard/bazel-go:{BUILD_EMBED_LABEL} was resolved to registry.gitlab.com/gauntletwizard/bazel-go:foo
F0430 20:27:05.873595 4586 fast_pusher_.py:194] Error publishing registry.gitlab.com/gauntletwizard/bazel-go:foo: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)

Users report they have made no change to their dependencies. Error is coming from containerregistry libraries so posting here. My suspicion is something changed in registry.gitlab.com that makes it so that containerregistry is no longer being able to validate certificates.

[s-garg]

@nlopezgi Is there a way to set PYTHONHTTPSVERIFY property in bazel, such that it trickles down to containerregistry? (found this thread to turn off cert verification in python: http://blog.pengyifan.com/how-to-fix-python-ssl-certificate_verify_failed/)

Also is it possible to figure out for which URL cert verification is failing? The SSL cert for https://registry.gitlab.com seems to be fine - current cert is valid from March 23rd, 2019 - May 22nd, 2020.

[s-garg]

The issue is coming from httplib2. You can reproduce the issue using the following python code:
import httplib2
h = httplib2.Http()
h.request('https://registry.gitlab.com')

And the issue can be resolved by adding the root certificate to the Http definition.

import httplib2
h = httplib2.Http('./USERTrustRSACertificationAuthority.crt')
h.request('https://registry.gitlab.com')

The root certificates used by httplib2 are coming from the cacerts.txt file.
(https://github.com/httplib2/httplib2/blob/master/python2/httplib2/cacerts.txt)

registry.gitlab.com probably switched the root CA last week and that has triggered the problem.

I don't see any option in the containerregistry registry code to specify a CACerts file.
I see three open PRs related to configuring CA Certs.

• Add new option cacerts. #52
• Bump httplib2 to 0.12.0 #134
• Add support for custom cacert #89

However, they have been open for a while. Is there any workaround for this problem that others have used?

[KaylaNguyen]

I'll try updating httplib2 for containerregistry.

[arcticwaters]

Until support is available in httplib2, use an http_archive patch: https://gist.github.com/arcticwaters/d3310d2b6e362a277501615c6ff07878. This patch works for gitlab.

Containerregistry won't register that repository if one is already defined so it has to go at the top of your WORKSPACE. Downside: you'll have to remember to update when containerregistry updates httplib2.

[s-garg]

I came up with my own hack to get it to work. But the patch approach seems better. Thanks for sharing!

[palvarez89]

This is still a problem. Is anybody looking into it?