Using cibuildwheel on a Gitlab instance behind a corporate proxy

[leducvin]

I am writing this to document how I could use cibuildwheel behind a corporate proxy (at least in Gitlab).

I couldn't find any information in the docs about this, nor in the Github issues or discussions.

Initially, I would get an HTTP 504 error (that was from my proxy). As I progressed with configuring this, I also got HTTP 502, and pip errors basically saying pip couldn't find packages.

Anyway, what I found is that I need to define the proxy environment variables and make sure they are passed to the container doing the build.

In particular, NO_PROXY must contain docker for the python container to be able to reach the dind container.

Also, I needed to implement docker-specific proxy configuration in .docker/config.json. I used CIBW_BEFORE_ALL for this.

# .gitlab-ci.yml

build-linux:
  image: python:3.8
  # make a docker daemon available for cibuildwheel to use
  services:
    - name: docker:dind
      entrypoint: ["env", "-u", "DOCKER_HOST"]
      command: ["dockerd-entrypoint.sh"]
  variables:
    DOCKER_HOST: tcp://docker:2375/
    DOCKER_DRIVER: overlay2
    # See https://github.com/docker-library/docker/pull/166
    DOCKER_TLS_CERTDIR: ""
    CIBW_BEFORE_ALL: >
      mkdir -p .docker &&
      echo "{"proxies":{"default":{"httpProxy": "http://10.9.8.7:8080", "httpsProxy": "http://10.9.8.7:8080", noProxy: "localhost,127.0.0.0/8,docker"}}}" > .docker/config.json
    CIBW_ENVIRONMENT_PASS_LINUX: HTTP_PROXY HTTPS_PROXY NO_PROXY
    HTTP_PROXY: http://10.9.8.7:8080
    HTTPS_PROXY: http://10.9.8.7:8080
    NO_PROXY: localhost,127.0.0.0/8,docker
  script:
    - curl -sSL https://get.docker.com/ | sh
    - python -m pip install cibuildwheel==2.16.2
    - cibuildwheel --output-dir wheelhouse
  artifacts:
    paths:
      - wheelhouse/

[gobater]

I have a similar issue and have not found a solution yet.
My corporate proxy is configured via PAC (and requires authentication). Since many linux tools/utilities rely on the environment variables (http_proxy, ...) for the proxy configuration and to avoid any security issue, we use proxydetox (https://proxydetox.colorto.cc/) which opens a connection in 127.0.0.1 (localhost)...

clearly I cannot access the host via 127.0.0.1 within the docker container, so I tried "host.docker.internal" but it does not work. I guess it's because the "NAT" does not work well with proxydetox listening only on 127.0.0.1. Not sure if using host networking would work, but there is no option in cibuildwheel to pass to docker run (just create)

Any Ideas on how to solve that?

PS: yes I know that injecting the credentials in clear text in an environment variable and using the real proxy IP works but it's not the intention...

[joerick]

docker create does accept an --expose option. You can pass arbitrary flags to docker using the CIBW_CONTAINER_ENGINE option - see create_args. You could use this to expose your proxy port of the local machine to the docker container... e.g. CIBW_CONTAINER_ENGINE=docker; create_args: --expose 8080.

Edit- Actually, the docker option you want might be --publish... But it too is supported by docker create.

[leducvin]

I'm not sure. It's just a wild idea, don't know if it makes sense, but if you create a Dockerfile that installs proxydetox and run this image as an extra service, maybe you can get the cibuildwheel container to communicate though that proxydetox container.

I'm not familiar with it, but I read that with Docker swarm it is possible to use so-called "secrets" to safely inject credentials in containers.

Maybe at some point an issue should be opened with the maintainers of cibuildwheel to see what they can suggest for users behind a proxy. Maybe they can provide official recommendations on how to proceed.

[mayeut]

It seems you can change the interface proxydetox listens to.
So instead of 127.0.0.1, you can use an IP that's reachable from the docker container.