Traefik cannot find the docker socket proxy

[pisaia01]

Hi I would appreciate your help on a problem I have. I am trying to add a socket proxy for my docker containers in order to increase the security. I am using socket-proxy but for some reason I do not know, Traefik cannot communicate with it. Here is the error is get

time="2024-01-01T18:19:29Z" level=debug msg="FIXME: Got an status-code for which error does not match any expected type!!!" error="error during connect: Get \"http://dockerproxy:2375/v1.24/version\": dial tcp 172.29.0.2:2375: connect: no route to host" module=api status_code=-1
time="2024-01-01T18:19:29Z" level=error msg="Failed to retrieve information of the docker client and server host: error during connect: Get \"http://dockerproxy:2375/v1.24/version\": dial tcp 172.29.0.2:2375: connect: no route to host" providerName=docker
time="2024-01-01T18:19:29Z" level=error msg="Provider connection error error during connect: Get \"http://dockerproxy:2375/v1.24/version\": dial tcp 172.29.0.2:2375: connect: no route to host, retrying in 2.318606147s" providerName=docker

here is my docker-compose file

version: '3'

services:
  dockerproxy:
    image: wollomatic/socket-proxy:1
    command:
      - '-loglevel=debug'
      - '-allowfrom=172.29.0.3/0'
      - '-allowGET=/v1\..{1,2}/(version|containers/.*|events.*)'
      - '-shutdowngracetime=5'
      - '-watchdoginterval=3600'
      - '-stoponwatchdog'
      - '-proxyport=2375'
    restart: unless-stopped
    read_only: true
    mem_limit: 64M
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges
    user: 65534:977
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      docker-proxynet:
        ipv4_address: 172.29.0.2

  traefik:
    image: traefik:2.10.7
    restart: unless-stopped
    read_only: true
    mem_limit: 2G
    cpus: 0.75
    depends_on:
      - dockerproxy
    user: "2000:2000"
    security_opt:
      - no-new-privileges:true
    volumes:
      - $DOCKERDIR/rules:/rules
      - $DOCKERDIR/acme/acme.json:/acme.json
      - $DOCKERDIR/logs:/logs
    command:
      - --global.checkNewVersion=true
      - --global.sendAnonymousUsage=true
      - --entryPoints.http.address=:80
      - --entryPoints.https.address=:443
      - --entryPoints.traefik.address=:8080
      - --entrypoints.https.forwardedHeaders.trustedIPs=$CLOUDFLARE_IPS,$LOCAL_IPS
      - --log=true
      - --log.filePath=/logs/traefik.log
      - --log.level=DEBUG
      - --accessLog=true
      - --accessLog.filePath=/logs/access.log
      - --accessLog.bufferingSize=100
      - --accessLog.filters.statusCodes=204-299,400-499,500-599
      - --providers.docker=true
      - --providers.docker.endpoint=tcp://dockerproxy:2375
      - --providers.docker.exposedByDefault=false
      - --providers.docker.network=traefik-servicenet
      - --providers.docker.swarmMode=false
      - --providers.file.directory=/rules
      - --providers.file.watch=true 
      - --api=true
      - --api.insecure=true
      - --api.dashboard=true
    labels:
      - "traefik.enable=true"
    environment:
      - CF_API_EMAIL=$CLOUDFLARE_EMAIL
      - CF_API_KEY=$CLOUDFLARE_API_KEY
      - DOMAINNAME_CLOUD_SERVER
    networks:
      docker-proxynet:
        ipv4_address: 172.29.0.3
      traefik-servicenet:
        ipv4_address: 192.168.64.2
      
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"


networks:
  traefik-servicenet:
    external: true
    name: traefik-servicenet
  docker-proxynet:
    driver: bridge
    internal: true
    ipam:
      config:
        - subnet: "172.29.0.0/24"
          gateway: "172.29.0.1"

I know it is missing the SSL part, but I removed that to upload it online. Thus SSL is not the issue here. Have you encountered such an issue before? Note that I am using AlmaLinux for OS.

[wollomatic]

Hello, thanks for reaching out. I'm always happy to get feedback.

Seems, that socket-proxy is missing one command line:

      - '-listenip=0.0.0.0'

Without this line, socket-proxy is only listening on 127.0.0.1 which ist only inside the container in a docker network.

Best regards,
Wolfgang

[pisaia01]

Hi, thanks a lot for the fast reply 👍 unfortunately it does not work. I have also added allowhealthcheck in order to check if the proxy is healty and no luck. I have even exposed 2375 just in case, but again it does not work.

dockerproxy:
    image: wollomatic/socket-proxy:1
    command:
      - '-loglevel=debug'
      - '-allowfrom=172.29.0.0/24'
      - '-listenip=0.0.0.0'
      - '-allowGET=/v1\..{1,2}/(version|containers/.*|events.*)'
      - '-allowHEAD=/_ping'
      - '-shutdowngracetime=10'
      - '-allowhealthcheck'
      - '-watchdoginterval=3600'
      - '-proxyport=2375'
      - '-stoponwatchdog'
    restart: unless-stopped
    expose:
      - "2375"
    read_only: true
    mem_limit: 64M
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges
    user: 65534:977
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      docker-proxynet:
        ipv4_address: 172.29.0.2
    healthcheck:
      test: ["CMD", "./healthcheck"]
      interval: 10s
      timeout: 5s
      retries: 2

[pisaia01]

I have also tried to ping socket-proxy from within traefik CLI, but it does not work. Even though socket-proxy says that ping is an allowed request in its log

time=2024-01-02T16:29:18.615Z level=INFO msg="starting socket-proxy" version=1.0.2 os=linux arch=amd64 runtime=go1.21.5 URL=github.com/wollomatic/socket-proxy
time=2024-01-02T16:29:18.615Z level=INFO msg="configuration info" socketpath=/var/run/docker.sock listenaddress=0.0.0.0:2375 loglevel=DEBUG logjson=false allowfrom=172.29.0.0/24 shutdowngracetime=10
time=2024-01-02T16:29:18.615Z level=INFO msg="watchdog enabled" interval=3600 stoponwatchdog=true
time=2024-01-02T16:29:18.615Z level=INFO msg="configured allowed request" method=GET regex=^/v1\..{1,2}/(version|containers/.*|events.*)$
time=2024-01-02T16:29:18.615Z level=INFO msg="configured allowed request" method=HEAD regex=^/_ping$
time=2024-01-02T16:29:18.615Z level=DEBUG msg="checking socket availability"
time=2024-01-02T16:29:28.696Z level=DEBUG msg="checking socket availability"
time=2024-01-02T16:29:38.780Z level=DEBUG msg="checking socket availability"

[wollomatic]

Hi pisaia01, thanks for your reply.

The Log file entry from socket-proxy looks good.

I'd suggest to remove the static IP configuration to test if it works then (just remove the ipam part of the network configuration and the dedicated ip addresses in the service configuration. Then, change the allowfrom to 0.0.0.0/0 (this is okay as long you do not expose the port outside the network - other socket proxies work with this setting by default).

[pisaia01]

still not working. I have even lowered traefik version to 2.10.0 but no luck. Ping is also not working.

since I am using AlmaLinux, is there any chance selinux is affecting it? Note that I have other containers as well that are connected on traefik-servicenet which I am able to ping from traefik

[pisaia01]

managed to solve it. I have changed the network part from this

docker-proxynet:
    driver: bridge
    internal: true
    ipam:
      config:
        - subnet: "172.29.0.0/24"
          gateway: "172.29.0.1"

to this

docker-proxynet:
    name: docker-proxynet
    driver: bridge
    ipam:
      config:
        - subnet: "172.29.0.0/24"

and now it works