Blank or Empty Storage Widget When network_mode Is Set to Host

[JMOdero]

Description

I have an issue where the storage widget does not display a value when my rootless homepage Podman container is set to network_mode: host. CPU and Memory resources work fine.

So your next question is why do you have it set to network_mode: host? If I set it to anything else (default, network_mode: container:vpn, etc) I get various patterns of API errors from the installed widgets. The only reliable way for me to have everything run, except the storage widget, is by using network_mode: host... and even that has another quirk. I have to launch the container and then restart it for the formatting to be correct. After that though everything is fine except the storage widget.

I really do not want a container running in network_mode: host on my server 24/7, which has limited my use of homepage. Ultimately I would like to use a more restrictive mode, but the API issues with the other modes I've tried render homepage unusable.

I originally had no desire to monitor my OS disk but I added it just to rule out any issues with USB direct attached storage. When adding my OS disk I noticed that I did not add the USB disk as a /mnt. I just used the /app/config volume that exists on that disk. I did the same with a volume called /test that is on my OS drive. I don't know if that makes any difference. With this configuration the internal OS drive has the same behavior as the USB HDD. No difference. Works perfectly with everything other than network_mode: host.

Anyway I thought it was really odd that changing the network mode would have such an effect on the storage widget.

My setup...

Apple M1 Mac mini
Storage in question is a single USB HDD, but I did test my internal OS drive and it has the same problem.
Running bare metal Linux headless...

Operating System: Fedora Linux Asahi Remix 40 (Server Edition)
     CPE OS Name: cpe:/o:fedoraproject:fedora-asahi-remix:40
          Kernel: Linux 6.10.9-402.asahi.fc40.aarch64+16k
    Architecture: arm64

Client:       Podman Engine
Version:      5.2.3
API Version:  5.2.3
Go Version:   go1.22.7
Built:        Mon Sep 23 19:00:00 2024
OS/Arch:      linux/arm64

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

My homepage yaml...

services:
  homepage:
    image: ghcr.io/gethomepage/homepage:latest
    container_name: homepage
    security_opt:
      - label=type:homepage.process
    network_mode: host
    ports:
      - 3000:3000
    volumes:
      - /mnt/blahblah/blahblah/blahblah/homepage/config:/app/config
      - /home/blahblah/blahblah/test:/test:ro
    restart: unless-stopped

Section of the widgets.yaml...

- resources:
    cpu: true
    memory: true
    disk:
      - /app/config
      - /test

From inside the homepage Podman container...

/app # df /app/config
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda             976762584 949561984  25926752  97% /app/config
/app # df /test
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/nvme0n1p6       139615232  17042036 120340412  12% /test
/app # 

I am using a vpn container as my network mode for most of my other containers. Something like gluetun which is why I tried network_mode: container:vpn in my homepage yaml and had poor results.

I noticed #2509 was a discussion about an "empty disk widget" but didn't mention network_mode at all. I wonder if they had the same issue.

homepage version

v0.9.10 (132d8f4, Sep 26, 2024)

Installation method

Docker

Configuration

Storage widget

Container Logs

Container logs are pristine when using network_mode: host...

Listening on port 3000

The logs are a disaster with any other type of network I've tried. Just a sample...

[2024-10-06T17:25:49.606Z] error: <httpProxy> [
  500,
  Error: connect ETIMEDOUT 192.168.1.69:32400
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16) {
    errno: -110,
    code: 'ETIMEDOUT',
    syscall: 'connect',
    address: '192.168.1.69',
    port: 32400
  }
]

[2024-10-06T17:25:49.739Z] error: <httpProxy> Error calling https://192.168.1.69:9443/api/endpoints/2/docker/containers/json...
[2024-10-06T17:25:49.739Z] error: <httpProxy> [
  500,
  Error: connect ECONNREFUSED 192.168.1.69:9443
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1555:16) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '192.168.1.69',
    port: 9443
  }
]
[2024-10-06T17:25:49.739Z] error: <credentialedProxyHandler> HTTP Error 500 calling https://192.168.1.69:9443/api/endpoints/2/docker/containers/json?all=1
[2024-10-06T17:25:52.425Z] error: <httpProxy> Error calling https://192.168.1.69:9443/api/endpoints/2/docker/containers/json...

Browser Logs

I get a 304 Not Modified response with network_mode: host.

GET
	http://192.168.1.69:3000/api/widgets/resources?type=disk&target=/app/config
Status               304 Not Modified
Version              HTTP/1.1
Transferred          206 B (2 B size)
Referrer Policy      strict-origin-when-cross-origin
DNS Resolution       System

{
	"GET": {
		"scheme": "http",
		"host": "192.168.1.69:3000",
		"filename": "/api/widgets/resources",
		"query": {
			"type": "disk",
			"target": "/test"
		},
		"remote": {
			"Address": "192.168.1.69:3000"
		}
	}
}
Status              304 Not Modified
Version             HTTP/1.1
Transferred         206 B (2 B size)
Referrer Policy     strict-origin-when-cross-origin
DNS Resolution      System

Response Header...

HTTP/1.1 304 Not Modified
Content-Type: application/json; charset=utf-8
ETag: "bwc9mymkdm2"
Vary: Accept-Encoding
Date: Sun, 06 Oct 2024 20:25:25 GMT
Connection: keep-alive
Keep-Alive: timeout=5

Warnings from browser...

Layout was forced before the page was fully loaded. If stylesheets are not yet loaded this may cause a flash of unstyled content. markup.js:250:53
unreachable code after return statement main-0ddda9b10a00d3ad.js:1:21512

When I delete the network_mode: host line from my yaml and let podman just do its thing I get a 200 code instead of 304...

Status                  200 OK
Version                 HTTP/1.1
Transferred             367 B (151 B size)
Referrer Policy         strict-origin-when-cross-origin
DNS Resolution          System

{
	"GET": {
		"scheme": "http",
		"host": "192.168.1.69:3000",
		"filename": "/api/widgets/resources",
		"query": {
			"type": "disk",
			"target": "/test"
		},
		"remote": {
			"Address": "192.168.1.69:3000"
		}
	}
}

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
ETag: "6o42qpq6vv47"
Content-Length: 151
Vary: Accept-Encoding
Date: Sun, 06 Oct 2024 20:46:39 GMT
Connection: keep-alive
Keep-Alive: timeout=5

Troubleshooting

N/A

[shamoon]

Podman != docker, its often billed as a drop-in replacement, but in my experience it often isn't.

Regarding the disks, you can enable debug and check the output in the logs, it'll give more info

[JMOdero]

Podman != docker, its often billed as a drop-in replacement, but in my experience it often isn't.

Regarding the disks, you can enable debug and check the output in the logs, it'll give more info

I added LOG_LEVEL=debug to my hompage yaml file...

services:
  homepage:
    image: ghcr.io/gethomepage/homepage:latest
    container_name: homepage
    security_opt:
      - label=type:homepage.process
    network_mode: host
    ports:
      - 3000:3000
    environment:
      - LOG_LEVEL=debug
    volumes:
      - /mnt/blahblah/blahblah/blahblah/homepage/config:/app/config
      - /home/blahblah/blahblah/test:/test:ro
    restart: unless-stopped

I went back and checked the homepage log file. I assume these are the lines that are pertinent to the the "resources" widgets...

[2024-10-06T23:11:33.087Z] debug: <resources> timeData: {"current":1728256293087,"uptime":94931.22,"timezone":"GMT+0000","timezoneName":"UTC"}
[2024-10-06T23:11:34.078Z] debug: <resources> fsSize: []
[2024-10-06T23:11:34.080Z] debug: <resources> fsSize: []
[2024-10-06T23:11:34.082Z] debug: <resources> memory: {"total":7917830144,"free":1137524736,"used":6780305408,"active":3573448704,"available":4344381440,"buffers":475136,"cached":4196663296,"slab":307986432,"buffcache":4505124864,"swaptotal":16507699200,"swapused":380633088,"swapfree":16127066112,"writeback":0,"dirty":573440}

The uptime and memory widgets seem to be working fine. I assume the "fsSize: []" entries are for the two storage drives. I couldn't find it earlier but fsSize was discussed before in #3795.

Anyway I thought I was on to something with that network thing. That must have been what I was messing with when the storage widget was working. I didn't connect the two at the time.

I am baffled how network_mode: host only seems to negatively affect the storage widget.

And you are right... Podman≠Docker. It works most of the time for my simple needs as a noob, but I can see where it can get problematic.

[shamoon]

Sounds like a complicated setup tbh, but yea, fsSize is the complete list of disks returned from https://systeminformation.io which is what we use for this.

You can try glances as an alternative

[JMOdero]

Totally forgot to try running the container with the --privileged flag. When I did with network_mode: host everything worked. So I guess I need to add some capability to the the container if I want that widget to work as intended.

Anyway I hope this information helps someone.

Thanks for the help.

[JMOdero]

Sorry for the delay in getting back.

As I said the solution to all the errors I was getting with my rootless Podman setup was eschewing Podman's default network settings for containers. I don't know if this is THE solution or if there are any other better solutions.

Unfortunately I am kind of new to the world of containers and I know bits and pieces, but I don't have a comprehensive grasp of the subject. I start setting up my containers using docker compose in portainer. Podman for the most part works fine with docker compose yaml files. Once I get the docker compose yaml files 95% of the way I want them I translate them into quadlets using podlet. This allows the containers to run as systemd services in fedora.

I say all that because I have no clue how to do networking with docker compose. I will explain what I did with quadlets and someone can figure out what the docker version is.

I will use the example of homepage. First I created a directory called homepage in /home/URUSERNAME/.config/containers/systemd. In that folder I put the quadlet file for homepage (homepage.container). I then created a file called homepage.network. The contents of that file are as follows:

[Unit]
Description=homepage network
After=network-online.target
 
[Network]
NetworkName=homepage
Subnet=10.95.0.0/16
Gateway=10.95.0.1
 
[Install]
WantedBy=default.target

As far as I can tell the key components are the subnet and gateway. WantedBy just ensures the network is always started at boot.

Then in my homepage.container quadlet I just reference homepage.network as follows:

[Container]
AutoUpdate=registry
ContainerName=homepage
Image=ghcr.io/gethomepage/homepage:latest
Network=homepage.network
PublishPort=3000:3000
SecurityLabelType=homepage.process
Volume=/blahblah/blahblah/blah:/app/config

[Service]
Restart=always

So I just added the line... Network=homepage.network. Ignore the SecurityLabelType= line. That was added just to deal with some SELinux issues. AutoUpdate=registry was added so that a timer can check for updates every few weeks and automatically update the container image... It is one of the benefits of using quadlets. No more watchtower.

If you are using pods the process is a bit different. Setting up the network file is the same, but instead of referencing the network in the container file you reference it in the pod file.

My pihole.pod file looks like this...

[Pod]
Network=pihole.network
PodName=pihole
PublishPort=53:53
PublishPort=53:53/udp
PublishPort=80:80
PublishPort=5353:5353/tcp
PublishPort=5353:5353/udp

The pi-hole pod contains pi-hole and unbound.

Once all your network, container, and pod files are set up the way you want...

$ systemctl --user daemon-reload

Then start your containers with...

$ systemctl --user start homepage.service