When the network share goes offline...

[PyrokineticDarkElf]

I noticed the other week, after goofing up my NAS temporarily, that when the network share for storage gets unmounted or isn't mounted at boot, Frigate will create a directory /mnt/frigate and use up all the host storage.

I've done some searching and from what I gather, the container could be prevented from booting using a script, if the mount directory is not already present. I'm a little out of my depth here as I've not tried setting up scripts in a Docker compose file before. Is there any documentation on this or a resource someone might recommend?

[phabibjr]

I had a race condition where Frigate would start before fstab would mount my SMB shares causing the host drive to fill.

I switched from mounting via fstab to a systemd.mount units and changed the docker ststemd file to require the systemd mount so docker wouldn’t start until the SMB shares mounted.

I also made the mount folder immutable using chattr +i command so if the share mount fails no files can be written to that folder. This will keep the host system disk from filling up with recordings.

[PyrokineticDarkElf]

Thanks for the quick response! Sorry I couldn't offer the same haha
This seems viable but I've decided to follow @erkexzcx's method as I find it easier to have all my config in the compose file.

[erkexzcx]

Some time ago I've been messing with network shares and fstab. I always ended up spending more time than I should, while the actual fix is very easy and always work.

The secret of getting it to work perfectly is:

1. Use NFS storage instead of Samba. It is generally faster.
2. Tune NFS parameters. Below is what I am using to store recordings/events to my Synology NAS.
3. Mount NFS storage using docker-compose (docker) and not host's fstab or host's mount. Docker will handle connection to NFS and personally I never had such case when NFS is not mounted, but application still works.

Here is how my (relevant parts of) docker-compose.yml look like:

services:
  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "2048mb" # I guess 2G Should be more than enough
    volumes:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral
      - /etc/localtime:/etc/localtime:ro
      - ./frigate/config.yml:/config/config.yml # my Frigate config
      - ./frigate/db:/db # This must be on local storage
      - frigate:/media/frigate # This is using below NFS volume
      - ./frigate/labelmap.txt:/labelmap.txt # I renamed/combined some object types
      - type: tmpfs # Optional: 4GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 4000000000
    ports:
      - "5000:5000" # WebUI
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp

volumes:
  frigate:
    driver_opts:
      type: "nfs4"
      o: "addr=192.168.0.123,rsize=32768,wsize=32768,hard,timeo=600,retrans=2,noresvport,nconnect=16,vers=4.1"
      device: ":/volume2/frigate"

[PyrokineticDarkElf]

This seems to be working for me! I'm not using NFS and don't plan to at the minute. That being said, if you think the speed improvements are worthwhile, I might have to give it a go.

I modified the compose file to work with cifs:

volumes:
  frigate:
    driver_opts:
      type: "cifs"
      o: "username=myUsername,password=myPassword,iocharset=utf8,file_mode=0777,dir_mode=0777"
      device: "//192.168.0.123/Frigate"

Really appreciate the help with this!

[Loic691]

Some time ago I've been messing with network shares and fstab. I always ended up spending more time than I should, while the actual fix is very easy and always work.

The secret of getting it to work perfectly is:

1. Use NFS storage instead of Samba. It is generally faster.
2. Tune NFS parameters. Below is what I am using to store recordings/events to my Synology NAS.
3. Mount NFS storage using docker-compose (docker) and not host's fstab or host's mount. Docker will handle connection to NFS and personally I never had such case when NFS is not mounted, but application still works.

Here is how my (relevant parts of) docker-compose.yml look like:

services:
  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "2048mb" # I guess 2G Should be more than enough
    volumes:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral
      - /etc/localtime:/etc/localtime:ro
      - ./frigate/config.yml:/config/config.yml # my Frigate config
      - ./frigate/db:/db # This must be on local storage
      - frigate:/media/frigate # This is using below NFS volume
      - ./frigate/labelmap.txt:/labelmap.txt # I renamed/combined some object types
      - type: tmpfs # Optional: 4GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 4000000000
    ports:
      - "5000:5000" # WebUI
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp

volumes:
  frigate:
    driver_opts:
      type: "nfs4"
      o: "addr=192.168.0.123,rsize=32768,wsize=32768,hard,timeo=600,retrans=2,noresvport,nconnect=16,vers=4.1"
      device: ":/volume2/frigate"

Hi Thanks
What's your settings in the NFS Share on Synology ?
I enabled the IP address of the LXC which host frigate but I have thir error log

<html>
<body>
<!--StartFragment-->
failed to deploy a stack: Network frigate_default Creating Network frigate_default Created Container frigate Creating Container frigate Created Container frigate Starting Error response from daemon: error while mounting volume '/var/lib/docker/volumes/frigate_frigate/_data': failed to mount local volume: mount :/volume2/frigate:/var/lib/docker/volumes/frigate_frigate/_data, data: addr=192.168.2.10,rsize=32768,wsize=32768,hard,timeo=600,retrans=2,noresvport,nconnect=16,vers=4.1: operation not permitted
--


<!--EndFragment-->
</body>
</html>

[trip5]

Just came here to add my 2 cents. I don't know much about NFS and just started using it (Radarr & Sonarr running outside of my Synology). NFS is weird. Mounting hard appears the way to go for this type of situation as soft may lose data.

But the timeout options as specified by @erkexzcx don't appear to be honored. As best as I understand it NFS is supposed to double the timeo=600 (60 seconds) with each retransmission attempt to the maximum specified by retrans=2 but I let mine disconnect for 10 minutes and it reconnected immediately after enabling NFS... at least according to Red Hat.

Pretty sure that means retrans is ignored when used with hard. That said, Docker NFS mounting probably works differently and I'm going to guess that both options are actually ignored.

Also, it seems like specifying rsize and wsize is not needed. I may be mistaken but it seems like the 2 different versions of NFS will negotiate with each other on connection.

Keeping noresvport allows better port usage and nconnect allows multiple TCP connections so these are both good. Adding file lock to prevent two apps from altering a file at the same time seems like a good idea too.

Final result is: o: "addr=192.168.1.99,hard,lock,noresvport,nconnect=16,vers=4.1" which appears to function perfectly. Started importing a movie, disabled NFS on the Synology, waiting for this to appear in the Radarr log:

linuxserver-radarr  | [Warn] DiskScanService: Movie's root folder (/movies/) doesn't exist. 
linuxserver-radarr  | [Error] DownloadedMovieImportService: Import failed, path does not exist or is not accessible by Radarr: /movies/!Radarr-downloads/Big (1988). Ensure the path exists and the user running Radarr has the correct permissions to access this file/folder

I re-enabled NFS on the Synology and Radarr picked up right where it left off within seconds.

The one problem is that if the Synology NFS is offline while the container is started, it shows this error:
Error response from daemon: error while mounting volume '/var/lib/docker/volumes/radarr_Movies/_data': failed to mount local volume: mount :/volume1/Movies:/var/lib/docker/volumes/radarr_Movies/_data, data: addr=192.168.1.99,nolock,soft,nfsvers=4: connection refused and the container simply doesn't start.

The solution to that little issue is simple: sudo crontab -e and add @reboot /home/me/wait-for-nas.sh and that script is:

#!/bin/bash
echo Testing connection to NAS...
ping 192.168.1.99 -c 1
if [ $? -eq 0 ]; then
	echo NAS is online.  No need to restart docker containers.
else
	echo not OK...
	until ping 192.168.1.99 -c 1; do
		echo "NAS not available.  Waiting a bit..." >&2
		done
	echo Restarting docker containers.
	docker restart radarr
	docker restart sonarr
fi

Edit: realized I was posting to the Frigate Github instead of Stackexchange. This is a top result when searching for "What happens when Docker container when NFS share goes offline" so I even updated this post a few times in case anyone else stumbles upon it like I did. Apologies if I'm stepping on Frigate's toes here. Even though my info is for Radarr/Sonarr, I hope it's useful to Frigate users and anyone else who uses NFS shares with Docker too.