-
-
Notifications
You must be signed in to change notification settings - Fork 170
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New files added to mergerfs exported over NFS not showing up on ubuntu/debian NFS client with kernel >=5.6, but accessible directly. #1036
Comments
I'm dealing with some issues with my main server so I'm a bit busy with that so I likely won't get a chance to look at this for a week or so but... can you trace mergerfs? See if it is getting requests from NFS when you say it doesn't return anything? Can you do other requests? find and ls only stress opendir/readdir. What about stat, open, chmod, etc.? |
Stat and chmod calls - no difference. Updated script: sudo mkdir -p /export/mergerfs /mnt/{disk1,disk2,nfs}
sudo chmod 777 /export/mergerfs /mnt/{disk1,disk2,nfs}
curl -LO https://github.com/trapexit/mergerfs/releases/download/2.33.5/mergerfs_2.33.5.ubuntu-jammy_amd64.deb
sudo dpkg -i mergerfs_2.33.5.ubuntu-jammy_amd64.deb
sudo mergerfs -o nonempty,allow_other,use_ino,noforget,inodecalc=path-hash,cache.files=off,moveonenospc=true,category.create=mfs,dropcacheonclose=true,minfreespace=1G,fsname=mergerfs /mnt/disk\* /export/mergerfs
sudo strace -fvTtt -s 256 -p $(pgrep mergerfs) -o ~/mergerfs.strace.txt &
sudo apt -y install nfs-common nfs-kernel-server
echo -e "/export localhost(ro,fsid=0,no_subtree_check)\n/export/mergerfs localhost(rw,fsid=1,root_squash,all_squash,no_subtree_check,anonuid=1000,anongid=1000)" | sudo tee /etc/exports
sudo exportfs -ra
sudo mount localhost:/ /mnt/nfs
nfsstat -m
echo -e "\nTEST"
echo -e "\nTEST - initial state (empty dir) of nfs ls or find"
ls -la /mnt/nfs/mergerfs
find /mnt/nfs/mergerfs
echo -e "\nTEST - Create file on nfs"
touch /mnt/nfs/mergerfs/test-on-nfs
sleep 2
echo -e "TEST - File (1) missing in ls or find"
ls -la /mnt/nfs/mergerfs
find /mnt/nfs/mergerfs
echo -e "\nTEST - Create local file on mergerfs"
touch /export/mergerfs/test-local-on-mergerfs
sleep 2
echo -e "TEST - Files (2) still missing in ls or find"
ls -la /mnt/nfs/mergerfs
find /mnt/nfs/mergerfs
echo -e "\nTEST - But this files (2) are actually accessible"
ls -la /mnt/nfs/mergerfs/test-on-nfs /mnt/nfs/mergerfs/test-local-on-mergerfs
chmod 666 /mnt/nfs/mergerfs/test-on-nfs /mnt/nfs/mergerfs/test-local-on-mergerfs
stat /mnt/nfs/mergerfs/test-on-nfs /mnt/nfs/mergerfs/test-local-on-mergerfs
echo -e "TEST - Files (2) not showing up in listing even after direct requests"
ls -la /mnt/nfs/mergerfs
echo -e "\nTEST - Create local file on branch"
touch /mnt/disk1/test-local-on-disk1
sleep 2
echo -e "TEST - Now all files (3) show up in ls or find"
ls -la /mnt/nfs/mergerfs
find /mnt/nfs/mergerfs
echo -e "\nTEST - Local ls (just in case)"
ls -la /export/mergerfs
find /export/mergerfs
sudo pkill -9 strace
cat ~/mergerfs.strace.txt
|
Preferably the trace would be only of the issue itself. I can't tell from the trace what's what otherwise. There are no timestamps on the client side to correlate with the trace. trace mergerfs, issue a touch, stat, and maybe ls to the nfs share. |
Hello. Is this issue still alive ? I'm considering using mergerfs on a Raspberry Pi 4 with 2 USB3 disks (10Tb and 12Tb both formatted NTFS) having filesystems merged with mergerfs and presented to Chromecast-for-google-tv devices (using VLC) via NFS. i.e. the Pi4 will be a home media server. This issue could make that scenario not very handy ;) |
It's open, yes, because there were no followups and no verification of claimed issue. |
OK I'll give it a whirl over the next week or two providing we survive just-acquired covid :) |
Having a relatively same setup, multiple drives with common directories, using mergerfs to pool them into one directory and then share the pooled directory via NFS. If I created the directory or file on the NFS client, it showed up instantly on both the client & server. When I created it on the server, it took a while to show up on the client side. I learned to work around this by creating everything on the client side. I had a chance to rebuild a NAS, using the same setup (MergerFS + NFS) and the trouble followed. Turned out the issue was with the NFS client mount options, I needed to include 'lookupcache=none':
This made changes on the NFS server appear instantly on the client side. Files & directories have their own respective min & max cache timeout values. ref: https://www.man7.org/linux/man-pages/man5/nfs.5.html#DATA_AND_METADATA_COHERENCE. Hope that helps. |
Thanks ! |
I'm having the same or similar issue. It is exactly as described here: https://serverfault.com/questions/680969/nfs-client-does-not-see-new-files-created-on-server-caching. Which seems to suggest it may be related to ctimes? It's kind of strange because if I mount with the previously suggested option microos-pi4-1:/var/kerbol> sudo mount -o lookupcache=none 192.168.50.9:/Media /var/kerbol/media
microos-pi4-1:/var/kerbol> touch media/test
microos-pi4-1:/var/kerbol> ls media/
Anime Anime_Movies Audio_Books Books Movies Music Pictures test TV Videos
microos-pi4-1:/var/kerbol> touch media/test2
microos-pi4-1:/var/kerbol> ls media/
Anime Anime_Movies Audio_Books Books Movies Music Pictures test TV Videos
microos-pi4-1:/var/kerbol> dd if=/dev/random of=media/test3 bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.397838 s, 26.4 MB/s
microos-pi4-1:/var/kerbol> ls media/
Anime Anime_Movies Audio_Books Books Movies Music Pictures test TV Videos
microos-pi4-1:/var/kerbol> curl gooogle.com -o media/test_file
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 220 100 220 0 0 2616 0 --:--:-- --:--:-- --:--:-- 2650
microos-pi4-1:/var/kerbol> ls media/
Anime Anime_Movies Audio_Books Books Movies Music Pictures test TV Videos
microos-pi4-1:/var/kerbol> touch media/
microos-pi4-1:/var/kerbol> ls media/
Anime Anime_Movies Audio_Books Books Movies Music Pictures test test2 test3 test_file TV Videos
microos-pi4-1:/var/kerbol> touch media/test4
microos-pi4-1:/var/kerbol> ls media/
Anime Anime_Movies Audio_Books Books Movies Music Pictures test test2 test3 test_file TV Videos
microos-pi4-1:/var/kerbol> vim media/test5
microos-pi4-1:/var/kerbol> ls media/
Anime Anime_Movies Audio_Books Books Movies Music Pictures test test2 test3 test4 test5 test_file TV Videos
microos-pi4-1:/var/kerbol> rm media/test*
microos-pi4-1:/var/kerbol> ls media/
Anime Anime_Movies Audio_Books Books Movies Music Pictures TV Videos Some, but not all file operations seem to force NFS to re-read the contents of the directory, while others it just ignores. Here is my mergerfs mount options if you're interested: |
New files added to mergerfs exported over NFS not showing up on ubuntu/debian NFS client with kernel >=5.6, but accessible directly.
Observed in real world after upgrade Proxmox from 6.4 to 7.2.
Tested nfs v3 and v4 mounted locally and over network with default mount opts and config of nfs client on Ubuntu 22.04 (5.15), Debian 11 (5.10), Ubuntu 20.04 (mainline 5.6). Kernels <=5.5 working as intended.
To Reproduce
On fresh Ubuntu 22.04 VM
Expected behavior
New files added to mergerfs exported over NFS must showing up on NFS client with kernel >=5.6.
System information:
Linux nfs-srv 5.15.0-27-generic #28-Ubuntu SMP Thu Apr 14 04:55:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
mergerfs version: 2.33.5
nonempty,allow_other,use_ino,noforget,inodecalc=path-hash,cache.files=off,moveonenospc=true,category.create=mfs,dropcacheonclose=true,minfreespace=1G,fsname=mergerfs /mnt/disk\* /export/mergerfs
The text was updated successfully, but these errors were encountered: