rg initially appears to hang, Ctrl+C without effect

[shoffmeister]

I have a rather peculiar scenario here, where running rg on a directory initially appears to hang, with Ctrl+C having no effect. A second run of the same command returns results instantaneously.

Context:

• VMware Workstation 17, running Fedora Linux 37 (btrfs), with 8 cores and 32 GB memory (Tiger Lake 45H CPU) on an SSD
• directory is 2 GB of git forge clones (i.e. plenty of repos, not too deep)

Thinking that things might be "cold" at the Linux kernel level, I tried running free && sync && echo 3 > /proc/sys/vm/drop_caches && free to get all caches inside Fedora Linux 37 dropped - but rg continues to perform well. I wouldn't know how to flush VMware Workstation caches, if any.

I am at a loss to explain this (mildly irritating) behaviour - in particular the inability to Ctrl+C / SIGINT the rg process?

What could be the trigger for this behaviour?

[BurntSushi]

Dunno. If I were in your shoes, my first instinct would be to run strace and see if it's getting hung on a syscall somewhere.

As far as ^C is concerned, ripgrep very specifically does not even attempt to handle it at all. On Unix systems, the default behavior of a process that does not handle a signal is for it to terminate. So if ripgrep isn't terminating, then that smells like something wrong outside of ripgrep to me.

VMware Workstation 17

Does this imply you're in a virtualized environment? If so, maybe the issue lurks there...

It might also be helpful if you can find a way to reproduce the problem with other tools, perhaps with vanilla grep.

You also mention that rg appears to initially hang, but then all subsequent runs are fine. And even clearing your page cache doesn't cause the problem to re-appear. So when does it re-appear? Is it only when you shutdown your VM and spin it back up that the first rg call hangs?

It might also help to see if rg hangs initially in any directory, or if it's just in this one specific thing.

Lots of variables here, but I can't think of anything that would point the blame at ripgrep specifically.

[shoffmeister]

Yes, Fedora Linux 37 is virtualized.

Rebooting that Linux guest (cycle through the virtual BIOS) does not bring back the hang. Turning off the virtual machine, and turning it on again, does not bring back the hang.

I'll pay attention to what happens the next time I reboot the host system - this is my next best chance to flush out whatever the various VMware services do.

FWIW, I have (now) read that VMware Workstation performs disk buffering of its own (cf https://communities.vmware.com/t5/VMware-Workstation-Pro/How-to-disable-virtual-disk-buffering-caching-used-in-15-x-but/td-p/473339); I have no idea how to get any dependable insight into that. If VMware Workstation does, in fact, block on some syscall, in some way (through the virtual hardware), then the SIGINT might not reach the rg process, hence Ctrl+C appearing to be ineffective. Strangely enough, the (virtual) mouse cursor is wonderfully smooth all the time.

FWIW, I by no means am insinuating that rg is to blame - it is just the trigger for something I don't understand (hence this post in Discussion)

[shoffmeister]

So a fresh boot of the virtual machine host ... did not bring back the hang.

Next best opportunity: The virtual machine is a development environment with (at times) plenty of build scope. I'll accumulate some uptime with builds and retry - perhaps the root cause is a full cache (somewhere) with unwanted content which needs to be evicted before rg can perform is fantastic magic.

I am trying to gather diagnostic data by running strace --follow-forks --relative-timestamps -o rg.strace.txt rg --files-with-matches and > /dev/null

[BurntSushi]

Nice, SGTM!

And yeah not being able to reliably reproduce this is going to make it a bugger to track down.

[shoffmeister]

I believe I have an answer, found via dmesg:

[  742.523489] nvme nvme0: I/O 4 QID 8 timeout, completion polled
[  742.523495] nvme nvme0: I/O 224 QID 11 timeout, completion polled
[  773.756016] nvme nvme0: I/O 96 QID 4 timeout, completion polled

Basically, this is the Linux nvme kernel driver talking to the VMware Workstation virtual hardware NVMe controller - and all that with the virtual disk files on a very fast PCI 4.0 SSD.

The symptom of the nvme timeout is basically a reasonably "locked" system, for 30 seconds at a time, on every single nvme timeout.

This only happens when "things" are very cold - where I suspect "things" maps to an inscrutable mix of Linux caches, VMware Workstation caches, and Windows 11 caches.

IOW, this seems really to be an artefact of VMware Workstation 17.0.2 on Microsoft Windows 11 running Fedora 38.

[BurntSushi]

Thank you for following up!!!