Ignore crate: API questions

[VorpalBlade]

Hi,

I'm looking for a quick way to traverse the entire the entire Linux file system (i.e. from /, running as root or at least with CAP_DAC_READ_SEARCH, crossing mount points) with support for custom ignore patterns (i.e. I will feed them programmatically and not care about .gitignore files in the file system). I want to use parallel traversal. This crate seems to be a good fit, but there are some things I am confused about and some things that seem missing:

• WalkBuilder::add_custom_ignore_filename: Given the name and documentation of this function I understand it as that it only takes file names? How do I do path glob ignores, rooted in /? I don't want to generate a dummy global ignore file, and store it to the file system (since add_ignore takes a path, not a impl Read unfortunately, otherwise I could possibly feed the ignores that way, though it seems silly to first generate a file then parse it again).
• Why are ParalellVisitorBuilder and ParallelVisitor handled as boxed dyn references? Is this just an optimisation for ripgreps usage patterns to prevent code bloat? Personally I will likely only have a single (or possibly two) ParallelVisitor type(s) in my program, so using dyn seems like a pessimisation. Would be nice to document the reasoning behind this.
• Am I correct in understanding that the directory walking itself is parallelised, not just the visiting (i.e. it is better than just using walkdir and doing a rayon parallel for over the results)? I found the docs to not be entirely clear on this.
• Small side note: Have you looked into io_uring for directory traversal? I know plocate uses it to build it's database, but I don't know how much it would actually speed things up.

[BurntSushi]

1. If you want to use globs specifically, use overrides.
2. I don't remember the specific reason, but I don't see any reason why it would impact perf to avoid Box. The marginal cost of virtual dispatch here seems very small.
3. Yes. That's entire point of this crate. It even parallelizes gitignore matching.
4. I doubt it would help. But someone should litigate that experiment. It probably won't be me, and it will need to be a new crate.

Note that overall, ignore is still a proof of concept. I just haven't gotten back around to incorporating all of the lessons I've learned into a refresh of the crate.

[VorpalBlade]

Thank you for your answer (very quick too!)

1. I didn't notice the override functionality before, but that seems like a good fit: I'm (planning to) build a tool for finding changes between package manager state and actual system state (i.e. which config files are changed) and then in the extension a tool for managing those changes (via user providing scripts). So glob ignores was indeed what I was planning to support.
2. It is interesting that you say dyn doesn't have that much overhead, I have more to learn here clearly (and I will probably do some benchmarking when I have some code built up)
3. Thank you, I haven't found any other crate that does this. In fact thank you for ripgrep and regex as well, you truly are a cornerstone of the rust ecosystem.
4. Won't be me either. There doesn't seem to be any nice abstractions on top of io_uring yet, and I have no interest in doing to low level bindings. I had enough of unsafe code through a career in C++.

There is already several tools outside ripgrep depending on ignore (fd-find for example), so it seems to be a pretty successful POC! Do you have a list somewhere of any of the more interesting lessons you learnt? I remember reading some blog posts you wrote on ripgrep back in the day, and those were quite interesting. Don't remember anything about ignore though.

[BurntSushi]

It is interesting that you say dyn doesn't have that much overhead, I have more to learn here clearly (and I will probably do some benchmarking when I have some code built up)

The keyword here is "marginal." I don't think in this specific instance the costs of virtual dispatch will matter. But of course there are cases where the costs of virtual dispatch are enormous. But in this case? I don't think so.

There is already several tools outside ripgrep depending on ignore (fd-find for example), so it seems to be a pretty successful POC! Do you have a list somewhere of any of the more interesting lessons you learnt? I remember reading some blog posts you wrote on ripgrep back in the day, and those were quite interesting. Don't remember anything about ignore though.

No list unfortunately. The main thing is that there are a ton of bugs on ripgrep's issue tracker related to incorrect handling of gitignore rules. But the code is in such a state (especially the gitignore matching) that I'm afraid to touch it.

The API is also IMO not so great and not that intuitive.

I basically built something for ripgrep's specific use case, but didn't really try to build a polished library that would be easy for others to use.

And yes, it does offer some pretty compelling functionality and performance advantages. Which is probably why others have used it, despite its shortcomings. :)