Increasing RipGrep speed for parsing large content size of various binary and text files

[ddk00]

First of all, thank you for this great tool!

I am currently conducting a research. One of the key components of that research is utilizing RipGrep at the highest possible performance level in order to parse thru enormous content size of various binary and text files.

My current command is: time rg -uuu -L <word> /<directory> > rg_output.txt
I am benchmarking it on the same OS across 3 different hardware sets with 3 different virtualization power specifications.

That performance test runs against a directory slightly less than 4 GB filled with various data and types. On average it completes in under 50 seconds.
In plans to run it against tens of TBs of data with sometimes unknown data types.

I assume utilization of pre-processors for some file types can help speed it up.

Would raising the amount of threads help in this situation? Or RG already uses the best and optimal amount of threads automatically?

For further research, I plan to run RG on a base Operating System as I noticed that some virtualized systems underperform due to VM software peculiarities. That new system will have elevated power specifications and I would like to make sure that I squeeze the best performance out of it while using RG.

Would you be able to help me and suggest if there are any best use combinations in order to increase speed, please?
And maybe if there are any known hardware and/or OS dependencies regarding the execution speed?

Thank you!

Sincerely,
Danil.

[BurntSushi]

First of all, thank you for this great tool!

I am currently conducting a research. One of the key components of that
research is utilizing RipGrep at the highest possible performance level in
order to parse thru enormous content size of various binary and text files.

I'll do my best to help, but I think there is a really important thing to
understand first: without a real world reproduction that I can myself run and
experiment with, the best I can do is wave my hands and make wild guesses.

The key thing to realize is that there are many variables that influence
ripgrep's performance, and it's quite literally impossible for you to
characterize all of them. For example, searching the single byte z on
terabytes of files all containing the letter a will have markedly different
performance characteristics than if you were searching for a. That's a
pathological example of how both the corpus and the query matter for
performance, but it's supposed to demonstrate that it matters. There are
many non-pathological examples too, and it's impossible for you to communicate
all of them without a real reproduction.

My current command is: time rg -uuu -L <word> /<directory> > rg_output.txt
I am benchmarking it on the same OS across 3 different hardware sets with 3
different virtualization power specifications.

For benchmarking, I would just make sure that <word> is representative of
what you'll actually be searching for.

I would generally expect best performance on native hardware, not virtualized.

That performance test runs against a directory slightly less than 4 GB filled
with various data and types. On average it completes in under 50 seconds.
In plans to run it against tens of TBs of data with sometimes unknown data
types.

Here's one example of me hand-waving but providing a concrete guess as to the
performance of a particular case with your corpus size:

• Assuming your directory tree is TBs big and you want to search all of it.
• The data in the directory tree is "normal" insomuch as it's not pathological.
  For example, it's not all completely random data or it's not just the same
  byte repeated over and over.
• The directory tree is in non-volatile storage, i.e., some HHD or SSD.
• Your query is a single long word, say, mushrooms. Not a regex pattern.
  Just a word.
• You run ripgrep with -jN where N is the number of physical CPU cores
  available in this environment.

In this case, I would generally expect you to saturated the bandwidth of your
non-volatile storage. It's not guaranteed to do so, but if it didn't, I would
be a little surprised and would need to dig in deeper to understand.

In contrast, if your query was, say, \w{9}, then I'd not expect you to
saturate the bandwidth of your non-volatile storage. In other words, your
search would be CPU bound.

I'd expect something like \wmushrooms\w to saturate I/O just like
mushrooms, because ripgrep is smart enough to search for mushrooms first
before running the regex engine. But with something like \w{9}, there are no
shortcuts. It's just the regex engine. And a regex engine is generally not
going to saturate, say, 6 GB/s read speeds on PCIE SSDs. Now, it might saturate
100 MB/s read speeds of a HHD. But you didn't say how your data is stored, so
it's hard to say!

See how the specific query (and haystack) matters? It really matters. And the
specific type of storage you're using matters too. If you're using SSDs, then
maybe throwing more CPUs at the problem will help if your queries are
complicated.

What this means is that some part of my advice to you is essentially "use
simple queries." How can you tell whether your query is "simple" enough for
ripgrep to make it go fast? I know how to do it. I don't know how to teach it
in a short GitHub comment though. The --trace flag can help because it will
tell you whether it extracted a fast set of literals. That doesn't always mean
ripgrep will be super fast, but it's a good indicator.

I assume utilization of pre-processors for some file types can help speed it
up.

Maybe. Hard to say without specifics. But you should be able to test this in
isolation.

Would raising the amount of threads help in this situation? Or RG already
uses the best and optimal amount of threads automatically?

ripgrep cannot know a priori what the optimal number of threads is for every
workload. That's not possible. It could probably do better than it does today.
But its heuristics are probably not suitable for, say, a 64 core CPU. In that
case, you probably want to use -j to increase the number of threads ripgrep
uses.

But this is a good example of a question where I'd say: why are you asking?
Why not try it yourself? Your benchmark only takes 50 seconds. Try some
different values of -j and see how it works!

For further research, I plan to run RG on a base Operating System as I
noticed that some virtualized systems underperform due to VM software
peculiarities. That new system will have elevated power specifications and I
would like to make sure that I squeeze the best performance out of it while
using RG.

Would you be able to help me and suggest if there are any best use
combinations in order to increase speed, please? And maybe if there are any
known hardware and/or OS dependencies regarding the execution speed?

Without being able to fully explain it, I would say:

• Use Linux.
• Avoid virtualization if you can.
• Avoid network attached storage (e.g., EBS on AWS).
• Use SSDs if you can. Ideally PCIE. You should be able to get 6 GB/s total
  read speeds. Or thereabouts.

With that said, there are some really critical aspects of your problem that
you've omitted. Here are some that I can see:

• How much money do you have to spend?
• How often does your corpus change?
• How often are you running queries?

It may very well be the case, for example, that you would be better served by
throwing all of your data into an Elasticsearch/SOLR/OpenSearch cluster. It
will take (much) longer to get it setup and operating, but it will respond with
results to queries orders of magnitude faster than ripgrep. It might not make
sense to do this if you just have to run one query or two on a corpus, but if
you need to run a lot of them? Hell yeah it will make sense to set up a real IR
system for this.