Out of memory VM oom-kill on large packages - 11k mutants

[ASuciuX]

Hello @sourcefrog ,

We've encountered a potential issue related to memory allocation when dealing with large packages, specifically observed with the stackslib package from the stacks repository. During a complete mutants run on stackslib, the process halted after processing 7355 out of 11k.

One hypothesis for this behavior is that the memory allocated for each mutant run is not being properly deallocated upon completion. This could result in an accumulated memory usage, leading to an excessive demand for RAM as the process progresses, particularly noticeable in larger packages like stackslib.

Can you please confirm if there's a memory deallocation issue with completed mutant runs? Any insights or suggestions on this matter would be highly appreciated.

We also want to run this for the remaining mutants, without re-running the mutants done already. Is there a straightforward method to run this?

Specs:
32vcpu, 64GB ram, 2TB nvme storage

This is the output:

[sourcefrog]

Hi,

The screenshot isn't a lot to go on. Can you tell me the command that you're using to run cargo-mutants, or do you have a log file from this run? Or if this is running under GitHub Actions, point me to the action run?

You previously experimented with high -j values. If you are using -j larger than 2 or 3 that could easily lead to memory exhaustion if the test process is very resource hungry, as it seems to be in your case. The first thing I would do is remove any -j value.

It would also help to get mutants.out/debug.log which you can attach here.

One hypothesis for this behavior is that the memory allocated for each mutant run is not being properly deallocated upon completion. This could result in an accumulated memory usage, leading to an excessive demand for RAM as the process progresses, particularly noticeable in larger packages like stackslib.

It's conceivable but it seems unlikely: the main cargo-mutants process keeps relatively little state about each mutant, and almost all the resource consumption should be in rustc, the test binaries, and anything that they launch. There is no specific resource deallocation in Rust, things are just dropped.

It would be highly useful to understand which processes are consuming the memory. For example you could periodically run and record to a (text) file the output of ps vax while the tests are running.

I think it's more likely that:

• you're just running too many parallel tasks, as above, and so it's thrashing, or
• your test suite leaks resources, e.g. spawning a new daemon or process for every test which never exits
• or possibly one generated mutant induces a memory leak in the code under test

We also want to run this for the remaining mutants, without re-running the mutants done already. Is there a straightforward method to run this?

Not exactly, at the moment: I plan to add an option for just this in #204 soon.

What I would suggest at the moment is: test one file or a few at a time using -f, then you can easily pick up if one of them fails, and we might learn more about what was going wrong.

[ASuciuX]

Update: The commit tested only had 7355 mutants, instead of 11k. It finalized all of them, and then broke the VM.

RUST_BACKTRACE=full STACKS_LOG_DEBUG=1 BITCOIND_TEST=1 cargo mutants --no-shuffle -j 10  -vV --package stackslib -E ": replace .{1,2} with .{1,2} in " --output ./ --test-tool=nextest --timeout 2500 -- --all-targets --test-threads 1

This was specifically for the stackslib package, limited to 10 jobs to prevent CPU/RAM overload. The setup was stable for 2-3 days before the crash. The timeout was strategically set to the test completion time plus an extra 15-20%, for efficiency.

Thanks to the suggestion to share the debug.log, it's clear there was no more work left, confirming the count of 7,355 mutants. We had fewer mutants as we excluded those with minor operations.( check exclude regex from the command shared).

debug.log

I'm very interested in understanding the exact cause of the crash after completing the mutants. Could your share your insights and feedback when you have time? Would be very helpful. Thanks!

[sourcefrog]

Well as I said my first suggestion would be to get rid of the -j or at least cut it to -j2. Run multiple VMs with --shards instead.

I think it's going to be hard to understand the memory leak unless you get some data on which process is consuming the memory, e.g. with periodic ps vax.

[sourcefrog]

@ASuciuX @wileyj Did you work out what was happening? Did reducing --jobs fix it?

[wileyj]

your test suite leaks resources, e.g. spawning a new daemon or process for every test which never exits

this is an interesting observation - since every test boots up a mock server that syncs for a specified amount of time. it's plausible that some of those processes aren't being stopped correctly (or some issue is preventing them from being killed). the amount of memory we allocated for this test should have been enough to accomodate a situation like that though, hopefully there's something in teh debug.log that can shed more light on this.

[sourcefrog]

That seems most likely to me. Bear in mind with -j10 you are running up to 10 copies of nextest and the test suite simultaneously, so if they leak or if they need more than say 5GB each an OOM would not be surprising.

[wileyj]

i was starting to think the same - run the tests like we already have, with a sidecar script recording point in time system resources along with which pid was using the most memory (i'm assuming the pid would contain the test(s) name @ASuciuX ?).