Option to skip or defer baseline tests

[sourcefrog]

Today, cargo-mutants always runs "baseline" tests on a copy of the unmutated tree before mutating anything. This is in general a good idea because all the results will be invalid if the tests aren't passing in a clean tree, which might happen for example if there are just some failing tests in the working tree, or if the tests don't pass when copied.

However in some edge cases it would be good to have an option to skip this and trust the developer that they're passing:

• If there's suspected to be a bug affecting specifically the baseline, like in Baseline always runs tests with --workspace, which may include unwanted tests #151
• When developing cargo-mutants itself or experimenting with options, and you just want it to go faster

[kpreid]

What about, by default, running the baseline only if there is a test that never passes in (any of | the first N) mutants? That way it would get “real results” faster, but a bad test will still be discovered eventually.

[sourcefrog]

Oh, interesting idea!

[sourcefrog]

What about, by default, running the baseline only if there is a test that never passes in (any of | the first N) mutants? That way it would get “real results” faster, but a bad test will still be discovered eventually.

The catch with that is that the desired state is that all the tests fail: in cargo-mutants's own tree for example, every mutant is caught so all the tests always fail (unless they are unviable.)

So we could in principle print a message that "either all your mutants are caught or your tree doesn't pass tests in a tmpdir." Or, if that happens, we could run clean tests to work out which it is.

However I'd anticipate that in CI setups it would be common to run all the tests separately or in advance of mutants testing, so running them again inside cargo-mutants is a waste of time.

Maybe an enum: --baseline=before/after/none.

[sourcefrog]

Thinking about the likely performance gains: the baseline is one clean build (which will have to be done eventually anyhow, so does not really count), plus one passing test run.

There's one test run per mutant so as first approximation is that this only saves about 1/n_mutants time, which might not be only 1% or less if a tree generates hundreds of mutants.

However, that might also be a few minutes in absolute time, which is not entirely negligible, especially to the extent a person is waiting for the result, in CI or locally.

Also, currently the baseline test is not parallelized with any other tests, and so the cost is proportionately higher. (I suppose this also could be changed, maybe with another option setting.)

Also, in general a test run that passes will be slower than a test run that fails, because the failure might be caught by a relatively cheap unit test, whereas the successful run has to run every test.

However, if someone is using filters or an upcoming incremental mode, they might run only a handful of mutants, and then the cost of the baseline test becomes relatively large.

[sourcefrog]

This would be pretty good with the new sharding feature #192, which currently runs the baseline tests on every VM. Running them only once won't save any elapsed time but it will save CPU seconds, and perhaps people would want to start the mutants run after another job that checks the tests.

The simple place to start seems to be

• Add --baseline=skip
• If this is set, you must specify a timeout

[sourcefrog]

Draft docs: https://github.com/sourcefrog/cargo-mutants/pull/247/files?short_path=7b21f7b#diff-7b21f7b87fccb9c8ac0f82e3b060588d83e189cf51c5be528270ac6f5189b633

[sourcefrog]

What about, by default, running the baseline only if there is a test that never passes in (any of | the first N) mutants? That way it would get “real results” faster, but a bad test will still be discovered eventually.

I think this might work well if, later, we run individual tests (like in nextest) rather than whole test binaries. Probably lots of trees have only a couple of test targets, so it wouldn't have much discrimination.

For now, I'm adding --baseline=skip and will just say that you have to make sure they actually are all passing...