Mutations tested unreliably: where to search for the bug? #280
-
I experienced on several different machines with two different code bases that the generated mutations are not always tested reliably. Sometimes, mutations are caught by the same test suite, sometimes not, causing the CI to fail apparently for no reason. Here is an example:
The above example originates from two subsequent PRs which both do not have changed the test suite. Thus, my question now is: have I called |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 8 replies
-
Running the failed workflow again seemed to have caught the mutant although I did not have changed the test suite either. To really catch it once and for all, I will need to create another test case. |
Beta Was this translation helpful? Give feedback.
-
I don't think you're calling it the wrong way, although you could try adding Also I'd suggest changing your CI to upload I'd also add What I think is most likely is that one of the tests is not quite deterministic. If you look in |
Beta Was this translation helpful? Give feedback.
-
OK I could easily reproduce the nondeterminicity here, so it's not something about CI:
I think I can see in your tests a place where they are not hermetic. Remember that Rust's test suite (regardless of cargo-mutants) runs multiple tests in parallel. So anything that affects global state of the process or the machine can cause interference between tests. In this particular case, interestingly, the non-hermicity never seems to cause a problem in normal runs, only under cargo-mutants, and presumably also if you had a naturally-occurring bug. |
Beta Was this translation helpful? Give feedback.
What I would suggest there is:
Make that change by hand in your working tree; see if the tests catch it.
I ran that here from the
feature/tar-handling-cli
branch and it says all the mutants were missed: