Support for mutating macro-generated source code.

[nordzilla]

Hi,

I tried cargo-mutants for the first time on the source code for one of my projects in which most of the mutation candidates are generated from declarative macros (primarily trait impls for various integer types).

When running cargo mutants in my unmodified source tree, I get this output:

❯ cargo mutants
Found 13 mutants to test
ok       Unmutated baseline in 15.1s build + 1.0s test
 INFO Auto-set test timeout to 20s
13 mutants tested in 1m 15s: 13 caught

However, when I run cargo expand on my library and fix any resulting compiler errors in the expanded code (due to direct use of unstable features etc.), I am able to get the following result:

Found 685 mutants to test
ok       Unmutated baseline in 8.6s build + 0.9s test
 INFO Auto-set test timeout to 20s
MISSED   src/lib.rs:1363:23: replace + with * in <impl CountDigits for i128>::count_digits_radix in 7.0s build + 0.9s test
MISSED   src/lib.rs:1424:23: replace + with * in <impl CountDigits for NonZeroI128>::count_digits_radix in 4.1s build + 0.9s test
MISSED   src/lib.rs:1497:23: replace + with * in <impl CountDigits for isize>::count_digits_radix in 4.1s build + 0.9s test
MISSED   src/lib.rs:1290:23: replace + with * in <impl CountDigits for NonZeroI64>::count_digits_radix in 4.3s build + 0.9s test
MISSED   src/lib.rs:1229:23: replace + with * in <impl CountDigits for i64>::count_digits_radix in 7.0s build + 0.9s test
MISSED   src/lib.rs:1558:23: replace + with * in <impl CountDigits for NonZeroIsize>::count_digits_radix in 7.3s build + 1.0s test
685 mutants tested in 1h 5m 15s: 6 missed, 679 caught

---

Overall, this seems like a difficult problem to solve for a number of reasons:

• At least in my case, I had all of my source code and test code in my lib.rs file, and simply replacing the lib.rs file with the expanded version would remove the tests, since the tests are also generated by macros. Using the --tests flag from cargo-expand could solve this, but that makes it a lot harder to get everything compiling again after the expansion.

• The number of edge-case modifications to the expanded source code in order to get it to compile are likely difficult to automate. Though, there may be better ways than the approach I took: I'm certainly not a cargo-expand expert.

I had a look through the existing issues and discussions, and I didn't see many conversations when searching for "macro", so I'm curious if any thought has been given to this type of functionality.

Lastly, thanks for the great work on this crate. It's both easy to use and incredibly useful out of the box with default settings.

[nordzilla]

Update

I just just found this paragraph here.

cargo-mutants typically can't do much to help with crates that primarily generate code using macros or build scripts, because it can't "see" the code that's generated. (You can still run it, but it's may generate very few mutants.)

That makes me feel like there is no intent to support this functionality (which is fine). I wonder if this section should mention that it is possible for the user to use cargo-expand prior to invoking the mutation testing.

I'd be happy to submit a PR to update those docs later, if desired.

[sourcefrog]

Lastly, thanks for the great work on this crate. It's both easy to use and incredibly useful out of the box with default settings.

Thanks so much! That is so encouraging!

I just just found this paragraph here.

Yes, it's a known and documented limitation today. I'm not opposed in principle to mutating macro-generated code, I just couldn't think of a way to make it work well automatically.

I would welcome either a PR that just adds a suggestion to manually run cargo expand, or more thoughts on how we could automate that.

[nordzilla]

Outlook

I did a bit more digging into this last night, and I think that the best path forward here is to update the documentation, rather than attempt to support macro expansion in cargo-mutants through the methods discussed above.

Exploration

My initial thought was that cargo-mutants could support something like a --macros flag that attempts to compile the macro-expanded code. I attempted to follow my manual process on a couple different crates and came up with this non-exhaustive list of feature attributes that may need to be added in order for expanded code to compile:

#![feature(rustc_attrs)]
#![feature(rustc_private)]
#![feature(panic_internals)]
#![feature(prelude_import)]

In theory, cargo-mutants could run cargo-expand and then add necessary attributes to the top of the file, but there are a number of issues that come with this:

0. Tests may be removed entirely by the expansion if they are in the same file as the source
   I mention this in the original post, but this would only work if tests are written in a different file. Replacing the source file with the cargo-expand output would remove tests unless you use the --tests flag. However, my experience with getting the expanded tests to compile with the --tests flag was not fun compared to the simple list of attributes needed without the flag.

1. These features are Nightly-only
   This, in and of itself, isn't necessarily an issue. It would just have to be well documented that one would have to run cargo +nightly mutants if Nightly is not their default.

2. These features are unstable
   This also isn't the end of the world, given that macro-expansion would be an opt-in experience. However, I think as a rule it might be best to have a bit of confidence in stability of the code features if your intent is to improve confidence in your test suite.

3. ⚠️ There is no guarantee that the expanded code will compile, or that it has the same behavior as the source code
   In the cargo-expand README it explicitly mentions that the expansion is a lossy process for debugging aid only. It even gives an example of a case in which the expanded code produces different behavior than the original macro.

Recommendation

If macro expansion is to be supported in the future, it should be through a lossless process with deterministic results and (hopefully) no use of unstable, Nightly-only features.

I think the best path forward here would be to update the documentation with a brief warning that, in addition to not being supported by cargo-mutants, manually expanding your code may yield inaccurate results and is not recommended.

I'd be happy to submit a docs PR for this later.

[sourcefrog]

I agree. A doc PR explaining in a bit more detail why it's not supported today would be welcome. We could also give some advice on how people can do it manually.