Allow specifying the degree of parallelism

[MatisseHack]

Description

I have several test projects written with xUnit that I want to run in CI. These unit tests are IO bound and run much more quickly when parallelized with dotnet vstest --parallel. However, machines in my CI system (GitHub Actions) only have two CPU cores, which limits test execution to two concurrent assemblies. I would like to speed up my tests when run in CI by increasing the degree of parallelism further.

I added the following xunit.runner.json file to my project, but it only affected parallelization within each assembly.

{
  "parallelizeAssembly": true,
  "maxParallelThreads": -1
}

I was very surprised that parallelizeAssembly seemingly has no effect when run with the dotnet CLI. I assume MSBuild and VSTest create separate xUnit test runners for each assembly, which is why this setting has no effect?

I was able to work around that issue by using dotnet test --no-build -maxCpuCount:8, but the maxCpuCount parameter does not work with dotnet vstest, which tends to give me cleaner test output. Would it make sense to add support for maxCpuCount (or a similar parameter) in VSTest?

Steps to reproduce

Run dotnet vstest --parallel on a machine with more test projects than CPU cores.

Expected behavior

The dotnet CLI should respect properties in my xunit.runner.json file. However, barring that, it would be nice if there was a parameter that would increase the degree of parallelism when running dotnet vstest --parallel.

Actual behavior

A higher degree of parallelism can only be achieved when running tests through MSBuild (dotnet test).

Diagnostic logs

N/A

Environment

Linux and macOS

$ dotnet --version
6.0.401

[nohwnd]

@MatisseHack I think all the options you set are propagated correctly but our code assumes that tests will run in parallel (doing CPU bound work at the same time), and hence it chooses the minimal number between logical processor count, the specified maxCpuCount and assembly count to avoid overwhelming the CPU with too much parallel work. In your case the maxParallelLevel will be 2, because you have 2 logical processors.

In your case this assumption is not correct, because your tests are IO bound, and hence they run concurrently on the processors (they don't always do CPU work, so more than 2 concurrent tests can easily be handled by the CPU).

We don't have an option for that, and I don't think we can easily add it as there is an existing behavior, and we don't want to add yet another very similar option.

[MatisseHack]

Do you mean that -maxCpuCount should work as a parameter to dotnet vstest? I got an error when I tried that:

The argument -maxCpuCount:1 is invalid. Please use the /help option to check the list of valid arguments.

Allowing this parameter and making it consistent with the behavior of dotnet test -maxCpuCount:8 would make the most sense to me, but I realize there is legacy behavior to consider when calling vstest.console.exe directly.

Is there any way to use the same test adapter for all assemblies then? I'm not familiar with the internals of VSTest, but am I correct in thinking that it's creating a separate xUnit test adapter for each assembly, which overrides xUnit's ability to run assemblies in parallel? Would it be possible to add an option to adjust this behavior instead?

[elpht]

I don't understand that having test frameworks, like xUnit, that are able to handle (and interested in doing that) parallelism, vstest opts for the no-no, "I don't support it, nor I let others do it". Woudn't be the easiest for vstest to hand over that responsibility to the test framework or, at least, allowing that option?

[nohwnd]

This is a new feature and won't be implemented, we are focusing on adding new features to Testing.Platform instead. https://aka.ms/testingplatform