xUnit Parallel Settings Not Being Respected By Test Runner in Visual Studio 2019

[abowen]

Followed the documentation for changing parallel settings using xunit.runner.json however the runner doesn't respect these values in Visual Studio.

Steps to reproduce

1. Created new .NET Core Console app
2. Added xUnit testing library + fake long running tests (3x)
3. Added xunit.runner.json with different parallel settings (2 assemblies enabled for parallel, 1 disabled for parallel)
4. Copy newest xunit.runner.json version in .csproj
5. Test Explorer -> Disable Run Tests In Parallel
6. Run tests, and all assemblies run sequentially (ignoring the xUnit settings)
7. Test Explorer -> Enable Run Tests In Parallel
8. Run tests, and all assemblies run in parallel (ignoring the xUnit settings)

https://github.com/abowen/xUnit.Parallel.Example

I opened the issue on Microsoft's Developer Community and they sent me here
https://developercommunity.visualstudio.com/content/problem/838033/xunit-parallel-settings-not-being-respected-by-vis.html

[rodrigoramirez93]

I'm having this issue too. With two different test assemblies that fight for the database and one end up with a deadlock.

[ALPEAUTUMNSG]

Is this issue has been fixed already? I'm having same issue to with regards to this.

[abowen]

It's not as far as I'm aware.

[MartyIX]

Any progress on this? It's a bit unfortunate.

[Rakshasas]

Just to add to this. I've been reading about how tests in the same class run in serial, but after splitting tests that I want to run in parallel I still didn't see difference in times.

So I decided to remove Visual Studio from the equation.

Approximate test time in Visual Studio: 30 seconds.
Approximate test time using "dotnet test" against the exact same project: 14 seconds

I do have one test that takes 12-14 seconds to run which I need to fix, but this does show that parallel is not respective in Visual Studio because that specific test is the runtime of all my tests.

[xperiandri]

They don't care about Visual Studio or supporting any of its features. Never ever recommend using xUnit.

[kkauffman-incomm]

Installing this package allows everything to run in parallel:
https://www.meziantou.net/parallelize-test-cases-execution-in-xunit.htm

[HypeillingerChillinger]

Same here - I use dotnet test.

I have two Xunit test projects which I want to run sequentially (for better understanding like merging the tests from both projects in your mind then running the merge sequentially).

So I followed the instructions and added a xunit.runner.json in both projects following the instructions here:
https://xunit.net/docs/configuration-files

And used this file (xunit.runner.json) in each project

{
  "$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
  "parallelizeAssembly": false,
  "parallelizeTestCollections": false
}

Copied it to build folder in each project:

  <ItemGroup>
    <Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />
  </ItemGroup>

A test from project A cleans the database while running test from project B. Invoking each project alone like dotnet test ./MyProject... works perfectly.

So the settings in the files xunit.runner.json are not respected.

[bradwilson]

@HypeillingerChillinger We have no control over when VSTest chooses to run two assemblies in parallel, because it isn't leaving that decision to us. In order for us to have control, it would have to run a single instance of our test adapter, and pass us both assemblies and say "run both". Instead, what it does is instantiate us twice, and tells each instance "go run this one assembly". So the setting is ignored, because we aren't in control of the decision.

[elpht]

@bradwilson does then using parallelizeAssembly option make sense at all? Do you know about any plans to be supported by MS in future? Is there any way to run xunit tests via script so that this setting is taken into account?

Thank you in advance!

[bradwilson]

@elpht wrote:

does then using parallelizeAssembly option make sense at all?

If you're using VSTest (whether through Visual Studio or the dotnet CLI), then: no.

I'm not aware of any plans on the VSTest side to modify their behavior in a way that would allow us to properly make fine-grained parallelization decisions.

That said, if your goal is to just turn off parallelism from the command line, that's something we can accomplish today. You can run multiple assemblies, with parallelism turned off, from the dotnet command line. You don't do it with dotnet test: you do it with dotnet vstest. Let's talk about the differences between the two command lines.

---

dotnet test is a convenience command line tool: it will perform restore and build if needed before running your tests, and can be used to either run every test project in your solution (dotnet test) or a single project (dotnet test <ProjectPath>). What it cannot do, though, is run "just some" of your test assemblies (in parallel), nor can you tell it to run all test assemblies but not in parallel.

Here's what it looks like when we run all projects via dotnet test:

You can see in this example that VSTest has created two instances of our adapter, and runs each individually without the other knowing. This is why xunit.runner.json does not have any effect: we literally have no idea that things are being run in parallel. It's happening at an opaque layer above us.

The total run time here is a the expected 10 seconds from running everything in parallel (plus the overhead performed by dotnet test in ensuring that it knows what to run, and whether it needs to be built first).

---

dotnet vstest is a lower level command. It does not have a "run all" feature, and it cannot restore or build if your binaries are out of date. To the point: it doesn't understand solutions or projects at all. It only understands assemblies. To run a test assembly with dotnet vstest, you build it first yourself (usually with dotnet build), and then you pass the path to that test assembly to the dotnet vstest command line.

The command line can accept multiple test assemblies, and it will run them sequentially by default:

Again, xunit.runner.json is ignored, because we're launched twice.

Unlike dotnet test, you can actually influence whether parallelism is on or not with the --parallel switch. So you can also use dotnet vstest to run several assemblies of your choosing in parallel, without having to run them all:

---

Hopefully this illustrates what's going on, and what control you do (and don't) have over everything.

If you're unhappy that xunit.runner.json is not supported here, I suggest you open an issue in the VSTest issue tracker to let them know. No guarantee they will fix it, but at least it can be an additional data point for them to consider.

In v3, we should be able to offer more command line flexibility here with our console runner. Today in v2, our console runner can run multiple test assemblies (with a command line style much like dotnet vstest, where you pass us already built assemblies), but it only supports .NET Framework. In v3, we will be able to support .NET Core/.NET as well as .NET Framework, and of course our console runner will fully respect xunit.runner.json.

[elpht]

Thank you for your detailed explanation @bradwilson.

I think this VSTest issue already tackles that matter.

[bradwilson]

I don't think it's exactly the same, but it would be interesting to see what their potential solution is.

[nohwnd]

Answering here, rather than in microsoft/vstest#4044 (comment) to keep the context.

I don't understand that having test frameworks, like xUnit, that are able to handle (and interested in doing that) parallelism, vstest opts for the no-no, "I don't support it, nor I let others do it". Woudn't be the easiest for vstest to hand over that responsibility to the test framework or, at least, allowing that option?

I don't think we (vstest / dotnet test) are limiting the parallelism that xunit can do (to serious degree). Both dotnet test and vstest can run all dlls in parallel, but they do it in a different manner:

• dotnet test relies on MSBuild to do the parallelism, and then hand out a single dll for vstest to run. Which we then hand down to xunit. MSBuild parallelizes by default, but you can suppress it by -n:1. Here we don't allow xunit to take all dlls and schedule them because 1) we don't have them, and 2) we cannot guarantee they will be able to run in the same process (e.g. they are mix of different target frameworks and dependencies).

• dotnet vstest (or dotnet test when used with dlls, or vstest.console), will take multiple dlls, and split them and then start 1 testhost per each. This can also happen in parallel (using the --parallel flag or MaxCpuCount setting). We hand down 1 dll per testhost to xunit, because we need to make sure that only compatible dlls are running in the same process, and that for .net framework all dlls are using compatible dependencies, or are isolated to appdomains.

• We don't limit the thread-level parallelism that xunit can do in any way (other than the internals of vstest having to keep up, which can be a little bit of back pressure).

VSTest side to modify their behavior in a way that would allow us to properly make fine-grained parallelization decisions.

In theory we would be able to hand out all the assemblies to xunit and let you parallelize on process-level. But that would also make the whole datacollector infra useless (which is our problem of course, not yours). What are the fine-grained decisions and optimizations you would be doing? As far as I am aware you need to start 1 process per .NET assembly because of deps and runtimeconfig json. And either share .NET Framework process and isolate in appdomains, or run 1 per each.

Are you talking more about a scheduling level? e.g. this assembly has 10k tests, lets run 5k in one process and 5k in other?

[bradwilson]

MSBuild parallelizes by default, but you can suppress it by -n:1

There is no -n switch listed under dotnet test --help.

In addition, this is what I see when trying to use -n:1:

The fact that MSBuild "does the parallelism" is an artifact that nobody here likely cares about. If you want to make it easy to tell people how to turn off parallelism, then I would assume you'd add a documented switch in the dotnet test help page.

(As an aside, it makes sense that MSBuild "does the parallelism" because the output when running multiple assemblies in parallel with dotnet test is interleaved in a quite bad way that makes for a poor user experience.)

What are the fine-grained decisions and optimizations you would be doing?

This entire discussion is about how we expose assembly parallelism to users via xunit.runner.config.json. I consider it to be "fine grained" because we would run all the ones that want to be run in parallel, in parallel, and then run the rest sequentially. If users are using dotnet test (or, IMO, the superior dotnet vstest) then they are forced to construct the matrix of commands that would be necessary to run the parallel ones in parallel, and the sequential ones in sequence, and even then it would be inferior to what we would offer today because if anything fails in any of the parallel assemblies, then none of the sequential ones would run (since you've forced them into at least two separate invocations of dotnet vstest).

That is the fine-grained decision I was talking about.

As far as I am aware you need to start 1 process per .NET assembly because of deps and runtimeconfig json.

This part will be irrelevant in xUnit.net v3 since everything is a process.

Are you talking more about a scheduling level? e.g. this assembly has 10k tests, lets run 5k in one process and 5k in other?

No.