[BUG] Coverlet Nightly Crashing when merging JSON results

[pinkfloydx33]

Describe the bug
When running the coverlet nightly builds the generation of results fails when combining multiple test runs using JSON format. Individual runs generate reports successfully. The exception thrown is:

/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error : Deserialization of types without a parameterless constructor, a singular parameterized constructor, or a parameterized constructor annotated with 'JsonConstructorAttribute' is not supported. Type 'Coverlet.Core.Method'. Path: $['MyProject.Auth.dll']['/datadisks/disk1/work/1/s/src/MyProject.Auth/Utils/AuthExtensions.cs']['MyProject.Auth.Utils.AuthExtensions']['System.Threading.Tasks.Task MyProject.Auth.Utils.AuthExtensions::ClearRolesForUser(MyProject.Auth.Services.IExternalAuthService,System.String,System.Threading.CancellationToken)'] | LineNumber: 4 | BytePositionInLine: 204. [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.ThrowHelper.ThrowNotSupportedException(ReadStack& state, Utf8JsonReader& reader, NotSupportedException ex) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.ThrowHelper.ThrowNotSupportedException_DeserializeNoConstructor(Type type, Utf8JsonReader& reader, ReadStack& state) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.Serialization.JsonDictionaryConverter`3.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TDictionary& value) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.Serialization.JsonDictionaryConverter`3.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TDictionary& value) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.Serialization.JsonDictionaryConverter`3.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TDictionary& value) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.Serialization.JsonDictionaryConverter`3.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TDictionary& value) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 utf8Json, JsonTypeInfo jsonTypeInfo, Nullable`1 actualByteCount) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.JsonSerializer.ReadFromSpan[TValue](ReadOnlySpan`1 json, JsonTypeInfo jsonTypeInfo) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options) [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at Coverlet.Core.Coverage.GetCoverageResult() in /_/src/coverlet.core/Coverage.cs:line 317 [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]
/home/AzDevOps/.nuget/packages/coverlet.msbuild/6.0.1-preview.31.g0db8de77cb/build/coverlet.msbuild.targets(72,5): error :    at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in /_/src/coverlet.msbuild.tasks/CoverageResultTask.cs:line 87 [/datadisks/disk1/work/1/s/test/MyProject.Distributed.Extensions.Tests/MyProject.Distributed.Extensions.Tests.csproj]

Depending on the order in which I run the projects, the first line of the message will include multiple Type and/or method names

To Reproduce
Run coverlet nightly build (6.0.1-preview.31.g0db8de77cb) against a project and use json output format. Then, run coverlet against another project (or even the same one!) with the MergeWith option.

dotnet test --configuration Release --logger "console" --logger xunit -p:CollectCoverage=true -p:CoverletOutputFormat="json" -p:CoverageRoot="${PWD}/../Coverage" -p:MergeWith="${PWD}/../Coverage/coverage.json" 

Expected behavior
Coverlet does not crash

Actual behavior
Coverlet crashes with the above exception

Configuration (please complete the following information):
Please provide more information on your .NET configuration:
* Which coverlet package and version was used? 6.0.1-preview.31.g0db8de77cb (also crashes on 6.0.1-preview.26.g71c202b19e)
* Which version of .NET is the code running on? .NET7 but .NET SDK 8.0.0 is installed and active
* What OS and version, and what distro if applicable? Windows 11 x64, Ubuntu 22.04
* What is the architecture (x64, x86, ARM, ARM64)? x64
* Do you know whether it is specific to that configuration? Occurs on at least two configurations mentioned

Additional context
Works fine with package versions 6.0.0

[Bertk]

Thanks for reporting 🙏

Please use the Packages (ZIP file) from PR build and try the MergeWith operation again.

Please share the results.

see also https://stackoverflow.com/questions/72268018/jsonconstructor-fails-on-ienumerable-property

[pinkfloydx33]

I will try later today if I can, otherwise it will likely need to wait until Monday.

Thanks for looking into this.

[pinkfloydx33]

@Bertk @MarcoRossignoli @daveMueller

Sorry for the delay. I just tested with the packages from the feed linked above and it does not correct the issue:

D:\Packages\nuget\coverlet.msbuild\6.0.1-preview.34.gfba427ba33\build\coverlet.msbuild.targets(72,5): error : Deserial
ization of types without a parameterless constructor, a singular parameterized constructor, or a parameterized constru
ctor annotated with 'JsonConstructorAttribute' is not supported. Type 'Coverlet.Core.Method'.

I think this fix only works for .NET8+ (from this documentation):

In .NET 7 and earlier versions, the [JsonConstructor] attribute can only be used with public constructors.

---

I assume this means Coverlet's target framework and not my project's. Coverlet looks to be running on net6.0:

coverlet/src/coverlet.core/coverlet.core.csproj

Line 5 in 1981476

<TargetFrameworks>netstandard2.0;net6.0</TargetFrameworks>

coverlet/src/coverlet.msbuild.tasks/coverlet.msbuild.tasks.csproj

Line 5 in 1981476

<TargetFrameworks>netstandard2.0;net6.0</TargetFrameworks>

coverlet/src/coverlet.collector/coverlet.collector.csproj

Line 3 in 1981476

<TargetFrameworks>netstandard2.0;net6.0</TargetFrameworks>

coverlet/src/coverlet.console/coverlet.console.csproj

Line 5 in 1981476

<TargetFramework>net6.0</TargetFramework>

---

I think another option would be updating the version of S.T.J to the 8.0.x band which should be compatible with .NET6

coverlet/src/coverlet.core/coverlet.core.csproj

Line 16 in 1981476

<PackageReference Include="System.Text.Json" VersionOverride="6.0.9"/>

[Bertk]

@pinkfloydx33 Could you please check the package from draft build

If this does not resolve the issue, I will update System.Text.Json version to 8.0.1.

[pinkfloydx33]

@Bertk I will test this morning and report back

[pinkfloydx33]

@Bertk I can confirm that the packages from the draft build you linked are working as expected

[pinkfloydx33]

@Bertk thanks! do you know when this will make it into the nightly builds?

[Bertk]

@pinkfloydx33 The PR is already merged and the new package should be available tomorrow.

[pinkfloydx33]

@Bertk great, thanks!

[pinkfloydx33]

@Bertk @MarcoRossignoli

I should have paid more attention. While it would appear that coverlet no longer crashes when merging JSON, it's not actually merging the results. Only the coverage results from the last test project to run appear in the final results.

I don't know when this started happening since the nightly build previously couldn't even run without crashing. Should I open a new issue? It feels like all part of the same thing.

This was using nightly 6.0.1-preview.36.g9295fca754.

(BTW the version you had me download from the Azure Devops pipeline was preview.37 so I'm not even sure I've got the right nightly, despite being built three days ago)

[Bertk]

Could you please execute dotnet test ... with --diag:"coverlet.msbuild.test.diag.log;tracelevel=verbose" and check whether the code coverage data is created without MergeWith option. Maybe some dependencies cannot be loaded.
e.g.

• System.Text.Json
• System.Collections.Immutable (1536)
• System.Reflection.Metadata
• Microsoft.Bcl.AsyncInterfaces (1517)

Tomorrow I will also look for error handling of MergeWith option e.g. path check -p:MergeWith="${PWD}/../Coverage/coverage.json"

[pinkfloydx33]

Coverage data is correctly generated for individual projects when not using MergeWith option. Unfortunately I need them merged

When using --diag:"coverlet.msbuild.test.diag.log;tracelevel=verbose" I see a bunch of the following in the logs:

Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.

It's just the Microsoft.Bcl.AsyncInterfaces that seems to not be loading, but I get the same errors with and without MergeWith

[Bertk]

Please add <PackageReference Include="Microsoft.Bcl.AsyncInterfaces" Version="7.0.0" /> to test project. Which Microsoft.Net.Test.Sdk version is used for the test project?

The path in MergeWith option is not sanitized or normalized. The documentation uses relative path but your example uses -p:MergeWith="${PWD}/../Coverage/coverage.json. Could you please try a relative path for MergeWith.

[pinkfloydx33]

Relative path is what I'm using locally. The above path was just for sake of the repro.

Though in CI we've always used MSBuild properties to dictate the output path, setting CoverageRoot to an absolute path (which we must do in our CI) and then using that to construct CoverletOutput and MergeWith in our .props files. It's always worked and has never been a problem. If it can't support absolute paths anymore I'd be compelled to open another bug.

I can try adding the AsyncInterfaces to my test project, but I don't see how that would impact coverlet and as a long term solution would probably not be possible (and feel a bit hacky).

I'm out of office the rest of the day.. I will give it a try tomorrow morning.

The project is using Microsoft.Net.Test.Sdk 17.8.0

[Bertk]

Looks like there is another serialization problem with Coverlet.Core.Instrumentation.BranchKey.

 Coverlet.Core.Tests.CoverageTests.TestCoverageWithTestAssembly
   Source: CoverageTests.cs line 60
   Duration: 14,1 sec

  Message: 
System.NotSupportedException : The type 'Coverlet.Core.Instrumentation.BranchKey' is not a supported dictionary key using converter of type 'System.Text.Json.Serialization.Converters.SmallObjectWithParameterizedConstructorConverter`5[Coverlet.Core.Instrumentation.BranchKey,System.Int32,System.Int32,System.Object,System.Object]'. Path: $.InstrumentedResults.Documents.Branches.
---- System.NotSupportedException : The type 'Coverlet.Core.Instrumentation.BranchKey' is not a supported dictionary key using converter of type 'System.Text.Json.Serialization.Converters.SmallObjectWithParameterizedConstructorConverter`5[Coverlet.Core.Instrumentation.BranchKey,System.Int32,System.Int32,System.Object,System.Object]'.

  Stack Trace: 
ThrowHelper.ThrowNotSupportedException(WriteStack& state, NotSupportedException ex)
JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
JsonTypeInfo`1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed)
JsonSerializer.WriteString[TValue](TValue& value, JsonTypeInfo`1 jsonTypeInfo)
JsonSerializer.Serialize[TValue](TValue value, JsonSerializerOptions options)
CoverageTests.TestCoverageWithTestAssembly() line 91
RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
----- Inner Stack Trace -----
ThrowHelper.ThrowNotSupportedException_DictionaryKeyTypeNotSupported(Type keyType, JsonConverter converter)
JsonConverter`1.WriteAsPropertyName(Utf8JsonWriter writer, T value, JsonSerializerOptions options)
JsonConverter`1.WriteAsPropertyNameCore(Utf8JsonWriter writer, T value, JsonSerializerOptions options, Boolean isWritingExtensionDataProperty)
DictionaryOfTKeyTValueConverter`3.OnWriteResume(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state)
JsonDictionaryConverter`3.OnTryWrite(Utf8JsonWriter writer, TDictionary dictionary, JsonSerializerOptions options, WriteStack& state)
JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
DictionaryOfTKeyTValueConverter`3.OnWriteResume(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state)
JsonDictionaryConverter`3.OnTryWrite(Utf8JsonWriter writer, TDictionary dictionary, JsonSerializerOptions options, WriteStack& state)
JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
ListOfTConverter`2.OnWriteResume(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state)
JsonCollectionConverter`2.OnTryWrite(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state)
JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)

[pinkfloydx33]

Hey @Bertk

I just wanted to let you know that I have tried all of the following (and combinations thereof):

• absolute paths
• relative paths
• adding Microsoft.Bcl.AsyncInterfaces to the test projects (8.x and 7.x)
• downgrading Microsoft.Net.Test.Sdk back several versions

While the coverlet run succeeds with the MergeWith option, it still doesn't actually merge the results.

I did a little further investigation to see if I could figure out when this behavior started. Unfortunately the crashing behavior begins before (or at least in tandem with) the failure to actually merge:

• 6.0.0: works as expected
• All nightly builds up to (and including) 6.0.1-preview.15.g85fa12e3ea: work as expected
• Nightly builds beginning with 6.0.1-preview.16.g2307f655bd: crash with MergeWith option
• Nightly builds beginning with 6.0.1-preview.36.g9295fca754 (up to current): don't crash with MergeWith, but also don't really merge

I hope that helps tell you something. 🤞

For now we are going to continue using the stable version. We hope we can start using the nightly again as there's several fixes around records and primary constructors that we are really looking forward to.

I appreciate your effort here. Thanks again.

[daveMueller]

@Bertk are you already working on this? Sounds interesting. However I will pin this issue as it could break some user pipelines and I think we should probably solve this before we do a new release.

[Bertk]

@daveMueller I agree this has to be fixed before release coverlet and I started with a draft PR #1601. Until now I could not identify the source of the serialization failure.

Currently I check the feature of latest System.Text.Json version which supports

• Disabling reflection defaults
• Populate read-only members
• Missing member handling
• Interface hierarchy support

This was updated Migrate from Newtonsoft.Json to System.Text.Json

By the way, I will be offline until Monday.

[pinkfloydx33]

Whatever other serialization issues exist, they don't seem to be occurring 100% of the time (given that coverlet doesn't crash for me anymore). In the cases where a crash does not occur, I'd expect the output of the merge operation to be correct. Or are the other serialization issues trapped/caught and thus don't manifest to the user? I just want to make sure that both sides of this issue are handled (crash and failure to merge on success)

[Bertk]

dotnet test --configuration Release --logger "console" --logger xunit -p:CollectCoverage=true -p:CoverletOutputFormat="json" -p:CoverageRoot="${PWD}/../Coverage" -p:MergeWith="${PWD}/../Coverage/coverage.json"

@pinkfloydx33 I try to reproduce the error and uses a small script. Could you please explain the parameter CoverageRoot?

cd C:\GitHub\coverlet
dotnet test --configuration debug /p:CollectCoverage=true /p:CoverletOutputFormat="json" --results-directory:"C:\GitHub\coverlet\artifacts\Reports\coverlet.core" --diag:"C:\GitHub\coverlet\artifacts\log\debug\coverlet.core.test.diag.log;tracelevel=diag"  test\coverlet.core.tests\coverlet.core.tests.csproj
dotnet test --configuration debug  -bl:test.binlog /p:CollectCoverage=true /p:CoverletOutputFormat="json" --results-directory:"C:\GitHub\coverlet\artifacts\Reports\coverlet.collector.tasks" /p:MergeWith="C:\GitHub\coverlet\test\coverlet.core.tests\coverage.json" --diag:"C:\GitHub\coverlet\artifacts\log\debug\coverlet.collector.test.diag.log;tracelevel=diag" test/coverlet.collector.tests/coverlet.collector.tests.csproj

By the way, MergeWith does not use --results-directory parameter. This is another gap which should be implemented.

[pinkfloydx33]

CoverageRoot is just an MSBuild property that we pass in during tests. We use the value of it (in a .props file) to dynamically construct the paths to MergeWith and CoverlerOutput so that they are at known, normalized locations that our CI can then use to attach the output to the build across any of our projects.

For the purposes of this issue, CoverageRoot is neither here nor there, and is just a red herring if you're thinking it's the root cause. In my example above, it was an accident in copy/paste (when dealing with multiple open files) into the issue. It should've been CoverletOutput.

Ignore any of the commands I pasted. Just try to merge the JSON results from any two seperate projects (passing the appropriate MergeWith option) and you'll see that it does not actually merge the results and just overwrites them.

[Bertk]

@pinkfloydx33 CoverageRoot is not related to this issue but was mentioned in "To Reproduce" section. I do not know a MSBuild property CoverageRoot and therefore I ask for more information.

I did some updates and the coverage.net7.0.json file was created using MergeWith parameter

Could you please use the nuget package from PR #1601 build and check this with your use case (PR nuget packages).

[pinkfloydx33]

Yes I will try later this morning. Sorry about the confusion, as stated it was just a bad copy/paste on my part.

[pinkfloydx33]

@Bertk It works, but it also doesn't. Let me explain:

If I pass MergeWith to the first test run, it fails with the following exception:

D:\Packages\nuget\coverlet.msbuild\6.0.1-preview.46.gacc19dfd25\build\coverlet.msbuild.targets(72,5): error : MergeWith - file 'D:/Source/MyProject.Platform/test/Coverage/coverage.js
on' does not exist [D:\Source\MyProject.Platform\test\MyProject.Platform.Auth.Tests\MyProject.Platform.Auth.Tests.csproj]
D:\Packages\nuget\coverlet.msbuild\6.0.1-preview.46.gacc19dfd25\build\coverlet.msbuild.targets(72,5): error :    at Coverlet.Core.Coverage.GetCoverageResult() in /_/src/coverlet.co
re/Coverage.cs:line 323 [D:\Source\MyProject.Platform\test\MyProject.Platform.Auth.Tests\MyProject.Platform.Auth.Tests.csproj]
D:\Packages\nuget\coverlet.msbuild\6.0.1-preview.46.gacc19dfd25\build\coverlet.msbuild.targets(72,5): error :    at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in /_/src/co
verlet.msbuild.tasks/CoverageResultTask.cs:line 87 [D:\Source\MyProject.Platform\test\MyProject.Platform.Auth.Tests\MyProject.Platform.Auth.Tests.csproj]

The only way it works is if I don't pass MergeWith to the first test project and then pass it to the remaining ones. This is a problem because CI runs the same command against every test project. I don't have the ability to run one command for the first project and another for the rest. We are using a normal Azure DevOps dotnet test task which runs against a glob of projects. We've always been able to use Coverlet in this fashion.

Running the projects individually with different arguments does produce a properly merged coverage results file.

[Bertk]

@pinkfloydx33 I replaced the exception with a information logging. Please check again PR nuget packages.

[pinkfloydx33]

@Bertk I can confirm that everything is now working as expected!
Thanks for all of your help with this!

[pinkfloydx33]

@Bertk I don't know what made me think it was working. I re-tested with the package mentioned in #1583 (comment) but the coverage data is not being merged. See my comments on PR #1601 for some more details

My apologies. I swear I tried running it multiple times to validate that it was working.... I think I was thrown off by an erroneous 100% in the results and by simply being distracted with work

[Bertk]

@pinkfloydx33 Thanks you for providing the results. I did one successful test with coverlet projects (see PR #1601) and this looks OK.

[pinkfloydx33]

@Bertk I can confirm that the latest nightly is working as expected! No more crashing and json results are properly merged. Thanks for all your work on this. It's much appreciated!

We should be able to close this (and unpin it @daveMueller )