Problem with xUnit and collecting coverage

[kfrajtak]

Hi,
I am having problem collecting results from multiple (2) xUnit runs.

My method is

1. Instrument the assemblies

"AltCover.exe" --single -i=Tests.xUnit\bin\Debug -s=FluentAssertions -s=xunit -s=System -s=Microsoft -s=Ninject 
   -s=Infragistics -s=Grape -s=ClosedXML -s=Connector -s=Humanizer -s=OpenTracing -s=WCFExtras

2. Run the tests

"vstest.console.exe" __Instrumented\Tests.xUnit.dll 
  /TestCaseFilter:"Category=Screenplay" // there is around 30 tests here
  /ResultsDirectory:C:\work\temp\cc

"vstest.console.exe" __Instrumented\Tests.xUnit.dll 
  /TestCaseFilter:"Category!=Screenplay" // and more than 200 tests here
  /ResultsDirectory:C:\work\temp\cc

3. Collect the data

"AltCover.exe" Runner --collect -r __Instrumented --outputFile=C:\work\temp\cc\output.xml

I am currently running the test suits (meaning the CLI command) it in parallel, but sequential run did not make any difference. The output XML files are of the same size.

Thanks for help,
Karel

[SteveGilham]

Having instrumented the assemblies of interested once, this creates one XML template report file. Running a set of tests will create one or more intermediate files of the form [report path].[index].acv and running a second set of tests will produce more such files, with incrementing index numbers. Finally collecting the results will process all those intermediate files and fill in the initial report file.

If you want a separate report for each of the two test sets, out of the one instrumentation pass, then the process has to be

1. instrument once
2. run test set 1
3. collect using -o, --outputFile=VALUE Optional: write the recorded coverage to this file rather than overwriting the original report file.
4. Create a new, empty file called [report path].acv
5. run test set 2
6. collect using --outputFile=Different Value

which is essentially an instance of the Pause/Resume coverage collection behaviour.

[kfrajtak]

Steve, thanks for the swift reply :)
I have tried the "serial" execution and collecting the coverage into separate report files (--o), which did not result in desired output (though without the new acv file).
Regarding the acv files - where are those created? In the directory with the instrumented assemblies?

[SteveGilham]

The .acv files are created in the same folder as the XML report. The full file paths are generated by appending .acv (for the control file) or .[index].acv (for the binary format coverage data) as appropriate to the full file path for the XML report defined at the time of instrumentation, and baked into the instrumented files.

[kfrajtak]

I am now running the test suites sequentially and collecting the coverage data after each run.
I guess there's still something wrong with my setup - the size of the files is not much different

28/01/2021  15:12       121,378,681 cov_output_xunit.xml
28/01/2021  15:07       128,484,071 cov_output_xunit_collected.0.xml
28/01/2021  15:08       128,478,104 cov_output_xunit_collected.1.xml
28/01/2021  15:12       128,484,622 cov_output_xunit_collected.2.xml

[SteveGilham]

By itself, the increase in size of each coverage report over the original simply reflects the number of times the string "0" representing a visit count or percentage gets replaced by a longer numeric string. It doesn't say, what, where or how much has been covered.

For that, there are the module, class and method level summary records in the XML, or you can get various graphical displays either by the ReportGenerator tool, or AltCover's own ConvertTo-BarChart cmdlet post-processing the XML.

[kfrajtak]

Even though those are files from separate test suites?

[SteveGilham]

The change from the template file to the filled in report in each case shows a slightly different change, even if on the 0.01% of the total file size level, so clearly different coverage data were filled in in each case. The tools for visualization of the report are agnostic as to how the report was generated.

Given that different sub-sets of the test code were run, then the first prediction that can be made of the output is that when analysed, the test suite that was run will show coverage and the ones that weren't will only show incidental coverage (e.g. subclassing of tests or other shared code).