dotnet test crashes with System.Exception: Unable to read beyond the end of the stream if code under tests calls System.Environment.Exit

[chamons]

Description

Any usage of System.Environment.Exit in code under test with dotnet test crashes the test runner with a mysterious stack:

The active test run was aborted. Reason: Test host process crashed

Test Run Aborted with error System.Exception: One or more errors occurred.
 ---> System.Exception: Unable to read beyond the end of the stream.
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action`1 errorHandler, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---.

Steps to reproduce

• dotnet new nunit
• Add:

    [Test]
    public void Test1()
    {
        System.Environment.Exit (1);
        Assert.Pass();
    }

• dotnet test

Expected behavior

Some error message that says the process under test exited unsuspectingly. The phrase Test host process crashed kinda got that across, but it didn't crash, it exit'ed.

Actual behavior

Test run for /Users/donblas/Programming/xamarin-macios/tools/nnyeah/repro/bin/Debug/net6.0/repro.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.2.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
The active test run was aborted. Reason: Test host process crashed

Test Run Aborted with error System.Exception: One or more errors occurred.
 ---> System.Exception: Unable to read beyond the end of the stream.
   at System.IO.BinaryReader.Read7BitEncodedInt()
   at System.IO.BinaryReader.ReadString()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.LengthPrefixCommunicationChannel.NotifyDataAvailable()
   at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.TcpClientExtensions.MessageLoopAsync(TcpClient client, ICommunicationChannel channel, Action`1 errorHandler, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---.

Environment

Microsoft (R) Test Execution Command Line Tool Version 17.2.0 (x64)
dotnet-sdk-6.0.301-rtm.22254.17-osx-x64/dotnet dotnet test

[nohwnd]

This is expected, any testhost disconnect, is considered a crash/abort, unless vstest.console told that testhost that it is okay to exit. Your test stops the testhost, and with that, it also stops the test framework, adapter, and other logic responsible for sending back the data. So that is a crash from the point of view of vstest.console.

Do you have a better suggestion how to handle this? Would you expect us to ignore the call to .Exit? And do nothing?

[chamons]

Hmm, is it possible from VSTests's perspective to detect an exit vs a crash?

If the line said something like:

"The active test run was aborted. Reason: Test host process exited with code 1"

when System.Environment.Exit was called instead of a crash, it might have made it more clear that some library multiple levels down was calling System.Environment.Exit (which I admit is a bad idea and I fixed).

[nohwnd]

Crash in vstest language is any unexpected stop of the testhost before the vstest.console tells it to stop (by sending the terminate request). This includes exiting with exit 0.

[riQQ]

Related issue #2952