Partition helix test runs using historical execution time data

[dibarbet]

Resolves #62036

This does a few different things to fix our test partitioning

1. Run real test discovery to find the real set of tests to run and puts that in a json file in prepare-tests. We must do it here because discovery must be run before test payload minimization.
2. During partitioning, we lookup the test execution history for the appropriate stage to estimate the execution time of each test. We then use this info to partition the tests into work items that run under a certain time (currently 2:30). There is one test that does not run in under 2:30 consistently - Microsoft.CodeAnalysis.CSharp.UnitTests.OverloadResolutionPerfTests.NestedLambdas_01 but that can be fixed as a followup
3. Run tests via vstest.console.dll directly. This is what already gets called when using dotnet test and passing a specific dll in. We run directly on vstest.console.dll so we can utilize RSP files to output a test filter that contains all fully qualified test method names (there is a bug where RSP files passed to dotnet test do not get passed down correctly, see Long response files don't work with dotnet test & dotnet vstest  microsoft/vstest#3513). RSP files get around max path limitations and avoid escaping issues on different platforms.

[dibarbet]

@jaredpar this is ready for round 2

[RikkiGibson]

great progress.

[RikkiGibson]

It looks like this phase takes maybe 30 seconds on desktop. Should we tinker with firing the all the tasks off immediately, pushing them into an array or something, then awaiting them one by one?

There's no need to spend time on it in this PR, but thought we should keep it in our back pocket.

[dibarbet]

Yup - we can play around with it (will do that separately). We have to be careful not to run too many though that we get rate limited.

[jaredpar]

Can we file a work item to track refactoring run tests and prepare tests to have the better separation of responsibilities? Essentially at this point I think we should just move all helix ops into prepare tests and dumb down run tests a bit (perhaps just remove it from our core infra loop).

[dibarbet]

Can we file a work item to track refactoring run tests and prepare tests to have the better separation of responsibilities? Essentially at this point I think we should just move all helix ops into prepare tests and dumb down run tests a bit (perhaps just remove it from our core infra loop).

Done - #63413

[dibarbet]

test failures look like a known outage uploading results
https://helix.dot.net/api/jobs/b0ef8a27-a0a8-466f-b918-0152d8be7506/workitems/Microsoft.Build.Tasks.CodeAnalysis.UnitTests_Microsoft.CodeAnalysis.CSharp.CodeStyle.UnitTests_1?api-version=2019-06-17

there is an ICM open for it right now. Will retry once it is closed.

[dibarbet]

looks like there are some new breakages that were not showing up before. They also aren't showing up in main so likely some main change/helix rollout + partitioning is breaking this.

[xUnit.net 00:02:10.41]     Microsoft.CodeAnalysis.Editor.UnitTests.CommentSelection.CommentUncommentSelectionCommandHandlerTests.Uncomment_AtBeginningOfEndOfBlockComment [FAIL]
  Failed Microsoft.CodeAnalysis.Editor.UnitTests.CommentSelection.CommentUncommentSelectionCommandHandlerTests.Uncomment_MatchesBlockComment [1 ms]
  Error Message:
   System.TypeInitializationException : The type initializer for 'Roslyn.Test.Utilities.StaTaskScheduler' threw an exception.
---- System.IO.FileLoadException : Could not load file or assembly 'file:///C:/h/w/B66909A9/w/A15A08F5/e/Microsoft.CodeAnalysis.EditorFeatures.UnitTests/Debug/net472/Microsoft.CodeAnalysis.XunitHook.DLL' or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)
-------- System.NotSupportedException : An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for more information.
  Stack Trace:
     at Roslyn.Test.Utilities.StaTaskScheduler.get_DefaultSta()

[dibarbet]

Going to merge this. Will followup on why exactly https://github.com/dotnet/roslyn/blob/main/src/Compilers/CSharp/Test/WinRT/Metadata/WinMdMetadataTests.cs#L141 causes the xunit hook to fail to load when run before any wpffact test.