Create full platform manifest in local build (don't require job join)

[dagood]

See #1362 for context on the platform manifest, and how the official build currently harvests every platform's artifacts from CoreCLR and Libraries to build full platform manifests.

Local builds (such as source-build) only have access to a single machine/platform, so harvesting is not possible. Without harvesting, the platform manifest ends up incomplete and can cause problems, in particular causing dotnet publish to include libraries that contradict the ones already included in the target platform's shared framework.

The idea I've been considering so far is a template approach. (Copying from #1362):

---

I think we need to stop using harvesting to populate this file, and instead calculate the contents. We probably need to keep a platform manifest template (list of files) in the source to act as a template.

This would normally be a maintenance hazard, but we can prevent that:

1. Each platform uses the template to generate an expected platform manifest, which is used when building targeting packs, runtime packs, installers, etc.
2. When running a platform-specific job, validate:
   • All sharedfx files generated on that platform are in the manifest template.
   • There are no files in the template that are expected to be built on the current platform but weren't.
3. By the time any CI/PR validation build ends, all platforms have run the validation step. This means that if all jobs succeed, we know the platform manifest is valid. Even if some platforms silently produced bad platform manifests, the build would fail overall.
4. Since official builds stop if some jobs failed, there is no risk of releasing an outdated platform manifest.

This does impact dev flows. Someone may add a new file to the shared framework and not know how to properly update the manifest template. To mitigate this, when the validation step fails, it should emit the corrected list of files and tell devs where they need to put it to fix up their PR.

[dagood]

I got reminded by a convo about dotnet/source-build-reference-packages that this actually probably isn't an active bug in source-build. Context: the platform manifest is delivered in two places in 3.0+: the targeting pack, and the runtime pack. Neither is source-built by dotnet/core-setup or dotnet/runtime:

• The targeting pack is ildasm'd and built from IL source, here: https://github.com/dotnet/source-build-reference-packages/tree/master/src/targetPacks/ILsrc/microsoft.aspnetcore.app.ref/3.1.0. This is done after the core-setup build completes, so we essentially are copy-pasting the complete platform manifest from the core-setup official build. (This will be the case with dotnet/runtime as well.)

• Source-build produces a runtime pack, but it doesn't ship as part of the source-built .NET Core Runtime/SDK. Whenever a source-built SDK user runs a self-contained publish command, the SDK downloads Microsoft-built runtime packs from NuGet. (Support building sdks that can build self-contained / ReadyToRun / NativeAOT applications without using pre-builts from nuget.org source-build#1215 tracks a way to deliver source-built runtime packs.) This means the platform manifest is also complete in this scenario.

---

I think we still need to track this work, though: IMO we should deliver source-built runtime packs, and ideally we should build targeting packs from this repo rather than duplicating the source elsewhere (opening a source-build issue about this).

[jkoritzinsky]

@dagood @NikolaMilosavljevic This is now important for our efforts to speed up the official build. The current solution from #1594 is causing our build to be slower than it can be since it waits for all CoreCLR and Libraries jobs.

We could likely save about 15 minutes to half an hour of build time on top of the time saved via parallelizing the CoreCLR and Libraries builds if we didn't have to wait for all platforms of CoreCLR and Libraries to be done before starting a given installer build.

If we want to get our official build times down, I think we need to do this.

[dagood]

Agreed. Renamed title to point out that bit more clearly and remove source-build. (It isn't an active source-build bug, and future work there to actually use the dotnet/runtime-generated platform manifest is tracked by dotnet/source-build#1523.)

[dagood]

We could likely save about a half hour of build time on top of the time saved via parallelizing the CoreCLR and Libraries builds if we didn't have to wait for all platforms of CoreCLR and Libraries to be done before starting a given installer build.

To drill in on this, this implies there is no single consistently longer platform, correct? E.g. if win-x64 were consistently the slowest, removing this join would have no impact because we would still have to wait for win-x64 coreclr, win-x64 libraries, and win-x64 mono before win-x64 installer.

[jkoritzinsky]

Yep. Part of that timing is signing fluctuation which will also go away. Windows is still longer most of the time, but sometimes some of the linux jobs can end up as a longer poll.

[NikolaMilosavljevic]

@jkoritzinsky is this still tracking for completion for 5.0?

[jkoritzinsky]

This won't make it for 5.0