Microsoft.NET.Build.Containers.IntegrationTests are pulling from Docker Hub

[akoeplinger]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=733130
Build error leg or test failing: Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.WriteToPrivateBasicRegistry
Pull request: https://github.com/dotnet/sdk.git/pull/42019

We need to stop pulling from upstream Docker Hub since we're hitting rate limits.

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "You have reached your pull rate limit",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=733130
Error message validated: [You have reached your pull rate limit]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/8/2024 7:20:41 PM UTC

Report

Build	Definition	Test	Pull Request
899373	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45559
897513	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45492
897504	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44611
895350	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45458
894962	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45419
894838	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45097
894363	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45427
893558	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45327
892151	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45392
893220	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45293
891423	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.EndToEnd_MultiProjectSolution	#45398
891375	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45354
891351	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45042
891345	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45391
891312	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45396
891307	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45395
888298	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45358
888033	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45293
888027	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45278
886062	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45293
884843	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#44974
884857	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45252
884539	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44540
884534	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45264
884377	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45255
884466	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45258
884409	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44570
884187	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44958
884160	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45179
883622	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45244
883604	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45245
883513	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45243
882355	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44974
882187	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.EndToEnd_NoAPI_ProjectType	#45043
881570	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation
881560	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45174
880538	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45169
880118	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45157
880099	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44570
880085	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45042
879699	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45143
879638	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateImageIndexTests.CreateImageIndex_Baseline	#43085
878388	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45108
878367	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45106
878361	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45105
878289	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateImageIndexTests.CreateImageIndex_Baseline	#43085
877637	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45071
877627	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44786
877502	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.ApiEndToEndWithLocalLoad	#45052
875785	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44786
875754	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45009
875752	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45015
875748	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.ApiEndToEndWithLocalLoad	#43085
875728	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45014
874542	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.EndToEnd_MultiProjectSolution	#44786
874047	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44786
873924	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44927
873400	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44805
873332	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44949
873330	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44948
873211	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44802
873191	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#43085

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	8	62

[akoeplinger]

@marcpopMSFT who owns Microsoft.NET.Build.Containers?

[baronfel]

@dotnet/sdk-container-builds-maintainers

[baronfel]

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

[nagilson]

Bumping this thread since it's affecting more PRs

[nagilson]

We don't have a path forward to fix this yet - Aspire moved to using their own ACR for storing the images they use, but we need an engineering-systems-managed solution for our entire stack. There's an issue tracking this request but I can't find it at the moment.

[baronfel]

Could be fixed at a root cause level by dotnet/dnceng#3389

[MichaelSimons]

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

@baronfel - It sounds like the test has this dependency and is what is pulling the image. One possible option to workaround scenarios where tests are pulling images directly from DockerHub would be to create a wrapper buildtools image. The Dockerfile would just be a FROM statement. The buildtools images are stored on MCR and therefore you would be able to get around this rate limiting issue.

[nagilson]

@dotnet/sdk-container-builds-maintainers Do you agree with @MichaelSimons suggestion? This has a pretty large impact on PR failure rates. If this is still an issue going forward a month or two down the line, I will sadly suggest that we just disable these tests

[marcpopMSFT]

@donJoseLuis @MichalPavlik I'm told by baronfel that there may be a solution to this of setting up our own ACR mirror of these so we don't get blocked by docker from pulling too often. He said to talk to @eerhardt about that. Is that something you can potentially drive getting set up?

As you can see above, this fails a couple of dozen PRs every week that have to be rerun. It's not a crazy blocker but that's high enough to be something we want eyes on to get fixed eventually.