-
Notifications
You must be signed in to change notification settings - Fork 352
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tracking Nuget 429s #10885
Comments
Issue on Nuget side NuGet/Home#10558 |
Created https://portal.microsofticm.com/imp/v3/incidents/details/344927997/home to ask for concurrency limit increases |
IcM got us 2x concurrency bump, and no new instances since 12:25 PDT thus far. Will keep an eye on it. |
@dougbu noted this hit again yesterday here: https://dev.azure.com/dnceng-public/public/_build/results?buildId=72672&view=logs&jobId=b72e85ab-3386-5aa9-6405-3837662d9688&j=8e9e2812-09db-5a5d-c249-d96adf9f47fa&t=51e8c4b1-e760-5f88-6965-6fecd74b90ed&s=6884a131-87da-5381-61f3-d7acc3b91d76 What we're doing:
|
/fyi @dotnet/aspnet-build |
I just created this PR to actually retry on HTTP 429 responses. But it doesn't (yet) honor the Anyway, if anyone would like to provide feedback: NuGet/NuGet.Client#4897 |
I had another thought. From reading the docs on nuget.config's maxHttpRequestsPerSource setting, it would appear that Therefore, if dotnet/* repos set |
@zivkan This would be an interesting experiment to run. Is there telemetry that could be enabled that would help illuminate the quantity of traffic from NuGet operations? |
System.Net.Http (including HttpClient) uses EventSource, and in the future (but not yet, unfortunately) NuGet will as well. So on Windows anything that can record ETW events, like PerfView, can collect the information. Docs say that on .NET Core EventSource uses EventPipe, which supports LTTng, as well as tools like But NuGet doesn't have any relevant telemetry (we only have telemetry in VS, nothing on command line. Even in VS, we need to limit telemetry quantity, so logging every HTTP request is out of the question, only aggregated telemetry is feasible). |
I think you'd just be interested in raw totals:
|
And perhaps total time spent in NuGet related tasks during restore? That would give you a good set of base statistics to determine whether improvements were being made without regressing perf. |
With a binlog, or text restore log at normal verbosity, we could calculate the number of gets and 404's with a grep of However, I wouldn't expect that If restore is run as a separate step in CI, then CI logs should tell us how long the restore was. Otherwise, if CI logs the timestamp (to sufficient accuracy) for each line, we could look at the time difference between the "Determining projects to restore" message and the last "Restored c:\path\to\my.csproj" message. We can (and I have in the past) written a program to use AzDO's REST APIs to download build info, even build logs or build artifacts. But as previously mentioned, we don't have any command line telemetry. So, not super easy to query/do the analysis, certainly nothing "at scale" (across multiple repos/build pipelines). |
As it has been a week since we have seen this issue, we believe we have addressed this issue. |
Build
https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=12817
Build leg reported
Microsoft.NET.Build.Tests.dll.1.WorkItemExecution
Pull Request
dotnet/sdk#27821
Action required for the engineering services team
To triage this issue (First Responder / @dotnet/dnceng):
If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:
Tracking Nuget 429s
Additional information about the issue reported
No response
Report
Summary
The text was updated successfully, but these errors were encountered: