Providers are unreliable

[spongessuck]

Functional impact

Builds fail because required lib files don't exist in project.

Minimal repro steps

1. Install files from a library (i.e. ladda@1.0.6, files: dist/ladda.min.js)
2. Ensure it works successfully
3. Later on, try to restore

Expected result

Files that restore successfully once should always restore successfully

Actual result

LibMan intermittently fails with error LIB002 or LIB018, or doesn't load the correct contents of the package (i.e., index.html instead of dist/**).

[justcla]

Thanks for raising the issue. Is there a way to reliably repro this?
Are you able to post the error output?

[Mamikel]

I don't know whether this provides any assistance, but FWIW, I'm finding that the libman restore command works reliably with cdnjs, but referencing the provider unpkg (which works within the IDE), consistently fails via the restore command, the reported errors are:
[LIB002]: The "bootstrap-wysiwyg@2.0.1" library could not be resolved by the "unpkg" provider
[LIB002]: The "material-design-icons-font@2.2.3" library could not be resolved by the "unpkg" provider
[LIB002]: The "roboto-fontface@0.4.5" library could not be resolved by the "unpkg" provider

I've attached my libman.json file:
libman.json.txt

[nickalbrecht]

I've ran into similar feedback messages from Libman a while ago when I tried to deploy an app that used it. I haven't looked into it at all, but at the time, if I rebuilt the solution first, then tried the deploy again it seemed to work (implying maybe one has caching issues)? I can't provide much more detail than that in my instance.

[blakepell]

I have a similar behavior. You add your package in via cdnjs (because unpkg almost never works, in fact I've never personally got it it to work). Some packages you select will come down, but others fail consistently. For instance, try to add the monaco-editor and it will consistently returns 404 for half of the files. I'm starting to think it's a CDN problem all together. Example:

Failed to download resource from "https://cdnjs.cloudflare.com/ajax/libs/monaco-editor/0.15.6/min-maps/vs/loader.js.map"

However, if I navigate to that URL manually and it worked. But then I did it again and it 404'd. I reloaded that exact 404 and it worked. I opened up the Chrome utilities and reloaded every few seconds and about 20% of the requests returned a 200 with the file, the rest 404'd with the same URL. This is the behavior seen when trying to restore packages with a fair amount of files. -Something- fails and then the whole thing fails. The more files there are, the more chance something 404's and it can be different every time which is making it impossible to restore some packages via this mechanism.

I haven't used this a ton but I've observed it over a serious of a few weeks.

[freemstr]

As of recent this problem became worse
Error LIB010 Failed to download resource from "https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.0/js/npm.js"
The 404 response from cdnjs is intermittent but it fails the build pipeline and costs man-hours in reaction to failed CI/CD build notifications and followup email notification about the build notifications
{
"version": "1.0",
"defaultProvider": "cdnjs",
"defaultDestination": "wwwroot/lib",
"libraries": [
{
"library": "twitter-bootstrap@3.4.0",
"destination": "wwwroot/lib/bootstrap/"
}]}

Are there any plans to implement a fallback mechanism?

Thank you.

[spongessuck]

I've just been downloading libraries with npm and adding filesystem references to files in node_modules.

…

On Mon, Dec 24, 2018, 11:04 freemstr ***@***.*** wrote: As of recent this problem became worse Error LIB010 Failed to download resource from " https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.0/js/npm.js" The 404 response from cdnjs is intermittent but it fails the build and costs man-hours in reaction to failed build notifications and followup email notification about the build notifications Are there any plans to implement a fallback mechanism? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#370 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGE0TkhVrJkc3kDmJydRdOfn6vuA-QS-ks5u8PsQgaJpZM4WbUgN> .

[freemstr]

@justcla
At 23:00 my Azure Pipeline build failed.
It succeeded several hours later with the same source project
Build consist of downloading clean source and building the solution
Error LIB002: The "@types/jquery@3.3.22" library could not be resolved by the "unpkg" provider
libman.json
{
"version": "1.0",
"defaultProvider": "cdnjs",
"defaultDestination": "JS/lib",
"libraries": [
{
"library": "@types/jquery@3.3.22",
"destination": "TS/lib/jquery/",
"provider": "unpkg"
}
]}

Could this be related?
#241

[srosam]

Are there any plans to implement a fallback mechanism?

I think this would be useful.
Builds failing because they cant get to the cdn could be a big problem.
What if someone was try to release a patch for a client who's having an issue... it needs to be sorted.

[KoalaBear84]

Here the same problems. But also when it is in production that unpkg.com is sometimes very slow and unreachable. For builds this is a lesser issue locally, because we can retry this, but when having CI it will fail the builds. In production it could take 30 seconds to load the page.. Something is wrong with unpkg, you should expect CloudFlare to keep it stable.

[ygoe]

Maybe the same bug here. Sometimes the packages can be restored, sometimes they cannot be resolved by "unpkg". I've read that unpkg is way better than cdnjs because of its architecture. While unpkg makes npmjs packages accessible, cdnjs is barely manageable anymore these days. If unpkg is just as bad, there are no more options and I'm keeping a copy of the JavaScript files in my Git repository again, just like in the past. It's not that they are too big for that. At least they're there where my code is, and not gone half of the time.

Also, why does LibMan have to restore these files from online at every build? The files are there, there's no reason to fail the build because they couldn't be downloaded. That's just stupid.

[gbjbaanb]

And strangely, ASPNETCDN is not one of the listed package providers. You'd think Microsoft have an option to use its own CDN.

[kjkrum]

@justcla I can frequently repro this when trying to restore material-design-icons, which contains a large number of files. It fails on a different file every time. It seems that restore does not retry individual files, and the entire operation fails if any file fails.

[saluton-mundo]

The LIB002 is most likely due to GET requests going via HTTP protocol - in such cases unpkg returns status code 301. That's the behaviour we're seeing as per #491

LIB018 seems to be present in larger packages - the meta related to it becomes truncated as confirmed with Fiddler.

Are there any timeouts in play within Libman to prevent hangs during the restore process. It could also be race conditions. If unpkg takes more than 30s to return the full package payload does Libman simply kill the request and throw?

[ygoe]

Oh, maybe it has to fetch on every build because of #490. It doesn't know what it's doing or what it has done. So it has to check every time. And it even fails either way. Can't detect new files and can't download anything, then blows up either the build or the application at runtime. It's on my list of things to throw out the window now.

[hempels]

We routinely see 'The ... library could not be resolved by the "unpkg" provider' errors. It causes continuous builds to fail. It causes local builds to fail. It's pretty much a fail.

I don't understand why the team opted to move to a library manager that supports fewer packages and is less reliable. Very frustrating.

[Ghosts]

We routinely see 'The ... library could not be resolved by the "unpkg" provider' errors. It causes continuous builds to fail. It causes local builds to fail. It's pretty much a fail.

I don't understand why the team opted to move to a library manager that supports fewer packages and is less reliable. Very frustrating.

Also running into the same error (also unpkg) and it's preventing production builds intermittently.

[Alex-Horlock]

We routinely see 'The ... library could not be resolved by the "unpkg" provider' errors. It causes continuous builds to fail. It causes local builds to fail. It's pretty much a fail.

I am having the same issue as hempels (Visual studio Community 2019 16.3.6). It would be great if this was solved as for my project Libman seems to be a generally good option.

[nickalbrecht]

I got fed up with it. All of the packages I used unpkg for originally are now obtainable from jsDelivr, I believe it was added after unpkg. So now I'm just using jsDelivr instead. Still using libman mind you, I just no longer using unpkg as a package source.

[ygoe]

@nickalbrecht jsDelivr is not available in the selection. How did you do that?

[nickalbrecht]

@ygoe Just update Library Manager to the most recent version, it was added at a later date so earlier version won't have it.

[jimmylewis]

@ygoe which version of VS are you using? The JSDelivr provider wasn't included in VS until 16.0, but you can build a private version of the VSIX. See more here.

[ygoe]

Tried with VS 2017. Not switched to VS 2019 yet. Waiting for a usable version (without several blocking bugs) with .NET Core 3 support (which should be 16.4).

[nickalbrecht]

You should also be able to use the CLI or the Build package if you don't have VS2019. I use Microsoft.Web.LibraryManager.Build so that my libraries restore as part of the build outside of VS. But we're kind of getting off topic :-P

[okosodovictor]

All my install library are failing to restore when i run restore client side library for my libma.js file. I am recieving this error " library could not be resolved by the "cdnjs" provider"
example:
{
"library": "twitter-bootstrap@3.3.6",
"destination": "wwwroot/lib/twitter-bootstrap/"
},
{
"library": "font-awesome@4.1.0",
"destination": "wwwroot/lib/font-awesome/"
},
{
"library": "toastr.js@2.1.4",
"destination": "wwwroot/lib/toastr.js/"
},
jquery works. others coud not be able to restore. I am using visual studio 2019.

[KoalaBear84]

Correct, around 8 hours ago this started happening. It is because of problems at cdnjs / Cloudflare. Looks like it should is resolved almost an hour ago:
https://status.cdnjs.com/incidents/0gdmkm4tl98d

[okosodovictor]

Issues solved. The package restore is working fine again.

[fairking]

I had a similar issue. The problem was with package name.
Instead of:

{
      "library": "twitter-bootstrap@4.4.1",
      "destination": "wwwroot/lib/bootstrap/"
    },

I was using

{
      "library": "bootstrap@4.4.1",
      "destination": "wwwroot/lib/bootstrap/"
    },

The message was confusing. And it wasn't installing any of my packages.

The message should be something like The package "bootstrap@4.4.1" was not found in "cdnjs" repository rather than The "bootstrap@4.4.1" library could not be resolved by the "cdnjs" provider.

[jimmylewis]

@denis-pujdak-adm-it If I were to rephrase your point, we should be emphasizing that the library name couldn't be found separately from the name@version? E.g.
The package "bootstrap@4.4.1" was not found in "cdnjs" repository.
vs.
The "bootstrap" library could not be resolved by the "cdnjs" provider (removing the version here to emphasize the name)

[fairking]

@jimmylewis Hm. Anyway. That was just my thought based on my experience with the instalation.

[kjkrum]

@jimmylewis I may have just assumed there were no retries because I didn't see any messages about them, just the final failure. When I first encountered these reliability issues, I set up a test project with a dependency on AdminBSB. I forget which providers offer that package. AdminBSB contains a huge number of files, thousands IIRC, and is never successfully restored.

[ardalis]

This continues to be very frustrating.

[nickalbrecht]

I had an error from an AzureDevOps job the other day from this. Basically had to re-queue the job a handful of times until it finally solved itself. Not sure if it finally went through because of the job getting assigned to a different machine that didn't have caching issues or what though.

[SmRiley]

I wonder if I can customize the mirror source like Nuget, which is too reliant on the cloud, and I think my hard disk is more reliable than the cloud.Libman is a good tool, but CDN isn't

[rwasef1830]

To any mortal soul in eternal torture by this shockingly unfixed bug, here's a workaround:

1. Remove the NuGet package Microsoft.Web.LibraryManager.Build from your project.

2. Install dotnet tool: dotnet tool install Microsoft.Web.LibraryManager.Cli

3. Put this in your csproj that contains libman.json:

<!-- Workaround https://github.com/aspnet/LibraryManager/issues/370 -->
  <Target Name="RestoreFrontEndLibraries" BeforeTargets="BeforeBuild">
    <Exec Command="dotnet tool restore" />
    <Exec Command="dotnet tool run libman restore" LogStandardErrorAsError="true" ContinueOnError="true">
      <Output TaskParameter="ExitCode" PropertyName="ErrorCode" />
    </Exec>
    <Exec Command="dotnet tool run libman restore" LogStandardErrorAsError="true" ContinueOnError="true" Condition="$(ErrorCode) != '0'">
      <Output TaskParameter="ExitCode" PropertyName="ErrorCode" />
    </Exec>
    <Exec Command="dotnet tool run libman restore" LogStandardErrorAsError="true" ContinueOnError="true" Condition="$(ErrorCode) != '0'">
      <Output TaskParameter="ExitCode" PropertyName="ErrorCode" />
    </Exec>
    <Exec Command="dotnet tool run libman restore" LogStandardErrorAsError="true" ContinueOnError="true" Condition="$(ErrorCode) != '0'">
      <Output TaskParameter="ExitCode" PropertyName="ErrorCode" />
    </Exec>
    <Exec Command="dotnet tool run libman restore" LogStandardErrorAsError="true" ContinueOnError="true" Condition="$(ErrorCode) != '0'">
      <Output TaskParameter="ExitCode" PropertyName="ErrorCode" />
    </Exec>
    <Exec Command="dotnet tool run libman restore" LogStandardErrorAsError="true" ContinueOnError="true" Condition="$(ErrorCode) != '0'">
      <Output TaskParameter="ExitCode" PropertyName="ErrorCode" />
    </Exec>
    <Exec Command="dotnet tool run libman restore" LogStandardErrorAsError="true" ContinueOnError="true" Condition="$(ErrorCode) != '0'">
      <Output TaskParameter="ExitCode" PropertyName="ErrorCode" />
    </Exec>
    <Exec Command="dotnet tool run libman restore" LogStandardErrorAsError="true" ContinueOnError="true" Condition="$(ErrorCode) != '0'">
      <Output TaskParameter="ExitCode" PropertyName="ErrorCode" />
    </Exec>
    <Exec Command="dotnet tool run libman restore" LogStandardErrorAsError="true" ContinueOnError="true" Condition="$(ErrorCode) != '0'">
      <Output TaskParameter="ExitCode" PropertyName="ErrorCode" />
    </Exec>
    <Exec Command="dotnet tool run libman restore" LogStandardErrorAsError="true" ContinueOnError="false" Condition="$(ErrorCode) != '0'" />
  </Target>

This code retries up to 10 times, you can copy / paste one of the middle <Exec> tags to add more retry attempts.
May you find peace at last.

[SmRiley]

对于任何因此令人震惊而无法修复的错误而遭受永远折磨的凡人灵魂，这里有一种解决方法：

1. 从您的项目中删除NuGet包Microsoft.Web.LibraryManager.Build。
2. 安装dotnet工具： dotnet tool install Microsoft.Web.LibraryManager.Cli
3. 将其放入包含libman.json的csproj中：

<！-解决方法https://github.com/aspnet/LibraryManager/issues/370- > 
  <目标 名称 = “ RestoreFrontEndLibraries ”  BeforeTargets = “ BeforeBuild ” >
    < Exec  命令 = “ dotnet工具还原” />
    < Exec  命令 = “ dotnet工具运行libman restore ”  LogStandardErrorAsError = “ true ”  ContinueOnError = “ true ” >
      <输出 TaskParameter = “ ExitCode ”  PropertyName = “ ErrorCode ” /> 
    </ </ Exec > 
    < Exec  命令 = “ dotnet工具运行libman restore ”  LogStandardErrorAsError = “ true ”  ContinueOnError = “ true ”  Condition = “ $（ErrorCode）！='0' “ > 
      < 输出 TaskParameter = ”ExitCode “  PropertyName= “ ErrorCode ” /> 
    </ Exec > 
    < Exec  命令 = “ dotnet工具运行libman restore ”  LogStandardErrorAsError = “ true ”  ContinueOnError = “ true ” 条件 = “ $（ErrorCode）！='0' ” > 
      < 输出 TaskParameter = “ ExitCode “  PropertyName = ” ErrorCode “ /> 
    </ 执行 >
    < Exec  命令 = “ dotnet工具运行libman restore ”  LogStandardErrorAsError = “ true ”  ContinueOnError = “ true ” 条件 = “ $（ErrorCode）！='0' ” > 
      < 输出 TaskParameter = “ ExitCode ”  PropertyName = “ ErrorCode ” /> 
    < / 执行 > 
    < 执行 命令 = “dotnet工具运行libman restore “ LogStandardErrorAsError = “ true ”  ContinueOnError = “ true ” 条件 = “ $（ErrorCode）！='0' ” > 
      < 输出 TaskParameter = “ ExitCode ”  PropertyName = “ ErrorCode ” /> 
    </ Exec > 
    < Exec  Command = “ dotnet工具运行libman恢复“  LogStandardErrorAsError = “ true ” ContinueOnError =“ true ” 条件 = “ $（ErrorCode）！='0' ” >
      <输出 TaskParameter = “ ExitCode ”  PropertyName = “ ErrorCode ” />
    </ exec >
    < Exec  命令 = “ dotnet工具运行libman restore ”  LogStandardErrorAsError = “ true “  ContinueOnError = ” true “ 条件 = ”$（ErrorCode）！='0' “> 
      < 输出 TaskParameter = “的ExitCode ” 属性名 = “错误代码” /> 
    </ Exec的 > 
    < Exec的 命令 = “ DOTNET工具运行利布曼恢复”  LogStandardErrorAsError = “真”  ContinueOnError = “真” 条件 = “ $（错误码）！=' 0' “ > 
      <输出 TaskParameter = “ ExitCode“  PropertyName = ” ErrorCode “ />
    </ </ Exec >
    < Exec  Command = ” dotnet工具运行libman restore “  LogStandardErrorAsError = ” true “  ContinueOnError = ” true “  Condition = ” $（ErrorCode）！='0' “ >
      <输出 TaskParameter = “ ExitCode ”  PropertyName = “错误代码“ />
    </执行 > 
    < 执行 命令 = “ dotnet工具运行libman restore ”  LogStandardErrorAsError = “ true ”  ContinueOnError = “ false ” 条件 = “ $（ErrorCode）！='0' ” /> 
  </ 目标 >

此代码最多重试10次，您可以复制/粘贴其中一个中间<Exec>标签以添加更多重试次数。
愿你终于找到和平。

I don't think it can solve the problem that cdnjs cannot connect

[rwasef1830]

It doesn't solve the problem of connect failure, but it just does retry, it works after some times. Better than build server random failure due to this issue.

[SmRiley]

It doesn't solve the problem of connect failure, but it just does retry, it works after some times. Better than build server random failure due to this issue.

I think it is more important for subsequent versions to customize the mirror source like Nuget. Now everything is built on the damn cloud

[gbjbaanb]

It doesn't solve the problem of connect failure, but it just does retry, it works after some times. Better than build server random failure due to this issue.

It won't help you one bit if the CDN providers rate-limit you because your build server is fetching from their servers too often. Its hardly a responsible practice either, if you're building using this service, then commit your libraries or the relevant files.

[nickalbrecht]

Typically when this happens for me, LibMan may claim it can't be reached, but if I navigate to the URL manually in a browser it will come up. I'm more inclined to believe it's an inconsistent behavior with the caching in LibMan versus the CDN refusing to fulfill the request. Every time that I can remember navigating to the URL manually, the resource was found.

[gbjbaanb]

Perhaps if libman reported the http status code it might be a little more obvious, if it is rate-limited rather than generally failed to fetch the data. I don't think the caching is at fault, as libman will be attempting to fetch something (that 'metadata') from the CDN to determine if its cache is up-to-date.

Stick wireshark on, and see for yourself what happens in the meantime.

[EdiWang]

Today on my computer, I have The "tinymce@5.7.1" library could not be resolved by the "cdnjs" provider, but my build server is fine.

So I download and debug Libman source code, in my case, it is caused by an exception when trying to parse JSON.

Unterminated string. Expected delimiter: ". Path 'assets[104].files[32]', line 1, position 2398115.

The requested URL was https://api.cdnjs.com/libraries/tinymce

The json string has an invalid ending:

I request the URL from Microsoft Edge browser and use notepad++ to view the JSON and found the same thing

But when using Google Chrome to request the JSON, to my surprise, it is fine:

I RDP into my build server, request the same JSON from Microsoft Edge, it is also fine.

So I guess it may be a network issue or CDN issue that give me a bad json. I try to switch to a different network using a VPN, but libman still fails. After debug, I found it is caching the old bad json.

The cached file is located in "C:\Users\Edi\AppData\Local\.librarymanager\cache\cdnjs\tinymce\metadata.json"

After clean up the local cache, everything works fine.

So I would suggest libman DO NOT cache bad json file when having an exception during the installation.

[gbjbaanb]

That suggests its not Libman at fault at all but the networking subsystem, which is not surprising as all Windows http stack is built on the same internet management stuff. So once you have a malformed download, it sticks in the browser cache (that libman is leveraging). The cache will contain this json as it will have assumed a correct download - I doubt any http get will cache a failed request!

So now you're looking at the duration these items stay in the cache for. The only thing Libman can do in this situation is to give a better error message that the file is corrupt. If it can (and this is doubtful) remove it from the cache so it gets refreshed, so much the better.

[kjkrum]

That's still Libman's fault, just in a different way. If the system cache doesn't work the way Libman needs it to work, Libman shouldn't be using the system cache.

[jimmylewis]

@gbjbaanb libman has its own cache, so we can be smarter here about not caching a broken response payload.

@EdiWang thanks for the detailed writeup, I've opened #630 to track this specific behavior.

[EdiWang]

@gbjbaanb @kjkrum It's not in the system cache or browser cache, it's libman's own cache located in "C:\Users\<Username>\AppData\Local\.librarymanager\cache\"

The problem is libman downloaded the broken json file (which http status code is still 200), did not validate it's format before caching it.

[EdiWang]

To clear the cache, we can either manually delete C:\Users\<Username>\AppData\Local\.librarymanager\cache or from libman CLI:

dotnet tool install -g Microsoft.Web.LibraryManager.Cli

then

libman cache clean

Reference: https://docs.microsoft.com/en-us/aspnet/core/client-side/libman/libman-cli?view=aspnetcore-5.0#manage-library-cache

[pynej]

https://api.cdnjs.com/libraries/ace is now broken. (the CDN version, not my local cache.)
I will reiterated the improved error messages, as this was giving "library could not be resolved" when it should be able to see that it's corrupted and have a more clear message indicating that it's not user error and the package proovider needs to be notified.
I made ticked with ace but I'm not sure if they are pushing the cdnjs package themselves, or if it is just a bad cached file in the cdn network or something.

[hoshinokanade]

A year passed the providers are still very unreliable: It randomly fails the CI and we can do nothing but to speed up plans to migrate away from Libman as soon as possible.

[maxt3r]

The issue is still happening:

libman.json : error LIB002: The "font-awesome@4.7.0" library could not be resolved by the "cdnjs" provider

Running libman cache clean seems to have helped.

[jimmylewis]

@maxt3r if this just recently started happening for cdnjs, it's probably due to #685

[vivekthangaswamy]

One of the actions that helped me solve this problem was updating project packages with stable releases. Some of my packages were running in depreciated version.