NPM lock file updating errors with 401 for custom artifactory

[Anonycoders]

How are you running Renovate?

Self-hosted Renovate

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

38.125

Please tell us more about your question or problem

Hi folks 👋🏼
We are using (self-hosted) renovate in Enterprise Github and for updating dependencies, everything works like a charm but when it comes to updating package-lock.json or any lock file, it fails with 401 error.

The weird part is that the very same registry with the same credentials is working for finding packages and updates!

I have tried almost all of the suggestions and workarounds discussed in other issues/discussions but so far no luck.
Can somebody please help me on this?

More Contexts:

Registry

Our registry is a self-hosted JFrog artifactory with custom domain:

https://artifactory.bar.com/artifactory/api/npm/foo-npm-virtual/

Config.js

module.exports = {
  binarySource: 'install',
  executionTimeout: 300, // minutes
  hostRules: [
    {
      hostType: 'npm',
      matchHost: 'https://artifactory.bar.com/artifactory/api/npm/foo-npm-virtual/',
      username: process.env.ARTIFACTORY_USER,
      password: process.env.ARTIFACTORY_NPM_TOKEN
    },
    {
      hostType: 'maven',
      ...
    },
  ],
  npmrc: `registry=https://artifactory.bar.com/artifactory/api/npm/foo-npm-virtual/`,
  logFileLevel: 'warn',
  repositories: [], // the list of repositories is now configured in the github action in .github/workflows/renovate.yaml
  onboardingConfig: {
    extends: [
      "config:base",
      "local>foo/renovate:java-libraries-versioning",
      "group:recommended",
      ":semanticCommitsDisabled",
      "default:preserveSemverRanges"
    ],
    lockFileMaintenance: {
        enabled: true
    },
    rebaseWhen :"conflicted",
    automergeSchedule: "after 09:00 and before 16:00 on Monday to Thursday",
    dependencyDashboard: true,
    ignoreDeps: [],
    assignees: [],
  }
};

---

.npmrc

strict-ssl=false
always-auth=true
registry=https://artifactory.bar.com/artifactory/api/npm/foo-npm-virtual/

I know that .npmrc won't work here since we use the inline npmrc in config.js but pasted here just in case.

Logs (if relevant)

Logs

DEBUG: manager.getUpdatedPackageFiles() reuseExistingBranch=false (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: updateArtifacts for lockFileMaintenanceFiles (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: npm.updateArtifacts(package.json) (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: No packageManager updates - returning null (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: No package files need updating (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Getting updated lock files (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Writing package.json files (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
       "packageFiles": ["package.json"]
DEBUG: Writing package-lock.json (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Writing any updated package files (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Found 1 npm host rule(s) (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Adding npmrc entry for //artifactory.bar.com/artifactory/api/npm/foo-npm-virtual/ with username/password (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: npmrc file .npmrc found in repository (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Writing updated .npmrc file to .npmrc (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Generating package-lock.json for . (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Spawning npm install to create ./package-lock.json (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Found npm constraint in package.json engines: 10.9.0 (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Updating lock file only (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Using node constraint "20" from .node-version (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Removing package-lock.json first due to lock file maintenance upgrade (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Setting CONTAINERBASE_CACHE_DIR to /tmp/renovate/cache/containerbase (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Using containerbase dynamic installs (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: Resolved stable matching version (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
       "toolName": "node",
       "constraint": "20",
       "resolvedVersion": "v20.18.0"
DEBUG: Executing command (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
       "command": "install-tool node v20.18.0"
DEBUG: exec completed (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
       "durationMs": 731,
       "stdout": "[11:33:07.029] INFO (167): Installing tool node@20.18.0...\n[11:33:07.033] INFO (167): tool already installed\n    tool: \"node\"\nv20.18.0\n10.8.2\n0.29.3\n[11:33:07.472] INFO (167): Install tool node succeeded in 447ms.\n",
       "stderr": ""
DEBUG: Executing command (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
       "command": "install-tool npm 10.9.0"
DEBUG: rawExec err (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
       "err": {
         "cmd": "/bin/sh -c install-tool npm 10.9.0",
         "stderr": "",
         "stdout": "[11:33:07.746] INFO (203): Installing npm npm@10.9.0...\n[11:33:08.441] WARN (203): Npm error:\nnpm info using npm@10.8.2\nnpm info using node@v20.18.0\nnpm http fetch GET 401 https://artifactory.bar.com/artifactory/api/npm/foo-npm-virtual/npm 38ms (cache skip)\nnpm error code E401\nnpm error Incorrect or missing password.\nnpm error If you were trying to login, change your password, create an\nnpm error authentication token or enable two-factor authentication then\nnpm error that means you likely typed your password in incorrectly.\nnpm error Please try again, or recover your password at:\nnpm error   https://www.npmjs.com/forgot\nnpm error\nnpm error If you were doing some other operation then your saved credentials are\nnpm error probably out of date. To correct this please try logging in again with:\nnpm error   npm login\nnpm error A complete log of this run can be found in: /tmp/containerbase-npm-lpXblx/_logs/2024-10-25T11_33_07_933Z-debug-0.log\n[11:33:08.450] ERROR (203): npm install command failed\n[11:33:08.451] FATAL (203): Install tool npm failed in 708ms.\n",
         "options": {
           "cwd": "/tmp/renovate/repos/github/foo/renovate-npm-test",
           "encoding": "utf-8",
           "env": {
             "NPM_CONFIG_CACHE": "/tmp/renovate/cache/others/npm",
             "npm_config_store": "/tmp/renovate/cache/others/pnpm",
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8",
             "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
           },
           "maxBuffer": 10485760,
           "timeout": 18000000
         },
         "exitCode": 1,
         "name": "ExecError",
         "message": "Command failed: install-tool npm 10.9.0\n",
         "stack": "ExecError: Command failed: install-tool npm 10.9.0\n\n    at ChildProcess.<anonymous> (/usr/local/renovate/lib/util/exec/common.ts:101:11)\n    at ChildProcess.emit (node:events:531:35)\n    at ChildProcess.emit (node:domain:488:12)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:294:12)"
       },
       "durationMs": 1220
DEBUG: lock file error (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
       "err": {
         "cmd": "/bin/sh -c install-tool npm 10.9.0",
         "stderr": "",
         "stdout": "[11:33:07.746] INFO (203): Installing npm npm@10.9.0...\n[11:33:08.441] WARN (203): Npm error:\nnpm info using npm@10.8.2\nnpm info using node@v20.18.0\nnpm http fetch GET 401 https://artifactory.bar.com/artifactory/api/npm/foo-npm-virtual/npm 38ms (cache skip)\nnpm error code E401\nnpm error Incorrect or missing password.\nnpm error If you were trying to login, change your password, create an\nnpm error authentication token or enable two-factor authentication then\nnpm error that means you likely typed your password in incorrectly.\nnpm error Please try again, or recover your password at:\nnpm error   https://www.npmjs.com/forgot\nnpm error\nnpm error If you were doing some other operation then your saved credentials are\nnpm error probably out of date. To correct this please try logging in again with:\nnpm error   npm login\nnpm error A complete log of this run can be found in: /tmp/containerbase-npm-lpXblx/_logs/2024-10-25T11_33_07_933Z-debug-0.log\n[11:33:08.450] ERROR (203): npm install command failed\n[11:33:08.451] FATAL (203): Install tool npm failed in 708ms.\n",
         "options": {
           "cwd": "/tmp/renovate/repos/github/foo/renovate-npm-test",
           "encoding": "utf-8",
           "env": {
             "NPM_CONFIG_CACHE": "/tmp/renovate/cache/others/npm",
             "npm_config_store": "/tmp/renovate/cache/others/pnpm",
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8",
             "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
           },
           "maxBuffer": 10485760,
           "timeout": 18000000
         },
         "exitCode": 1,
         "name": "ExecError",
         "message": "Command failed: install-tool npm 10.9.0\n",
         "stack": "ExecError: Command failed: install-tool npm 10.9.0\n\n    at ChildProcess.<anonymous> (/usr/local/renovate/lib/util/exec/common.ts:101:11)\n    at ChildProcess.emit (node:events:531:35)\n    at ChildProcess.emit (node:domain:488:12)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:294:12)"
       },
       "type": "npm"
DEBUG: No updated lock files in branch (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: No changes to package files, skipping post-upgrade tasks (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: PR has no releaseTimestamp (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)
DEBUG: No files to commit (repository=foo/renovate-npm-test, branch=renovate/lock-file-maintenance)

[rarkins]

Do you have a .npmrc in your repo? If not, where is that .npmrc with strict-ssl and always-auth from? It looks like you left those settings out of your config.js npmrc

[Anonycoders]

Sorry for that.
The .npmrc is located in the root of the renovate repository where the configurations and workflows reside.
Also each target repository (to be updated with renovate) has a separate .npmrc in the root of its repository too.

Something like:

always-auth=true
email=${ARTIFACTORY_USER}
registry=https://artifactory.bar.com/artifactory/api/npm/foo-npm-virtual/
//artifactory.bar.com/artifactory/api/npm/foo-npm-virtual/:_auth=${ARTIFACTORY_NPM_SECRET}

[Anonycoders]

Hi @rarkins , any update on this issue? :(

[rarkins]

The failure in the logs comes from a failure of install-tool npm and not npm install. In other words, it's from "containerbase", which Renovate uses, trying to install npm before running npm install, and it's due to your need to route things through Artifactory, which is not easy to do with Containerbase dynamic installs (which is binarySource=install in Renovate terms).

Getting binarySource=install to work with private registries is quite a challenge. There's documentation about it here but we don't have the time to troubleshoot through it all with users.

Instead, you should get Renovate running with binarySource=global and a Renovate "full" image, which comes with one version of each package manager preinstalled.

My recommendation is to update to renovate/renovate:39-full and retest, because it defaults to binarySource=global. If doing that solves the problem above, please comment here and start a new/separate discussion for any subsequent problems.

[Anonycoders]

Thank you for the explanation @rarkins 🙇🏼

I will test the renovate/renovate:39-full but my concern is that it comes with the latest npm,yarn and other tools. Would it cause any issue later for services using different version of for example npm and node?

Getting binarySource=install to work with private registries is quite a challenge. There's documentation about it here but we don't have the time to troubleshoot through it all with users.

This seems worthy to try but for the starter, can we use the suggested envs from the link by passing them to renovate or we need to have some sort of custom dockerfile?
If you have some sample or instructions would be great but if not, I'll explore later no pressure.

[rarkins]

Yes, using the full image comes with its downsides. But this is as far as I can help you here - please see the disclaimer to follow

[github-actions[bot]]

Hi there,

The maintainers have labeled this discussion as a use case specific to "enterprise" users. Enterprise users typically self-host Renovate and have advanced requirements like custom certificates, HTTP proxies, private registries, air gapping, etc.

Such cases usually need a lot of back-and-forth to resolve, because the problem is related to the user's environment. Usually the user's environment can not be reproduced publicly, which means that maintainer debugging is not possible.

The maintainers decided their time is best spent on issues that help large numbers of users. This means enterprise-specific issues affecting one user (or a small number of users), are not a priority. If we can help you quickly, we will. But if you see this message, then a maintainer believes this is not a quick fix. The maintainers will not actively work on this issue, unless new information is found.

If you are an enterprise user, please consider the following options:

• Troubleshoot and contribute a fix yourself, or
• Hire a consultant to help you, or
• Consider an Enterprise support contract from Mend.io, or
• Wait for a community member to help you, if they have the same problem

If you choose to wait, then this discussion can remain open indefinitely. But please be patient, do not nudge or bump this discussion just to get attention.

Thanks, the Renovate team