Occasionally getting candidates is not iterable error

[jonathonlacher]

Description:
I'm occasionally getting the following error when this Action is ran:

Run actions/setup-python@v5
Installed versions
  Version 3.12.3 was not found in the local cache
  Error: candidates is not iterable

Action version:
v5.1.0

Platform:

• Ubuntu
• macOS
• Windows

Runner type:

• Hosted
• Self-hosted

Tools version:

3.12.3

Repro steps:
Repo is on GHES, so not able to link to it.

1. I have a reusable workflow that is used by a number of other projects.
2. The reusable workflow has a step that uses the setup-python action.

   ...
   runs:
     using: "composite"
     steps:
     - uses: actions/setup-python@v5
       with:
         python-version: '3.12.3'

3. On occasion, I think starting with in the last day, I sometimes see the error. A re-run typically works successfully.

Not sure if the complexities of GHES are the cause, though I have seen some indications searching all of github.com that at least 1 other person has gotten this error before when using this action.

They potentially say it's related to rate limiting, though the error, at least to me, isn't clear enough to indicate that.

This also might be relevant: setup-python is one of the default bundled Actions with GHES, so my understanding is the that the runner should only be calling out to github.com for accessing the python-versions maniest and corresponding release.

Expected behavior:

Here is what I expect, for the Action to download the release tar.

Actual behavior:

The action fails to find the version available for downloading.

[jonathonlacher]

At the very least, it seems the error message isn't clear enough to understand the actual issue.

[HarithaVattikuti]

Hello @jonathonlacher
Thank you for creating this issue. We will investigate it and get back to you as soon as we have some feedback.

[mahabaleshwars]

Hello @jonathonlacher, Thank you once again for bringing this issue to our attention. We attempted to reproduce the problem but were unsuccessful. Upon further investigation, we identified the following potential causes for the failure:

• Network Interruptions: Intermittent network issues might cause failures in downloading the specified Python version. These issues could be transient and may not occur during subsequent attempts.
• GHES Complexities: Specific complexities or configurations related to GitHub Enterprise Server (GHES) might influence the action's behavior.
  It would be very helpful if we could obtain the detailed logs of the failed run or a link to a public repository that reproduces the problem . As we understand from your description, the issue is not consistent, and these logs would aid us in identifying the root cause more effectively.

[aparnajyothi-y]

Hello @jonathonlacher, we attempted to reproduce the issue on the GHES server but were unsuccessful even after more than 150 runs. Please review the above feedback, which likely identifies the cause of the issue. I have attached a screenshot for reference.

[jonathonlacher]

@aparnajyothi-y @mahabaleshwars do you have any suggestions on how to capture more logs or which logs would be helpful?

What specifically is throwing the error ( Error: candidates is not iterable)?

[HarithaVattikuti]

Hello @jonathonlacher,
To capture more detailed logs in your GitHub Actions workflow, you can enable additional debugging by setting the following environment variables:

env:
  ACTIONS_RUNNER_DEBUG: true
  ACTIONS_STEP_DEBUG: true

To check the logs in the runner, follow these steps:

1. Navigate to the Runner Directory: Go to the directory where the self-hosted runner is installed.
2. Check the _diag Directory: Within the runner installation directory, locate the _diag directory. It contains diagnostic logs.

These settings will generate detailed logs to help diagnose issues. However, we currently cannot pinpoint the exact cause of the "Error: candidates is not iterable" message.

[jonathonlacher]

@HarithaVattikuti I've setup a repo that runs the Action on a 5 minute cron with the debug logs enabled, will let you know when I encounter the issue.

As a note, we use ephemeral runners in containers, and don't shell access to them, so I'm not able to grab any files from the runner file system.

[jonathonlacher]

Ok I have debug logs from the setup-python failure:

##[debug]Evaluating condition for step: 'Run actions/setup-python@v5'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Run actions/setup-python@v5
##[debug]Register post job cleanup for action: actions/setup-python@v5
##[debug]Loading inputs
##[debug]Evaluating: (((github.server_url == 'https://github.com') && github.token) || '')
##[debug]Evaluating Or:
##[debug]..Evaluating And:
##[debug]....Evaluating Equal:
##[debug]......Evaluating Index:
##[debug]........Evaluating github:
##[debug]........=> Object
##[debug]........Evaluating String:
##[debug]........=> 'server_url'
##[debug]......=> 'https://git.myorg.com'
##[debug]......Evaluating String:
##[debug]......=> 'https://github.com'
##[debug]....=> false
##[debug]..=> false
##[debug]..Evaluating String:
##[debug]..=> ''
##[debug]=> ''
##[debug]Expanded: ((('https://git.myorg.com' == 'https://github.com') && github['token']) || '')
##[debug]Result: ''
##[debug]Loading env
Run actions/setup-python@v5
  with:
    python-version: 3.10
    check-latest: false
    update-environment: true
    allow-prereleases: false
##[debug]Python is expected to be installed into /opt/hostedtoolcache
::group::Installed versions
Installed versions
  ##[debug]Semantic version spec of 3.10 is 3.10
  ##[debug]isExplicit: 
  ##[debug]explicit? false
  ##[debug]evaluating 0 versions
  ##[debug]match not found
  Version 3.10 was not found in the local cache
  ##[debug]Getting manifest from actions/python-versions@main
  Error: candidates is not iterable
  ##[debug]Node Action run completed with exit code 1
  ##[debug]Finishing: Run actions/setup-python@v5

Here is my workflow:

name: Run Python on cron

on:
  schedule:
    - cron: '*/5 * * * *'
  workflow_dispatch: {}

jobs:
  Just-run-python:
    runs-on: [ self-hosted, arc-dind-rootless-enterprise ]

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.10' 

      - name: Run Python script
        run: python main.py

[Lavaerius]

Hi there. I've done some testing with this, by making a duplicate of the setup-python action and injecting some more debug statements in a few areas.

    core_1.debug(`listed candidates: ${JSON.stringify(candidates)}`)
    return __awaiter(this, void 0, void 0, function* () {
        const platFilter = os.platform();
        let result;
        let match;
        let file;
        for (const candidate of candidates) {
            const version = candidate.version;
        .....

With this, I found that there are times when a run will hit the github api rate limit issue, but instead of falling back to
getManifestFromURL() it receives valid json, and fails at iterating over it.

improper response, no fallback

  ##[debug]manifestUrl: https://api.github.com/repos/actions/python-versions/git/blobs/5418fd77743bd877e972056787b3ee67a5725566
  ##[debug]listed candidates: {"message":"API rate limit exceeded for 52.255.212.108. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}
  Error: candidates is not iterable
  ##[debug]Node Action run completed with exit code 1

proper response, proper fallback:

  ##[debug]API rate limit exceeded for 52.255.212.108. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)
  ##[debug]Falling back to fetching the manifest using raw URL. https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json

I'm not certain if this fix should be in the action code, or if this is a bug in how github responds to api limits, but this does appear to be the culprit for this particular issue.

[Lavaerius]

on advice, I've submitted:
actions/toolkit#1832

[mahabaleshwars]

Hello @jonathonlacher, thanks to the efforts of @Lavaerius, we have identified the root cause of the issue. It is related to the API rate limit being exceeded and the inability to fallback to the raw API. We're currently investigating it.