Tons of rate limit errors / sccache-actions can be slow because of github storage

[yujincheng08]

I found that my action's cache hit rate is too low (especially for c/c++). I enable debug log and saw tons of rate limits. How to solve this?

data close failed: RateLimited (temporary) at Writer::close => {"$id":"1","innerException":null,"message":"Request was blocked due to exceeding usage of resource 'Count' in namespace ''.","typeName":"Microsoft.TeamFoundation.Framework.Server.RequestBlockedException, Microsoft.TeamFoundation.Framework.Server","typeKey":"RequestBlockedException","errorCode":0,"eventId":3000}

[yujincheng08]

FYI, the first build:
https://github.com/LSPosed/Metagisk/actions/runs/5209660209/attempts/1

And the second build (re-trigger):
https://github.com/LSPosed/Metagisk/actions/runs/5209660209

And there are 1440 C/C++ misses!

Here you are the log from the first build, and you can see there are tons of rate limits after a specific step.

sccache_log.zip

That drastically drops the hit rate of the second build.

[Xuanwo]

There are very little thing we can do for this. Sccache & Ghac are designed like this: too much small object will lead to rate limit.

Try setup a s3 bucket if the speed is critical to you.

[yujincheng08]

Ok. Then I will use sccache locally and then upload to GitHub action cache like I previously did.

[Xuanwo]

topjohnwu/Magisk#7052 seems a nice idea! We can add this feature in sccache-action too:

• Start sccache with local cache
• Pack and unpack cache from ghac

[sylvestre]

@Xuanwo is that something you could implement ? :)

[sylvestre]

See mozilla/sccache#1818 & https://github.com/orgs/community/discussions/55049

[mwestphal]

In order to work around this, I had to implement my own sccache logic and not rely on this action, here it is if it helps anyone:

  - name: Initialize sccache environnement
    shell: bash
    run: |
      echo SCCACHE_CACHE=$(sccache --show-stats | grep Local | cut -d '"' -f2) >> $GITHUB_ENV
      echo DATE_STRING=$(date +'%Y%m%d') >> $GITHUB_ENV
      sccache --stop-server

  - name: Recover sccache cache
    uses: actions/cache@v3
    with:
      path: ${{env.SCCACHE_CACHE}}
      key: sccache-cache-${{runner.os}}-0-${{env.DATE_STRING}}
      restore-keys: sccache-cache-${{runner.os}}-0

  - name: Start sccache
    shell: bash
    working-directory: ${{github.workspace}}
    run: sccache --start-server

[...]    

  - name: Cleanup sccache
    working-directory: ${{github.workspace}}
    shell: bash
    run: sccache --stop-server

• First step run the server, recover cache location and current data, then stop the server
• Second step recover a github action cache based on the runner.os AND the date. If the cache for today has not been created, it will grab the cache from yesterday thanks to the restore-key logic. A single cache file is used, not hundreds.
• Third step start the server
• Last step stop the server and report stats
• On cleanup, cache action upload today's cache if it was not available

A 2h C++ build now takes less than 1 hour.

Hth,

See it live here: https://github.com/f3d-app/f3d-superbuild/blob/main/.github/actions/f3d-superbuild/action.yml

[sylvestre]

what about integrating this into sccache action itself ? :)
maybe as an option ?

[mwestphal]

what about integrating this into sccache action itself ? :)

That would be great, but I've never written a line of typescript in my life. Maybe the maintainer will want to integrate such a change.

[sylvestre]

@Xuanwo would you be up for this task ? :)

[mwestphal]

(added a link to the action on our repo)

[0o-de-lally]

In order to work around this, I had to implement my own sccache logic and not rely on this action, here it is if it helps anyone:

Hi all, seeing the same issue with the file fragmentation. I've also run into the mtime issue with large rust builds forcing long rebuilds.

Note we've also tried using a remote S3, though since github secrets can't be used on public Pull Request events, that option isn't viable for actually open source projects.

This seems like a straightforward and useful implementation.

First step run the server, recover cache location and current data, then stop the server.

Question to @mwestphal: Why is there a need to start the sccache server (given that it will start on the first compile request)?

Question for the sccachecore maintainers: Is there any reason that this behavior (saving local sccache files, and restoring) is not the default for github actions?

[mwestphal]

Why is there a need to start the sccache server (given that it will start on the first compile request)?

I need to recover the location sccache will look for the its own cache (SCCACHE_CACHE), I need to start the server for that. I then stop it, and recover the cache from gha, then start it again.

[0o-de-lally]

@sylvestre here's a toy I made on a fork of this action. https://github.com/0o-de-lally/sccache-action/tree/local
It sets an input "local" which defaults to true in my version.

It will just cache the contents into a bundle called "sccache". It's minimalist, there are no key options. It will instead delete the cache every time and overwrite it (github doesn't allow just updating a cache).

This suits my needs.

Anyone else that wants to try it it's a zero config:

    - name: enable sccache
      uses: 0o-de-lally/sccache-action@local

cc. @mwestphal