Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

propose a different caching strategy in sccache-action #81

Open
sylvestre opened this issue Nov 17, 2023 · 1 comment
Open

propose a different caching strategy in sccache-action #81

sylvestre opened this issue Nov 17, 2023 · 1 comment

Comments

@sylvestre
Copy link
Collaborator

Currently,
sccache-action uses the strategy described here:
https://github.com/mozilla/sccache/blob/main/docs/Architecture.md
the storage being GitHub action storage.
it is storing a artifact per file.

It causes some issues like:

  • rate limiting (we have often artifact not stored because GHA is rejecting them)
  • we store a lot of things

We should consider having a new configuration parameter to use a local copy of the cache instead.

The process would be:

  1. if it exists, download the cache storage (file single) and unpack it
  2. set up sccache to use the local unpacked cache directory - with direct access (by default in 0.7.2)
  3. after the build, upload the new local cache directory in the github storage

If there is no cache available, skip step 2, just start sccache as local
@sylvestre sylvestre mentioned this issue Nov 17, 2023
Closed
@russkel
Copy link

russkel commented Jan 25, 2024

I run sscache from within Docker on Actions so this would be a good idea.

@Alphare Alphare mentioned this issue Apr 3, 2024
Open
Alphare added a commit to Alphare/sccache that referenced this issue Apr 23, 2024
@Alphare
feat: add a way of using GHA cache locally
80ac943 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 23, 2024
@Alphare
feat: add a way of using GHA cache locally
ae6430a 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 24, 2024
@Alphare
feat: add a way of using GHA cache locally
33b7587 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 24, 2024
@Alphare
feat: add a way of using GHA cache locally
7b240de 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 24, 2024
@Alphare
feat: add a way of using GHA cache locally
38d71fa 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 24, 2024
@Alphare
feat: add a way of using GHA cache locally
e46b0fd 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 24, 2024
@Alphare
feat: add a way of using GHA cache locally
c8aee11 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 24, 2024
@Alphare
feat: add a way of using GHA cache locally
b4d7561 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 24, 2024
@Alphare
feat: add a way of using GHA cache locally
9f7147e 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 24, 2024
@Alphare
feat: add a way of using GHA cache locally
4f069b1 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 24, 2024
@Alphare
feat: add a way of using GHA cache locally
37eba01 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 24, 2024
@Alphare
feat: add a way of using GHA cache locally
a05dc47 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 24, 2024
@Alphare
feat: add a way of using GHA cache locally
5056ed1 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Alphare added a commit to Alphare/sccache that referenced this issue Apr 24, 2024
@Alphare
feat: add a way of using GHA cache locally
464138d 
This is the first implementation of uploading the local cache
as a single file to a remote cache for reuse in a future build.

Right now it is only done for GHA as that was the intended scope¹,
but one could adapt this system to other remote caches.

Because of the immutability of GHACache, this commit only adds support
for re-using the cache for the same version (as defined by the user
through the `SCCACHE_GHA_VERSION` environment variable).
A way of reusing incremental build within a given version or even
across versions could be devised, but it falls outside the scope of
this particular effort, and it's probably not trivial.

[1] Mozilla-Actions/sccache-action#81
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sylvestre @russkel