sccache-action usage: Use GHA as local cache #2142
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2142 +/- ##
==========================================
- Coverage 30.91% 30.22% -0.69%
==========================================
Files 53 53
Lines 20112 20590 +478
Branches 9755 10029 +274
==========================================
+ Hits 6217 6224 +7
- Misses 7922 8281 +359
- Partials 5973 6085 +112 ☔ View full report in Codecov by Sentry. |
Alphare
force-pushed
the
gha-local
branch
2 times, most recently
from
9a8cc59
to
37bf416
Compare
|
Fascinating idea! Actually, I believe it could be a broader solution applicable to all storage services. For instance, downloading from S3/GCS/AZBlob should also be useful. What do you think? |
Indeed, I even mentioned this briefly in this PR's first message. I think this can make sense, but I didn't want to start with a more generic implementation before I could get a sense of how correct my solution is to this problem. Is there any chance you could answer the few questions I've left as comments in the code? |
|
I think this is ready for a first review. I'll try to somehow get |
sylvestre
changed the title
|
I guess you saw that some jobs are failing |
Alphare
force-pushed
the
gha-local
branch
2 times, most recently
from
80ac943
to
ae6430a
Compare
|
After a long battle I finally managed in a (very very) hacky way to exercise this path¹. I'll have to make adjustments here when debugging. [1] https://github.com/Mozilla-Actions/sccache-action/actions/runs/8803242248/job/24160865310 |
Alphare
force-pushed
the
gha-local
branch
2 times, most recently
from
33b7587
to
7b240de
Compare
src/cmdline.rs
Outdated
| @@ -274,6 +282,15 @@ pub fn try_parse() -> Result<Command> { | |||
| Ok(Command::DistAuth) | |||
| } else if matches.get_flag("dist-status") { | |||
| Ok(Command::DistStatus) | |||
| } else if matches.get_flag("upload-cache") { | |||
There was a problem hiding this comment.
Hi, I'm considering whether we can add this feature without altering the existing workflow. For instance, we could introduce a new configuration value in sccache, such as remote_cache. When the sccache daemon starts, it would load from the remote cache and then write back to it upon shutdown or at regular intervals.
This approach would eliminate the need for users to manually download or upload the cache.
What do you think?
There was a problem hiding this comment.
In this design, we just need to add a new input in sccache-action without changing the workflow.
There was a problem hiding this comment.
I can definitely see the "upon shutdown" version work and I could possibly already use it in this PR. I don't think it could work for regular intervals since GHA for example does not allow re-uploading to the same cache version+path.
There was a problem hiding this comment.
I can definitely see the "upon shutdown" version work and I could possibly already use it in this PR. I don't think it could work for regular intervals since GHA for example does not allow re-uploading to the same cache version+path.
You are right. I'm guessing regular intervals is not that useful too. We can remove this part.
There was a problem hiding this comment.
I can definitely see the "upon shutdown" version work and I could possibly already use it in this PR.
The key point I want to emphasize is not to add new command lines or alter the existing workflow that utilizes sccache. WDYT?
There was a problem hiding this comment.
Yep, agreed, I think it's a better design anyway. :)
Alphare
force-pushed
the
gha-local
branch
10 times, most recently
from
5056ed1
to
464138d
Compare
This is the first implementation of uploading the local cache as a single file to a remote cache for reuse in a future build. Right now it is only done for GHA as that was the intended scope¹, but one could adapt this system to other remote caches. Because of the immutability of GHACache, this commit only adds support for re-using the cache for the same version (as defined by the user through the `SCCACHE_GHA_VERSION` environment variable). A way of reusing incremental build within a given version or even across versions could be devised, but it falls outside the scope of this particular effort, and it's probably not trivial. [1] Mozilla-Actions/sccache-action#81
|
I have failed to fully get this to a working state because of an unknown error (in the I have detailed my thoughts about the remaining work in a TODO inside the code to make it easier to contextualize for a further effort, as agreed to with @sylvestre in a private chat. |
This is a draft PR to implement the base for a fix for Mozilla-Actions/sccache-action#81.
Some users have been implementing their own fix by going around the
sccache-actionand making their own cache in GHAc (Mozilla-Actions/sccache-action#50), but I think that this allows us to be more generic by moving more of the complexity intosccacheitself, which makes it easier for end-users and allows us to re-use the same principle for other remote storage systems in the future.This will need to be accompanied by a separate PR in
sccache-actionto make use of the new config option(s) and the new--update-cachecommand.I've listed TODOs in the code as questions for reviewers, hence the draft state.