feat(buildless): add support for buildless (https://less.build)

- feat(buildless): add configuration structures for buildless
- feat(buildless): implement cache backend for buildless, using webdav
- feat(buildless): implement cache backend for buildless, using redis
- feat(buildless): implement agent detection and use (where enabled)
- feat(buildless): decode agent config and use specified port
- fix(buildless): don't use control port for cache traffic
- docs(buildless): add `docs/Buildless` and root README references

Signed-off-by: Sam Gammon <sam@elide.ventures>

Cargo.toml

@@ -2,7 +2,7 @@
 edition = "2021"
 name = "sccache"
 rust-version = "1.67.1"
-version = "0.7.4"
+version = "0.7.5-buildless"
 
 categories = ["command-line-utilities", "development-tools::build-utils"]
 description = "Sccache is a ccache-like tool. It is used as a compiler wrapper and avoids compilation when possible, storing a cache in a remote storage using various cloud storage."
@@ -146,10 +146,13 @@ all = [
   "memcached",
   "gcs",
   "azure",
+  "buildless",
   "gha",
   "webdav",
 ]
 azure = ["opendal", "reqsign"]
+buildless = ["opendal", "redis", "webdav"]
+buildless-client = ["buildless", "dist-client", "native-zlib", "vendored-openssl"]
 default = ["all"]
 gcs = ["opendal", "reqsign", "url", "reqwest/blocking", "trust-dns-resolver"]
 gha = ["opendal"]

README.md

@@ -42,6 +42,7 @@ Table of Contents (ToC)
   * [Azure](docs/Azure.md)
   * [GitHub Actions](docs/GHA.md)
   * [WebDAV (Ccache/Bazel/Gradle compatible)](docs/Webdav.md)
+  * [Buildless (Agent, Cloud, or both)](docs/Buildless.md)
 
 ---
 
@@ -287,3 +288,4 @@ Storage Options
 * [Azure](docs/Azure.md)
 * [GitHub Actions](docs/GHA.md)
 * [WebDAV (Ccache/Bazel/Gradle compatible)](docs/Webdav.md)
+* [Buildless (Agent, Cloud, or both)](docs/Buildless.md)

docs/Buildless.md

@@ -0,0 +1,112 @@
+# Buildless
+
+[Buildless][0] is a suite of build caching tools and a remote build caching cloud. It can be used with nearly any build
+toolchain which supports caching, including SCCache, Gradle, Maven, Bazel, and TurboRepo.
+
+Near-caching with the [Buildless Agent][1] is free forever for every user. Then, to share caches, link your CLI to the
+[Buildless Cloud][2], where it can be shared with your teammates, vendors, customers, and more.
+
+## Using SCCache with Buildless
+
+The Buildless adapter for SCCache supports several transports and endpoint types:
+
+- **[Buildless Agent][1]:** Local Buildless agent and near-cache
+- **[Buildless Cloud][2]:** Cloud services for caching and configuration
+- **[HTTPS][3]**, **[Redis][4]**, and **[GitHub Actions][5]**
+
+Usually, you will want to use this in tandem with the [Buildless Agent][1], which acts as a near-cache and optimized
+backhaul router when cloud services are enabled.
+
+## Installation & Setup
+
+Obtain a [release of `sccache`][6] from the Buildless team, or clone the fork and compile it with the instructions
+below.
+
+### Obtaining a release
+
+Releases are [published on GitHub][6], and additionally bundled with the [Buildless CLI][1] (see `buildless sccache`).
+Verification of releases can be performed via [Sigstore][9] and similar tools; for more details, see the
+[CLI release notes][10].
+
+1. Download a release from one of the sources above
+2. Place it somewhere on your machine, make sure it is executable
+3. (**Optional**): Install the [Buildless CLI][1] and run the [Buildless Agent][8]
+
+### Building yourself
+
+1. Clone or add the fork: `git clone git@github.com:buildless/sccache.git`
+2. Checkout your desired branch
+3. Build with: `cargo build --release --features=buildless-client`
+4. Check that Buildless is enabled: `./target/release/sccache --help`, which should show:
+
+```
+Usage: sccache ...
+
+Enabled features:
+    S3:        false
+    Redis:     true
+    Memcached: false
+    GCS:       false
+    GHA:       false
+    Azure:     false
+    Buildless: true
+```
+
+> Native zlib and TLS are built-in when using this configuration.
+
+## Usage
+
+If you have an API key set at `BUILDLESS_APIKEY` ([docs][7]), the module will **activate automatically.**
+The [Buildless Agent][8] is detected, if running, and used (unless disabled -- see _Agent Negotation_ below). Otherwise,
+**you can set `SCCACHE_BUILDLESS`** to any value to force-enable the adapter.
+
+See other environment variables below for customizing `sccache`'s behavior as it relates to Buildless.
+
+| Environment Variable | Description                                                                                  |
+|----------------------|----------------------------------------------------------------------------------------------|
+| `SCCACHE_BUILDLESS`  | Force-enables the Buildless backend, even with no API key or other credentials.              |
+| `BUILDLESS_APIKEY`   | Automatically detected and set as the user's API key. Enables the module if detected.        |
+| `BUILDLESS_ENDPOINT` | Sets a custom endpoint for use by `sccache`. This should only be used in advanced scenarios. |
+| `BUILDLESS_NO_AGENT` | Instructs `sccache` not to ever use the [Buildless Agent][8].                                |
+
+### Agent Negotiation
+
+If the [Buildless Agent][8] is running on the local machine, it will be detected and used instead of the public
+[Buildless Cloud service][2]. The agent can be configured to use edge services or not, and **does not require a license
+or payment of any kind**. An account with Buildless Cloud is not required for local use.
+
+Agent detection works with a "rendezvous file," defined at known path on each operating system. The file is typically
+encoded in JSON and includes agent connection and protocol details.
+
+### Buildless Cloud
+
+If no agent is available, or if your agent is configured for Buildless Cloud services, it will automatically upload
+cached objects asynchronously and pull cached objects from the cloud, via an optimized long-living connection.
+
+### Transport Selection
+
+Several transport types are available for use via the agent and via the Buildless Cloud, including **HTTPS**, **Redis**
+(RESP), **gRPC**, and more. This adapter can use HTTPS or Redis.
+
+Since the agent only supports HTTP at this time, it is used automatically when the agent is enabled. When using Cloud
+servics directly, Redis is used, with HTTPS as a fallback. Buildless Cloud traffic is [always encrypted][11].
+
+## Docs & Support
+
+Find more documentation, including API reference docs, via the [Buildless Docs][12]. Support is also available for
+paying users, and for free users on a best-effort basis, via the [Buildless Support][13] site.
+
+[0]: https://less.build
+[1]: https://docs.less.build/cli
+[2]: https://less.build/resources/network
+[3]: https://docs.less.build/reference/supported-api-interfaces
+[4]: https://docs.less.build/docs/redis
+[5]: https://docs.less.build/docs/github-actions
+[6]: https://github.com/buildless/sccache
+[7]: https://docs.less.build/docs/auth
+[8]: https://docs.less.build/agent
+[9]: https://sigstore.dev
+[10]: https://github.com/buildless/cli/releases
+[11]: https://www.ssllabs.com/ssltest/analyze.html?d=edge.less.build
+[12]: https://less.build/docs
+[13]: https://less.build/support

src/cache/buildless.rs

@@ -0,0 +1,186 @@
+// Copyright 2016 Mozilla Foundation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use std::path::Path;
+
+use opendal::layers::LoggingLayer;
+use opendal::services::Webdav;
+use opendal::Operator;
+
+use cache::redis::RedisCache;
+
+use serde::{
+    Deserialize,
+};
+
+use crate::cache;
+use crate::config::BuildlessTransport;
+use crate::errors::*;
+
+pub struct BuildlessCache;
+
+// Constants used by the Buildless cache module.
+const BUILDLESS_LOCAL: &str = "local.less.build";
+const BUILDLESS_LOCAL_PORT_CONTROL: u16 = 42010;
+const BUILDLESS_LOCAL_PORT_HTTP: u16 = 42011;
+const BUILDLESS_LOCAL_PORT_RESP: u16 = 42012;
+const BUILDLESS_GLOBAL: &str = "global.less.build";
+const BUILDLESS_GLOBAL_PORT_HTTPS: u16 = 443;
+const BUILDLESS_GLOBAL_PORT_RESP: u16 = 6379;
+const BUILDLESS_HTTP_PREFIX_GENERIC: &str = "/cache/generic";
+const BUILDLESS_HTTP_APIKEY_USERNAME: &str = "apikey";
+
+#[derive(Debug, PartialEq, Eq, Deserialize)]
+pub struct BuildlessAgentEndpoint {
+    pub port: u16,
+    pub socket: Option<String>,
+}
+
+#[derive(Debug, PartialEq, Eq, Deserialize)]
+pub struct BuildlessAgentConfig {
+    pub pid: u32,
+    pub port: u16,
+    pub socket: Option<String>,
+    pub control: Option<BuildlessAgentEndpoint>,
+}
+
+fn read_agent_config(path: &Path) -> Option<BuildlessAgentConfig> {
+    let config: serde_json::Result<BuildlessAgentConfig> =
+        serde_json::from_str(path.to_str().unwrap());
+    return if config.is_ok() {
+        Some(config.unwrap())
+    } else {
+        None
+    }
+}
+
+fn validate_port(port: u16, _transport: BuildlessTransport) -> bool {
+    return port != BUILDLESS_LOCAL_PORT_CONTROL
+}
+
+fn build_https(use_agent: bool, endpoint: &Option<String>, apikey: &Option<String>, agent: &Option<BuildlessAgentConfig>) -> Result<Operator> {
+    let mut builder = Webdav::default();
+
+    // setup https endpoint or use global
+    if let Some(endpoint) = endpoint {
+        builder.endpoint(endpoint);
+    } else {
+        if use_agent && agent.is_some() {
+            let agent_config = agent
+                .as_ref()
+                .unwrap();
+            let effective_port: u16;
+            if validate_port(agent_config.port, BuildlessTransport::HTTPS) {
+                effective_port = agent_config.port;
+            } else {
+                effective_port = BUILDLESS_LOCAL_PORT_HTTP;
+            }
+            builder.endpoint(&*format!("http://{BUILDLESS_LOCAL}:{effective_port}"));
+        } else {
+            builder.endpoint(&*format!("https://{BUILDLESS_GLOBAL}:{BUILDLESS_GLOBAL_PORT_HTTPS}"));
+        }
+    }
+
+    // set default key path
+    builder.root(BUILDLESS_HTTP_PREFIX_GENERIC);
+
+    // if we have an explicit API key, use it
+    if let Some(apikey) = apikey {
+        builder.username(BUILDLESS_HTTP_APIKEY_USERNAME);
+        builder.password(apikey);
+    }
+    let op = Operator::new(builder)?
+        .layer(LoggingLayer::default())
+        .finish();
+    return Ok(op)
+}
+
+fn build_resp(use_agent: bool, endpoint: &Option<String>, apikey: &Option<String>) -> Result<Operator> {
+    return if endpoint.is_some() {
+        // build with custom redis URL endpoint
+        RedisCache::build(endpoint
+            .as_ref()
+            .unwrap()
+            .as_str()
+        )
+    } else {
+        let protocol: &str;
+        let endpoint_target: String;
+        if !use_agent {
+            protocol = "rediss";
+            endpoint_target = format!("{BUILDLESS_LOCAL}:{BUILDLESS_LOCAL_PORT_RESP}");
+        } else {
+            protocol = "redis";  // do not need TLS wrapping with local agent
+            endpoint_target = format!("{BUILDLESS_GLOBAL}:{BUILDLESS_GLOBAL_PORT_RESP}");
+        }
+
+        if apikey.is_some() {
+            let apikey_value: &String = apikey.as_ref().unwrap();
+            let auth = format!("{BUILDLESS_HTTP_APIKEY_USERNAME}:{apikey_value}@");
+
+            // build with apikey or other auth
+            RedisCache::build(&[
+                protocol,
+                "://",
+                &auth,
+                &endpoint_target,
+            ].concat())
+        } else {
+            // build with implied authorization, or no authorization
+            RedisCache::build(&[
+                protocol,
+                "://",
+                &endpoint_target,
+            ].concat())
+        }
+    }
+}
+
+impl BuildlessCache {
+    pub fn build(use_agent: bool, transport: &BuildlessTransport, endpoint: &Option<String>, apikey: &Option<String>) -> Result<Operator> {
+        // resolve agent state file path
+        let configpath: &str;
+        let instancepath: &str;
+        if cfg!(windows) {
+            configpath = "C:\\ProgramData\\buildless\\buildless-agent.json";
+            instancepath = "C:\\ProgramData\\buildless\\buildless-service.id";
+        } else if cfg!(unix) {
+            // darwin path
+            configpath = "/var/tmp/buildless/buildless-agent.json";
+            instancepath = "/var/tmp/buildless/buildless-service.id";
+        } else {
+            unimplemented!("Buildless caching is only supported on macOS, Linux, and Windows.");
+        }
+        let instance_exists = Path::new(instancepath).exists();
+        let agent_config_exists = Path::new(configpath).exists();
+        let do_use_agent =
+            use_agent &&  // agent is enabled
+            instance_exists &&  // instance is installed on this machine
+            agent_config_exists;  // agent is running (rendezvous file exists)
+
+        let agent_config: Option<BuildlessAgentConfig>;
+        if do_use_agent {
+            agent_config = read_agent_config(Path::new(configpath));
+        } else {
+            agent_config = None;
+        }
+
+        return match transport {
+            BuildlessTransport::AUTO => build_https(do_use_agent, endpoint, apikey, &agent_config),
+            BuildlessTransport::HTTPS => build_https(do_use_agent, endpoint, apikey, &agent_config),
+            BuildlessTransport::RESP => build_resp(do_use_agent, endpoint, apikey),
+            BuildlessTransport::GHA => unimplemented!("GHA protocol is not implemented yet for Buildless SCCache integration.")
+        }
+    }
+}