From 34cb4297f891f9cc101643049adc1e9be9d266f9 Mon Sep 17 00:00:00 2001 From: bohendo Date: Thu, 1 Feb 2024 08:41:25 -0500 Subject: [PATCH 1/4] overhaul README --- README.md | 62 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 61 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index a69c495..11b0f28 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,66 @@ # CloudExec -CloudExec is a command-line tool for easily running cloud-based jobs on DigitalOcean. Start, manage and pull the results of jobs from your terminal. +CloudExec is a management tool for running computation jobs on DigitalOcean via the command line. It is general purpose; Cloudexec can set up the server with arbitrary dependencies and then run an arbitrary workload process, but it is designed to run a single long-running code analysis job such as a fuzz testing or mutation testing campaign. Output data and runtime logs are uploaded to DigitalOcean's S3-style object storage for later analysis and, when the job is complete, the server is automatically destroyed. The client-side management engine of CloudExec is written in Golang and the server itself is managed by a Bash script. + +Features: + +- 1Password CLI support for secure DigitalOcean API key management. CloudExec will help you configure these credentials and verify that they are valid. +- Launch config file allows specification of: + - An input folder which is uploaded to the runtime server and also to DigitalOcean's S3-style object storage for later reference. This folder is zipped for speedy uploads. + - A job name, providing human-readable tags for each job. + - A timeout, after which the workload process will be terminated if it hasn't finished already, output will be uploaded to persistent storage, and the server will be destroyed so you will stop being charged for it. + - A setup command which uses bash to install dependencies and prepare the server to run the workload process. + - A run command which will kick off the workload process +- An `init` subcommand for creating a new, default launch config file +- Output is periodically uploaded to DigitalOcean's S3-style object storage so you can pull results incrementally from a running job +- Jobs can be cancelled early if the workload process hasn't completed or the timeout hasn't been reached yet +- Monitoring the runtime logs of a specific job or the status of all jobs +- Automatic `ssh_config` additions allowing you to access a running server by executing `ssh cloudexec` +- Tracks the cumulative costs incurred for running processes and the total cost of completed processes + +Run `cloudexec help` to list available subcommands or `cloudexec --help` for information regarding a specific subcommand: + +```bash +$ cloudexec help +NAME: + cloudexec - easily run cloud based jobs + +USAGE: + cloudexec [global options] command [command options] [arguments...] + +COMMANDS: + check, c Verifies cloud authentication + configure Configure credentials + init Create a new cloudexec.toml launch configuration in the current directory + launch, l Launch a droplet and start a job + logs Stream logs from a running job + cancel Cancels any running cloudexec jobs + clean Cleans up any running cloudexec droplets and clears the spaces bucket + pull Pulls down the results of the latest successful job + status, s Get status of running jobs + state Manage state file + attach, a Attach to a running job + version, v Gets the version of the app + help, h Shows a list of commands or help for one command + +GLOBAL OPTIONS: + --help, -h show help +``` + +Example job status output: + +```bash +$ cloudexec status --all ++--------+----------------+-----------+----------------+---------+------+-------+---------------------+---------------------+--------------+-------------+------------+ +| JOB ID | JOB NAME | STATUS | DROPLET IP | MEMORY | CPUS | DISK | STARTED AT | UPDATED AT | TIME ELAPSED | HOURLY COST | TOTAL COST | ++--------+----------------+-----------+----------------+---------+------+-------+---------------------+---------------------+--------------+-------------+------------+ +| 1 | medusa fuzz | completed | 12.34.56.78 | 4096 MB | 4 | 50 GB | 2024-01-01 13:55:53 | 2024-01-02 14:05:29 | 2 days | $0.125 | $6.0100 | ++--------+----------------+-----------+----------------+---------+------+-------+---------------------+---------------------+--------------+-------------+------------+ +| 2 | experiment | failed | 12.34.56.79 | 2048 MB | 2 | 25 GB | 2024-01-01 14:04:50 | 2024-01-01 14:08:03 | 3 minutes | $0.0625 | $0.0034 | ++--------+----------------+-----------+----------------+---------+------+-------+---------------------+---------------------+--------------+-------------+------------+ +| 3 | slither-mutate | running | 12.34.56.80 | 2048 MB | 2 | 25 GB | 2024-01-02 10:04:50 | 2024-01-02 12:08:03 | 2 hours | $0.0625 | $0.125 | ++--------+----------------+-----------+----------------+---------+------+-------+---------------------+---------------------+--------------+-------------+------------+ +``` ## Getting Started From cdfd7d7ba49ff02bb357a205a051713b6b93238b Mon Sep 17 00:00:00 2001 From: bohendo Date: Thu, 1 Feb 2024 08:41:25 -0500 Subject: [PATCH 2/4] fix syntax highlighting --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 11b0f28..6953376 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ Features: Run `cloudexec help` to list available subcommands or `cloudexec --help` for information regarding a specific subcommand: -```bash +```text $ cloudexec help NAME: cloudexec - easily run cloud based jobs @@ -49,7 +49,7 @@ GLOBAL OPTIONS: Example job status output: -```bash +```text $ cloudexec status --all +--------+----------------+-----------+----------------+---------+------+-------+---------------------+---------------------+--------------+-------------+------------+ | JOB ID | JOB NAME | STATUS | DROPLET IP | MEMORY | CPUS | DISK | STARTED AT | UPDATED AT | TIME ELAPSED | HOURLY COST | TOTAL COST | From 32f760d9a22fab44705952aa60376dc36f8bf601 Mon Sep 17 00:00:00 2001 From: bohendo Date: Fri, 8 Mar 2024 15:50:11 -0400 Subject: [PATCH 3/4] expand launch config section of readme --- README.md | 35 +++++++++++++++++++++++++---------- 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 6953376..0c9dada 100644 --- a/README.md +++ b/README.md @@ -146,7 +146,7 @@ Note what your [1Password secret references](https://developer.1password.com/doc These references generally follow the format: `op:////`. For example, if you saved your keys to a vault called `Private`, in an item called `DigitalOcean` and the api key field is called `ApiKey`, then the secret reference to use is `op://Private/DigitalOcean/ApiKey`. -#### Configure CloudExec +#### Configure CloudExec credentials ```bash cloudexec configure @@ -161,30 +161,43 @@ DIGITALOCEAN_SPACES_SECRET_ACCESS_KEY DIGITALOCEAN_SPACES_REGION ``` -Remember, if you save secret values to a `.env` file, never commit it to any version control system. Add such `.env` files to your project's `.gitignore` file to help prevent making such mistakes. +Remember, if you save secret values to a `.env` file, never commit it to any version control system. Add such `.env` files to your project's `.gitignore` file to help prevent mistakes. -### Check CloudExec access - -Confirm `cloudexec` has access to DigitalOcean. +Confirm `cloudexec` is authorized to access to DigitalOcean. ```bash cloudexec check ``` -### Launch a new remote job +### Configure the new job -Generate a cloudexec.toml configuration file in the current directory. +Generate a `cloudexec.toml` configuration file in the current directory. ```bash cloudexec init ``` -Update the `cloudexec.toml` as needed. +Update this `cloudexec.toml` file as needed. The following fields are available: + +`[input]`: + +- `jobName`: an arbitrary, human-readable label that can help identify this job +- `directory`: the path to the input directory which will be uploaded to the cloud runner and from which the run command will be executed +- `timeout`: a string specifying a maximum duration for which the job can run. After this timeout is reached, results will be uploaded to s3-style storage and the server will be destroyed. For example, "6h" for six hours or "3d" for three days. + +`[commands]`: + +- `setup`: A bash string that can be used to instal arbitrary software prior to the start of the job. These setup commands are run at the beginning of each job and time elapsed does not count towards the timeout. +- `run`: A bash string that executes the workload command + +### Launch a new remote job + +Run `cloudexec launch` from the directory containing the launch config. ```bash # default nyc3 region and c-2 size droplet, using a cloudexec.toml file in the current directory cloudexec launch -# custom region and droplet size +# Or, specify a custom region and droplet size cloudexec launch --size c-4 --region sfo2 ``` @@ -226,6 +239,8 @@ cloudexec status cloudexec status --all ``` +The DigitalOcean dashboard will also provide helpful info including the droplet status, cpu and memory usage, and more; look for a droplet with a name that starts with `cloudexec-`. + ### Sync files from a completed job to a local path ```bash @@ -248,7 +263,7 @@ cloudexec cancel cloudexec clean ``` -Note that there is often a delay while deleting files from Digital Ocean Spaces buckets. +Note that there is a delay of up to 2 weeks while deleting files from Digital Ocean Spaces buckets. Files cannot be accessed or overwritten while such delays are pending, so plan ahead. This delay is enforced by Digital Ocean, cloudexec is not able to influence it. ## Optional: Create a CloudExec DigitalOcean image From 20f8c7688c4996ec50ac9f1264fb8b17f6565b1c Mon Sep 17 00:00:00 2001 From: bohendo Date: Wed, 13 Mar 2024 11:16:37 -0400 Subject: [PATCH 4/4] clarify warnings regarding .env secrets and deletion delays --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0c9dada..58a44f0 100644 --- a/README.md +++ b/README.md @@ -161,7 +161,7 @@ DIGITALOCEAN_SPACES_SECRET_ACCESS_KEY DIGITALOCEAN_SPACES_REGION ``` -Remember, if you save secret values to a `.env` file, never commit it to any version control system. Add such `.env` files to your project's `.gitignore` file to help prevent mistakes. +Remember, if you save secret values to a `.env` file, never commit it to any version control system. Add such `.env` files to your project's `.gitignore` file to help prevent mistakes. Even when not committed, plaintext secrets in a `.env` file can pose security risks so we recommend using a dedicated secret management tool such as 1Password. Confirm `cloudexec` is authorized to access to DigitalOcean. @@ -263,7 +263,7 @@ cloudexec cancel cloudexec clean ``` -Note that there is a delay of up to 2 weeks while deleting files from Digital Ocean Spaces buckets. Files cannot be accessed or overwritten while such delays are pending, so plan ahead. This delay is enforced by Digital Ocean, cloudexec is not able to influence it. +Note that there is a delay of up to 2 weeks while deleting files from Digital Ocean Spaces buckets. Be aware that during this delay, the files will continue to occupy space in your bucket and may incur storage costs. If you need to manage your storage usage or costs, consider this delay when planning your data management strategy. This delay is managed by Digital Ocean, cloudexec is not able to influence it. ## Optional: Create a CloudExec DigitalOcean image