docs: updated & improved README

mento-protocol · Jul 24, 2024 · b81aa66 · b81aa66
1 parent 0a8a6e7
commit b81aa66
Showing 1 changed file with 110 additions and 96 deletions.
diff --git a/README.md b/README.md
@@ -8,18 +8,18 @@ A monorepo for our governance watchdog, a system that monitors Mento Governance
 
 ![Architecture Diagram](arch-diagram.png)
 
-- [Requirements](#requirements)
-- [Local Development of Cloud Function Code](#local-development-of-cloud-function-code)
+- [Requirements for local development](#requirements-for-local-development)
+- [Local Infra Setup (when project is deployed already)](#local-infra-setup-when-project-is-deployed-already)
+- [Running and testing the Cloud Function locally](#running-and-testing-the-cloud-function-locally)
 - [Testing the Deployed Cloud Function](#testing-the-deployed-cloud-function)
-- [Infra Setup (when project is deployed already)](#infra-setup-when-project-is-deployed-already)
-- [First Time Infra Deployment via Terraform](#first-time-infra-deployment-via-terraform)
+- [Updating the Cloud Function](#updating-the-cloud-function)
+- [Infra Deployment via Terraform](#infra-deployment-via-terraform)
   - [Google Cloud Permission Requirements](#google-cloud-permission-requirements)
   - [Deployment from scratch](#deployment-from-scratch)
   - [Migrate Terraform State to Google Cloud](#migrate-terraform-state-to-google-cloud)
-- [Updating the Cloud Function](#updating-the-cloud-function)
 - [Teardown](#teardown)
 
-## Requirements
+## Requirements for local development
 
 1. Install the `gcloud` CLI
 
@@ -51,6 +51,15 @@ A monorepo for our governance watchdog, a system that monitors Mento Governance
    # For other systems, see https://developer.hashicorp.com/terraform/install
    ```
 
+1. Install `jq` (used in a few shell scripts)
+
+   ```sh
+   # On macOS
+   brew install jq
+
+   # For other systems, see https://jqlang.github.io/jq/
+   ```
+
 1. Authenticate with Google Cloud default credentials in your local shell
 
    ```sh
@@ -70,45 +79,8 @@ A monorepo for our governance watchdog, a system that monitors Mento Governance
 1. A Telegram group to send notifications to
 
 1. A Telegram bot must be in the group to receive the notifications.
-   If you're doing this from scratch, here's how to create a bot
-
-   - Open a new chat with @BotFather
-   - Use the `/newbot` command to create a new bot
-   - Copy the API key printed out at the end of the prompt and store it in your `terraform.tfvars`
-
-     ```hcl
-     telegram_bot_token = "<bot-api-key>"
-     ```
-
-   - Get the Chat ID by inviting @MissRose_bot to the group and then using the `/id` command
-   - Add the Chat ID to your `terraform.tfvars`
 
-     ```hcl
-     telegram_chat_id = "<group-chat-id>"
-     ```
-
-   - Remove @MissRose_bot after you got the Chat ID
-
-## Local Development of Cloud Function Code
-
-- `npm install` (couldn't use `pnpm` because Google Cloud Build failed trying to install pnpm at the time of writing)
-- `npm start` to start a local cloud function
-- `npm test` to call the local cloud function with a mocked payload, this will send a real Discord message into channel belonging to the webhook in `.env`:
-
-  ```sh
-  curl -H "Content-Type: application/json" -d @src/proposal-created.fixture.json localhost:8080
-  ```
-
-## Testing the Deployed Cloud Function
-
-You can test the deployed cloud function manually by using the `proposal-created.fixture.json` which contains a similar payload to what a QuickAlert would send to the cloud function:
-
-```sh
-./test-deployed-function.sh
-# or `npm run test-in-prod` if you prefer npm to call this script
-```
-
-## Infra Setup (when project is deployed already)
+## Local Infra Setup (when project is deployed already)
 
 1. Set your local `gcloud` project to the watchdog project:
 
@@ -127,7 +99,7 @@ You can test the deployed cloud function manually by using the `proposal-created
 
    ```sh
    touch ./infra/terraform.tfvars
-   # This file should be `.gitignore`d to avoid accidentally leaking sensitive data
+   # This file is `.gitignore`d to avoid accidentally leaking sensitive data
    ```
 
 1. Add the following values to your `terraform.tfvars`, you can look up all values in the Google Cloud console (or ask another dev to share his local `terraform.tfvars` with you)
@@ -146,27 +118,83 @@ You can test the deployed cloud function manually by using the `proposal-created
    group_billing_admins = "<our-billing-admins-group>"
    ```
 
-1. Add the Discord Webhook URL from Google Cloud Secret Manager into your local `terraform.tfvars`:
+1. Add the Discord Webhook URL from Google Cloud Secret Manager to your local `terraform.tfvars`:
 
    ```sh
-   # You will need the "Secret Manager Secret Accessor" IAM role for this command to succeed
+   # You need the "Secret Manager Secret Accessor" IAM role for this command to succeed
    echo "discord_webhook_url = \"$(gcloud secrets versions access latest --secret discord-webhook-url)\"" >> terraform.tfvars
    ```
 
+1. Add the Telegram Bot Token and Chat ID to your local `terraform.tfvars`
+
+   ```sh
+   # Get the chat ID from cloud function's terraform state
+   echo "\ntelegram_chat_id = \"$(terraform state show "google_cloudfunctions2_function.watchdog_notifications" | grep TELEGRAM_CHAT_ID | awk -F '= ' '{print $2}' | tr -d '"')\"" >> terraform.tfvars
+
+   # Get the bot token from secret manager (you need the "Secret Manager Secret Accessor" IAM role for this command to succeed)
+   echo "telegram_bot_token = \"$(gcloud secrets versions access latest --secret telegram_bot_token)\"" >> terraform.tfvars
+   ```
+
 1. [Get our QuickNode API key from the QuickNode dashboard](https://dashboard.quicknode.com/api-keys) and add it to your local `terraform.tfvars`:
 
    ```sh
    # ./infra/terraform.tfvars
-   discord_webhook_url = "<discord-webhook-url>"
    quicknode_api_key   = "<your-quicknode-api-key>"
    ```
 
    This is necessary for Terraform to be able to create & destroy QuickAlerts as part of `terraform apply`
 
-## First Time Infra Deployment via Terraform
-
 1. Auto-generate a local `.env` file by running `npm run generate:env`
 
+## Running and testing the Cloud Function locally
+
+- Make sure you generated a local `.env` file via `npm run generate:env` earlier
+- `npm install` (couldn't use `pnpm` because Google Cloud Build failed trying to install pnpm at the time of writing)
+- `npm start` to start a local cloud function
+- `npm test` to call the local cloud function with a mocked payload, this will send a real Discord message into the channel belonging to the configured Discord Webhook:
+
+  ```sh
+  curl -H "Content-Type: application/json" -d @src/proposal-created.fixture.json localhost:8080
+  ```
+
+## Testing the Deployed Cloud Function
+
+You can test the deployed cloud function manually by using the `proposal-created.fixture.json` which contains a similar payload to what a QuickAlert would send to the cloud function:
+
+```sh
+./test-deployed-function.sh
+# or `npm run test:prod` if you prefer npm to call this script
+```
+
+## Updating the Cloud Function
+
+You have two options, using `terraform` or the `gcloud` cli. Both are perfectly fine to use.
+
+1. Via `terraform` by running `npm run deploy:via:tf`
+   - How? The npm task will:
+     - Call `terraform apply` which re-deploys the function with the latest code from your local machine
+   - Pros
+     - Keeps the terraform state clean
+     - Same command for all changes, regardless of infra or cloud function code
+   - Cons
+     - Less familiar way of deploying cloud functions (if you're used to `gcloud functions deploy`)
+     - Less log output
+     - Slightly slower because `terraform apply` will always fetch the current state from the cloud storage bucket before deploying
+2. Via `gcloud` by running `npm run deploy:via:gcloud`
+   - How? The npm task will:
+     - Look up the service account used by the cloud function
+     - Call `gcloud functions deploy` with the correct parameters
+   - Pros
+     - Familiar way of deploying cloud functions
+     - More log output making deployment failures slightly faster to debug
+     - Slightly faster because we're skipping the terraform state lookup
+   - Cons
+     - Will lead to inconsistent terraform state (because terraform is tracking the function source code and its version)
+     - Different commands to remember when updating infra components vs cloud function source code
+     - Will only work for updating a pre-existing cloud function's code, will fail for a first-time deploy
+
+## Infra Deployment via Terraform
+
 ### Google Cloud Permission Requirements
 
 In order to create this project from scratch using the [terraform-google-bootstrap](https://github.com/terraform-google-modules/terraform-google-bootstrap) module, you will need the following permissions in our Google Cloud Organization:
@@ -177,28 +205,47 @@ In order to create this project from scratch using the [terraform-google-bootstr
 
 ### Deployment from scratch
 
+1. Outcomment the `backend` section in `main.tf` (because this bucket doesn't exist yet, it will be created by the first `terraform apply` run)
+
+   ```hcl
+   # backend "gcs" {
+   #   bucket = "governance-watchdog-terraform-state-<random-suffix>"
+   # }
+   ```
+
+1. Run `terraform init` to install the required providers and init a temporary local backend in a `terraform.tfstate` file
+
 <!-- markdown-link-check-disable -->
 
 1. [Create a Discord Webhook URL](https://support.discord.com/hc/en-us/articles/228383668-Intro-to-Webhooks) for the channel you want to receive notifications in <!-- markdown-link-check-enable -->
 
-2. Add the Discord Webhook URL to your local `terraform.tfvars`:
+1. Add the Discord Webhook URL to your local `terraform.tfvars`:
 
    ```sh
    # This will be stored in Google Secret Manager upon deployment via Terraform
    echo "discord_webhook_url = \"<discord-webhook-url>"" >> terraform.tfvars
    ```
 
-3. Outcomment the `backend` section in `main.tf` (because this bucket doesn't exist yet, it will be created by the first `terraform apply` run)
+1. Create a Telegram group and invite a new bot into it
 
-   ```hcl
-   # backend "gcs" {
-   #   bucket = "governance-watchdog-terraform-state-<random-suffix>"
-   # }
-   ```
+   - Open a new telegram chat with @BotFather
+   - Use the `/newbot` command to create a new bot
+   - Copy the API key printed out at the end of the prompt and store it in your `terraform.tfvars`
 
-4. Run `terraform init` to install the required providers and init a temporary local backend in a `terraform.tfstate` file
+     ```hcl
+     telegram_bot_token = "<bot-api-key>"
+     ```
 
-5. **Deploy the entire project via `terraform apply`**
+   - Get the Chat ID by inviting @MissRose_bot to the group and then using the `/id` command
+   - Add the Chat ID to your `terraform.tfvars`
+
+     ```hcl
+     telegram_chat_id = "<group-chat-id>"
+     ```
+
+   - Remove @MissRose_bot after you got the Chat ID
+
+1. **Deploy the entire project via `terraform apply`**
 
    - You will see an overview of all resources to be created. Review them if you like and then type "Yes" to confirm.
    - This command can take up to 10 minutes because it does a lot of work creating and configuring all defined Google Cloud Resources
@@ -207,21 +254,17 @@ In order to create this project from scratch using the [terraform-google-bootstr
 
      **Often a simple retry of `terraform apply` helps**. Sometimes a dependency of a resource has simply not finished creating when terraform already tried to deploy the next one, so waiting a few minutes for things to settle can help.
 
-6. Set your local `gcloud` project to our freshly created one:
+1. Set your local `gcloud` project ID to our freshly created one:
 
    ```sh
-   # If that `awk` magic fails, just look up the project ID manually via `gcloud projects list`
-   project_id=$(terraform state show "module.bootstrap.module.seed_project.module.project-factory.google_project.main" | grep 'project_id' | awk -F '"' '{print $2}')
-
-   gcloud config set project $project_id
-   gcloud auth application-default set-quota-project $project_id
+   ./set-project-id.sh
    ```
 
-7. Check that everything worked as expected
+1. Check that everything worked as expected
 
    ```sh
    # 1. Call the deployed function via:
-   npm run test-in-prod # or call the script directly via ./test-deployed-function.sh
+   npm run test:prod # or call the script directly via ./test-deployed-function.sh
 
    # 2. Monitor the configured Discord channel for a message to appear
    open https://discord.com/channels/966739027782955068/1262714272476037212
@@ -272,35 +315,6 @@ For all team members to be able to manage the Google Cloud infrastructure, you n
    rm terraform.tfstate.backup
    ```
 
-## Updating the Cloud Function
-
-You have two options, using `terraform` or the `gcloud` cli. Both are perfectly fine to use.
-
-1. Via `terraform` by running `npm run deploy:via:tf`
-   - How? The npm task will:
-     - Compile TS to JS
-     - Zip the `./dist` folder into `function-source.zip`
-     - And then call `terraform apply` which re-deploys the function with the new source code from the zip file
-   - Pros
-     - Keeps the terraform state clean
-     - Same command for all changes, regardless of infra or cloud function code
-   - Cons
-     - Less familiar way of deploying cloud functions (if you're used to `gcloud functions deploy`)
-     - Less log output
-     - Slightly slower because `terraform apply` will always fetch the current state from the cloud storage bucket before deploying
-2. Via `gcloud` by running `npm run deploy:via:gcloud`
-   - How? The npm task will:
-     - Generate a temporary `.env.yaml` (because for some reason gcloud does not support normal `.env` files)
-     - Look up the service account used by the cloud function
-     - Call `gcloud functions deploy` with the correct parameters
-   - Pros
-     - Familiar way of deploying cloud functions
-     - More log output making deployment failures slightly faster to debug
-     - Slightly faster because we're skipping the terraform state lookup
-   - Cons
-     - Will lead to inconsistent terraform state (because terraform is tracking the function source code and its version)
-     - Different commands to remember when updating infra components vs cloud function source code
-
 ## Teardown
 
 Before destroying the project, you'll need to migrate the terraform state from the cloud bucket backend onto your local machine.