feat: expand the troubleshooting section of the canary-checker docs

canary-checker/docs/concepts/troubleshooting.md

@@ -1,12 +1,8 @@
 # Troubleshooting
 
-## Run the check from the CLI
+## Increase the log verbosity of a Canary using `trace` annotation
 
-   The easiest way of troubleshooting is to run the `canary-checker run` command with a copy of the canary CRD locally, this enables rapid feedback loops.
-
-## Enable trace and debug
-
-  To increase the amount of logs for a particular trace add a `debug: true` or `trace: true` annotation:
+  To add detailed debug information related to a specific Canary to the log output, add a `debug: true` or `trace: true` annotation to the Canary resource:
 
   ```yaml trace.yaml
   apiVersion: canaries.flanksource.com/v1
@@ -34,17 +30,48 @@
 | `trace` | - HTTP Request and Response Header <br/>- HTTP Response Body <br />- Custom Metrics |
 
 
-## Run checks immediately using `next-runtime`
+## Run a Canary immediately using `next-runtime` annotation
 
 To run a canary outside of its normal schedule add the annotation:
 
 ```bash
  kubectl annotate canary <canary> next-runtime=$(date -Iseconds)
  ```
 
-## Temporarily pause a canary using `suspend`
+## Temporarily pause a Canary using `suspend` annotation
 
 ```bash
  kubectl annotate canary <canary> supend=true
  ```
 
+## Run a Canary from the CLI
+
+   The easiest method of troubleshooting is to run the `canary-checker run` command with a copy of the canary resource locally.  This increases velocity of troubleshooting iteration, and also prevents the possibility of sensitive information in verbose output being leaked to logging systems by the canary pod.
+
+### Local permissions
+
+   The local canary-checker execution may need to be provided with the permissions required to run the check.  
+   Depending on the check this might include:
+   - The ability to read secrets or other objects in the cluster
+   - Access to cloud APIs, such as AWS, GCP or Azure API interfaces
+   - Locally installed binaries invoked by `exec` scripts
+   The following local settings will affect the permissions available for use by the canary-checker executable:
+   - The active kubectl context as specified in `KUBECONFIG`
+   - The AWS profile specified in `AWS_DEFAULT_PROFILE` or `AWS_PROFILE`, or the AWS credentials specified by `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
+   - The GCP credentials configured through `gcloud auth application-default login`
+   - The `$PATH` configuration
+
+### Local Canary execution
+
+   Install the CLI as per the [Installation instructions](../../cli/)
+   Generate a local resource file with the trace annotation enabled:
+
+```bash
+    kubectl get canary test-canary -n test-namespace -o yaml | yq '.metadata.annotations.trace = "true"' > test-canary.yaml
+```
+
+    Or generate a resource file by hand, then run the canary-checker CLI:
+
+```bash
+    canary-checker run -v test-canary.yaml
+```