diff --git a/docs/cli-quickstart.md b/docs/cli-quickstart.md index 6e6a444..e2a4dad 100644 --- a/docs/cli-quickstart.md +++ b/docs/cli-quickstart.md @@ -314,6 +314,6 @@ Or Continue learning how to use Quix CLI to develop you pipeline locally. - [Quix CLI local development :octicons-arrow-right-24:](./cli-local-debug.md) + [Quix CLI local development :octicons-arrow-right-24:](./local-development/local-debug.md) diff --git a/docs/images/yaml-variables.png b/docs/images/yaml-variables.png new file mode 100644 index 0000000..993438d Binary files /dev/null and b/docs/images/yaml-variables.png differ diff --git a/docs/cli-local-debug.md b/docs/local-development/local-debug.md similarity index 100% rename from docs/cli-local-debug.md rename to docs/local-development/local-debug.md diff --git a/docs/cli-local-secrets.md b/docs/local-development/local-secrets.md similarity index 99% rename from docs/cli-local-secrets.md rename to docs/local-development/local-secrets.md index ae0b9ac..f833928 100644 --- a/docs/cli-local-secrets.md +++ b/docs/local-development/local-secrets.md @@ -2,7 +2,7 @@ In any development environment, securing sensitive information like API tokens is essential. Quix CLI provides robust tools for managing these secrets within your data pipelines. This tutorial will walk you through the steps to securely add, update, and manage secrets in your Quix pipeline, ensuring that your sensitive data remains protected throughout the development process. -Before diving into the details, make sure you've set up your local environment correctly. If you haven't already, complete the [Quickstart](./cli-quickstart.md) to get Quix CLI up and running. For detailed instructions on setting up your local development environment, refer to our [Local Development](./cli-local-debug.md) tutorial. +Before diving into the details, make sure you've set up your local environment correctly. If you haven't already, complete the [Quickstart](./cli-quickstart.md) to get Quix CLI up and running. For detailed instructions on setting up your local development environment, refer to our [Local Development](./local-debug.md) tutorial. ## Step 1: Add the Secret to Your `.env` File diff --git a/docs/local-development/local-yaml-variables.md b/docs/local-development/local-yaml-variables.md new file mode 100644 index 0000000..c569c3d --- /dev/null +++ b/docs/local-development/local-yaml-variables.md @@ -0,0 +1,80 @@ + +# Managing YAML variables + +In the `quix.yaml` file, users can define variables to customize both deployment configurations and topics settings. These variables help in setting specific values for different environments (e.g., production, staging, development) without hardcoding values in the YAML file. + +### Cloud-Based Variable Management + +This feature is cloud-only, meaning that variable values for each environment (such as production, staging, or development) are set in the cloud. The cloud environment allows you to configure variables dynamically based on your deployment needs. + +You can load the variables from the cloud using the CLI command [quix use](../cli-reference/use.md). This command allows you to select an environment (e.g., production or development) and load the variables from the cloud for that environment. + +![Cloud Yaml Variables](../images/yaml-variables.png) + +### Local Development with `.quix.yaml.variables` + +For local development, instead of setting the values in the cloud, you can use a `.quix.yaml.variables` file to manage variables locally. This file allows you to simulate the cloud-based variable management in your local environment. + +If variables are defined in both the cloud and the `.quix.yaml.variables` file, the local file values will override those loaded from the cloud. + +The `.quix.yaml.variables` file is a simple text file with `key=value` format. + +Example of a `.quix.yaml.variables` file: + +```bash +CPU=200 +RAM=500 +REPLICAS=2 +PARTITIONS=2 +``` + +By using this local file, you can test different configurations locally without hardcoding them into the `quix.yaml` file itself. + +## Defining YAML Variables + +To define variables in `quix.yaml`, use a placeholder format like `{{VAR_NAME}}`. These variables allow users to customize the configuration for both deployments and topics based on different environments. + +Here’s an example: + +```yaml +resources: + cpu: {{CPU}} + memory: {{RAM}} + replicas: {{REPLICAS}} + +# This section describes the Topics of the data pipeline +topics: + - name: trades + persisted: false + configuration: + partitions: {{PARTITIONS}} +``` + +In this example, the `CPU`, `RAM`, `REPLICAS`, and `PARTITIONS` variables are set based on the specific environment. + +## Managing YAML Variables Locally + +For local development, the `.quix.yaml.variables` file is used to set the values for the defined variables in `quix.yaml`. Example: + +```bash +# .quix.yaml.variables file +CPU=200 +RAM=500 +REPLICAS=2 +PARTITIONS=2 +``` + +When running the application locally, these values are automatically picked up to simulate the cloud-based variable management. + +## Best Practices for Using YAML Variables + +- **Use Clear and Descriptive Names**: Always use descriptive names for your variables (e.g., `CPU`, `MEMORY`, `REPLICAS`, `PARTITIONS`) to avoid confusion. +- **Separate Cloud and Local Configurations**: Keep your cloud-based variable management separate from local configurations by using the `.quix.yaml.variables` file for local development. +- **Version Control**: Avoid hardcoding sensitive values like API keys or tokens directly in the YAML file. Use environment variables or secret management tools. +- **Validation**: Ensure that required variables are properly set before deploying. Failing to do so can lead to unexpected behavior in different environments. + +## Running the pipeline locally + +When you run the [quix pipeline up](../cli-reference/pipeline/up.md) command, the CLI will process both cloud and local variables and then run the pipeline with the correct configuration for the selected environment. + +By managing variables this way, you ensure that your deployments and topics are flexible and can adapt to various environments without needing to change the core YAML configuration. diff --git a/mkdocs.yml b/mkdocs.yml index bcd7b61..d565731 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -54,8 +54,10 @@ plugins: nav: - 'Overview': 'overview.md' - 'Quickstart': 'cli-quickstart.md' -- 'Local Development': 'cli-local-debug.md' -- 'Local Secrets': 'cli-local-secrets.md' +- 'Local Development': + - 'Running applications locally': 'local-development/local-debug.md' + - 'Managing secrets locally': 'local-development/local-secrets.md' + - 'Managing YAML variables': 'local-development/local-yaml-variables.md' - 'Commands Summary': 'cli-commands-summary.md' - 'How-To guides': - 'Using the CLI with GitHub Actions': 'How-To/cli-github-actions.md'