Update documentation

README.md

@@ -3,8 +3,8 @@
 **[Documentation](https://kencx.github.io/homelab)**
 
 This repository contains infrastructure-as-code for the automated deployment and
-configuration and management of a Hashicorp (Nomad + Consul + Vault) cluster.
-The cluster is hosted on Proxmox as a personal, private homelab.
+configuration, and management of a Hashicorp (Nomad + Consul + Vault) cluster on
+Proxmox.
 
 ## Disclaimer
 
@@ -17,13 +17,12 @@ actions that are irreversible!
 
 ## Overview
 
-This project aims to provision a full Hashicorp cluster in a semi-automated
+This project aims to provision a full Hashicorp cluster in a **semi-automated**
 manner. It utilizes Packer, Ansible and Terraform:
 
-- Packer creates base Proxmox VM templates from cloud images and ISOs
-- Terraform provisions cluster nodes by cloning existing VM templates
-- Ansible installs and configures Vault, Consul, Nomad on cluster
-  nodes
+1. Packer creates base Proxmox VM templates from cloud images and ISOs
+2. Terraform provisions cluster nodes by cloning existing VM templates
+3. Ansible installs and configures Vault, Consul, Nomad on cluster nodes
 
 It comprises minimally of one server and one client node with no high
 availability (HA). The nodes run Vault, Consul and Nomad as a cluster.
@@ -41,15 +40,47 @@ physical machines.
 - [x] Consul service discovery
 - [x] Secure node communication via mTLS
 - [x] Personal Certificate Authority hosted on Vault
+- [x] Secrets management, retrieval and rotation with Vault
 - [x] Automated certificate management with Vault and consul-template
 - [x] Let's Encrypt certificates on Traefik reverse proxy
-- [x] Scheduled, automated backups with Restic and Autorestic
 
 ## Getting Started
 
 See the [documentation](https://kencx.github.io/homelab/getting_started) for more
 information on the concrete steps to configure and provision the cluster.
 
+## Folder Structure
+
+```bash
+.
+├── ansible/
+│   ├── roles
+│   ├── playbooks
+│   ├── inventory    # inventory files
+│   └── goss         # goss config
+├── bin              # custom scripts
+├── packer/
+│   ├── base         # VM template from ISO
+│   └── base-clone   # VM template from existing template
+└── terraform/
+    ├── cluster      # config for cluster
+    ├── dev          # config where I test changes
+    ├── minio        # config for Minio buckets
+    ├── modules      # tf modules
+    ├── nomad        # nomad jobs
+    ├── postgres     # config for Postgres DB users
+    ├── proxmox      # config for Proxmox accounts
+    └── vault        # config for Vault
+```
+
+## Limitations
+
+- Manual Vault unseal on reboot
+- Inter-job dependencies are [not supported](https://github.com/hashicorp/nomad/issues/545) in Nomad
+- Vault agent is run as root
+
+See [issues]() for more information.
+
 ## Acknowledgements
 
 - [CGamesPlay/infra](https://github.com/CGamesPlay/infra)

docs/src/SUMMARY.md

@@ -29,10 +29,7 @@
             - [Unseal Vault](ansible/roles/unseal_vault.md)
             - [Vault](ansible/roles/vault.md)
 
-    <!-- - [Development Server]() -->
-
 <!-- - [CICD]() -->
-<!-- - [Monitoring]() -->
 <!-- - [Vault]() -->
 <!-- - [Consul]() -->
 <!-- - [Nomad]() -->
@@ -45,8 +42,6 @@
     - [Diun](apps/diun.md)
     - [Registry](apps/registry.md)
 
-- [Backups](backups.md)
-
 # References
 
 - [Known Issues](references/issues.md)

docs/src/ansible/roles/unseal_vault.md

@@ -1,3 +1,5 @@
+# Unseal Vault
+
 >**Work in Progress**: This role is unfinished and untested.
 
 This role unseals an initialized but sealed Vault server. The unseal key shares

docs/src/getting_started.md

@@ -1,66 +1,93 @@
 # Getting Started
 
-This documents provides an overview for provisioning and installing the cluster.
+Our goal is to provision a Nomad, Consul and Vault cluster with one server node
+and one client node. The basic provisioning flow is as follows:
 
->**Note**: It is assumed that all nodes are running on Proxmox as Debian 11 VMs.
->Please fork the project and make the necessary configuration changes should you
->choose to run the cluster with LXCs or an alternative distro.
+1. Packer creates base Proxmox VM templates from cloud images and ISOs
+2. Terraform provisions cluster nodes by cloning existing VM templates
+3. Ansible installs and configures Vault, Consul, Nomad on cluster nodes
 
-## Prerequisites
+### Assumptions
 
-See [Prerequisites](prerequisites.md) for the full requirements.
+The following assumptions are made in this guide:
 
->**Note**: Use the `bin/generate-vars` script to quickly generate variable files
->in `packer` and `terraform` subdirectories.
+- All [prerequisites](./prerequisites.md) are fulfilled
+- The cluster is provisioned on a Proxmox server
+- All nodes are running Debian 11 virtual machines (not LXCs)
+
+Please make the necessary changes if there are any deviations from the above.
 
 ## Creating a VM template
 
-There are two methods to create a VM template:
+The Proxmox builder plugin is used to create a new VM template. It supports two
+different builders:
 
-- From an [ISO file](./images/packer.md#proxmox-iso) (WIP)
-- From an [existing cloud image](./images/packer.md#proxmox-clone) (recommended)
+- `proxmox-clone` - From an [existing VM template](./images/packer.md#proxmox-clone) (recommended)
+- `proxmox-iso` - From an [ISO file](./images/packer.md#proxmox-iso) (incomplete)
 
-We will be building the template from an existing cloud image.
+We will be using the first builder. If you have an existing template to
+provision, you may [skip to the next section](#provisioning-with-terraform).
+Otherwise, assuming that we are lacking an existing, clean VM template, we will
+import a cloud image and turn it into a new template.
 
->**Note**: See [Cloud Image](images/cloud_image.md) for how to import an
->existing cloud image into Proxmox.
+>**Note**: It is important that the existing template [must
+>have](https://pve.proxmox.com/wiki/Cloud-Init_Support#_preparing_cloud_init_templates):
+>
+>    - An attached cloud-init drive for the builder to add the SSH communicator
+>      configuration
+>    - cloud-init installed
+>    - qemu-guest-agent installed
 
-1. Navigate to `packer/base-clone`.
-2. Populate the necessary variables in `auto.pkrvars.hcl`:
+1. (Optional) Run the `bin/import-cloud-image` [script](./images/cloud_image.html#script) to import a new cloud image:
+
+```bash
+$ import-cloud-image [URL]
+```
+
+2. Navigate to `packer/base-clone`
+
+>**Tip**: Use the `bin/generate-vars` script to quickly generate variable files
+>in `packer` and `terraform` subdirectories.
+
+3. Populate the necessary variables in `auto.pkrvars.hcl`:
 
 ```hcl
-proxmox_url      = "https://${PVE_IP}:8006/api2/json"
-proxmox_username = "user@pam"
-proxmox_password = "password"
+proxmox_url      = "https://<PVE_IP>:8006/api2/json"
+proxmox_username = "<user>@pam"
+proxmox_password = "<password>"
 
-clone_vm = "cloud-image-name"
-vm_name  = "base-template"
+clone_vm = "<cloud-image-name>"
+vm_name  = "<new-template-name>"
 vm_id    = 5000
 
 ssh_username = "debian"
 ssh_public_key_path = "/path/to/public/key"
 ssh_private_key_path = "/path/to/private/key"
 ```
 
-3. Build the image:
+4. Build the image:
 
 ```bash
 $ packer validate -var-file="auto.pkrvars.hcl" .
 $ packer build -var-file="auto.pkrvars.hcl" .
 ```
 
-Packer will create a new base image that has common configuration and
-software installed (eg. Docker). For more information, refer to
-[Packer](./images/packer.md#proxmox-clone).
+Packer will create a new base image and use the Ansible post-provisioner to
+install and configure software (eg. Docker, Nomad, Consul and Vault). For more
+details, see [Packer](./images/packer.md#proxmox-clone).
 
 ## Provisioning with Terraform
 
-1. Navigate to `terraform/cluster`.
+We are using the
+[bpg/proxmox](https://registry.terraform.io/providers/bpg/proxmox/latest/docs)
+provider to provision virtual machines from our Packer templates.
+
+1. Navigate to `terraform/cluster`
 2. Populate the necessary variables in `terraform.tfvars`:
 
 ```hcl
-proxmox_ip        = "https://${PVE_IP}:8006/api2/json"
-proxmox_api_token = "${API_TOKEN}"
+proxmox_ip        = "https://<PVE_IP>:8006/api2/json"
+proxmox_api_token = "<API_TOKEN>"
 
 template_id = 5000
 ip_gateway  = "10.10.10.1"
@@ -94,11 +121,8 @@ ssh_private_key_file = "/path/to/ssh/private/key"
 ssh_public_key_file  = "/path/to/ssh/public/key"
 ```
 
->**Note**: To create a Proxmox API token, see [Access
->Management](./terraform/proxmox.md#access-management).
-
->**Note**: Any template to be cloned by Terraform must have `cloud-init` and
->`qemu-guest-agent` installed.
+<!-- >**Note**: To create a Proxmox API token, see [Access -->
+<!-- >Management](./terraform/proxmox.md#access-management). -->
 
 3. Provision the cluster:
 
@@ -118,30 +142,57 @@ Client node: VMID 111 at 10.10.10.111
 An Ansible inventory file `tf_ansible_inventory` should be generated in the same
 directory with the given VM IPs in the `server` and `client` groups.
 
-For more information, refer to the [Terraform configuration for
+For more details, refer to the [Terraform configuration for
 Proxmox](terraform/proxmox.md).
 
 ## Configuration with Ansible
 
-1. Navigate to `ansible`.
+At this stage, there should be one server node and one client node running on
+Proxmox that is reachable by SSH. These nodes should have Nomad, Consul and
+Vault installed. We will proceed to use Ansible (and Terraform) to configure
+Vault, Consul and Nomad (in that order) into a working cluster.
+
+1. Navigate to `ansible`
 2. Ensure that the Terraform-generated Ansible inventory file is being read:
 
 ```bash
 $ ansible-inventory --graph
 ```
 
-3. Populate and check the `group_vars` file in
+3. Populate and check the `group_vars` files in
    `inventory/group_vars/{prod,server,client}.yml`
 
 ```bash
 $ ansible-inventory --graph --vars
 ```
 
+>**Note**: The `nfs_share_mounts` variable in `inventory/group_vars/client.yml`
+>should be modified or removed if not required
+
 4. Run the playbook:
 
 ```bash
 $ ansible-playbook main.yml
 ```
 
-This will configure and start Vault, Consul and Nomad in both nodes with mTLS
-and gossip encryption.
+The playbook will perform the following idempotently:
+
+1. Create a root and intermediate CA for Vault
+2. Configure Vault to use new CA
+3. Initialize Vault roles, authentication and PKI with Terraform with
+   [configuration](./terraform/vault.md) in `terraform/vault`
+4. Configure Vault-agent and consul-template in server node
+5. Configure Consul and Nomad in server node. These roles depend on Vault being
+   successfully configured and started as they require Vault to generate a
+   gossip key and TLS certificates
+6. Repeat 4-5 for client node
+
+### Note on Data Loss
+
+When re-running the playbook on the same server, Vault will not be
+re-initialized. However, if the playbook is run on a separate server (eg. for
+testing on a dev cluster), the Vault role will permanently delete any
+existing state in the `terraform/vault` subdirectory if a different
+`vault_terraform_workspace` is not provided. This WILL result in permanent data
+loss and care should be taken when running the role (and playbook) on multiple
+clusters or servers.

docs/src/images/packer.md

@@ -10,27 +10,27 @@ Proxmox.
 
 ## Proxmox-clone
 
-The `proxmox-clone` builder creates a new VM template from an existing one. This
-is best used with an [uploaded cloud image](./cloud_image.md) which has been
-converted into a VM template.
+The `proxmox-clone` builder creates a new VM template from an existing one. If
+you do not have an existing VM template or want to create a new template, you
+can [upload a new cloud image](./cloud_image.md) and convert it into a new VM template.
 
-This existing template [must
+Note that this existing template [must
 have](https://pve.proxmox.com/wiki/Cloud-Init_Support#_preparing_cloud_init_templates):
 
 - An attached cloud-init drive for the builder to add the SSH communicator
-  configuration.
-- `cloud-init` installed.
+  configuration
+- `cloud-init` installed
 
-The builder will do the following:
+After running the builder, it will do the following:
 
-1. Clone existing template.
-2. Add a SSH communicator configuration via cloud-init.
+1. Clone existing template by given name
+2. Add a SSH communicator configuration via cloud-init
 3. Connect via SSH and run the shell provisioner scripts to prepare the VM for
-   Ansible.
-4. Install and start `qemu-guest-agent`.
-5. Run the Ansible provisioner with the `ansible/common.yml` playbook.
+   Ansible
+4. Install and start `qemu-guest-agent`
+5. Run the Ansible provisioner with the `ansible/common.yml` playbook
 6. Stop and convert the VM into a template with a new (and empty) cloud-init
-   drive.
+   drive
 
 ### Variables