Talos Kubernetes on Proxmox

Terraform module for creating a Kubernetes Cluster

• on Proxmox Virtual Environment
• using Talos OS
• and bootraping it with fluxcd

Prerequisites

• Proxmox Virtual Environment 8.2.5
• Terraform v1.9.8
• Talos v1.8
• Kubernetes
• fluxcd 2.4.0

Before running the module, you need to have an up and running Proxmox cluster configured for Terraform

Usage

cat main.tf
module "talos_k8s_cluster" {
  source  = "vdupain/talos-k8s-cluster/proxmox"
  version = "1.0.0-rc8"

  cluster = {
    name     = "cluster-demo"
    gateway  = "192.168.10.1"
    cidr     = 24
    endpoint = "192.168.10.210"
  }

  vms = {
    "k8s-cp-0" = {
      host_node      = "pve1"
      machine_type   = "controlplane"
      ip             = "192.168.10.210"
      cpu            = 2
      ram_dedicated  = 4096
      os_disk_size   = 10
      data_disk_size = 10
      datastore_id   = "local-lvm"
    }
    "k8s-cp-1" = {
      host_node      = "pve1"
      machine_type   = "controlplane"
      ip             = "192.168.10.211"
      cpu            = 2
      ram_dedicated  = 4096
      os_disk_size   = 10
      data_disk_size = 10
      datastore_id   = "local-lvm"
    }
    "k8s-cp-2" = {
      host_node      = "pve1"
      machine_type   = "controlplane"
      ip             = "192.168.10.212"
      cpu            = 2
      ram_dedicated  = 4096
      os_disk_size   = 10
      data_disk_size = 10
      datastore_id   = "local-lvm"
    }
  }

  proxmox = {
    endpoint     = "https://pve.domain"
    insecure     = true
    username     = "user"
    password     = "password"
    api_token    = "user@pve!terraform=secret"
  }

  gitops = {
    repository   = "https://github.com/vdupain/gitops.git"
    token        = "github_pat"
    cluster_name = "my-cluster"
  }

}

terraform init
...
terraform apply
...
module.talos_k8s_cluster.module.fluxcd[0].flux_bootstrap_git.this: Still creating... [50s elapsed]
module.talos_k8s_cluster.module.fluxcd[0].flux_bootstrap_git.this: Still creating... [1m0s elapsed]
module.talos_k8s_cluster.module.fluxcd[0].flux_bootstrap_git.this: Creation complete after 1m0s [id=flux-system]

Apply complete! Resources: 13 added, 0 changed, 0 destroyed.

Using cluster

Configuration files are store in output folder

$ ls -l output
total 8
-rw------- 1 devbox devbox 2295 Nov  2 17:23 kube-config.yaml
-rw------- 1 devbox devbox 1653 Nov  2 17:23 talos-config.yaml

Kubernetes cluster

$ kubectl --kubeconfig output/kube-config.yaml get nodes
NAME   STATUS     ROLES           AGE   VERSION
cp-0   NotReady   control-plane   43s   v1.31.1
cp-1   NotReady   control-plane   43s   v1.31.1
cp-2   NotReady   control-plane   43s   v1.31.1

Talos OS cluster

$ export CONTROL_PLANE_IP=192.168.10.210
$ export WORKER_IP=192.168.10.211
$ export TALOSCONFIG="output/talos-config.yaml"
$ talosctl config endpoint $CONTROL_PLANE_IP
$ talosctl config node $WORKER_IP
$ talosctl health
discovered nodes: ["192.168.10.210" "192.168.10.211" "192.168.10.212"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for no diagnostics: ...
waiting for no diagnostics: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: OK
waiting for all control plane static pods to be running: ...
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: ...
waiting for all control plane components to be ready: OK
waiting for all k8s nodes to report ready: ...
waiting for all k8s nodes to report ready: OK
waiting for kube-proxy to report ready: ...
waiting for kube-proxy to report ready: SKIP
waiting for coredns to report ready: ...
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK

Flux bootstrap

$ flux --kubeconfig output/kube-config.yaml get kustomization -A
NAMESPACE  	NAME       	REVISION          	SUSPENDED	READY	MESSAGE
flux-system	flux-system	main@sha1:5902d505	False    	True 	Applied revision: main@sha1:5902d505

Requirements

Name	Version
flux	>=1.4.0
kubernetes	2.31.0

Providers

Name	Version
local	n/a

Modules

Name	Source	Version
gitops_k8s	./modules/gitops_k8s	n/a
init_k8s	./modules/init_k8s	n/a
talos_k8s	./modules/talos_k8s	n/a
vms_proxmox	./modules/vms_proxmox	n/a

Resources

Name	Type
local_file.kube_config	resource
local_file.talos_config	resource

Inputs

Name	Description	Type	Default	Required
certificate	Certificate for k8s sealed-secrets	object({ cert = string key = string })	null	no
cluster	Cluster configuration	object({ name = string talos_version = optional(string, "v1.8.2") network_dhcp = optional(bool, false) gateway = optional(string) cidr = optional(number) vlan_id = optional(number, null) endpoint = optional(string) })	n/a	yes
gitops	GitOps configuration	object({ repository = string token = string cluster_name = string })	null	no
pci	Mapping PCI configuration	map(object({ name = string id = string iommu_group = number node = string path = string subsystem_id = string }))	null	no
proxmox	Proxmox configuration	object({ endpoint = optional(string) insecure = optional(bool) username = optional(string) password = optional(string) api_token = optional(string) ssh_agent = optional(string, false) })	n/a	yes
vms	VMs configuration	map(object({ host_node = string machine_type = string datastore_id = optional(string, "local-lvm") ip = optional(string) cpu = number ram_dedicated = number os_disk_size = number data_disk_size = number install_disk = optional(string, "/dev/sda") gpu = optional(string) }))	n/a	yes

Outputs

Name	Description
cluster_name	Retrieves the name for a k8s Talos cluster
config_ipv4_addresses	Retrieves VM names with IPv4 address for a k8s Talos cluster
kube_config	Retrieves the kubeconfig for a k8s Talos cluster
qemu_ipv4_addresses	Retrieves VM names with IPv4 address for a k8s Talos cluster
talos_config	Retrieves the talosconfig for a k8s Talos cluster
vm_ipv4_address_vms	Retrieves IPv4 address for a k8s Talos cluster