Mainframe

This repository consists of software that runs on a Raspberry Pi Cluster.

Goals

1. Learn Terraform
2. Learn Kubernetes
3. Run a few services in my home network (Home Assistant, AdGuard, maybe an app) that are currently running via other means.

Setup

There is a little bit of manual work that needs to be done for each Pi.

1. Generate a new ssh key for communicating with the pi nodes: ssh-keygen -t ed25519 -f ~/.ssh/mainframe -C "terraform@mainframe.local"
2. Download Balena Etcher
3. Download the DietPI .img
4. For each PI, grab the SD card, use Balena Etcher to flash the above image, and then replace dietpi.txt in the root of the SD Card with the one found in this repo. Lastly, adjust the hostname accordingly.

Automation

Terraform is used as much as possible for automation. It may not be perfectly suited for this task, but see goal #1.

Ensure terraform is installed by running:

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

The terraform automation is broken up into 3 stages (bootstrap, kubernetes, services), all explained in their respective README's.

Updating K3S

This process is not automated yet, but you can SSH into each of the nodes and perform upgrades manually.

On the server, you can run curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 --secrets-encryption --disable local-storage --disable servicelb, and on the agent, you can run curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.27.6+k3s1 K3S_URL=https://192.168.50.22:6443 K3S_TOKEN=PRIVATE sh -

Networking

By default, nothing is exposed to the outside world as the router firewall blocks everything.

Static IP

The cluster currently uses two hostnames: home.mnara.ca and ha.mnara.ca. The cluster is setup to run a task every 5 minutes to update the IP address incase it changes.

HTTP(s)

Run kubectl get ingress to get the IP of the ingress service and then port-forward 80 and 443 so that those services are available externally.

DNS

AdGuard Home will be exposed on 192.168.50.53 (hardcoded, can let it auto assign), and once its up, adjust the router settings to use that as the DNS.

Home Assistant

Home assistant doesn't work without modifications to its configuration file, it complains about a reverse proxy error.

In order to fix this:

1. Figure out which node longhorn is using for its storage volume by going to https://longhorn.internal.mnara.ca, then go to the volumes tab and click on the home assistant volume. Somewhere on that page it should mention which mainframe node its on.
2. SSH into the node
3. Run lsblk
4. cd into the directory returned by lsblk (cd /var/lib/kubelet/pods/9514a649-ebd0-4b6e-84ff-ad74a17f5a7f/...)
5. Add the following configuration to configuration.yaml:

   http:
     server_host: 0.0.0.0
     ip_ban_enabled: true
     login_attempts_threshold: 5
     use_x_forwarded_for: true
     trusted_proxies:
       - 10.42.0.0/16
       - 192.168.0.0/16
   

6. Restart the home-assistant pod by scaling up and then down: kubectl scale --replicas=0 deployment/home-assistant

Todo

1. Backup longhorn storage so we don't lose configuration for home-assistant and adguard.
2. Use wildcard cert in traefik and nginx by default. Currently doesn't seem to be a way to re-configure Traefik, so we would have to re-initialize k3s without traefik first.
3. Automate grabbing the kubernetes config from coordinator node.
4. Write a script that generates the DietPi configuration from some input parameters. Most of the information in the config is defaults, but DietPi doesn't work with a partial configuration.