Self-hosting RedwoodJS on Kubernetes

Hi all 👋

After seeing some posts about self-hosting on Heroku and Render I got inspired and decided to take a swing at writing about Self-hosting RedwoodJS on Kubernetes. If you are a serverfull person who likes to get your hands dirty with Docker, Kubernetes with GitHub Actions - this read might be just for you. Heads-up though; It's quite a lot of config. 🤓

Also; while this is a working implementation that currently supports a production application (maybe a future #show-tell), it leaves some decisions to make on your part. That being said, let me know if you want me to elaborate on some topics and I'm definitely down to make this implementation better.

Table of Contents

• Docker: Build RedwoodJS, migrate and seed
• Kubernetes: Tools and objects
• GitHub: GitHub Actions and GitHub Container Registry
• Result: The screenshots
• Conclusion: The after thoughts
• Resources: The relevant files

Docker

We will use Docker to containerize our Redwood application, and in this implementation we will have two images; one for api and one for web. These Dockerfiles are pretty straight-forward, with some comments to the instructions.

api/Dockerfile

###########################################################################################
# Runner: node
###########################################################################################

FROM node:14 as runner

# Node
ARG NODE_ENV
ARG RUNTIME_ENV

ENV NODE_ENV=$NODE_ENV
ENV RUNTIME_ENV=$RUNTIME_ENV

ARG DATABASE_URL
ENV DATABASE_URL=$DATABASE_URL

# Set workdir
WORKDIR /app

COPY api api
COPY .nvmrc .
COPY graphql.config.js .
COPY package.json .
COPY redwood.toml .
COPY yarn.lock .

# Install dependencies
RUN yarn install --frozen-lockfile

# Build
RUN yarn rw build api

# Migrate database
# This has been commented out in this example post
# RUN yarn rw prisma migrate deploy

# Seed database
# This has been commented out in this example post
# RUN yarn rw prisma db seed

# Clean up
RUN rm -rf ./api/src

# Set api as workdirectory
WORKDIR /app/api

# Expose RedwoodJS api port
EXPOSE 8911

# Entrypoint to @redwoodjs/api-server binary
ENTRYPOINT [ "yarn", "rw", "serve", "api", "--port", "8911", "--rootPath", "/api" ]

Before we move over to the web side of things, did you notice how we were using yarn rw serve api in the entrypoint along with a --rootPath argument? Without going into much depth in this post, head over to Add rootPath to api-server to read about the motivation behind this. For now, make sure that your redwoodjs.toml's [web].apiProxyPath directive is set to /api, i.e. like so;

[web]
  port = 8910
  apiProxyPath = "/api"

web/Dockerfile

###########################################################################################
# Builder: node
###########################################################################################

FROM node:14 as builder

# Node
ARG NODE_ENV
ARG RUNTIME_ENV

ENV NODE_ENV=$NODE_ENV
ENV RUNTIME_ENV=$RUNTIME_ENV

# Set workdir
WORKDIR /app

# COPY api .
COPY web web
COPY .nvmrc .
COPY graphql.config.js .
COPY package.json .
COPY redwood.toml .
COPY yarn.lock .

# Install dependencies
RUN yarn install --frozen-lockfile

# Build
RUN yarn rw build web

# Clean up
RUN rm -rf ./web/src

###########################################################################################
# Runner: Nginx
###########################################################################################

FROM nginx as runner

# Copy dist
COPY --from=builder /app/web/dist /usr/share/nginx/html

# Copy nginx configuration
COPY web/config/nginx/default.conf /etc/nginx/conf.d/default.conf

# List files
RUN ls -lA /usr/share/nginx/html

# Expose RedwoodJS web port
EXPOSE 8910

As we are running nginx as our web server, lets also bring in a nginx config. It's nothing fancy and mostly adding some caching of static assets. We add header X-Awesomeness because we can, not because we need to.

web/config/nginx/default.conf

server {
    listen 8910 default_server;
    root /usr/share/nginx/html;

    # Add global header
    add_header X-Awesomeness 9000;

    # 1 hour cache for css and js
    location ~* \.(?:css|js)$ {
        expires 1h;
        add_header Pragma public;
        add_header Cache-Control "public";
        access_log off;
    }

    # 7 days cache for image assets
    location ~* \.(?:ico|gif|jpe?g|png)$ {
        expires 7d;
        add_header Pragma public;
        add_header Cache-Control "public";
        access_log off;
    }

    location / {
        try_files $uri $uri/ /index.html;
    }
}

Kubernetes

Now for the fun part (at least for some): Kubernetes.

I suggest using Kustomize to build and generate your Kubernetes manifest file. However, in the spirit of keeping complexity down, I will just document some of the generated Kubernetes objects. Feel free to DM me for the relevant Kustomize files I use. Furthermore, I have removed the Resource Limiting for brevity and using Kubernetes Nginx Controller for the Ingress.

Secrets

As our Docker images contains sensitive information and likely will be hosted in a private Container Registry, we need to create a Docker Registry config secret that Kubernetes can use to pull down the images. To export the secret and store it in a persistent file, run kubectl get secret <my-container-registry-secret> -o yaml. This should result in a secret like below.

apiVersion: v1
data:
  .dockerconfigjson: |
    [obfuscated]
kind: Secret
metadata:
  name: my-container-registry-secret
type: kubernetes.io/dockerconfigjson

Deployments

In these examples we are pulling the images from GitHub Container Registry (the ghcr.io part), defining the usual suspects (ports, match labels, replicas etc) and we are passing in relevant environment variables, such as DATABASE_URL to the api.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 2
  selector:
    matchLabels:
      platform: api
  template:
    metadata:
      labels:
        platform: api
    spec:
      containers:
        - env:
            - name: NODE_ENV
              value: development
            - name: RUNTIME_ENV
              value: dev
            - name: DATABASE_URL
              value: postgres://username:password@dbserver:5432/dbname
          image: ghcr.io/<your-org>/redwoodjs-api-main:latest
          name: api
          ports:
            - containerPort: 8911
      imagePullSecrets:
        - name: my-container-registry-secret
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      platform: web
  template:
    metadata:
      labels:
        platform: web
    spec:
      containers:
        - env:
            - name: NODE_ENV
              value: production
            - name: RUNTIME_ENV
              value: production
          image: ghcr.io/<your-org>/redwoodjs-web-main:22784ba
          name: web
          ports:
            - containerPort: 8910
      imagePullSecrets:
        - name: my-container-registry-secret

Services

We use NodePort to expose our pods as a service.

apiVersion: v1
kind: Service
metadata:
  labels:
    platform: api
  name: api
spec:
  ports:
    - port: 8911
  selector:
    platform: api
  type: NodePort
---
apiVersion: v1
kind: Service
metadata:
  labels:
    platform: web
  name: web
spec:
  ports:
    - port: 8910
  selector:
    platform: web
  type: NodePort

Ingress

This ingress will route / to the web pods and /api to the api pods, and make sure we issue an SSL/TLS certificate using cert-manager.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: nginx
  name: app
spec:
  rules:
    - host: jeliasson-redwoodjs-on-kubernetes.20.61.155.112.nip.io
      http:
        paths:
          - backend:
              serviceName: web
              servicePort: 8910
            path: /
          - backend:
              serviceName: api
              servicePort: 8911
            path: /api
  tls:
    - hosts:
        - jeliasson-redwoodjs-on-kubernetes.20.61.155.112.nip.io
      secretName: domain-tld-tls

GitHub

So we got all this fancy Docker and Kubernetes stuff defined. Cool. How do we automate and deploy all this? Personally I'd give GitHub my money any day. Ironically, and a large thanks to Microsoft acquiring GitHub and the compute power that came with that, all the stuff we need is free. Regardless, here is this posts meme to illustrate my point.

@mojombo - So... if you have any pull at GitHub, @jeliasson would not say no to an invite to Codespaces wink wink

Anyway;

We will use GitHub Container Registry to store our Docker images, and we use GitHub Actions to build- push- and deploy our application to Kubernetes. Before we jump into the GitHub Actions part of things, I have a confession to make; I also use ArgoCD since about a year back. If you are running Kubernetes and not running ArgoCD (or something equivalent) I'm not sure what you are doing 😅 Just kidding! It's great though. Point being, we will use a ArgoCD deployment in the examples below. I will briefly explain what you could do as a alternative.

GitHub Container Registry

GitHub offers Container Registry pretty much free of charge. You just need to opt-in for Preview and the images are published to your GitHub account/organisation, not to a specific repository.

GitHub Actions

Unfortunately, I have not had the time to look into having the same GitHub Action Workflow for handling different environments. So for the example below, we are using one workflow for the main environment/branch. I'll come back and update this post if I find a pretty way to do this.

What the workflow essentially does;

1. Set some environment variables
2. Checkout source code
3. Login to Container Registry (GitHub Container Registry)
4. Build and push our image (which also migrate and seed the database)
5. Checkout a deployment repository (see ArgoCD best practices)
6. Use Kustomize to update our placeholder image to the image just built
7. Generate a latest.yaml file (debugging and could be used for manual deploy)
8. Commit changes back to the deployment repository

What happens after (in ArgoCD) that is not covered in this post;

1. ArgoCD will sync with deployment repository and pickup the repository changes
2. Use Kustomize to build our Kubernetes manifests
3. Sync these manifests with a target Kubernetes cluster

Alternative: What you could do if you are not running ArgoCD or something similar;

1. After step 4 (build and push our image) you want to use e.g. a GitHub Action for Kubernetes to;
2. Login to the Kubernetes cluster using a kubeconfig
3. Deploy the latest.yaml file generated in step 7.

I have explained these steps further in comments below.

.github/workflows/redwoodjs-app-main.yaml

# Force redeploy

name: 'redwoodjs-app-main'

on:
  push:
    branches:
      - main

env:
  # Build
  NODE_ENV: 'development'
  RUNTIME_ENV: 'main'

  # Container Registry
  CONTAINER_REGISTRY_HOSTNAME: ghcr.io
  CONTAINER_REGISTRY_USERNAME: jeliasson
  CONTAINER_REGISTRY_PASSWORD: ${{ secrets.__GITHUB_ACCESS_TOKEN }}
  CONTAINER_REGISTRY_REPOSITORY: jeliasson
  CONTAINER_REGISTRY_IMAGE_PREFIX: redwoodjs-app

  # Repository
  GIT_DEPLOY_REPOSITORY_NAME: jeliasson/redwoodjs-on-kubernetes-deploy
  GIT_DEPLOY_REPOSITORY_BRANCH: main
  GIT_DEPLOY_REPOSITORY_AUTHOR_NAME: jeliasson
  GIT_DEPLOY_REPOSITORY_AUTHOR_EMAIL: jeliasson@users.noreply.github.com
  GIT_DEPLOY_REPOSITORY_AUTHOR_TOKEN: ${{ secrets.__GITHUB_ACCESS_TOKEN }}

jobs:
  #
  # Build
  #
  build:
    name: Build
    runs-on: ubuntu-20.04
    timeout-minutes: 10
    strategy:
      fail-fast: true
      matrix:
        platform: [api, web]
        include:
          - platform: api
            DATABASE_URL: __MAIN_DATABASE_URL
          - platform: web

    steps:
      # Checkout source code
      - name: Checkout source code
        uses: actions/checkout@v2

      # Setup Docker using buildx-action
      - name: Setup Docker
        uses: docker/setup-buildx-action@v1

      # Login to Docker Container Registry
      - name: Docker login
        uses: docker/login-action@v1
        with:
          registry: ${{ env.CONTAINER_REGISTRY_HOSTNAME }}
          username: ${{ env.CONTAINER_REGISTRY_USERNAME }}
          password: ${{ env.CONTAINER_REGISTRY_PASSWORD }}

      # Build Docker image with a :latest and :<git sha> tag
      - name: Docker build
        uses: docker/build-push-action@v2
        with:
          push: true
          context: .
          file: ./${{ matrix.platform }}/Dockerfile
          build-args: |
            NODE_ENV=${{ env.NODE_ENV }}
            RUNTIME_ENV=${{ env.RUNTIME_ENV }}
            DATABASE_URL=${{ secrets[matrix.DATABASE_URL] }}
          tags: |
            ${{ env.CONTAINER_REGISTRY_HOSTNAME }}/${{ env.CONTAINER_REGISTRY_REPOSITORY }}/${{ env.CONTAINER_REGISTRY_IMAGE_PREFIX }}-${{ matrix.platform }}-${{ env.RUNTIME_ENV }}:latest
            ${{ env.CONTAINER_REGISTRY_HOSTNAME }}/${{ env.CONTAINER_REGISTRY_REPOSITORY }}/${{ env.CONTAINER_REGISTRY_IMAGE_PREFIX }}-${{ matrix.platform }}-${{ env.RUNTIME_ENV }}:${{ github.sha }}

  #
  # Configure
  #
  configure:
    name: Configure
    needs: [build]
    runs-on: ubuntu-20.04
    timeout-minutes: 10
    strategy:
      max-parallel: 1
      fail-fast: true
      matrix:
        platform: [api, web]
        include:
          - platform: api
          - platform: web

    steps:
      # Checkout deployment repository
      - name: Checkout source code
        uses: actions/checkout@v2
        with:
          submodules: recursive
          repository: ${{ env.GIT_DEPLOY_REPOSITORY_NAME }}
          ref: ${{ env.GIT_DEPLOY_REPOSITORY_BRANCH }}
          token: ${{ env.GIT_DEPLOY_REPOSITORY_AUTHOR_TOKEN }}

      # Login to Docker Container Registry
      - name: Docker login
        uses: docker/login-action@v1
        with:
          registry: ${{ env.CONTAINER_REGISTRY_HOSTNAME }}
          username: ${{ env.CONTAINER_REGISTRY_USERNAME }}
          password: ${{ env.CONTAINER_REGISTRY_PASSWORD }}

      # Save these Docker credentials to the deployment repository
      # It will be used in Kustomize to generate a Kubernetes secret for the container registry
      - name: Save Container Registry credentials
        run: |
          cat $HOME/.docker/config.json |                                         \
            jq 'del(.credsStore) | del(.HttpHeaders)' >                           \
            kubernetes/overlays/${RUNTIME_ENV}/secrets/.dockerconfigjson

      # Setup Kustomize
      - name: Setup Kustomize
        uses: imranismail/setup-kustomize@v1

      # Use Kustomize to update the image placeholder in the Kustomize manifest file
      - name: Set Docker image
        run: |
          cd kubernetes/overlays/${RUNTIME_ENV}
          kustomize edit set image placeholder/${{ matrix.platform }}=${CONTAINER_REGISTRY_HOSTNAME}/${CONTAINER_REGISTRY_REPOSITORY}/${CONTAINER_REGISTRY_IMAGE_PREFIX}-${{ matrix.platform }}-${{ env.RUNTIME_ENV }}:${GITHUB_SHA}
          cat kustomization.yml

      # For debugging purposes, create a latest.yaml file
      - name: Generate Kubernetes latest manifest
        run: |
          cd kubernetes/overlays/${RUNTIME_ENV}
          kustomize build -o latest.yaml
          printf '%s\n%s\n' "# Generated with Kustomize at $(date)" "$(cat latest.yaml)" > latest.yaml

      # Commit our changes (e.g. the updated image tag generated by Kustomize)
      - name: Commit and push changes
        uses: EndBug/add-and-commit@v6
        with:
          author_name: ${{ env.GIT_DEPLOY_REPOSITORY_AUTHOR_NAME }}
          author_email: ${{ env.GIT_DEPLOY_REPOSITORY_AUTHOR_EMAIL }}
          branch: ${{ env.GIT_DEPLOY_REPOSITORY_BRANCH }}
          message: '[ci] Deployed ${{ github.repository }}@${{ github.sha }}: ${{ github.event.head_commit.message }}'
          pull_strategy: '--no-ff'
          push: true
          token: ${{ env.GIT_DEPLOY_REPOSITORY_AUTHOR_TOKEN }}

Result

GitHub Actions

Our workflow pipeline ran successfully and looks ✅

GitHub Container Registry

Here we can see our built api image in GitHub Container Registry. Want to give it a test go?

docker run -it --rm \
       -p 8910:8910 \
       ghcr.io/jeliasson/redwoodjs-app-web-main:3a567f545902d35472b6c8d334439cb7b8a47c01

ArgoCD

This is the web interface of Argo CD with an overview of the application we just deployed. In this cluster we have two ingress controllers, load balanced on the very edge of the Kubernetes cluster.

Are we live yet?

Yes, we are live. And for a simple api health.

Conclusion

Well, this became much longer than I thought it would be. If you are still reading this, thank you for bearing with me. Surely this is not an easy undertaking for someone not working with, or interested in, DevOps, and there are some missing pieces here. Wth did ArgoCD do? Deployment repository? Wat?

Anyway, we got ourself a RedwoodJS application running in Kubernetes and it costs me absolutely nothing (as I already have a cluster for other stuff) besides some sweat and tears along the way to make it play nice.

What can be done better

• Make the Github workflow less complex and handle multiple environments.
• We are migrating and seeding our database after build, before image push and deployment to Kubernetes. This is not ideal, as the new database schema will be up before the new api is deployed. A better way of doing this would be e.g. an Init Container that would have a small footprint with Prisma installed.

Going forward from here

I will be the first to acknowledge that this is quite a tedious setup, but at the same time confirm that this (in some variation) what is required to deploy a application to a scalable cluster. There are many moving parts. For that, I salute all the amazing SaaS platforms out there that makes our lives easier that way. For me, this is part work and part a fun challange.

• ArgoCD: Make a documentation about the setup and how it works
• CDN: Make sure the web is served by a cdn. The logic for deployment and purging old cache could be done with a GitHub Action or in an Init Container.
• ...not sure yet.

Resources

If you want the latest versions of the files described above, head over to the repository. The most important ones;

• .github/workflows/redwoodjs-app-main.yaml
• api/Dockerfile
• web/Dockerfile
• web/config/nginx/default.conf
• redwood.toml

A shoutout to @Tobbe, @ajcwebdev, @dac09 and @pi0neerpat for their input. 💌

I hope you found this read intresting. 👋