A web system about reference letter handling in the context of DIT HUA Thesis "Use of devops methodologies and tools in development and production environment of web applications"
- Table Of Contents
- Setup & Run Projects Locally (Installation)
- Deploy fastapi and vuejs projects to a VM (Virtual Machine)
3.1. CI/CD tool configuration (Jenkins Server)
3.1.1. Step 1: Configure Shell
3.1.2. Step 2: Add webhooks to system repository
3.1.3. Step 3: Add the credentials needed
3.1.4. Create Job
3.1.4.1. Build stage
3.1.4.2. Docker Deployment
3.1.4.3. Kubernetes Deployment
3.2. Deployment with Docker and docker-compose using Ansible
3.3. Deployment with Kubernetes using a piece of Ansible
3.3.1. Using Multiple Namespaces
git clone --recurse-submodules https://github.com/panagiotis-bellias-it21871/reference-letters-system.git
cd reference-letters-systemFor Python FastAPI project see here and for JavaScript VueJS project see here
We are going to need 3 VMs. One for the jenkins server and one for each execution environment (docker and kubernetes)
Make sure service is running
sudo systemctl status jenkins
netstat -anlp | grep 8080 # needs package net-toolsGo to Dashboard / Manage Jenkins / Configure System / Shell / Shell Executable and type '/bin/bash'
Dublicate repositories for easier configuration.
- Add SSH keys & SSH Agent plugin with id 'ssh-docker-vm' to access docker-vm
- Add Secret Texts for every environmental variable we need to define in our projects during deployment, like below
# ID What is the value?
app-db-user The backend database user
app-db-pass The password for the above user
app-db-name The database we have for the backend functionality
auth-secret The authentication secret that backend is using
mail-username Mail account's username
mail-password Mail account's app password
mail-from Mail account address
mail-port Mail account's server port (e.g. 587 for gmail)
mail-server Mail account's provider (e.g. smtp.gmail.com for gmail-google)
mail-from-name Mail account's name
base-endpoint-prefix REST API's base prefix
rl-letters-endpoint REST API's base endpoint for reference letters requests
auth-endpoint-prefix REST API's prefix for authentication routes
auth-login-endpoint REST API's signin endpoint
docker-db-url The database url where our postgres db is running using docker
docker-backend-url The url where backend application is running in docker
docker-username The username we have in the container registry we use to push images
docker-push-secret The secret we have in the container registry we use to push images
docker-container-registry The container registry you use (e.g. ghcr.io for Github Container Registry). If you use DockerHub, create it with value 'https://index.docker.io/v1/'
docker-backend-prefix-image The docker image name for the backend application
docker-frontend-prefix-image The docker image name for the frontend application
k8s-db-url The database url where our postgres db is running using kubernetes
k8s-backend-url The url where backend application is running in kubernetes
In the job the pipeline will be the fastapi.vue.Jenkinsfile
Takes the code from the git repository and its submodules
sudo su
sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository --yes --update ppa:ansible/ansible
sudo apt install ansible
ansible-galaxy collection install community.dockerAnsible connects to the docker-vm through ssh and runs a playbook that it will define the sensitive parameters and will use docker-compose module to do docker-compose up the containers according to docker-compose.yml
So, to deploy our app we need a docker image updated. So we build the images according to nonroot.Dockerfile and Dockerfile, we are logging in Github Container Registry (we will talk about it later) and push the image there to be public available.
After we have configure connection between jenkins user and our k8s cluster, we update secrets and configmaps using also some Ansible to populate ~/.env values and create all the needed entities such as persistent volume claims, deployments, cluster IPs, ingress, services.
Secrets and ConfigMaps could be just prepared from earlier. This is applied to the https ingress, we will see later in SSL configuration
In order to be able to use Ansible for automation, there is the ansible-reference-letter-project. There is installation and usage guide.
Now, In order to deploy our project in Docker environment, we use a playbook that uses an Ansible role to run the application with docker-compose according to the docker-compose.yml. In that file, we have defined three services, the postgres container with its volume in order to be able to store data, the fastapi container and the vuejs container for our system taking environmental variables from local .env files (it's ready when we run the playbook from jenkins-server where the sensitive values from environmental variables are parametric). The fastapi container is built according to the nonroot.Dockerfile as a nonroot process for safety reasons. The vuejs container is built according to the Dockerfile.
For the HTTPS part we will talk about later.
In order to deploy our project in Kubernetes cluster, we first need to connect to that VM so as to configure a better connection between local PC or jenkins server and deployment vm's:
- Installing microk8s
- Do this trick to write less in terminal
echo "alias k='microk8s.kubectl' " >> ~/.bashrcThe permanent alias will be applied only if you reconnect to your VM.
sudo usermod -a -G microk8s <your-username>
sudo chown -f -R <your-username> ~/.kube
microk8s enable dns dashboard hostpath-storage ingress
microk8s status# VM's terminal
k config view --raw > kube-config
cat kube-config
# Local terminal
mkdir ~/.kube
scp <vm-name>:/home/<vm-username>/kube-config ~/.kube/configEdit ~/.kube/config to replace the 127.0.0.1 with the VM's public ip and the certificate line in clusters section with the below line (not used this way in a real production environment)
insecure-skip-tls-verify: true- Don't forget to add a firewall rule for the port specified in the ~/.kube/config file With
kubectl get podsyou can ensure that the connection is established.
If you use CI/CD tool and mostly Jenkins do the following (for better deployment fork the repository to be able to change code where needed):
# Jenkins terminal
sudo su
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
su jenkins
cd
# Local terminal
scp ~/.kube/config <jenkins-vm-name>:/tmp/config
# Jenkins terminal
mkdir -p .kube
cp /tmp/config ~/.kube/With
kubectl get podsyou can ensure that the connection is established.
Either manually or via jenkins server using Jenkinsfile and secret texts the following will do the trick! The code is located in the k8s folder of each project, so every time we must change directory to be located in the correct application folder. The code is in the .yaml format.
- Don't forget to have the docker images in Github Container Registry because the deployment entities use them. You can follow the logic located in fastapi.vue.Jenkinsfile in the 'Docker Deployment' stage. You must have docker installed in your local machine (or jenkins server)
- In projects' README.md files you will find information about the docker image each application is dockerized in.
# Secret (for the postgresql database)
kubectl create secret generic pg-users \
--from-literal=PGUSER=<put user name here> \
--from-literal=PGPASSWORD=<put password here> \
--from-literal=FAUSER=<put backend username here> \
--from-literal=FAPASSWORD=<put backend password here> \
--from-literal=FADBNAME=<put backend database name here>
# Continue from here
cd reference-letters-fastapi-server
# Config Map (for .env variables)
cp ref_letters/.env.k8s.example ref_letters/.env
nano ref_letters/.env # change to the correct values
kubectl create configmap fastapi-config --from-env-file=ref_letters/.env
cd k8s
# Persistent Volume Claim
kubectl apply -f db/postgres-pvc.yaml
# Postgres Deployment
kubectl apply -f db/postgres-deployment.yaml
# Postgres Initialization
kubectl get pods
kubectl exec -it <POD-NAME> -- psql -h localhost -U postgres -p 5432 < ../../assets/init_db/*
# FastAPI Deployment
kubectl apply -f fastapi/fastapi-deployment.yaml
# Services (Cluster IPs)
kubectl apply -f db/postgres-clip.yaml
kubectl apply -f fastapi/fastapi-clip.yaml
cd ../..
cd reference-letters-vuejs-client
# Config Map (for .env variables)
cp .env.k8s.example .env
nano .env # change to the correct values
kubectl create configmap vuejs-config --from-env-file=.env
cd k8s
# Deployments
kubectl apply -f vuejs/vuejs-deployment.yaml
# Services (Cluster IPs)
kubectl apply -f vuejs/vuejs-clip.yaml
# Ingress (For just HTTP - Edit file changing host to your own dns name)
kubectl apply -f vuejs/vuejs-ingress.yamlTo change to the correct values the .env file we use some Ansible running this playbook. This is also used by Jenkins server and Jenkinsfile. See more here. *