Skip to content

Jenkins on Kubernetes (Legacy)

Jump to bottom
Abhi Yerra edited this page Feb 21, 2020 · 1 revision

Jenkins on Kubernetes {#jenkins-on-kubernetes-1 .graf .graf--h3 .graf--leading .graf--title} One of the benefits of Kubernetes is the ability to completely use the underlying instances saving money. One of the tasks that we recently explored was running Jenkins in Kubernetes as a Docker image to, well, build Docker images of the services that were were deploying onto Kubernetes. Meta, I know. To our pleasant surprise it actually turned out to not be difficult at all. Kubernetes Cluster {#kubernetes-cluster .graf .graf--h4 .graf-after--p} The first task for this is to have a running Kubernetes Cluster on AWS which is what we are using to demonstrate Jenkins on. To install Kubernetes on AWS check out this article{.markup--anchor .markup--p-anchor}. EBS Volume {#ebs-volume .graf .graf--h4 .graf-after--p} Once the Kubernetes Cluster is built the second task is create a persistent volume. This volume is what will be used to store $JENKINS_HOME{.markup--code .markup--p-code} Create a volume using the following command: aws ec2 create-volume --size 100 --region us-east-1 --availability-zone us-east-1a --volume-type gp2{.markup--code .markup--p-code} Make sure that the availability-zone{.markup--code .markup--p-code} is in a place that has your Kubernetes node is running. If you make the volume in an availability zone that a node is not running on Kubernetes will not be able to mount it. Peristent Volume {#peristent-volume .graf .graf--h4 .graf-after--p} Once we have created the volume the next step is to create a PersistentVolume in Jenkins. Make sure to change the volumeID in the manifest.

kind: PersistentVolume
apiVersion: v1
metadata:
  name: jenkins-home-pv
  labels:
type: amazonEBS
spec:
  capacity:
storage: 100Gi
  accessModes:
ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  claimRef:
namespace: default
name: jenkins-home
  awsElasticBlockStore:
volumeID: aws://us-east-1a/vol-1234
fsType: ext4
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins-home
  labels:
type: amazonEBS
spec:
  accessModes:
ReadWriteOnce
  resources:
requests:
  storage: 100Gi

Service/Deployment {#servicedeployment .graf .graf--h4 .graf-after--pre}

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  template:
metadata:
  labels:
app: jenkins
spec:
  volumes:
  • name: docker-sock
  • hostPath:
  • path: /var/run
  • name: jenkins-home
  • persistentVolumeClaim:
  • claimName: jenkins-home
  • containers:
  • name: jenkins
  • image: 6891.dkr.ecr.us-east-1.amazonaws.com/opszero/jenkins
  • imagePullPolicy: Always
  • env:
  • name: JENKINS_HOME
  • value: /jenkins_home/jenkins_home
  • name: AWS_ACCESS_KEY_ID
  • valueFrom:
  • secretKeyRef:
  • name: opszero
  • key: AWS_ACCESS_KEY_ID
  • name: AWS_SECRET_ACCESS_KEY
  • valueFrom:
  • secretKeyRef:
  • name: opszero
  • key: AWS_SECRET_ACCESS_KEY
  • name: AWS_DEFAULT_REGION
  • value: us-east-1
  • volumeMounts:
  • mountPath: /var/run
  • name: docker-sock
  • mountPath: /jenkins_home
  • name: jenkins-home
  • ports:

  • containerPort: 8080

    kind: Service apiVersion: v1 metadata: name: jenkins annotations: service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:1234:certificate/key-name-from-cert-manager spec: type: LoadBalancer selector: app: jenkins ports:
  • name: https
  • port: 443
  • targetPort: 8080

Makefile {#makefile .graf .graf--h3 .graf-after--li}

run:
cd .. && sudo docker run  -p 8888:8080 -p 50000:50000 -v /var/run/docker.sock:/var/run/docker.sock -v $$PWD/jenkins_home:/var/jenkins_home opszero/jenkins
cd .. && sudo docker run  -p 8888:8080 -p 50000:50000 -v /var/run/docker.sock:/var/run/docker.sock opszero/jenkins

build:
docker build -t opszero/jenkins .
eval $(shell aws ecr get-login --profile opszero --region us-east-1)
docker tag opszero/jenkins 123456.dkr.ecr.us-east-1.amazonaws.com/opszero/jenkins
docker push 518940969811.dkr.ecr.us-east-1.amazonaws.com/opszero/jenkins

keys:
openssl genrsa -out key.pem  # creates key.pem
openssl req -new -key key.pem -out csr.pem
you need to put the dns name of your website, testweb.local
for the 'Common Name' question
other questions, you can just accept defaults
actually, you can accept defaults for all, will work ok too
openssl x509 -req -days 9999 -in csr.pem -signkey key.pem -out cert.pem
rm csr.pem

Dockerfile {#dockerfile .graf .graf--h3 .graf-after--pre}

FROM jenkins

USER root

RUN apt-get update \
  && apt-get install -y sudo  \
apt-transport-https \
ca-certificates \
curl \
software-properties-common \
build-essential \
git-core \
python \
python-pip \
jq

RUN pip install --upgrade awscli

ENV DOCKER_BUCKET=get.docker.com
ENV DOCKER_VERSION=17.04.0-ce
ENV DOCKER_SHA256=c52cff62c4368a978b52e3d03819054d87bcd00d15514934ce2e0e09b99dd100

RUN set -x \
&& curl -fSL "https://${DOCKER_BUCKET}/builds/Linux/x86_64/docker-${DOCKER_VERSION}.tgz" -o docker.tgz \
&& echo "${DOCKER_SHA256} *docker.tgz" | sha256sum -c - \
&& tar -xzvf docker.tgz \
&& mv docker/* /usr/local/bin/ \
&& rmdir docker \
&& rm docker.tgz \
&& docker -v

RUN echo "jenkins ALL=NOPASSWD: ALL" >> /etc/sudoers

RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && \
chmod +x ./kubectl && \
mv ./kubectl /bin/kubectl

RUN mkdir -p /keys
COPY ./cert.pem /keys/cert.pem
COPY ./key.pem /keys/key.pem

ENV JENKINS_OPTS --httpPort=8080 --httpsPort=8081 --httpsCertificate=/keys/cert.pem --httpsPrivateKey=/keys/key.pem
Clone this wiki locally