With the growth of pro­gram­ming pro­jects and the rise of ser­vice ori­ented archi­tec­tures, the par­ti­tion­ing of the devel­op­ment pro­cess into more spe­cial­ized tasks becomes inev­it­able. These tasks will usu­ally be solved by dif­fer­ent teams in par­al­lel in order to increase effi­ciency, lead­ing to faster pro­gress in the devel­op­ment of the source code. How­ever, it cre­ates an addi­tional task of com­bin­ing the pro­gress of the dif­fer­ent teams. Changes intro­duced by one team might affect the imple­ment­a­tion of another, cre­at­ing con­flicts. While small con­flicts are usu­ally resolved quickly, a belated iden­ti­fic­a­tion might lead to a quick growth and let the developers enter “integ­ra­tion hell”. This term describes the point where mer­ging par­al­lel devel­op­ments of the code and resolv­ing asso­ci­ated con­flicts will take up more time than the ori­ginal code devel­op­ment itself.

To avoid “integ­ra­tion hell”, con­tinu­ous code integ­ra­tion needs to be done to ensure a timely iden­ti­fic­a­tion and res­ol­u­tion of con­flicts. There­fore, auto­mated con­tinu­ous integ­ra­tion tools like Jen­kins have developed to be an indis­pens­able tool for soft­ware devel­op­ment teams. They offer the pos­sib­il­ity to auto­mat­ic­ally build and test integ­rated code. This can be done peri­od­ic­ally or event driven, for example every time a developer pushes new code into a repos­it­ory. The CI tool then reports back the res­ults to the developers which are then able to resolve pos­sible con­flicts immediately.

The devel­op­ment of CI soft­ware like Jen­kins enables to quickly setup a con­tinu­ous integ­ra­tion server that can be con­figured to the spe­cific needs of a pro­ject eas­ily. How­ever, hard­ware require­ments for the server might be dif­fi­cult to estim­ate upfront. Pro­gress in code devel­op­ment is rarely con­stant over time: Upcom­ing soft­ware releases will largely increase the amount of code pro­duced and there­fore will increase the demands on the con­tinu­ous integ­ra­tion server. The asso­ci­ated urgency demand that the server will still be cap­able to per­form timely builds and tests of new code. This will lead to higher hard­ware require­ments. On the other hand, a pro­ject might hit a period of less devel­op­ment, as developers are con­cerned with other projects/problems. If this is the case, idling hard­ware will lead to unne­ces­sary expenses.

The vary­ing demand makes the CI pipeline a per­fect tar­get for elastic com­put­ing tech­no­lo­gies, as for example the Kuber­netes con­tainer orches­tra­tion tool in com­bin­a­tion with cloud-com­put­ing tech­no­lo­gies. This will enable users to auto­mat­ic­ally spin up serv­ers that build and test newly pro­duced code and ter­min­ate them as soon as they are not needed anymore.

In this short tutorial we will cre­ate a min­imal setup of an elastic CI pipeline using Jen­kins within a Kuber­netes cluster. This cluster can be used loc­ally, or within a scal­able envir­on­ment like AWS, Azure or Google cloud ser­vices.  Both Kuber­netes and Jen­kins offer a large amount of addi­tional setup options for fur­ther cus­tom­iz­a­tion and can be com­bined with addi­tional tools to com­plete the devel­op­ment pipeline, for example code repos­it­or­ies like Git and deploy­ment repos­it­or­ies as for example Nexus. The fol­low­ing steps will be per­formed on AWS-EC2 instances run­ning the Amazon Linux 2 AMI. How­ever, the steps are adapt­able to other oper­at­ing sys­tems (but might need a few adjustments).

The Kuber­netes Cluster

Our Kuber­netes cluster will con­sist of one mas­ter node and a vari­able num­ber of worker nodes. As net­work plu­gin, we will use Flan­nel. We will start by installing the Kuber­netes tools on all nodes. After this pre­par­a­tion is done, we will ini­tial­ize the cluster on the mas­ter nodes. The worker nodes can then be joined con­sec­ut­ively. The fol­low­ing com­mands should all be run as the root user (not just sudo, as some com­mands will not work) on every node.

sudo su

Assum­ing the oper­at­ing sys­tem and repos­it­or­ies have been brought up to date via

yum update -y

the setup starts with adding the Kuber­netes repos­it­ory to yum

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

To be able to run the Kuber­netes cluster, we need to dis­able SELinux on our node

setenforce 0sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

For yum to accept the Kuber­netes repos­it­ory sig­na­ture, GnuPG has to be updated

yum install -y http://mirror.centos.org/centos/7/os/x86_64/Packages/gnupg2-2.0.22-5.el7_5.x86_64.rpm

Fur­ther, a few ipt­ables set­tings need to be changed for Flan­nel to work correctly

cat <<EOF >  /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1EOF

sysctl --system

Now, install the Kuber­netes tools

yum install -y docker kubelet kubeadm kubectl --disableexcludes=kubernetes

At this point, it should be checked if the br_netfilter is running

lsmod | grep br_netfilter

If this com­mand does not give any out­put, br_netfilter needs to be star­ted with the fol­low­ing command

modprobe br_netfilter

For the ser­vices needed by Kuber­netes to start auto­mat­ic­ally when the sys­tem boots, the respect­ive ser­vices need to be enabled in the sys­tem daemon

systemctl enable kubelet.service systemctl enable dockersystemctl start docker

The Mas­ter Node

After run­ning the pre­par­a­tion steps of the last sec­tion on every node, it is now time to spin up the cluster by running

kubeadm init --pod-network-cidr=10.244.0.0/16

on the node that will serve as the Kuber­netes mas­ter. The com­mand will pro­duce an out­put show­ing how to join addi­tional nodes to the cluster. The respect­ive join com­mand can be recovered at a later stage by running

kubeadm token create --print-join-command

Note that this com­mand also cre­ates a new token. The tokens expire after 24h and thus need to be cre­ated on a reg­u­lar basis.

The Kuber­netes cluster should be admin­is­trated by a user dif­fer­ent than root, so we need to exit the root shell

exit

and then run

mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/config

to enable the default user to admin­is­trate our Kuber­netes cluster.

Finally, the Flan­nel net­work plu­gin needs to be star­ted within the cluster

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml

Note that this com­mand will not be run­nable as root user within the AWS EC2-Instance, as out­bound traffic is dis­abled for the root user.

Join­ing Worker Nodes

To join addi­tional EC2 instances as worker nodes, we only need to run the kubeadm join com­mand obtained in the mas­ter node setup on each node. It will look some­thing like this:

kubeadm join 172.31.9.52:6443 --token a9og1w.p9ju926ld39aoagb --discovery-token-ca-cert-hash  sha256:aba42be8add85f3f6bbd8fa76e765a14ab226038b90fc5a967bb3e7e58d716fa

with IP and tokens spe­cific to every setup. The asso­ci­ated IP is the Kuber­netes mas­ter IP and will be needed later. Note that the join com­mand tries to access port 6443 on the Kuber­netes mas­ter node, so that port should be opened for the Kuber­netes worker nodes. We can check if everything worked cor­rectly by running

kubectl get nodes

on the Kuber­netes mas­ter node. It should show two nodes (or more if you decided to join addi­tional worker nodes):

Run­ning the Jen­kins Mas­ter within Kubernetes

Run­ning a web server like Jen­kins within Kuber­netes is simple in prin­ciple but can get very com­plic­ated if all fea­tures of Kuber­netes should be used. In order to main­tain the sim­pli­city of this tutorial, we will restrict ourselves to use a ded­ic­ated host within the Kuber­netes cluster to launch the Jen­kins mas­ter pod. This means we will know the IP address of that spe­cific host in order to com­mu­nic­ate with the Jen­kins mas­ter. We start by defin­ing a Kuber­netes deploy­ment object on the Kuber­netes mas­ter node by pre­par­ing a con­fig­ur­a­tion file jenkins-master-deployment.yaml. Cre­ate a file named jenkins-master-deployment.yaml with the fol­low­ing content:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
        - name: jenkins
          image: schniedersdavid/jenkinsonkubernetes:jenkins
          env:
            - name: JAVA_OPTS
              value: -Djenkins.install.runSetupWizard=false
          ports:
            - name: http-port
              containerPort: 8080
            - name: jnlp-port
              containerPort: 50000
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-home
          emptyDir: {}

apply­ing this deploy­ment object using

kubectl apply -f jenkins-master-deployment.yaml

will launch a pod run­ning a jen­kins con­tainer on one of the worker nodes. Using

kubectl describe pods

we can identify which node that is and note down the IP address under which it is reach­able. This will be the Jen­kins mas­ter IP. It will be needed later.

Note that this IP is the AWS private IP. If we want to access this node from out­side the AWS Cloud, we need to extract the pub­lic IP of that node from the AWS con­sole. For me, this is 3.16.218.24 (this will be dif­fer­ent for every EC2 instance).

Tak­ing a look at the ports sec­tion of the deploy­ment con­fig­ur­a­tion file above, we can see that the con­tainer will accept con­nec­tions through ports 8080 and 50000. The former is the http port for the Jen­kins web UI, while the lat­ter will be used for the Java Net­work Launch­ing Pro­tocol (JNLP), which will allow the Jen­kins mas­ter to run applic­a­tion on the Jen­kins slaves later. For these ports to be access­ible within our Kuber­netes cluster, we will add two Node­Port ser­vices, open­ing these spe­cific ports of the Jen­kins mas­ter pod for applic­a­tion out­side of the Kuber­netes cluster. This will be done by cre­at­ing two ser­vice con­fig­ur­a­tion files, jenkins-master-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
  selector:
    app: jenkins

 and jenkins-slave-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins-jnlp
spec:
  type: NodePort
  ports:
    - port: 50000
      targetPort: 50000
  selector:
    app: jenkins

and apply­ing them to the cluster

kubectl apply -f jenkins-master-service.yamlkubectl apply -f jenkins-slave-service.yaml

By doing this, we will map the ports 8080 and 50000 within the Jen­kins mas­ter pod to two ran­domly gen­er­ated ports between 30000 and 32767. We can identify these ports by running

kubectl get services

After doing so, we can access the Jen­kins mas­ter UI using the IP of the node host­ing the port (we iden­ti­fied it earlier using kubectl describe pods) and the Node­Port mapped to the Jen­kins html port 8080 (here: 32710). Remem­ber to open that port to your cli­ent IP.

Within this UI, we can now con­fig­ure Jen­kins as we need it. In this tutorial, we will restrict ourselves to add the Kuber­netes cluster as a cloud to run build jobs within Jen­kins. To do so, we will visit the “Man­age Jen­kins” tab and nav­ig­ate to “Con­fig­ure Sys­tem”. At the very bot­tom of that page, we see the entry “Cloud”. In the asso­ci­ated drop-down menu, we choose Kuber­netes, which will spawn a large set of new con­fig­ur­a­tion options. Here, we will only change a few settings:

• Kuber­netes URL: This is the (private) IP of the Kuber­netes mas­ter (obtain­able via the AWS con­sole) on port 6443. For this com­mu­nic­a­tion to work, the port 6443 needs to be open for other nodes within the cluster.
  
• Jen­kins URL: This is the (private) IP of the Jen­kins mas­ter (obtain­able from kubectl describe pods, see above) on the Jen­kins http port (obtain­able from kubectl get ser­vices). Here, the port 32710 needs to be open for other nodes within the cluster.
  
• Jen­kins tun­nel: This is the (private) IP of the Jen­kins mas­ter (see above) on the Jen­kins-jnlp port (obtain­able from kubectl get ser­vices). Also this port needs to be open for all nodes within the cluster.
  

In addi­tion to that, we  check the box “Dis­able https cer­ti­fic­ate check”. We fur­ther need to set the cor­rect cre­den­tials to allow Jen­kins to access the Kuber­netes cluster. To do so, we click on “add” in the cre­den­tials row, we choose “Jen­kins”, then “Kuber­netes Ser­vice Account” for “kind”. We click on add and then choose the newly added “Secret text” in the cre­den­tials drop-down menu. For this ser­vice account to have the cor­rect access rights, we need to run the command

kubectl create clusterrolebinding default-admin --clusterrole cluster-admin --serviceaccount=default:default

on the Kuber­netes mas­ter node. Note that this com­mand will give admin rights to this ser­vice account, which will be a secur­ity issue if your Kuber­netes cluster is access­ible by others.

Next, we choose “Add Pod Tem­plate” at the bot­tom of the page. This will again gen­er­ate addi­tional fields. Here, we will change “Usage” to “Use this node as much as pos­sible”. We will fur­ther define the raw yaml for the Pod as

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: jnlp
    image: jenkins/jnlp-slave:3.27-1-alpine

After sav­ing this con­fig­ur­a­tion, Jen­kins is able to start Pods at will in order to run a pro­ject. This can be tested by cre­at­ing a test pro­ject. On the Jen­kins main page, we click on “new pro­ject” and choose free­style. We con­fig­ure the pro­ject to run a single shell command:

cat /dev/zero > /dev/null

This com­mand will run until it is can­celed by user interaction.

In order to test the Kuber­netes-Jen­kins integ­ra­tion, it makes sense to pro­hibit builds on the Jen­kins mas­ter node. We can do this by vis­it­ing “Man­age Jen­kins -> Con­fig­ure Sys­tem” and set­ting the num­ber of build pro­cessors to zero. If we now start a build of our test pro­ject, the Jen­kins server should spawn a Jen­kins slave on any avail­able worker node after some time, which will then run the shell com­mand entered before.

What’s next?

With the elastic Jen­kins CI pipeline now run­ning smoothly, scal­ing hard­ware to its cur­rent needs, it is time to integ­rate it into the devel­op­ment pipeline. A large set of plu­gins for Jen­kins make it easy to con­nect it to code repos­it­or­ies like Git­Lab, Git­Hub, Bit­bucket, and more, allow­ing for cus­tom­ized event driven builds. On the other end, plu­gins towards repos­it­or­ies as for example Son­a­type Nexus or Arti­fact­ory allows to auto­mat­ic­ally deploy suc­cess­fully built soft­ware. Con­tain­er­iz­ing these applic­a­tions as well makes up for a con­sist­ent, eas­ily main­tain­able and scal­able devel­op­ment plat­form. We will dis­cuss these fur­ther steps in an upcom­ing blog art­icle, so stay tuned…

We also only scratched the sur­face of Kuber­netes’ poten­tial in this tutorial. The Jen­kins mas­ter is still run­ning on a ded­ic­ated node that we need to know the IP of to access the web UI. Advanced net­work­ing options within Kuber­netes will help us to address the Jen­kins mas­ter, regard­less of which node it is run­ning on. Per­sist­ent volume tech­niques will allow to keep the server con­fig­ur­a­tion in cases of fail­over, where the Jen­kins mas­ter pod might be restar­ted on a dif­fer­ent node. These top­ics present a more involved deep dive into the com­plex mech­an­isms of Kuber­netes and Docker, which has been in large parts covered in our sem­inar series “Oper­a­tion­al­is­ier­ung von AI-Work­loads durch Con­tainer-basierte App­lika­tionen“. Our cur­rent webinars you can find here.