Setup a GitLab Runner in Kubernetes using MinIO for caching with the Helm 2 package manager.
Connecting a Kubernetes cluster to GitLab is pretty simple. Even simpler, the installation of a GitLab Runner when you do it from the GitLab admin area in Kubernetes Applications. Install “Helm Tiller” (older GitLab versions) is one click and “GitLab Runner” the second. That works, but …
The so installed runner has a simple, default configuration and is not using any cache. That’s ok for just trying CI in GitLab but for a serious installation you want to tweak the settings and use a cache.
There’re several instructions for setting up GitLab Runner in Kubernetes but it took me some time to figure out how it works and finally make it work. I hope this step-by-step guide provides some help or hints. Still I’m not a Kubernetes expert. So, if you find something to improve, I’m happy to get feedback.
We use MinIO because the GitLab Runner in Kubernetes needs a distributed cache which is either S3 or GCS. If you want a cache within your cluster, the MinIO service emulates the S3 cache type. So in the gitlab-runner settings we configure S3 pointing to our minio-service.
* A Kubernetes cluster (or minikube). I used a cluster with version 1.14.5
* Install Helm client https://helm.sh/docs/using_helm/
* Ensure that kubectl is using the right cluster (helm uses kubectl context)
First I’ll create an own namespace and a role for that namespace that is used by the service account. Using a namespace and limited service account adds a bit complexity. If you don’t need the namespace, things might get easier.
kubectl create namespace gitlab-runner
Create a service account
gitlab-runner in namespace
kubectl create serviceaccount gitlab-runner -n gitlab-runner
Create the role with permission in namespace and bind it to service account defined in file
kubectl create -f role-runner.yaml
Init helm for the namespace and let it install tiller in the namespace. Using the service account created before.
helm init --service-account gitlab-runner --tiller-namespace gitlab-runner
Now with helm and tiller ready, we can install MinIO and GitLab Runner.
Let’s start with MinIO. Because the service account of the namespace is limited, it can’t create a persistent volume. Usually this would be done by helm but now we have to create it manually upfront and tell helm to use the existing volume.
minio-standalone-pvc.yaml defines the config for the PersitentVolume and the claim for the volume. In this example a size of 4 GB is used. If you plan to use more, increase the value.
kubectl create -f minio-standalone-pvc.yaml
Now we install MinIO with helm and add some custom settings we put inside a
minio/values.yaml file. For more details on the MinIO helm chart see: https://github.com/helm/charts/tree/master/stable/minio
Important is here, that we set the
secretKey that will later be used by the GitLab Runner installation. Also the default bucket is created that the GitLab Runner will use to store the cache files.
helm --tiller-namespace gitlab-runner install --namespace gitlab-runner --name minio -f minio/values.yaml stable/minio
This step is optional: If you wanna access the MinIO browser from outside Kubernetes, you can create a NodePort or a temporary port-forward.
The NodePort is defined in the
minio-standalone-service.yaml as an own service.
kubectl create -f minio-standalone-service.yaml
Find out your cluster IP:
and the port the minio-service is mapped to:
kubectl get services -n gitlab-runner
The two commands will output the following where you can get the IP and the port, that is exposed.
The minio-service-ext service is mapped to port
32028. Open in a browser: http://10.159.0.133:32028 and you should see the MinIO login page.
The login accessKey and secretKey we set in the values.yaml for MinIO install to “minio” and “minio123”.
As mentioned, there’s also the temporary solution of creating a port-forward. Just run:
kubectl port-forward service/minio -n gitlab-runner 9000:9000
localhost:9000 in your browser.
Ok, finally lets install the GitLab Runner itself. This is described in more details here: https://docs.gitlab.com/runner/install/kubernetes.html#configuring-gitlab-runner-using-the-helm-chart.
Helm should have created a secret named “minio” in the namespace which is used by gitlab-runner. If this is not the case, create one.
Check if secret exists:
kubectl get secrets -n gitlab-runner
Create it if secret doesn’t exist:
kubectl create secret generic minio --from-literal=accesskey=minio -from-literal=secretkey=minio123 -n gitlab-runner
The gitlab-runner is not part of the helm repo so we have to add it to helm.
helm repo add gitlab https://charts.gitlab.io
We also set some custom values for this helm chart in
On the GitLab site you’ll find more details about all the config values in the helm chart. This is the minimal set including the settings for the cache.
Now install it to the namespace. Take care, you might want to install an older chart version depending on your GitLab server version.
helm --tiller-namespace gitlab-runner install --namespace gitlab-runner --name gitlab-runner -f runner/values.yaml gitlab/gitlab-runner
Now you should see the created runner in your GitLab Admin Area for further configuration inside GitLab.
Please note that we’ve not really considered security and performance in this setup. You might also need different settings depending on your Kubernetes version and installation. The example above runs a Kubernetes v1.14.5.