Continuous deployment with Gitlab-CI and Kubernetes
Create a simple, secure and scalable continuous deployment chain with Gitlab-CI for applications running in Kubernetes. No need for dedicated machines or authentication token inside Gitlab-CI configuration file(s).

Table of contents
Pre-requisite
- App source code resides inside a Gitlab instance, container registry activated
- A dedicated Kubernetes namespace already exist for the app
- The latest version of Helm has been installed. Here is the Helm installation documentation
Register the deployment runner on Gitlab
First, follow this documentation Gitlab project runner authentication token to create an authentication token for the runner we are going to register on the Gitlab project.
We will use that token inside the Gitlab runner's configuration in order to add it to the project.
The next step is to use Helm to register the runner. Here is how we achieve that:
- Add Gitlab Helm charts repository
$ helm repo add gitlab https://charts.gitlab.io
- Create the 'values.yml' configuration file for the chart:
gitlabUrl: http://gitlab.your-domain.com/
runnerRegistrationToken: "$runner_registration_token"
rbac:
create: true
clusterWideAccess: false
serviceAccountName: myapp-deployer
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
runners:
name: myapp
serviceAccountName: myapp-deployer
tags: myapp
- To create the default values file containing all available configuration settings and default values that we can use as a reference to adapt the previous 'values.yml' file:
$ helm show values gitlab/gitlab-runner > values-default.yml
Here is an explanation of the configuration parameters we put inside 'values.yml':
gitlabUrl
: the URL of the Gitlab instance where the project containing the app source code residesrunnerRegistrationToken
: the specific runner registration token previously retrieved inside the Runners settings of the app source code Gitlab's projectrbac.create
: wheather to create a dedicated Kubernetes service account (and associated RBAC resources) that will be used by the Gitlab runners manager pod(s) (the pod(s) that is registered to the Gitlab project and dynamically create other pods to run Gitlab-CI jobs)rbac.clusterWideAccess
: wheather to limit permissions of the runner service account to its namespace, and therefore create RBAC role and rolebinding resources instead of clusterrole and clusterrolebindingrbac.serviceAccountName
: the name of the service account that will be created for use by the runners manager podsrbac.rules
: define the RBAC rules to apply to the service account specified atrbac.serviceAccountName
runners.tags
: the tag(s) associated with the runner. Can be used inside deployment jobs to explicitely select the runner we want the job to run onrunners.serviceAccountName
: the name of the Kubernetes service account that will be used by the runners (pods that will run the deployment jobs). We set that value to the same service account used by the runners manager pods to give the runners pods (pods that actually run CI jobs) the permissions defined atrbac.rules
Once 'values.yml' properly configured, we can use the following command to register the runner. The runners resources will be created inside the app namespace that we call myapp
:
$ helm upgrade --install --namespace myapp gitlab-runner -f values.yml gitlab/gitlab-runner
When we make changes to the 'values.yml' configuration file, we can also use that same command to update the runner. We should now see that the deployment runner is registered on the project by going again to the runners settings.
The runner is now registered and ready to be used to run the deployment jobs.
Affine deployment runners permissions
In the previous section, we have specified the 'myapp-deployer' Kubernetes service account inside the 'values.yml' Gitlab Helm chart configuration, as the one that will be used by the Gitlab-CI jobs runners pods to deploy the app.
We gave that service account permissions to create any resources inside the app namespace only, using the 'rbac' configuration section inside the chart 'values.yml' configuration.
Inside the 'rbac.rules' section, anything not explicitly allowed will be denied. We can therefore easily reduce the service account permissions if needed.
When trying to do something we are not allowed to with the deployment runners, an error message will tell us precisely which permission is missing. For instance, we get this output from Gitlab-CI when trying to delete a 'secrets' resource while RBAC rules forbid us to do so:
And this one when trying to get 'secrets' resources while RBAC rules forbid us to do so:
As you can see, the 3 pieces required to add the missing permissions rules at 'rbac.rules' inside the 'values.yml' file are clearly mentionned in the error message: the API group of the resource, the resource name and the verb (delete, get...), and can be very helpful to debug RBAC rules problems when trying to give the runners only the permissions they need.
Configure Gitlab-CI for continuous deployment
We need to create a file called '.gitlab-ci.yml' inside the root directory of the Gitlab project containing the app source code. Inside that file, we will define the continuous deployment jobs. The first job we call 'build' will be used to create a container image from the app source code and push that image to the Gitlab container registry of the project. The second one we call 'deploy' will be used to create the Kubernetes resources required to run the app.
To release a new version of the app through the continuous deployment pipeline, we need to create a Git tag in the project, named with the new version number of the app. The app container image created during the 'build' job will be tagged with the new version number of the app defined by the Git tag. That way, the 'deploy' job will be able to retrieve the right container image version of the app to deploy inside the Kubernetes cluster. For info about the '.gitlab-ci.yml' configuration syntax and keywords, see .gitlab-ci.yml keyword reference.
Here is the content of the '.gitlab-ci.yml' file:
build:
stage: build
image:
name: gcr.io/kaniko-project/executor:v1.9.0-debug
entrypoint: [""]
script:
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}"
- /kaniko/executor
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
rules:
- if: $CI_COMMIT_TAG
deploy:
stage: deploy
image:
name: registry.gitlab.com/hackerstack/k8s-deployer:v1.25.3
script:
- cat ${CI_PROJECT_DIR}/deploy.yml | envsubst | kubectl apply -f -
tags:
- myapp
rules:
- if: $CI_COMMIT_TAG
The build job
We use Kaniko to build and push the app container image to the Gitlab container registry of the project. That way, we can build the container image from inside another container without using docker in docker and privileged containers which reduces complexity and improve security. The app container image is created from a Dockerfile present at the root directory of the app source code inside a Gitlab project. To make Kaniko authenticate to the Gitlab container registry of the project in order to push the app container images, we create the '/kaniko/.docker/config.json' file, containing the following:
{
"auths":{
"registry.gitlab.com":{
"auth":"<base64_encoded_credentials>"
}
}
}
We generate that file during the 'build' job thanks to the following environment variables automatically set by Gitlab-CI:
CI_REGISTRY
: the address of the Gitlab container registry. 'registry.gitlab.com' in this caseCI_REGISTRY_USERNAME
: the username required to push container images into the project’s Gitlab container registryCI_REGISTRY_PASSWORD
: the password required to push container images into the project’s Gitlab container registry.
The <base64_encoded_credentials>
represents the 'base64' encoding of ${CI_REGISTRY_USERNAME}:${CI_REGISTRY_PASSWORD}
.
The following environment variables used during the 'build' are also automatically set by Gitlab-CI:
CI_PROJECT_DIR
: path to the current Gitlab project directory inside the runner file systemCI_REGISTRY_IMAGE
: the address of container image inside the project's container registry. In this case, 'registry.gitlab.com/hackerstack/myapp'CI_COMMIT_TAG
: the name of the Git tag that launched the Gitlab-CI pipeline. In this case, will contain the version number of the app. We use that variable to tag the container image with the app version number defined in the Git tag. The resulting container image URL will therefore looks something likeregistry.gitlab.com/hackerstack/myapp:<app_version>
. That variable is finally used in the 'rules' section of the 'build' job definition in order to make that job run only when a Git tag is created on the project.
For a complete list of environment variables that are automatically set for Gitlab-CI jobs, have a look at Gitlab-CI predifined variables.
The deploy job
During the 'deploy' job, we create Kubernetes resources required to run the app inside the Kubernetes cluster. To achieve that, we use the following elements:
- a container image containing the 'kubectl' and 'envsubst' utilities
- the 'deploy.yml' file containing Kubernetes resources declarations for the app
Here is the content of the 'deploy.yml' file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
labels:
app: myapp
spec:
replicas: 1
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
imagePullSecrets:
- name: myapp
containers:
- name: myapp
image: registry.gitlab.com/hackerstack/myapp:$CI_COMMIT_TAG
ports:
- name: http
containerPort: 80
resources:
limits:
memory: "500Mi"
requests:
memory: "500Mi"
There are two things inside that file we want to highlight:
-
we use an 'imagePullSecrets' called 'myapp'. That secret will be used to authenticate to the project's Gitlab container registry in order to pull the app container image, as the project is not publicly accessible. If the project was publicly accessible, we won't need that 'imagePullSecrets'.
-
To create that secret, you first need to Create a Gitlab deploy token on the project, and get the resulting 'gitlab_deploy_token_username' and 'gitlab_deploy_token_password'. Then create the app container image pull secret inside the namespace containing the app resources inside the Kubernetes cluster using the following command:
$ kubectl -n <namespace> create secret docker-registry <secret-name> --docker-server <registry_address> --docker-username=<gitlab_deploy_token_username> --docker-password=<gitlab_deploy_token_password>
Here is an example:
$ kubectl -n myapp create secret docker-registry myapp --docker-server registry.gitlab.com --docker-username=registry-ro --docker-password=mysuperpassword
secret/myapp created
$ kubectl describe secret myapp -n myapp
Name: myapp
Namespace: myapp
Labels: <none>
Annotations: <none>
Type: kubernetes.io/dockerconfigjson
Data
====
.dockerconfigjson: 148 bytes
- The container image URL we use contains the
$CI_COMMIT_TAG
variable. That variable needs to be replaced during the resource deployment otherwise that won't work. That's why we passed the content of the 'deploy.yml' file to the 'envsubst' utility in order to replace the$CI_COMMIT_TAG
variable inside the file by that same variable value taken from the job runner environment variables. Remember, that environment variable is automatically set by Gitlab-CI and corresponds to the app new release version in this case. Once the$CI_COMMIT_TAG
replaced inside the 'deploy.yml' file, we pass that file content to the input of the 'kubectl apply -f' command in order to create the resource inside the Kubernetes cluster.
To run the 'deploy' job, we make sure we use the deployment runner we have previously created and configured by specifying the runner tag 'myapp' inside the job configuration.
Here is the Gitlab-CI logs for the 'deploy' job. We have added two other commands to the 'script' section of the job to show the created deployment resource and the container image used (the $CI_COMMIT_TAG
has been replaced as expected in the container image URL):
Kubectl configuration
Let's have a look at the 'kubectl' configuration inside the deployment container.
As you can see there is no specific 'kubectl' configuration. You may be wondering how are we able to run 'kubectl' commands to create resources inside the Kubernetes cluster then ?
Good question... That's because the runner pod we use is deployed inside the cluster (so it already know how to communicate with it through its internal API endpoint) and uses the 'myapp-deployer' service account's (we previously configured) token for authentication, as that service account is associated with the job runner pod.
To communicate with the Kubernetes cluster API the runner pod uses the KUBERNETES_*
environment variables. See the 'env' command in the following picture.
Also in that same picture, the 'ca.crt' file shown in the result of the 'ls' command is the certificate of the Kubernetes cluster's root certificate authority. That certificate will be used by the runner pod during HTTPS communications with the cluster API, to verify the certificate of the API server.
The 'namespace' file contains the name of the Kubernetes namespace inside which the pod is running and the 'token' one contains the token of the 'myapp-deployer' service account associated to the job runner pod.