Continuous deployment with Gitlab-CI and Kubernetes

Pre-requisite

• App source code resides inside a Gitlab instance, container registry activated
• A dedicated Kubernetes namespace already exist for the app
• The latest version of Helm has been installed. Here is the Helm installation documentation

Register the deployment runner on Gitlab

First go to Settings -> CI/CD inside the app source code project on Gitlab and expand the Runners tab. Then, grab the runners registration token inside the Specific runners section. We will use that token inside our Gitlab runner configuration in order to add it to the project.

The next step is to use Helm to register the runner. Here is how we achieve that:

• Add Gitlab Helm charts repository

$ helm repo add gitlab https://charts.gitlab.io

• Create the values.yml configuration file for the chart:

gitlabUrl: http://gitlab.your-domain.com/
runnerRegistrationToken: "$runner_registration_token"
rbac:
  create: true
  clusterWideAccess: false
  serviceAccountName: myapp-deployer
  rules:
    - apiGroups: ["*"] 
      resources: ["*"]
      verbs: ["*"] 
runners:
  name: myapp
  serviceAccountName: myapp-deployer
  tags: myapp

• To create the default values file containing all available configuration settings and default values that we can use as a reference to adapt the previous values.yml file:

$ helm show values gitlab/gitlab-runner > values-default.yml

Here is an explanation of the configuration parameters we put inside values.yml:

• gitlabUrl: the URL of the Gitlab instance where the app source code project resides
• runnerRegistrationToken: the specific runner registration token previously retrieved inside the Runners settings of the app source code Gitlab's project
• rbac.create: wheather to create a dedicated Kubernetes service account (and associated RBAC resources) that will be used by the Gitlab runners manager pod(s) (the pod(s) that is registered to the Gitlab project and dynamically create other pods to run Gitlab-CI jobs)
• rbac.clusterWideAccess: wheather to limit permissions of the runner service account to its namespace, and therefore create RBAC role and rolebinding resources instead of clusterrole and clusterrolebinding
• rbac.serviceAccountName: the name of the service account that will be created for use by the runners manager pod
• rbac.rules: define the RBAC rules to apply to the service account specified at rbac.serviceAccountName
• runners.tags: the tag(s) associated with the runner. Can be used inside deployment jobs to explicitely select the runner we want the job to run on
• runners.serviceAccountName: the name of the Kubernetes service account that will be used by the runners (pods that will run the deployment jobs). We set that value to the same service account as the runner manager pods to give the runners pods (pods that actually run CI jobs) the permissions defined at rbac.rules

Once values.yml properly configured, we can use the following command to register the runner. The runners resources will be created inside the app namespace that we call myapp:

$ helm upgrade --install --namespace myapp gitlab-runner -f values.yml gitlab/gitlab-runner

When we make changes to the values.yml configuration file, we can also use that same command to update the runner. We should now see that the deployment runner is registered on the project by going again to the runners settings.

The runner is now registered and ready to be used to run the deployment jobs.

Affine deployment runners permissions

In the previous section, we specify the myapp-deployer Kubernetes service account inside the values.yml Gitlab Helm chart configuration, as the one that will be used by the Gitlab-CI jobs runners pods to deploy the app.

We gave that service account permissions to create any resources inside the app namespace only, using the rbac configuration section inside the chart values.yml configuration.

Inside the rbac.rules section, anything not explicitly allowed will be denied. We can therefore easily reduce the service account permissions if needed.

When trying to do something we are not allowed to with the deployment runners, an error message will tell us precisely which permission is missing. For instance, we get this output from Gitlab-CI when trying to delete a secrets resource while RBAC rules forbid us to do so:

And this one when trying to get secrets resources while RBAC rules forbid use to do so:

As you can see, the 3 pieces required to add the missing permission rule inside the values.yml rbac.rules configuration section are clearly mentionned in the error message: the API group of the resource, the resource name and the verb (delete, get...), and can be very helpful to debug RBAC rules problems when trying to give the runners only the permissions they need.

Configure Gitlab-CI for continuous deployment

We need to create a file called .gitlab-ci.yml inside the root directory of the Gitlab project containing the app source code. Inside that file, we will define the continuous deployment jobs. The first job we call build will be used to create a container image from the app source code and push that image to the Gitlab container registry of the project. The second one we call deploy will be used to create the Kubernetes resources required to run the app.

To release a new version of the app through the continuous deployment pipeline, we need to create a Git tag on the project, named with the new version of the app. The app container image created during the build job will be tagged with the new version of the app defined by the Git tag. That way, the deploy job will be able to retrieve the right container image version of the app to deploy inside the Kubernetes cluster. For infos about the .gitlab-ci.yml configuration syntax and keywords, see .gitlab-ci.yml keyword reference.

Here is the content of the .gitlab-ci.yml file:

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.9.0-debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}"
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
  rules:
    - if: $CI_COMMIT_TAG

deploy:
  stage: deploy
  image: 
    name: registry.gitlab.com/hackerstack/k8s-deployer:v1.25.3
  script:
    - cat ${CI_PROJECT_DIR}/deploy.yml | envsubst | kubectl apply -f -
  tags:
    - myapp
  rules:
    - if: $CI_COMMIT_TAG

The build job

We use Kaniko to build and push the app container image to the Gitlab container registry of the project. That way, we can build the container image from inside another container whithout using docker in docker and privileged containers which reduce complexity and improve security. The app container image is created from a Dockerfile present at the root directory of the app source code Gitlab project. To make Kaniko authenticate to the Gitlab container registry of the project in order to push the app container images, we create the /kaniko/.docker/config.json file, containing the following:

{
   "auths":{
      "registry.gitlab.com":{
         "auth":"<base64_encoded_credentials>"
      }
   }
}

We generate that file during the build job thanks to the following environment variables automatically set by Gitlab-CI:

• CI_REGISTRY: the address of the Gitlab container registry. registry.gitlab.com in this case
• CI_REGISTRY_USERNAME: the username required to push containers to the project’s Gitlab container registry
• CI_REGISTRY_PASSWORD: the password required to push containers to the project’s Gitlab container registry.

The <base64_encoded_credentials> represents the base64 encoding of ${CI_REGISTRY_USERNAME}:${CI_REGISTRY_PASSWORD}.

The following environment variables used during the build are also automatically set by Gitlab-CI:

• CI_PROJECT_DIR: path to the current Gitlab project directory inside the runner file system
• CI_REGISTRY_IMAGE: the address of the project container registry. In this case, registry.gitlab.com/hackerstack/myapp
• CI_COMMIT_TAG: the name of the Git tag that launched the Gitlab-CI pipeline. In this case, will contain the version of the app. We use that variable to tag the container image with the app version defined in the Git tag. The resulting container image URL will therefore looks something like registry.gitlab.com/hackerstack/myapp:<app_version>. That variable is finally used in the rules section of the build job definition in order to make that job run only when a Git tag is created on the project.

Have a look at Gitlab-CI predifined variables for a complete list of environment variables that are automatically set for Gitlab-CI jobs.

The deploy job

During the deploy job, we create Kubernetes resources required to run the app inside the Kubernetes cluster. To achieve that, we use the following elements:

• a container image containing the kubectl and envsubst utilities
• the deploy.yml file containing the app Kubernetes resources declaration

Here is the content of the deploy.yml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      imagePullSecrets:
        - name: myapp
      containers:
      - name: myapp
        image: registry.gitlab.com/hackerstack/myapp:$CI_COMMIT_TAG
        ports:
          - name: http
            containerPort: 80
        resources:
          limits:
            memory: "500Mi"
          requests:
            memory: "500Mi"

There are two things inside that file we want to highlight:

• we use an imagePullSecrets called myapp. That secret will be used to authenticate to the project's Gitlab container registry in order to pull the app container image, as the project is not publicly accessible. If the project was publicly accessible, we won't need that imagePullSecrets. To create that secret, you first need to Create a Gitlab deploy token on the project, and get the resulting gitlab_deploy_token_username and gitlab_deploy_token_password. Then create the app container image pull secret inside the namespace containing app resources in the Kubernetes cluster using the following command:

$ kubectl -n <namespace> create secret docker-registry <secret-name> --docker-server <registry_address> --docker-username=<gitlab_deploy_token_username> --docker-password=<gitlab_deploy_token_password> 

Here is an example:

$ kubectl -n myapp create secret docker-registry myapp --docker-server registry.gitlab.com --docker-username=registry-ro --docker-password=mysuperpassword
secret/myapp created

$ kubectl describe secret myapp -n myapp
Name:         myapp
Namespace:    myapp
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/dockerconfigjson

Data
====
.dockerconfigjson:  148 bytes

• The container image URL we use contains the $CI_COMMIT_TAG variable. That variable needs to be replaced during the resource deployment otherwise that won't work. That's why we passed the content of the deploy.yml file to the envsubst utility in order to replace the $CI_COMMIT_TAG variable inside the file with its value taken from the job runner environment variables. Remember, that environment variable is automatically set by Gitlab-CI and corresponds to the app new release version in this case. Once the $CI_COMMIT_TAG replaced inside the deploy.yml file, we pass that file content to the input of the kubectl apply -f command in order to create the resource inside the app Kubernetes cluster.

To run the deploy job, we make sure we use the deployment runner we have previously created and configured by specifying the runner tag myapp inside the job configuration.

Here is the Gitlab-ci logs for the deploy job. We have added two other commands to the script section of the job to show the created deployment resource and the container image used (the $CI_COMMIT_TAG has been replaced as expected in the container image URL):

Kubectl configuration

Let's have a look at the kubectl configuration inside the deployment container.

As you can see there is no specific kubectl configuration. You may be wondering how we are able to run kubectl commands to create resources inside the app Kubernetes cluster then ?

Good question... That's because the runner pod we use is deployed inside the cluster (so it already know how to communicate with it through its internal API endpoint) and uses the myapp-deployer service account's (we previously configured) token for authentication, as that service account is associated with the job runner pod.

To communicate with the Kubernetes cluster API the runner pod uses the KUBERNETES_* environment variables. See the env command in the following picture. Also on that same picture, the ca.crt file shown in the result of the ls command is the certificate of the Kubernetes cluster root certificate authority. That certificate will be used by the runner pod during HTTPS communications with the cluster API, to verify the certificate of the API server.

The namespace file contains the name of the Kubernetes namespace inside which the pod is running and the token one contains the token of the myapp-deployer service account associated to the job runner pod.