Sign in Subscribe
Cloud

DRY Kubernetes manifests with Kustomize

Stop repeating yourself when creating Kubernetes manifests thanks to Kustomize. What is Kustomize, how it works and what is it capable of ? Let's answer those questions in this post.

Gmk Ziz 👓

• 8 min read
DRY Kubernetes manifests with Kustomize

What is Kustomize

Kustomize is a command-line tool that can be used to declaratively transform / customize Kubernetes manifests in a way that help avoid repetitions and ease maintenance and reuse of existing manifests.

It works with plain YAML and native Kubernetes resources. It is available as a standalone binary and also partially integrated into kubectl through kubectl kustomize and kubectl apply -k.

Kustomize is easy to learn and get started with compared to templating tools like helm and helmfile. Here is the Kubernetes blog post introducing Kustomize: Kustomize announcement.

How Kustomize works

  • Kustomize works with raw Kubernetes manifests that it takes as input
  • It then transforms the input manifests based on its configuration and output the result
  • Kustomize configurations are defined inside the kustomization.yml file
  • That's where we tell Kustomize which Kubernetes manifests files it should take as input and what transformations should be applied on them
  • On the Kustomization file page, you will find infos about the format and syntax of the kustomization.yml file and detailed explanations on every fields it supports

Example: Kustomizing Kubernetes deployment manifest

In this example, we will create two variants of a Kubernetes deployment resource manifest for staging and production environments.

The environments specific manifests will be generated from a common base using Kustomize. That way, we avoid repeating ourselves by duplicating Kubernetes resources manifests.

  • Example directory tree:
$ tree kustomize-example/
kustomize-example/
├── common
│   ├── deploy.yml
│   └── kustomization.yaml
├── prod
│   └── kustomization.yaml
└── staging
    └── kustomization.yaml
  • Files contents:
# Common

$ cat common/deploy.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: ubuntu
        resources:
          limits:
            memory: "300Mi"
          requests:
            memory: "300Mi"
            
$ cat common/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml

# Staging

$ cat staging/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../common/
commonLabels:
  env: staging

# Prod

$ cat prod/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../common/
commonLabels:
  env: prod
replicas:
- count: 4
  name: myapp
  • Resulting Kustomized manifests
# Staging

$ kustomize build staging/
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    env: staging
  name: myapp
  namespace: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
      env: staging
  template:
    metadata:
      labels:
        app: myapp
        env: staging
    spec:
      containers:
      - image: ubuntu
        name: myapp
        resources:
          limits:
            memory: 300Mi
          requests:
            memory: 300Mi
            
# Prod

$ kustomize build prod/
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    env: prod
  name: myapp
  namespace: myapp
spec:
  replicas: 4
  selector:
    matchLabels:
      app: myapp
      env: prod
  template:
    metadata:
      labels:
        app: myapp
        env: prod
    spec:
      containers:
      - image: ubuntu
        name: myapp
        resources:
          limits:
            memory: 300Mi
          requests:
            memory: 300Mi

More about installing and easily cutomizing Kubernetes resources manifests using Kustomize in next sections.

Install Kustomize

  • Sometimes, it can be useful to install Kustomize for development as some of its useful CLI features are not available from the kubectl kustomize command at the time of writing
  • Kustomize releases assets can be found here. To install Kustomize on Linux, do the following:
$ kustomize_version=v5.2.1 # choose version
$ linux_arch=arm64 # choose OS arch

# Get Kustomize binary
$ wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${kustomize_version}/kustomize_${kustomize_version}_linux_${linux_arch}.tar.gz

# Install Kustomize binary
$ sudo tar xzvf kustomize_v5.2.1_linux_amd64.tar.gz -C /usr/local/bin/

# Verify
$ kustomize version
{Version:kustomize/v4.5.4 GitCommit:cf3a452ddd6f83945d39d582243b8592ec627ae3 BuildDate:2022-03-28T23:12:45Z GoOs:linux GoArch:amd64}

Create and update kustomization file

  • To get usage help and example for any Kustomize CLI command, follow that command by -h or --help
  • Use the CLI or directly update the kustomization.yml file with the desired configurations settings

Initialize the kustomization file

This can be useful to quickly initialize a kustomization.yml file with autodetected yaml files from the current directory, added as Kustomize configuration resources.

$ tree k8s-resources/
k8s-resources/
├── deploy.yml
├── vpa.yml
├── ingress.yml
└── service.yml

$ kustomize create --autodetect

$ cat kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- vpa.yml
- ingress.yml
- service.yml

Add or remove resources from the kustomization file

# Remove resource
$ kustomize edit remove resource vpa.yml

$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- service.yml
- ingress.ym

# Add resource after creating the hpa.yml manifest
$ kustomize edit add resource hpa.yml

$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- service.yml
- ingress.yml
- hpa.yml

Add or remove common labels and annotations from the kustomization file

# Add labels and annotations
# Multiple labels and annotations values are separeted by whitespace
$ kustomize edit add label app:myapp env:staging
$ kustomize edit add annotation myapp/version:0.0.1

$ cat kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- service.yml
- ingress.yml
- hpa.yml
commonAnnotations:
  myapp/version: 0.0.1
commonLabels:
  app: myapp
  env: staging
  
# Remove labels and annotations
# Only use the keys to remove labels or annotations
# Multiple keys values are separeted by comma
$ kustomize edit remove label app,env
$ kustomize edit remove annotation myapp/version

$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- service.yml
- ingress.yml
- hpa.yml

Add or remove patches from the kustomization file

# Add patches
$ kustomize edit add patch --name myapp --kind Deployment --group apps --version v1 --path deploy-patch.yml

$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- service.yml
- ingress.yml
- hpa.yml
patches:
- path: deploy-patch.yml
  target:
    group: apps
    kind: Deployment
    name: myapp
    version: v1
 
# Remove patches
$ kustomize edit remove patch --name myapp --kind Deployment --group apps --version v1 --path deploy-patch.yml

$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- service.yml
- ingress.yml
- hpa.yml

As seen previously, Kustomize patches configuration has a target field that uniquely identifies the resource we want to patch (using at least one of the following resources elements: group, version, kind, name, namespace, labelSelector and annotationSelector) and a path field where we specify the path (relative to the patch file) to the file containing the patches. The patches files contents can be in different forms that are shown below.

Patches using path json6902
# Syntax
- op: <operation type> # Possible values: add, remove, replace
  path: <path to the field to modified on the resource>
  value: <value we want for the field selected with path>

# Examples

# Patching container image
# File: deploy-patch.yml
- op: replace
  path: /spec/template/spec/containers/0/image
  value: nginx:1.25.3
Patches using path strategic merge
  • In this case, the target field from Kustomize patches configuration can be omitted
  • The target resource is matched using the apiVersion, kind and metadata.name from the patches files
# Syntax

apiVersion: <apiVersion of the resource>
kind: <kind of the resource>
metadata:
  name: <name of the resource>
spec:
  <patches>

# Examples

# Patching container name and image on a Deployment named nginx
# File: deploy-patch.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  template:
    spec:
      containers:
        - name: nginx
          image: nginx:1.25.3

Set prefix and suffix for resources names

$ kustomize edit set nameprefix -- prefix-
$ kustomize edit set namesuffix -- -suffix

$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
(...)
- deploy.yml
(...)
namePrefix: prefix-
nameSuffix: -suffix

# Resulting Kubernetes resources
$ kustomize build
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefix-myapp-suffix
(...)

Set resources namespace

$ kustomize edit set namespace myapp-namespace

$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
(...)
- deploy.yml
(...)
namespace: myapp-namespace

# Resulting Kubernetes resources
$ kustomize build
apiVersion: apps/v1
kind: Deployment
metadata:
  (...)
  namespace: myapp-namespace
(...)

Set replicas

$ kustomize edit set replicas myapp=3

$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
(...)
- deploy.yml
(...)
replicas:
- count: 3
  name: myapp
  
# Resulting Kubernetes resources
$ kustomize build
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  (...)
spec:
  replicas: 3

Set containers images

# Original manifest
$ cat deploy.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  (...)
spec:
  (...)
    spec:
      containers:
      - name: myapp
        image: ubuntu
        (...)
        
# Edit images matching ubuntu as original name
# Set the new image name to debian and new tag to latest
$ kustomize edit set image ubuntu=debian:latest

$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
(...)
images:
- name: ubuntu
  newName: debian
  newTag: latest
  
# Resulting Kubernetes resources
$ kustomize build
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  (...)
spec:
  (...)
    spec:
      containers:
      - image: debian:latest
        name: myapp
        (...)

Add configmaps into the kustomization file

# From literals
$ kustomize edit add configmap myapp-literals --from-literal key1=value1 --from-literal key2=value2

# From file
$ cat config.yml 
hello
$ kustomize edit add configmap myapp-files --from-file=config.yml

# From env file
$ cat .env
db_host=127.0.0.1
db_user=myapp
db_name=myapp
$ kustomize edit add configmap myapp-envs --from-env-file=.env

# Resulting kustomization file
$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
(...)
configMapGenerator:
- literals:
  - key1=value1
  - key2=value2
  name: myapp-literals
- files:
  - config.yml
  name: myapp-files
- envs:
  - .env
  name: myapp-envs
  
# Resulting Kubernetes resources
$ kustomize build
apiVersion: v1
data:
  db_host: 127.0.0.1
  db_name: myapp
  db_user: myapp
kind: ConfigMap
metadata:
  name: myapp-envs-kgh2b6hhdc
---
apiVersion: v1
data:
  config.yml: |
    hello
kind: ConfigMap
metadata:
  name: myapp-files-k472b254hd
---
apiVersion: v1
data:
  key1: value1
  key2: value2
kind: ConfigMap
metadata:
  name: myapp-literals-ch29b7t2m8
  • Note that a suffix hash has been added to the name of the resulting Kubernetes resources. To avoid adding that suffix hash, the --disableNameSuffixHash CLI option can be used. Here is an example:
$ kustomize edit add configmap myapp-envs-no-suffix --from-env-file=.env --disableNameSuffixHash

$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configMapGenerator:
(...)
- envs:
  - .env
  name: myapp-envs-no-suffix
  options:
    disableNameSuffixHash: true
    
$ kustomize build
(...)
---
apiVersion: v1
data:
  db_host: 127.0.0.1
  db_name: myapp
  db_user: myapp
kind: ConfigMap
metadata:
  name: myapp-envs-no-suffix

Add secrets into the kustomization file

# From literals
$ kustomize edit add secret myapp-literals --from-literal key1=value1 --from-literal key2=value2

# From files
$ kustomize edit add secret myapp-files-from-dir --from-file=secret-files/*

# From env file
$ cat .env
db_password=superpassword
$ kustomize edit add secret myapp-envs --from-env-file=.env

# Resulting kustomization file
$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
(...)
secretGenerator:
- literals:
  - key1=value1
  - key2=value2
  name: myapp-literals
  type: Opaque
- files:
  - secret-files/secret1
  - secret-files/secret2
  name: myapp-files-from-dir
  type: Opaque
- envs:
  - .env
  name: myapp-envs
  options:
    disableNameSuffixHash: true
  type: Opaque

# Resulting Kubernetes resources
apiVersion: v1
data:
  db_password: c3VwZXJwYXNzd29yZA==
kind: Secret
metadata:
  name: myapp-envs
type: Opaque
---
apiVersion: v1
data:
  secret1: c2VjcmV0MQo=
  secret2: c2VjcmV0Mgo=
kind: Secret
metadata:
  name: myapp-files-from-dir-52tfht2h62
type: Opaque
---
apiVersion: v1
data:
  key1: dmFsdWUx
  key2: dmFsdWUy
kind: Secret
metadata:
  name: myapp-literals-kk658chggc
type: Opaque

Dynamically inject data into configmaps and secrets from env

In real world projects, our Kustomized Kubernetes manifests sources will be put into a source code versioning system in order to ease team work and keep track of changes.

Inside the versioning system, we don't want to put secret values like the app database password. Suppose our app needs specific environment variables in order to connect to its database (host, username, password...).

Here is how we could generate the Kubernetes configmap or secret resource containing required app environment variables whitout setting secret variables values inside manifests sources files:

  • Set the non secret variables key=value pairs inside a file and only set key for secrets:
$ cat .env
db_host=127.0.0.1
db_name=myapp
db_user=myapp
db_password
  • Create a secret or configmap generator Kustomize config using that file as environment variables file:
$ kustomize edit add configmap myapp-envs --from-env-file=.env
  • Set the db_password=<password_value> environment variable before building Kustomized manifests:
$ export db_password=superpassword

This will preferably be accomplished inside a CI/CD pipeline after retrieving the secret from a secret manager or other appropriate places.

  • After building Kustomized manifests, the secret or configmap data will be automatically populated with the previous exported db_password environment variable:
$ kustomize build
apiVersion: v1
data:
  db_host: 127.0.0.1
  db_name: myapp
  db_password: superpassword
  db_user: myapp
kind: ConfigMap
(...)

Deploy Kustomized resources

Here is how we can deploy Kubernetes resources from manifests generated with Kustomize:

# Deploy all resources
$ kubectl apply -k /path/to/kustomized/dir

# Deploy only resources with specific labels
$ kubectl apply -k /path/to/kustomized/dir -l app=myapp

It is also possible to do it like that:

$ kustomize build | kubectl apply -f -

Creating variants using overlays

Have a look at Kustomize announcement overlay exmaple for an overview about Kustomize idea of overlay.

Sign up for more like this.

Enter your email
Subscribe