Containers

DRY Kubernetes manifests with Kustomize

Stop repeating yourself when creating Kubernetes manifests thanks to Kustomize. What is Kustomize, how it works and what is it capable of ? Let's answer those questions in this post.

Gmk Ziz 👓

26 Dec 2023 • 8 min read

What is Kustomize

Kustomize is a command-line tool that can be used to 'declaratively' transform / customize Kubernetes manifests in a way that help avoid repetitions and ease maintenance and reuse of existing manifests.

It works with plain YAML and native Kubernetes resources. It is available as a standalone binary and also partially integrated into 'kubectl' through 'kubectl kustomize' and 'kubectl apply -k'.

'Kustomize' is easy to learn and get started with compared to templating tools like helm and helmfile. Here is the Kubernetes blog post introducing 'Kustomize': Kustomize announcement.

How Kustomize works

'Kustomize' works with raw Kubernetes manifests that it takes as input
It then transforms the input manifests based on its configuration and output the result
Kustomize configurations are defined inside the 'kustomization.yaml' file
That's where we tell 'Kustomize' which Kubernetes manifests files it should take as input and what transformations should be applied on them
You will find information about the format and syntax of the 'kustomization.yaml' file and detailed explanations on every fields it supports, in the Kustomization file page.

Example: Kustomizing Kubernetes deployment manifest

In this example, we will create two variants of a Kubernetes deployment resource manifest for staging and production environments.

The environments specific manifests will be generated from a common base using 'Kustomize'. That way, we avoid repeating ourselves by duplicating Kubernetes resources manifests.

Example directory tree:

$ tree kustomize-example/
kustomize-example/
├── common
│   ├── deploy.yml
│   └── kustomization.yaml
├── prod
│   └── kustomization.yaml
└── staging
    └── kustomization.yaml

Files contents:

# Common

$ cat common/deploy.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: ubuntu
        resources:
          limits:
            memory: "300Mi"
          requests:
            memory: "300Mi"
            
$ cat common/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml

# Staging

$ cat staging/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../common/
commonLabels:
  env: staging

# Prod

$ cat prod/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../common/
commonLabels:
  env: prod
replicas:
- count: 4
  name: myapp

Resulting Kustomized manifests

# Staging

$ kustomize build staging/
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    env: staging         # <= specific config for staging
  name: myapp
  namespace: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
      env: staging
  template:
    metadata:
      labels:
        app: myapp
        env: staging
    spec:
      containers:
      - image: ubuntu
        name: myapp
        resources:
          limits:
            memory: 300Mi
          requests:
            memory: 300Mi
            
# Prod

$ kustomize build prod/
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    env: prod         # <= specific config for prod
  name: myapp
  namespace: myapp
spec:
  replicas: 4         # <= specific config for prod
  selector:
    matchLabels:
      app: myapp
      env: prod
  template:
    metadata:
      labels:
        app: myapp
        env: prod
    spec:
      containers:
      - image: ubuntu
        name: myapp
        resources:
          limits:
            memory: 300Mi
          requests:
            memory: 300Mi

More about installing 'Kustomize' and easily cutomizing Kubernetes resources manifests using in next sections.

Install Kustomize

Sometimes, it can be useful to install 'Kustomize' for development as some of its useful CLI features are not available with the 'kubectl kustomize' command at the time of writing
'Kustomize' releases assets can be found here
To install 'Kustomize' on Linux, do the following:

$ kustomize_version=v5.4.2 # choose version
$ linux_arch=amd64 # choose OS arch

# Get Kustomize binary
$ wget https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${kustomize_version}/kustomize_${kustomize_version}_linux_${linux_arch}.tar.gz

# Install Kustomize binary
$ sudo tar xzvf kustomize_${kustomize_version}_linux_${linux_arch}.tar.gz -C /usr/local/bin/

# Verify
$ kustomize version

Create and update kustomization file

To get usage help and examples for any 'Kustomize' CLI command, make the commands followed by '-h' or '--help'
Use the CLI or directly update the 'kustomization.yaml' file with the desired configurations settings

Initialize the kustomization file

This can be useful to quickly initialize a 'kustomization.yaml' file with autodetected YAML files from the current directory, added as 'Kustomize' configuration resources.

$ tree k8s-resources/
k8s-resources/
├── deploy.yml
├── vpa.yml
├── ingress.yml
└── service.yml

$ kustomize create --autodetect

$ cat kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- vpa.yml
- ingress.yml
- service.yml

Add or remove resources from the kustomization file

k8s-kustomization-resources for more

# Remove resource
$ kustomize edit remove resource vpa.yml

$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- service.yml
- ingress.ym

# Add resource after creating the hpa.yml manifest
$ kustomize edit add resource hpa.yml

$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- service.yml
- ingress.yml
- hpa.yml

Add or remove common labels and annotations from the kustomization file

k8s-kustomization-commonlabels and k8s-kustomization-commonannotations for more

# Add labels and annotations
# Multiple labels and annotations values are separeted by whitespace
$ kustomize edit add label app:myapp env:staging
$ kustomize edit add annotation myapp/version:0.0.1

$ cat kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- service.yml
- ingress.yml
- hpa.yml
commonAnnotations:
  myapp/version: 0.0.1
commonLabels:
  app: myapp
  env: staging
  
# Remove labels and annotations
# Only use the keys to remove labels or annotations
# Multiple keys values are separeted by comma
$ kustomize edit remove label app,env
$ kustomize edit remove annotation myapp/version

$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- service.yml
- ingress.yml
- hpa.yml

Add or remove patches from the kustomization file

k8s-kustomization-patches for more

# Add patches
$ kustomize edit add patch --name myapp --kind Deployment --group apps --version v1 --path deploy-patch.yml

$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- service.yml
- ingress.yml
- hpa.yml
patches:
- path: deploy-patch.yml
  target:
    group: apps
    kind: Deployment
    name: myapp
    version: v1
 
# Remove patches
$ kustomize edit remove patch --name myapp --kind Deployment --group apps --version v1 --path deploy-patch.yml

$ cat kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
- service.yml
- ingress.yml
- hpa.yml

As seen previously, 'Kustomize' patches configuration has a 'target' field that uniquely identifies the resource we want to patch (using at least one of the following resources elements: 'group', 'version', 'kind', 'name', 'namespace, 'labelSelector' and 'annotationSelector') and a 'path' field where we specify the path (relative to the 'kustomization.yaml' file) to the file containing the patches. The patches files contents can be written using two formats that are shown below.

Patches using the json6902 standard

Kustomize patchesJson6902

# Syntax
- op: <operation type> # Possible values: add, remove, replace
  path: <path to the field to modified on the resource>
  value: <value we want for the field selected with path>

# Examples

# Patching container image
# File: deploy-patch.yml
- op: replace
  path: /spec/template/spec/containers/0/image
  value: nginx:1.25.3

Patches using the strategic merge standard

Kustomize patchesStrategicMerge

In this case, the 'target' field from 'Kustomize' patches configuration can be omitted
The target resource is matched using the 'apiVersion', 'kind' and 'metadata.name' fields from the patches files

# Syntax

apiVersion: <apiVersion of the resource>
kind: <kind of the resource>
metadata:
  name: <name of the resource>
spec:
  <patches>

# Examples

# Patching container name and image on a Deployment named nginx
# File: deploy-patch.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  template:
    spec:
      containers:
        - name: nginx
          image: nginx:1.25.3

Set prefix and suffix for resources names

k8s-kustomization-nameprefix and k8s-kustomization-namesuffix for more

$ kustomize edit set nameprefix -- prefix-
$ kustomize edit set namesuffix -- -suffix

$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
(...)
- deploy.yml
(...)
namePrefix: prefix-
nameSuffix: -suffix

# Resulting Kubernetes resources
$ kustomize build
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefix-myapp-suffix
(...)

Set resources namespace

k8s-kustomization-namespace for more

$ kustomize edit set namespace myapp-namespace

$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
(...)
- deploy.yml
(...)
namespace: myapp-namespace

# Resulting Kubernetes resources
$ kustomize build
apiVersion: apps/v1
kind: Deployment
metadata:
  (...)
  namespace: myapp-namespace
(...)

Set replicas

k8s-kustomization-replicas for more

$ kustomize edit set replicas myapp=3

$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
(...)
- deploy.yml
(...)
replicas:
- count: 3
  name: myapp
  
# Resulting Kubernetes resources
$ kustomize build
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  (...)
spec:
  replicas: 3

Set containers images

k8s-kustomization-images for more

# Original manifest
$ cat deploy.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  (...)
spec:
  (...)
    spec:
      containers:
      - name: myapp
        image: ubuntu
        (...)
        
# Edit images matching ubuntu as original name
# Set the new image name to debian and new tag to latest
$ kustomize edit set image ubuntu=debian:latest

$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deploy.yml
(...)
images:
- name: ubuntu
  newName: debian
  newTag: latest
  
# Resulting Kubernetes resources
$ kustomize build
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  (...)
spec:
  (...)
    spec:
      containers:
      - image: debian:latest
        name: myapp
        (...)

Add configmaps into the kustomization file

k8s-kustomization-configmapgenerator and k8s-kustomization-generatoroptions for more

# From literals
$ kustomize edit add configmap myapp-literals --from-literal key1=value1 --from-literal key2=value2

# From file
$ cat config.yml 
hello
$ kustomize edit add configmap myapp-files --from-file=config.yml

# From env file
$ cat .env
db_host=127.0.0.1
db_user=myapp
db_name=myapp
$ kustomize edit add configmap myapp-envs --from-env-file=.env

# Resulting kustomization file
$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
(...)
configMapGenerator:
- literals:
  - key1=value1
  - key2=value2
  name: myapp-literals
- files:
  - config.yml
  name: myapp-files
- envs:
  - .env
  name: myapp-envs
  
# Resulting Kubernetes resources
$ kustomize build
apiVersion: v1
data:
  db_host: 127.0.0.1
  db_name: myapp
  db_user: myapp
kind: ConfigMap
metadata:
  name: myapp-envs-kgh2b6hhdc
---
apiVersion: v1
data:
  config.yml: |
    hello
kind: ConfigMap
metadata:
  name: myapp-files-k472b254hd
---
apiVersion: v1
data:
  key1: value1
  key2: value2
kind: ConfigMap
metadata:
  name: myapp-literals-ch29b7t2m8

Note that a suffix hash has been added to the name of the resulting Kubernetes resources. To avoid adding that suffix hash, the '--disableNameSuffixHash' CLI option can be used. Here is an example:

$ kustomize edit add configmap myapp-envs-no-suffix --from-env-file=.env --disableNameSuffixHash

$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configMapGenerator:
(...)
- envs:
  - .env
  name: myapp-envs-no-suffix
  options:
    disableNameSuffixHash: true
    
$ kustomize build
(...)
---
apiVersion: v1
data:
  db_host: 127.0.0.1
  db_name: myapp
  db_user: myapp
kind: ConfigMap
metadata:
  name: myapp-envs-no-suffix

Add secrets into the kustomization file

k8s-kustomization-secretgenerator and k8s-kustomization-generatoroptions for more

# From literals
$ kustomize edit add secret myapp-literals --from-literal key1=value1 --from-literal key2=value2

# From files
$ kustomize edit add secret myapp-files-from-dir --from-file=secret-files/*

# From env file
$ cat .env
db_password=superpassword
$ kustomize edit add secret myapp-envs --from-env-file=.env

# Resulting kustomization file
$ cat kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
(...)
secretGenerator:
- literals:
  - key1=value1
  - key2=value2
  name: myapp-literals
  type: Opaque
- files:
  - secret-files/secret1
  - secret-files/secret2
  name: myapp-files-from-dir
  type: Opaque
- envs:
  - .env
  name: myapp-envs
  options:
    disableNameSuffixHash: true
  type: Opaque

# Resulting Kubernetes resources
apiVersion: v1
data:
  db_password: c3VwZXJwYXNzd29yZA==
kind: Secret
metadata:
  name: myapp-envs
type: Opaque
---
apiVersion: v1
data:
  secret1: c2VjcmV0MQo=
  secret2: c2VjcmV0Mgo=
kind: Secret
metadata:
  name: myapp-files-from-dir-52tfht2h62
type: Opaque
---
apiVersion: v1
data:
  key1: dmFsdWUx
  key2: dmFsdWUy
kind: Secret
metadata:
  name: myapp-literals-kk658chggc
type: Opaque

Dynamically inject data into configmaps and secrets from env

In real world projects, our 'Kustomized' Kubernetes manifests sources will be put into a source code versioning system (like Git) in order to ease team work and track changes.

Inside the versioning system, we don't want to put secret values like the app database password. Suppose our app needs specific environment variables in order to connect to its database (host, username, password...).

Here is how we could generate the Kubernetes 'configmap' or 'secret' resource containing the required app environment variables without setting secret variables values inside manifests sources files:

Set the non secret variables 'key=value' pairs inside a file and only set the 'key' for secrets:

$ cat .env
db_host=127.0.0.1
db_name=myapp
db_user=myapp
db_password

Create a 'Kustomize' secret or configmap generator configuration using that file as an environment variables file:

$ kustomize edit add configmap myapp-envs --from-env-file=.env

Set the db_password=<password_value> environment variable before building 'Kustomized' manifests:

$ export db_password=superpassword

This will preferably be accomplished inside a CI/CD pipeline after retrieving the secret from a secret manager (like Vault) or other appropriate places.

After building 'Kustomized' manifests, the 'secret' or 'configmap' data will be automatically populated with the previously exported 'db_password' environment variable:

$ kustomize build
apiVersion: v1
data:
  db_host: 127.0.0.1
  db_name: myapp
  db_password: superpassword
  db_user: myapp
kind: ConfigMap
(...)

Deploy resources managed with Kustomize

Here is how we can deploy Kubernetes resources from manifests generated with 'Kustomize':

# Deploy all resources
$ kubectl apply -k /path/to/kustomized/dir

# Deploy only resources with specific labels
$ kubectl apply -k /path/to/kustomized/dir -l app=myapp

It is also possible to do it like this:

$ kustomize build | kubectl apply -f -

Creating variants using overlays

For an overview about the 'Kustomize' idea of 'overlays', have a look at Kustomize overlays example.