Sign in Subscribe
Containers

Understanding secrets in Kubernetes

Highlights some essential Kubernetes knowledges that will make us comfortable when dealing with secret objects for our applications workloads.

Gmk Ziz 👓

• 6 min read
Understanding secrets in Kubernetes

Overview

  • Kubernetes 'secret' object can be used to store sensitive informations
  • The sensitive informations could then be securely transferred to pods or directly used by other Kubernetes resources
  • Sensitive informations inside 'secret' object are by default 'base64 encoded'
  • For other infos and cautions see Kubernetes Secrets

Secret types

Asking for help on 'secret' object creation shows 3 types of secrets:

$ kubectl create secret -h
Create a secret using specified subcommand.

Available Commands:
  docker-registry   Create a secret for use with a Docker registry
  generic           Create a secret from a local file, directory, or literal
value
  tls               Create a TLS secret
  • We use the 'docker-registry' secret type to create secrets for authenticating with container registries from different providers, not only Docker. A typical use case of this type of secret is pulling/pushing private container images from/to container registries. To create a secret of that type, you need:
    • the address of the container registry
    • the name of the authenticating user
    • the associated password for the authenticating user
  • The 'generic' secret type will be used to store 'key:value' pair data. The data could then be retrieved inside pods as files or environment variables
  • The 'tls' secret type will be used to store TLS certificates and keys. The certificate and associated private key data must be 'PEM' encoded. Could be used for instance with ingress controllers to make our apps accessible in HTTPS

Creating secrets

Docker-registry

Here is the creation command:

$ kubectl create secret docker-registry my-docker-registry-secret --docker-server=my-registry.example.com --docker-username=myuser --docker-password=mypassword

Let's have a look at the created secret:

$ kubectl get secret my-docker-registry-secret
NAME                        TYPE                             DATA   AGE
my-docker-registry-secret   kubernetes.io/dockerconfigjson   1      12s

$ kubectl get secret my-docker-registry-secret -o yaml
apiVersion: v1
data:
  .dockerconfigjson: eyJhdXRocyI6eyJteS1yZWdpc3RyeS5leGFtcGxlLmNvbSI6eyJ1c2VybmFtZSI6Im15dXNlciIsInBhc3N3b3JkIjoibXlwYXNzd29yZCIsImF1dGgiOiJiWGwxYzJWeU9tMTVjR0Z6YzNkdmNtUT0ifX19
kind: Secret
metadata:
  creationTimestamp: "****-**-**T**:**:**Z"
  name: my-docker-registry-secret
  namespace: juiceshop
  resourceVersion: "546844"
  uid: fd824b18-aacb-4341-a056-c3beb24b1162
type: kubernetes.io/dockerconfigjson

Let's see the 'base64' decoded value of the '.dockerconfigjson' data key:

$ echo "eyJhdXRocyI6eyJteS1yZWdpc3RyeS5leGFtcGxlLmNvbSI6eyJ1c2VybmFtZSI6Im15dXNlciIsInBhc3N3b3JkIjoibXlwYXNzd29yZCIsImF1dGgiOiJiWGwxYzJWeU9tMTVjR0Z6YzNkdmNtUT0ifX19" | base64 -d | jq
{
  "auths": {
    "my-registry.example.com": {
      "username": "myuser",
      "password": "mypassword",
      "auth": "bXl1c2VyOm15cGFzc3dvcmQ="
    }
  }
}
The Google Container Registry case

For the Google Container Registry, we can't generate a couple username/password for authentication. Instead, we have to create a Google 'service account' and give that 'service account' the permission to read data from GCS (Google Cloud Storage).

We can then use that 'service account' token to create the 'docker-registry' secret. Assuming that the '~/gcr-image-pull-token.json' file contains the 'service account' token, we will use the following values for the registry username and password within the 'docker-registry' secret creation command:

  • --docker-username=_json_key
  • --docker-password="$(cat ~/gcr-image-pull-token.json)"

TLS

Here is the creation command:

kubectl create secret tls my-tls-secret --key=tls-key.pem --cert=tls-cert.pem

Here is a truncated output of the content of the TLS certificate and private key files we used:

  • tls-cert.pem
-----BEGIN CERTIFICATE-----
MIIDzDCCArSgAwIBAgIUOiPOFvdtyD/wOCrgaoGNsV6CuIEwDQYJKoZIhvcNAQEL
BQAwYTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM
(...)
jJBYxC4Hi6ZNnb2/UUliumVmmQZvLLcyNK+rcbGGPtP0wB+QoXhDUrKAsrM/McTK
f1IAy/XYLdk9UUcLxpN2Iw==
-----END CERTIFICATE-----
  • tls-key.pem
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA2OmXD24qOvorrGqB3GOdeNdHF1v6QIb3CSA4tNfqH3fqQ22F
iEsXHt8s2dfHlSqwZ4J5bwwzjp9eXqwwNRPDwWIaFzesWh1qsWxJIvh9Z2b/9TUD
(...)
5C9jvJzMZRAqQ2dWd0yO00a6kf/Uyi/dzVpx96iwf0nwj0R0dEndudHkH0awBAwi
JZvsLB7gYCceCXs+S/OROtFh0322pBTHwmBTSyUARRF2VOfC3bQj
-----END RSA PRIVATE KEY-----

Now let's have a look at the created secret:

$ kubectl get secret my-tls-secret
NAME            TYPE                DATA   AGE
my-tls-secret   kubernetes.io/tls   2      12m

$ kubectl get secret my-tls-secret -o yaml
apiVersion: v1
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR6RENDQXJTZ0F3SUJBZ0lVT2lQT0Z2ZHR5RC93T0
  (...truncated)
hoRFVyS0Fzck0vTWNUSwpmMUlBeS9YWUxkazlVVWNMeHBOMkl3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBMk9tWEQyNHFPdm9yck
  (...truncated)
Z1lDY2VDWHMrUy9PUk90RmgwMzIycEJUSHdtQlRTeVVBUlJGMlZPZkMzYlFqCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==       
kind: Secret
metadata:
  creationTimestamp: "****-**-**T**:**:**Z"
  name: my-tls-secret
  namespace: juiceshop
  resourceVersion: "651996"
  uid: b5913af1-5fab-45b2-be2d-252b7a829ea7
type: kubernetes.io/tls

Generic

The 'kubectl create secret generic -h' command shows examples for creating generic secrets. Let's try some of them and see the result.

  • Create a new secret named 'my-secret' with 'key1=supersecret' and 'key2=topsecret'
$ kubectl create secret generic my-secret --from-literal=key1=supersecret --from-literal=key2=topsecret
secret/my-secret created

$ kubectl get secret my-secret -o yaml
apiVersion: v1
data:
  key1: c3VwZXJzZWNyZXQ=
  key2: dG9wc2VjcmV0
kind: Secret
metadata:
  creationTimestamp: "****-**-**T**:**:**Z"
  name: my-secret
  namespace: juiceshop
  resourceVersion: "655183"
  uid: 4bec3a1f-f7b9-4d5f-bd5e-f218e94a218f
type: Opaque
  • Create a new secret named 'my-secret' using a combination of a file and a literal
$ cat bar/db_password
superpass

$ kubectl create secret generic my-secret --from-file=bar/db_password --from-literal=passphrase=topsecret
secret/my-secret created

$ kubectl get secret my-secret -o yaml
apiVersion: v1
data:
  db_password: c3VwZXJwYXNzCg==
  passphrase: dG9wc2VjcmV0
kind: Secret
metadata:
  creationTimestamp: "****-**-**T**:**:**Z"
  name: my-secret
  namespace: juiceshop
  resourceVersion: "655590"
  uid: 5fe8546b-489c-4986-a711-69a4155e7f11
type: Opaque
  • Create a new secret named 'my-secret' with specified key name. From the previous example, we will set the key name to 'database_password' instead of the default 'db_password' taken from the filename
$ cat bar/db_password
superpass

$ kubectl create secret generic my-secret --from file=database_password=bar/db_password
secret/my-secret created

$ kubectl get secret my-secret -o yaml
apiVersion: v1
data:
  database_password: c3VwZXJwYXNzCg==
kind: Secret
metadata:
  creationTimestamp: "****-**-**T**:**:**Z"
  name: my-secret
  namespace: juiceshop
  resourceVersion: "655654"
  uid: 362c9267-7e35-4101-a659-ae7e56c0e420
type: Opaque
  • Create a new secret named 'my-secret' from a file containing a bunch of lines with 'key=value' pairs
$ cat app.env 
db_user=user
db_password=superpass

$ kubectl create secret generic my-secret --from-env-file=app.env
secret/my-secret created

$ kubectl get secret my-secret -o yaml
apiVersion: v1
data:
  db_password: c3VwZXJwYXNz
  db_user: dXNlcg==
kind: Secret
metadata:
  creationTimestamp: "****-**-**T**:**:**Z"
  name: my-secret
  namespace: juiceshop
  resourceVersion: "653254"
  uid: 0297bafd-fffa-4af2-adbf-97cd3147de1f
type: Opaque
  • Create a new secret named 'my-secret' with keys taken from each filename in folder bar and values from the associated files contents
$ ls bar/
db_password  db_user

$ cat bar/db_user 
user

$ cat bar/db_password 
superpass

$ kubectl create secret generic my-secret --from-file=bar/
secret/my-secret created

$ kubectl get secret my-secret -o yaml
apiVersion: v1
data:
  db_password: c3VwZXJwYXNzCg==
  db_user: dXNlcgo=
kind: Secret
metadata:
  creationTimestamp: "****-**-**T**:**:**Z"
  name: my-secret
  namespace: juiceshop
  resourceVersion: "654889"
  uid: ea7d7829-240c-4cbe-a3ec-9bb6750377ea
type: Opaque

Using secrets

From pods as file

Here is how we make existing Kubernetes secrets available inside pods as files:

  • In our Kubernetes manifests used to create pods, we need to declare a 'volume' inside pods 'spec' section, that will be supplied with one or many of our existing secrets
  • According to Kubernetes documentation, volumes containing secrets are backed by tmpfs (a RAM-backed filesystem) so they are never written to non-volatile storage
  • We then have to declare a 'volumeMounts' inside pods 'spec.containers' section, to mount the secrets as files to specific paths inside the pods containers. Here is an example:
apiVersion: apps/v1
kind: Deployment
(...)
spec:
  (...)
  template:
   (...)
    spec:
      containers:
      - name: db
        image: mariadb
        env:
          (...)
          - name: MYSQL_PASSWORD_FILE
            value: "/run/secrets/mysql/user_password"
          - name: MYSQL_ROOT_PASSWORD_FILE
            value: "/run/secrets/mysql/root_password"
        volumeMounts:
        (...)
        - name: secret-volume
          mountPath: /run/secrets/mysql
        (...)
      volumes:
      (...)
      - name: secret-volume
        secret:
          secretName: mysql-secret
  • The data from the existing Kubernetes secret named 'mysql-secret' will be made available to pods into the '/run/secrets/mysql' directory
  • All data keys from the 'mysql-secret' secret will be name of files inside the '/run/secrets/mysql' directory and the associated key values the files contents

Here is an example. The following key/value pairs data from the 'mysql-secret' secret:

user_password: O3hndfn!dSZu
root_password: kHj483iYh?d2

will be available inside the pods containers as the following files:

  • /run/secrets/mysql/user_password containning O3hndfn!dSZu
  • /run/secrets/mysql/root_password containning kHj483iYh?d2

We could also specify a specific file path from where we want a specific secret data to be available. That path is relative to the path specified inside the 'mountPath' field for the secret volume ('/run/secrets/mysql' in this case).

Here is an example. If we wanted to make the 'user_password' secret available from '/run/secrets/mysql/other_dir/other_user_password_filename' instead of '/run/secrets/mysql/user_password', we would have used:

        (...)
        volumeMounts:
        - name: secret-volume
          mountPath: /run/secrets/mysql
        (...)
      volumes:
      - name: secret-volume
        secret:
          secretName: mysql-secret
          items:
          - key: user_password
            path: other_dir/other_user_password_filename

From pods as environment variables

  • The Kubernetes generic secret named 'mysecret' exists and has been created with the '--from-env-file' option of the 'kubectl create secret generic command, from the following file content:
username=myusername
password=mypassword
  • The following Kubernetes 'Deployment' ressource manifest will make 'username' and 'password' data from 'mysecret' available to pod containers as environment variables:
apiVersion: apps/v1
kind: Deployment
(...)
spec:
  (...)
  template:
   (...)
    spec:
      containers:
      - name: my-container-name
        image: my-container-image
        env:
          - name: USERNAME
            valueFrom:
              secretKeyRef:
                name: mysecret
                key: username
          - name: PASSWORD
            valueFrom:
              secretKeyRef:
                name: mysecret
                key: password
  • The created environment variables inside the pod containers are:
USERNAME=myusername
PASSWORD=mypassword

To pull private container images

Simply use the 'spec.imagePullSecrets' to specify the name of the 'docker-registry' secret to use for authentication when pulling private container images defined inside Kubernetes manifests files. Here is an example:

apiVersion: apps/v1
kind: Deployment
(...)
spec:
  (...)
  template:
   (...)
    spec:
      imagePullSecrets:
        - name: my-docker-registry-secret
      containers:
      - name: my-container-name
        image: myregistry.example.com/my-private-container-image
        (...)

Sign up for more like this.

Enter your email
Subscribe