Exposing Kubernetes apps with ingress-gce in GKE

Overview

When working with Google Kubernetes Engine (GKE), we don't neccessarily have to deploy a custom 'ingress controller' to expose our apps. There is already one called ingress-gce that is running on the cluster master nodes.

As we don't have any control over a GKE cluster master nodes, we can't see what is running on them, and therefore won't see the ingress-gce pods.

When we create ingess objects without specific configurations indicating for which 'ingress controller' they are created for, the ingress-gce will inspect them and apply the rules they define.

Be aware that each time you create an ingress object that is successfully processed by the ingress-gce, a Google Cloud HTTP(S) loadbalancer (external to the cluster) is also created.

HTTP(S) traffic arriving to that loadbalancer will then be automatically forwarded to the ingress-gce pods running inside the cluster.

When an HTTP(S) request Host header value (for instance 'www.hackerstack.org') matches againts an ingress rule processed by the ingress-gce pods, that HTTP(S) request is forwarded to one of the cluster internal services as defined by the ingress rule.

Since our apps pods are reachable through those cluster internal services, they will then be reached from the outside of the cluster through the external HTTP(S) loadbalancer.

Creating the ingress object

Here is the manifest for creating an 'ingress' object that is meant to be processed by the ingress-gce 'ingress controller' in GKE:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  namespace: myapp
  annotations:
    # Use an already existing global static IP as the IP address of the HTTP/HTTPS
    # loadbalancer that will be created for this ingress object
    kubernetes.io/ingress.global-static-ip-name: "my-static-public-ip"
    # Use an already existing managed certificate for rules that use TLS
    networking.gke.io/managed-certificates: "my-managed-certificate"
    # Use the frontendconfig called 'redirect-http-to-https' to redirect http to https
    networking.gke.io/v1beta1.FrontendConfig: "redirect-http-to-https" 
spec:
  rules:
  - host: myapp.example.local
    http:
      paths:
      - backend:
          service:
            name: myapp
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
    secretName: "my-managed-certificate"

• We will see how to create the managed TLS certificate and frontendconfig used inside the 'ingress' object manifest next, and explain few things about them
• Regarding the global static IP address, here is the documentation for creating it: gcp-static-external-ip-address-reservation
• Here is a list of the 'ingress' object related annotations we can use inside the manifest for additional features gce-ingress-object-annotations
• For infos about all available fields we can use inside the 'ingress' object manifest have a look at kubernetes-api-ref-ingress-v1
• As you can see inside the 'ingress' object manifest, we defined a rule for HTTP(S) requests with Host header corresponding to 'myapp.example.local'. The 'backend' section defines a cluster internal service name and port on which the requests matching the rule will be forwarded.
• In order for this to work properly, the backend service must be of type 'LoadBalancer'. Next we will show the manifest for creating that backend service.

Creating the backend service

• The backend service on which the ingress-gce will route eligible HTTP(S) traffic must be of type 'LoadBalancer' when using GKE (can’t be of type 'ClusterIP' for instance, which is supported by other 'ingress controllers'.

apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: myapp
  annotations:
    cloud.google.com/load-balancer-type: "Internal"
spec:
  selector:
    app: myapp
  ports:
    - name: http
      port: 80           # port on which the service will be reachable
      targetPort: 8080   # backend pods port on which the associated service will forward resquests 
  type: LoadBalancer

• Here is a list of GKE gce-ingress related annotations we can use inside the backend service manifest for additional features: gke-ingress-gce-backend-service-annotation

Creating the managed certificate

Here is the manifest we use to create the managed TLS certificate for the public domain name from which our Kubernetes apps will be reached:

apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: "my-managed-certificate"
  namespace: myapp
spec:
  domains:
    - mydomain.example.local

Before creating the managed certificate object, we have to make sure the the domain name for which we are generating the TLS certificate point to the public IP address (the global static IP) of the cluster external HTTP(S) load balancer we have previously mentionned.

Once the pre-requisite met, the TLS certificate will be generated and available for use by the HTTP(S) load balancer. It will be automatically renewed before expiration.

Frontendconfig and backendconfig

Frontendconfig and backendconfig are custom resources available only in GKE. They can be used to enable additional features when using ingress-gce (the default 'ingress controller' in GKE).

Frontendconfig

When we enable some frontendconfig features, they are available for all the backend services defined inside the 'ingress' objects and therefore for all applications reachable through those backend services.

Here is an example manifest we use to create frontendconfig resources:

apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata:
  name: redirect-http-to-https
  namespace: myapp
spec:
  redirectToHttps:
    enabled: true
    responseCodeName: "302"

This frontendconfig, once indicated (we will see how next) inside the manifest of an 'ingress' object that is dedicated to be used by the ingress-gce 'ingress controller', will enable HTTP to HTTPS redirection for all hosts specified inside that ingress object rules. This will automatically create a rule for the cluster external HTTP(S) load balancer associated to the ingress-gce 'ingress controller' in order to make the redirection.

In addition to HTTP to HTTPS redirection, there are others 'ingress' features that can be enabled through frontendconfig resources. For a complete list, have a look at the 'FrontendConfig' section in gke-gce-ingress-features.

To use an existing frontendconfig resource to enable specific features, we specify the name of that frontendconfig inside the metadata.annotations field of the 'ingress' manifest as follows:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  (...)
  annotations:
    (...)
    # Redirect HTTP to HTTPS
    networking.gke.io/v1beta1.FrontendConfig: "redirect-http-to-https" 
    (...)
spec:
  rules:
  - host: myapp.example.local
    (...)
  tls:
    secretName: "my-managed-certificate"

Backendconfig

When we enable some backendconfig features, they are available for specific backend services we choose and therefore available only for applications reachable through the chosen backend services.

Here is an example manifest we use to create backendconfig resources:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  namespace: myapp
  name: my-cloud-armor-rules
spec:
  securityPolicy:
    name: "cloud-armor-default" # an already existing cloud armor security policy

This backendconfig, once used by one or more of the 'ingress' resources backend services, will enable an existing Google Cloud Armor security policy on those backend services, for instance to restrict access only to specific IP addresses or block some well known web applications attacks.

In addition to enabling Google Cloud Armor features for specific 'ingress' backends, there are others 'ingress' features that can be enabled through backenconfig resources. For a complete list, have a look at the 'BackendConfig' section in gke-gce-ingress-features.

To use an existing backendconfig resource to enable specific features for chosen ingress backend services, we specify the name of that backendconfig inside the metadata.annotations field of the chosen backend services manifests as follows:

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"default": "my-cloud-armor-rules"}'
  (...)

That's all. We now know how to publicly expose our Kubernetes applications in GKE using the default 'ingress controller' and use frontendconfig and backendconfig custom resources to enable additional features.