Kubernetes YAML to rule them all

Declarative manifests written in YAML have become synonymous with Kubernetes. They no longer only deal with standard workloads in Kubernetes, but have branched out to allow managing basically everything using YAMLs and the Kubernetes API. In this post I want to show how Custom Resources and Kubernetes Operators can be used to extend the Kubernetes API and manage everything from complex distributed applications and cloud resources to Kubernetes clusters themselves, all just with Kubernetes YAML manifests.

What are Custom Resources

Kubernetes consists of a multitude of resources. From Secrets, Pods to Ingresses. They all come together to deploy, manage, and run workloads on Kubernetes. All of these types are managed via the Kubernetes API, an HTTP-based API with JSON payloads.

Custom Resources (CR) are a way to extend the Kubernetes API with your own custom types (and therefore endpoints). To do this you write a Custom Resource Definition (CRD), which is a resource in Kubernetes to declare other resources. A CRD defines some meta information (names, API group) and a schema for the custom type. The schema is defined using OpenAPI v3 (sometimes still known under its old name Swagger).

A very simple example:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: foobars.maibornwolff.de
spec:
  group: maibornwolff.de
  names:
    kind: Foobar
    plural: foobars
    singular: foobar
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          spec:
            properties:
              description:
                description: A description for a foobar
                nullable: true
                type: string

This defines a new type/resource Foobar in the API group maibornwolff.de with an API version v1alpha1.

If I apply this manifest just like any other Kubernetes manifest, the new custom resource is created in the API and I can use it afterwards. The following is a valid manifest (called a custom object as it is a specific object of a custom resource) for the new Foobar CR and the API will happily accept it:

apiVersion: maibornwolff.de/v1alpha1
kind: Foobar
metadata:
  name: myfirstfoobar
  namespace: default
spec:
  description: My first foobar

What to use CRs for

A custom resource alone does not have any effect. The Kubernetes control plane will not gain any new logic for such types. It will store them, but will not do anything with them.

This is where Kubernetes Operators come in. They are custom components that automate the management of application-specific operational tasks and integrate with the Kubernetes API for easier control. In essence they codify operational knowledge and processes that classically would be part of manual runbooks or procedures for an operations team.

An example is the HiveMQ Platform Operator. It deploys and manages HiveMQ MQTT Brokers in Kubernetes. Without an operator, this would be a real hassle to automate. To keep a broker available during restarts, configuration changes and upgrades, these need to be done pod by pod and need to wait after each step for the broker to resynchronize. The Operator helps here because it knows how HiveMQ works (the application-specific logic and procedures) and can talk to the broker, so it can properly perform such tasks without human intervention.

And CRs are a way to interface with Operators. The HiveMQ Operator for example has a CR with which I can define things like the broker version to deploy, the resource limits, extensions, configuration, and so on.

By using CRs I can integrate such higher abstractions into my other Kubernetes deployments and manage them just like any other manifest. So if I deploy my Kubernetes workloads with a GitOps approach with FluxCD, I can deploy and manage my HiveMQ broker the same way and just as easily as I can deploy a ConfigMap.

Using CRs for resources outside of Kubernetes

Many Operators are used to abstract and automate workloads inside Kubernetes, like the aforementioned HiveMQ Brokers. But more and more Operators are also used to manage resource that sit outside Kubernetes. For example, databases deployed with cloud providers (AWS RDS, Azure Database for PostgreSQL).

But why would we want to do that?

Anything but trivial workloads running in Kubernetes will have external dependencies. Be it a PostgreSQL database, an S3 bucket for file storage, a Redis instance for caching, or many others. And to reduce the operational burden it makes sense to deploy these components as managed services with a cloud provider like AWS or Azure. In a more traditional setup these resources would be deployed using Terraform with definitions writtn in HCL and provisioned using CI/CD pipelines (e.g. GitHub Actions, Gitlab CI/CD). But this leads to a break in how the application itself (using Kubernetes YAML manifests) is deployed (via GitOps with FluxCD or ArgoCD) vs how its dependencies are deployed. This means more complexity, different languages to learn, harder synchronization and therefore a higher risk of problems.

Deploying Kubernetes Operators that allow to manage external cloud resources via CRs unifies both approaches. It gives application developers just one common interface/API, namely Kubernetes YAML manifests. And instead of mixing deployment approaches, from a developer perspective everything is done via GitOps. It leads to better integration and less complexity, everything required for an application to run (its actual Pods, along with external resources) are all bundled into one GitOps deployment.

In the following sections I want to go into detail on some existing Operators for such purposes.

Crossplane

Crossplane is a project that generically provides CRs to deploy and manage cloud provider resources from Kubernetes. Under the hood it actually uses Terraform, so it can be thought of as a Kubernetes interface for Terraform.

A simple example from the Crossplane Get Started Guide:

apiVersion: s3.aws.m.upbound.io/v1beta1
kind: Bucket
metadata:
  name: crossplane-bucket-example
  namespace: default
spec:
  forProvider:
    region: us-east-2

This provisions an S3 Bucket in AWS. Crossplane does support full lifecycle management. Meaning it will update resources if their Kubernetes manifest changes and it will also delete them in the cloud provider if the custom object is deleted in Kubernetes.

Crossplane is a very powerful tool, and basically brings the power of Terraform to Kubernetes manifests. This can make it quite complex as deploying a managed service in the cloud can involve a multitude of resources.

For this scenario Crossplane offers a feature called Composition, a way to define a set of resources as one entity (you can think of it as an analog to Terraform modules). Platform teams can use this to provide higher level abstractions of cloud resources to their users. Crossplane itself also promotes the Composition feature as a way to create pre-defined bundles of "normal" Kubernetes resources (like Deployment or Service), but I prefer to handle this with other Kubernetes/FluxCD mechanisms like Helm Charts or Kustomizations. And I let Crossplane only deal with external resources.

Crossplane needs some setup effort, but if properly integrated, it can provide a powerful tool for use case teams to easily manage cloud resources.

Hybrid-cloud operators

The hybrid-cloud operators are a series of Kubernetes Operators I developed as open-source software for MaibornWolff a few years ago. They provision and manage cloud resources. But unlike Crossplane they provide a higher abstraction level, and try to abstract between multiple cloud providers. The most mature one of these is the hybrid-cloud-postgresql-operator. The idea is that you just request a PostgreSQL database server via CR. And depending on in which cloud your Kubernetes cluster runs, the Operator will provision a database instance with that provider. So if you run on an AWS EKS cluster, the Operator will provision an AWS RDS instance. And if you run in Azure AKS, the Operator will deploy an Azure Database for PostgreSQL. This allows for multi and hybrid cloud deployments without application developers having to handle the concrete environment. The application just deals with generic PostgreSQL, the Operator takes care of the specifics. The hybrid in the name is aimed at Kubernetes setups running in the cloud and on-premise and the Operator is also capable of deploying self-managed PostgreSQL instances. But I have to admit that feature is still in the prototype stage.

The hybrid-cloud operators were an experiment to provide true multi and hybrid cloud deployments for workloads and allow teams to easily shift their applications from the cloud to on-premise and back. But it turned out that most customer projects stay with just one cloud, so they don't need the multi-cloud abstraction. And moving workloads between cloud and on-premise in most cases needs functional and technical changes on the application side anyway, so providing that abstraction is not that big an advantage. Instead, it makes more sense to use a combination of Crossplane and Kubernetes-native modularization with Helm and Kustomize that have a lower complexity overall.

Cluster API

The idea behind Cluster API (or CAPI for short) is simple: What if we manage Kubernetes clusters via Kubernetes? CAPI provides a set of declarative APIs as Custom Resources to manage the complete lifecycle of Kubernetes clusters.

The actual work is done by Cluster API controllers (Kubernetes Operators) that either deal with infrastructure (like provisioning nodes), managing the control plane or bootstrapping the cluster. Controllers exist for a lot of cloud and on-premise infrastructure providers. From the big cloud providers like AWS and Azure to smaller ones like Hetzner or bare-metal management via Metal3. These controllers run in a separate cluster called a management cluster whose job it is to manage other clusters (called workload clusters).

To define a cluster you need to write a set of YAML manifests (or let clusterctl generate them from standard templates) that define the cluster itself, its infrastructure, its control plane and its nodes. CAPI controllers then take care of provisioning the cluster. They also deal with reconfiguring or upgrading clusters, and can delete them.

Not only can CAPI manifests be deployed and managed using a GitOps approach with FluxCD, CAPI can also be integrated with FluxCD to then provision workloads on newly created clusters. This integration is dead simple: Once a cluster is created, CAPI controllers will store a Kubeconfig (connection information and credentials) for the new cluster in a Kubernetes Secret. FluxCD Kustomizations and HelmReleases can be instructed to perform their work not on the local cluster but in a cluster pointed to by a kubeconfig in a Secret:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo-mycluster
  namespace: default
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: podinfo
  path: "./kustomize"
  kubeConfig:
    secretRef:
      name: mycluster-kubeconfig

This way we can handle the entire lifecycle of our Kubernetes platform using declarative manifests and GitOps: We describe our clusters using Kubernetes resources that we deploy via FluxCD. We then deploy platform services like HiveMQ Brokers using special Operators and FluxCD and then finally deploy our custom applications again using manifests via FluxCD. It's Kubernetes YAML and GitOps all the way down.

One solution that I've been using at my job at MaibornWolff is Giant Swarm. They provide a complete managed Kubernetes platform and fully use CAPI and the GitOps approach with FluxCD. And they nicely encapsulate the CAPI manifests with higher level cluster apps that abstract much of the complexity away.

Conclusion

Kubernetes has moved on from being "just" a container orchestration solution. The seemingly simple concepts Custom Resources and Operators have given way to a landscape where Kubernetes and its declarative approach sit at the heart of platforms. Platforms that not only manage workloads, but also themselves and any external services and dependencies they require. Kubernetes has become a universal integration layer.