K8s Extensibility

Kubernetes Extensibility: Custom Resources, CRDs, and Operators

Kubernetes has become the de facto standard for container orchestration, but its true power lies in its extensibility. This blog post will dive into Kubernetes extensibility, exploring when and why it was introduced, and focusing on key concepts like Custom Resources (CRs), Custom Resource Definitions (CRDs), and Operators.

The Birth of Kubernetes Extensibility

Kubernetes extensibility was introduced early in the project’s life, with the concept of Third Party Resources (TPRs) appearing in Kubernetes 1.2 (March 2016). This was later replaced by Custom Resource Definitions (CRDs) in Kubernetes 1.7 (June 2017).

The need for extensibility arose from the realization that while Kubernetes provided excellent core functionality, different users and organizations had unique requirements that couldn’t be met by the standard Kubernetes API objects alone. Extensibility allows users to define their own API objects and extend Kubernetes to manage these custom resources, effectively turning Kubernetes into a platform for building platforms.

Custom Resources (CRs)

A Custom Resource (CR) is an extension of the Kubernetes API that represents a customized installation or a specific configuration of a Kubernetes object. CRs allow you to store and retrieve structured data in Kubernetes, just like built-in resources such as Pods or Services.

Custom Resource Definitions (CRDs)

A Custom Resource Definition (CRD) is a Kubernetes object that defines a new type of Custom Resource. It tells Kubernetes about the structure and behavior of your custom objects.

Creating a CRD

Here’s an example of creating a simple CRD for a “Database” resource:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                engine:
                  type: string
                version:
                  type: string
                storageSize:
                  type: string
  scope: Namespaced
  names:
    plural: databases
    singular: database
    kind: Database
    shortNames:
    - db

This CRD defines a new resource type called “Database” with properties for engine, version, and storage size.

Operators

An Operator is a method of packaging, deploying, and managing a Kubernetes application. It puts operational knowledge into software to reliably manage an application.

Operators use CRDs to define the desired state of an application and then continuously monitor and adjust the actual state to match the desired state. This mechanism is called operator pattern. More info on this url

Creating an Operator

Creating an Operator typically involves several steps:

1. Define the CRD for your custom resource.
2. Implement the control loop logic (usually in Go).
3. Package the Operator as a container image.
4. Deploy the Operator to your Kubernetes cluster.

Here’s a simple example of an Operator’s main loop in Go:

package main

import (
    "context"
    "fmt"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "sigs.k8s.io/controller-runtime/pkg/client/config"
    "sigs.k8s.io/controller-runtime/pkg/controller"
    "sigs.k8s.io/controller-runtime/pkg/manager"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"
    "sigs.k8s.io/controller-runtime/pkg/source"
)

func main() {
    // Get a config to talk to the apiserver
    cfg, err := config.GetConfig()
    if err != nil {
        log.Fatal(err)
    }

    mgr, err := manager.New(cfg, manager.Options{})
    if err != nil {
        log.Fatal(err)
    }

    // Create a new controller
    c, err := controller.New("database-controller", mgr, controller.Options{
        Reconciler: &reconcileDatabase{client: mgr.GetClient()},
    })
    if err != nil {
        log.Fatal(err)
    }

    // Watch Database objects
    err = c.Watch(&source.Kind{Type: &DatabaseV1{}}, &handler.EnqueueRequestForObject{})
    if err != nil {
        log.Fatal(err)
    }

    // Start the controller
    if err := mgr.Start(signals.SetupSignalHandler()); err != nil {
        log.Fatal(err)
    }
}

type reconcileDatabase struct {
    client client.Client
}

func (r *reconcileDatabase) Reconcile(request reconcile.Request) (reconcile.Result, error) {
    // Fetch the Database instance
    database := &DatabaseV1{}
    err := r.client.Get(context.TODO(), request.NamespacedName, database)
    if err != nil {
        return reconcile.Result{}, client.IgnoreNotFound(err)
    }

    // Your reconciliation logic here
    fmt.Printf("Reconciling Database %s\n", database.Name)

    return reconcile.Result{}, nil
}

This is a basic structure for an Operator that watches for Database custom resources and reconciles them.

Tools for Creating Operators

Several tools can help in creating Operators:

1. Operator SDK: Part of the Operator Framework, it provides tools to build, test, and package Operators.

2. Kubebuilder: A framework for building Kubernetes APIs using CRDs.

3. KUDO (Kubernetes Universal Declarative Operator): Allows you to create Operators using declarative YAML, without writing Go code.

4. Metacontroller: A tool for writing controllers in any language.