I wrote my first Kubernetes operator maybe four years ago. That CRD has gone through three API versions (v1alpha1, v1beta1, v1) and two field-rename migrations. Here is what I got wrong and how I would do it differently.

Mistake 1: putting runtime state in spec

My first CRD had spec.desiredReplicas and spec.currentReplicas. Yes, I know, obvious. The operator would reconcile and update spec.currentReplicas to match actual replicas. Any user editing the YAML saw both fields, got confused, and sometimes tried to change currentReplicas to scale the thing (which of course did nothing because the reconciler would just overwrite it).

The fix is boring. Runtime state goes in .status. A reconciler never writes to .spec of the resource it is managing. I now follow the rule that if field X is something the operator observes or derives, it must be in .status. If field X is something the user asserts, it must be in .spec.

There is a second subtle reason: writing to .status goes through the status subresource, which has separate RBAC and doesn’t bump .metadata.generation. This means watchers that trigger only on generation changes (which is usually what you want) don’t fire on status updates. Much less thrashing. I wrote about this in see my post on an operator reconcile loop that wouldn’t quit.

Mistake 2: over-nested spec

I had a spec that looked like:

spec:
  config:
    networking:
      service:
        ports:
          - name: http
            port: 80

Five levels deep. When users wrote this, they got indentation wrong. When I updated defaults, the path was .spec.config.networking.service.ports, which is painful for JSON patch. When I needed to deprecate config.networking, I had to migrate data through three nested objects.

Now I flatten aggressively. One level of nesting at most, unless there is a strong reason (e.g., a list of things that each have their own fields). The spec is shorter, the defaulting is cleaner, and the migration path is simpler.

Mistake 3: mixing required and optional without discipline

Early on, I had optional fields with default values baked into the controller. Users would omit the field, the controller would apply its default, and on the next reconcile the status would show the defaulted value. But the spec did not. So kubectl get -o yaml did not match what was happening. Confusing.

Now I use OpenAPI default: in the schema. Kubernetes fills in the defaults at admission time, so the spec reflects the effective value. This is also pretty helpful for GitOps, because the stored manifest has no surprises:

spec:
  replicas:
    type: integer
    default: 1
    minimum: 0
    maximum: 10

Mistake 4: enum as string is better than enum as struct

I had a field mode that started as a string with schedule|online|drain values. Later I wanted schedule to take an additional parameter. I “cleverly” turned it into:

mode:
  schedule:
    interval: 1h

This was an API change. Now users had:

mode:
  type: object
  properties:
    schedule:
      type: object
      properties:
        interval: { type: string }
    online: { type: object }
    drain: { type: object }

If I had started with a discriminator pattern (a field modeKind plus a sibling field with the parameters for the active kind), migration would have been one-step. Lesson: plan for additional parameters on enum-valued fields.

Mistake 5: status without condition stanza

My early status was a flat list of fields: ready, message, lastTransitionTime, observedGeneration. It worked. Then I wanted to report that Reconcile was in progress but HealthCheck had failed, and I did not have a multi-dimensional way to express that.

The idiomatic pattern is a list of conditions:

status:
  conditions:
    - type: Reconciled
      status: "True"
      reason: ReconcileSucceeded
      lastTransitionTime: 2024-11-03T18:00:00Z
    - type: Ready
      status: "False"
      reason: HealthCheckFailed
      message: upstream returned 503

controller-runtime has meta.SetStatusCondition as a helper. Use it. Every k8s-native tool knows how to render conditions. Your users get a uniform experience.

Mistake 6: printer columns as an afterthought

When I rolled out v1, I did not set additionalPrinterColumns. kubectl get things showed only NAME and AGE. When I added conditions, users still had to do -o yaml to see status. Now I set columns explicitly:

additionalPrinterColumns:
  - name: Ready
    type: string
    jsonPath: .status.conditions[?(@.type=="Ready")].status
  - name: Phase
    type: string
    jsonPath: .status.phase
  - name: Age
    type: date
    jsonPath: .metadata.creationTimestamp

Two lines of yaml per column, enormous quality-of-life win.

Mistake 7: not versioning from day one

I shipped v1 as v1 because “this is simple, I do not need alpha/beta”. Two months later I needed to change a field. Migration was disruptive because everything was v1 and everything expected v1 forever.

Now I ship v1alpha1 when there is any chance the schema might evolve. Users can still run it in dev; the contract is just weaker. When the API stabilizes, promote to v1beta1 and set up conversion webhooks. The conversion webhook work is not free, but it is far cheaper than a forced in-place migration.

Reflection

Most of these mistakes come from wanting to be clever or wanting to ship the first version fast. The durable patterns are:

  • Spec is what the user asserts. Status is what the operator observes.
  • Conditions for status. Nothing bespoke.
  • OpenAPI defaults, not controller defaults.
  • Printer columns from day one.
  • v1alpha1 first, promote when you are confident.

I wish I had a mentor explaining this to me four years ago. Write CRDs like you will have to migrate them twice, because you will.

Related: see my post on an admission webhook that crashed my cluster for another “Kubernetes extension points: also footguns”.