Observability

Enough formulas to write real alerts without spending a weekend in a textbook.

Structured logging is not the hard part. The hard part is everything around it.

I conflated these for years. Here is a cleaner way to think about them.

A retrospective on how our team finally beat alert fatigue.

I used to be scared of perf record in prod. Then I wasn't.

One innocuous request_id label, 18M active series, and a very bad Friday.

Head sampling is simple. Tail sampling works. Here is a config we run in production without sadness.

When $var=All, Grafana was sending literally the string 'All' to my query. There's a setting for that.

Forgetting the rate() inside histogram_quantile gives you weird-looking percentiles. Here's why.

How to read the output of runtime/trace and actually understand why your service is slow.

Logs-within-a-span are a first-class OpenTelemetry concept and I had been putting them in attributes.

A short bpftrace script that pinpoints which process and peer are responsible for TCP retransmits on a noisy box

Span attributes vs baggage — they're different and I had been confusing them.

An operator kept thrashing at 300 reconciles per second, and the bug was a single annotation I was setting on the managed resource

Shows exactly how many rows have changed since the last ANALYZE. Great for 'do I need to analyze?'