Posts
Things I’ve written that are long enough to deserve an introduction. Mostly backend engineering, occasional detour into photography and espresso machines.
-
A TCP RST that took a week to track down
A long-lived HTTP connection got RST every 12 hours, and the answer lived in the intersection of conntrack, a load balancer, and a very patient test
-
An operator reconcile loop that wouldn't quit
An operator kept thrashing at 300 reconciles per second, and the bug was a single annotation I was setting on the managed resource
-
systemd timers and the clock drift that ate our backups
Our backups stopped running for nine days and the cause was a quiet combination of OnCalendar, RandomizedDelaySec, and a drifting RTC
-
Why my homelab Pi-hole kept forgetting its DNS override
A local DNS override on my Pi-hole would come back for a day and then mysteriously vanish, and the culprit was a docker-compose restart policy
-
Three days of debugging a cgroup memory accounting bug
A service kept getting OOM-killed with plenty of memory headroom, and the trail led into the cgroup v2 memory controller and its file-backed accounting
-
nftables rule ordering surprised me
A two-hour outage caused by a harmless-looking rule insertion into the wrong chain position, and what I learned about nftables evaluation
-
A compiler optimization I didn't know existed until it broke
Tail call optimization, manual trampolines, and a crash that took me three days to trace to the optimizer
-
SIMD in Rust without losing my mind
Portable SIMD is real and it's good; a walkthrough of a checksum I vectorized and what surprised me
-
PLT and GOT: the indirection I never noticed
Dynamic linking on ELF is a two-step dance through the PLT and the GOT, and once you see it you cannot unsee it
-
Stack vs heap: what escape analysis taught me
Go's escape analysis decides where your variables live, and reading its output changed how I write performance-sensitive code
-
Moving a service from Go to Rust, an honest report
Four months, one service rewrite, the wins were not the ones I expected and the costs were larger than I thought
-
The panic in a goroutine that took down prod
A nil deref in a background worker crashed our whole service because Go's panic model doesn't care that you spawned a goroutine to handle errors safely
-
CPU caches explained through a slow hash table
A hash table that benchmarked beautifully in isolation was five times slower under realistic load, and L2 cache misses explained why
-
Linking errors on Linux: an afternoon with ld
Undefined symbols, multiple definitions, and the weekend I finally understood how --as-needed and --no-undefined interact
-
Unsafe code boundaries I drew and regretted
A cautionary tale about wrapping unsafe primitives in safe abstractions, and the time my unsafe was more unsafe than I realized