Had a span representing a batch job. The job has several phases: “loaded config”, “fetched rows”, “transformed”, “pushed to sink”. I’d been setting attributes like phase_loaded_at=..., phase_fetched_at=.... Worked but read weird.

Turns out OTel has span events for exactly this:

span.AddEvent("loaded config")
// ...
span.AddEvent("fetched rows", trace.WithAttributes(
    attribute.Int("row_count", len(rows)),
))

Each event has a timestamp, a name, and optional attributes. It shows up as a point on the span’s timeline. In Jaeger/Tempo UIs, events are rendered as markers on the span bar, which is way clearer than reading attribute values.

Think of events as “log messages attached to a span.” Not standalone logs, not span attributes — timestamped in-span annotations.

Good uses:

  • Phase boundaries in a long operation.
  • Non-fatal issues (a retry happened, a cache miss occurred).
  • Key state changes that would be useful in a postmortem.

Less-good uses:

  • Anything where the event itself should be searchable across traces — that’s a log line or a metric.
  • High-frequency events in a tight loop — each event has overhead, and too many events per span makes the UI unreadable.

Default limit is 128 events per span in the OTel SDK; configurable, but if you’re hitting it, you’re probably using events wrong.

Refactoring our batch job instrumentation to use events instead of attribute-per-phase made the traces dramatically easier to read.