Grafana Tempo

Tempo renders execution traces as waterfall diagrams, showing parallel target execution and per-target phase breakdowns.

Setup

Export traces in OTLP format and send them to Tempo’s OTLP HTTP endpoint:

grog traces export --format=otel --limit 1 --output /tmp/trace-otel.json

curl -X POST http://tempo:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d @/tmp/trace-otel.json

Note

The OTLP export maps each build to a root span with child spans for every target. Span attributes include all phase timing fields (grog.command_duration_ms, grog.queue_wait_ms, etc.) so you can filter and search by them in the Tempo UI.

What you get

Once traces are in Tempo, you can:

• Waterfall view: See all targets laid out on a timeline, showing which ran in parallel and where time was spent
• Search by attributes: Find traces where grog.cache_result = CACHE_MISS or where grog.command_duration_ms > 30000
• Compare traces: Side-by-side comparison of two builds to see what changed
• Link to Loki: If you also ship logs to Loki, Grafana can correlate trace spans with build output logs