Distributed tracing

6 min read

The third pillar alongside metrics and logs. The API ships W3C-standard spans to Tempo via OTLP. Grafana queries Tempo by trace_id. The same trace_id is on every Pino log record and every Sentry/GlitchTip event. Result: a slow request becomes a waterfall you can read, not a number you have to guess about.

Your first trace

Generate a slow request and find it in Grafana. Open Grafana → Explore → Tempo. Search by service.name = boringstack-api and http.target = /your/endpoint. Open a representative trace. You’ll see a waterfall: parent span (the HTTP request) with child spans nested below it (auth plugin, database queries, external API calls, queue operations). Click a span to see its attributes and duration. The slowest span is usually the culprit. From the trace view, click “Logs for this span” to jump to every log line emitted while that span was active.

What’s traced

The API initializes an OpenTelemetry SDK (src/config/otel/) before any other module imports. Auto-instrumentation patches several libraries at load time. The resulting spans flow through OTLP/HTTP to Tempo.

Incoming HTTP (Elysia routes)

Every request to the API becomes a parent span with method, route, status, and duration attributes. This is the root of every other span on the request’s path.

Outgoing fetch / undici

Calls to Stripe, Resend, OpenAI, OAuth providers, or anything you fetch each become a child span. You can see exactly how long Stripe took before blaming your own code.

ioredis / Valkey

BullMQ’s underlying ioredis driver is instrumented, so queue enqueue and lock operations show up as spans. Useful when “why was this job slow to start?” is the question.

BullMQ job processing (manual wrap)

Every worker’s processJob is wrapped in withQueueSpan (see src/lib/tracing/), producing a queue.<name>.process span per job with messaging attributes (queue name, job id, attempt number). Failed jobs record the exception on the span.

Database queries (opt-in)

postgres-js (Drizzle’s underlying driver) has no upstream OTel auto-instrumentation, so DB query spans are opt-in via a small withDbSpan helper. Wrap hot-path queries:

import { withDbSpan } from "@/lib/tracing";

const user = await withDbSpan(
  "users.findById",
  { "db.statement": "select id, email from users where id = $1" },
  () => db.query.users.findFirst({ where: eq(users.id, userId) })
);

Skip it for one-off queries. Use it when a service method is critical-path or you’re benchmarking.

Browser to API trace continuation

The UI’s Sentry SDK (@sentry/react with browserTracingIntegration()) adds sentry-trace and traceparent headers to every /api/* fetch. The API’s OTel SDK reads them and continues the trace, so a single trace ID spans the browser action and everything it triggered server-side.

Sentry’s browser tracesSampleRate is 0 and the API’s SENTRY_TRACES_SAMPLE_RATE defaults to 0. Sentry is error-capture-only. OTel is the single tracer that ships spans to Tempo. @sentry/bun v10+ is built on @sentry/opentelemetry, so any non-zero Sentry sample rate registers a second tracer on top of the OTel SDK and double-instruments HTTP, fetch, and DB. Flip it back if you want transactions in Sentry as well.

Common use cases

Find why an endpoint got slow: open Grafana, Explore, Tempo. Query by service.name = boringstack-api and endpoint name. Open a representative slow trace. The waterfall shows which child span dominates: DB, external API, queue, or your own code.
Detect N+1 queries: a trace with 30 identical SELECT * FROM accounts WHERE id = ? spans stacked is unmistakable. Refactor to a single WHERE id IN (...) and the trace flattens.
Diagnose background job lag: a user reports “my welcome email took an hour.” Search Tempo by userId (set as a span attribute by the auth plugin), find the queue.email-delivery.process span. The gap between enqueue and process tells you whether the queue was backed up or the email provider was slow.
Benchmark before and after a refactor: capture trace IDs from representative requests before your change. After deploying, capture the same requests again. Compare waterfalls side by side. See exactly which span you made faster (or slower).
Pivot from a log line to its trace: in Grafana Loki, expand any API log line. The trace_id field has a clickable “View trace in Tempo” link (configured in the Loki datasource’s derivedFields). One click opens the trace view in the same Grafana tab.
Pivot from a trace to its logs: in Tempo’s trace view, click a span. “Logs for this span” opens Loki filtered to {compose_service=~"api-dev|api"} | json | trace_id="...". Every log line emitted while that span was active appears.

Adding spans to your own code

The auto-instrumentation covers infrastructure (HTTP, fetch, redis, queues). Your business logic isn’t auto-traced. For hot paths or operations you’re benchmarking, add manual spans:

import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("apps/api/<your-domain>");

const result = await tracer.startActiveSpan(
  "expensiveComputation",
  { attributes: { "user.count": users.length } },
  async (span) => {
    try {
      return await doWork();
    } finally {
      span.end();
    }
  }
);

The two ready-to-use helpers in src/lib/tracing/:

withQueueSpan(queueName, job, handler): already applied to every BullMQ worker. Pattern to copy when adding a new queue.
withDbSpan(name, attributes, handler): opt-in DB query wrapper. Useful around db.query.X.findMany(...) calls in service methods you’re scrutinising.

Storage + retention

Tempo runs in single-binary mode (no separate distributor / ingester / querier processes): fine for a single-host BoringStack deployment. Spans land on local disk at /var/tempo (a Docker volume named tempo_data). Default retention is 24 hours; tune in compose/tempo/tempo.yml under compactor.block_retention. Traces are denser than logs per unit-debugging-value, so 48h–7d is reasonable in production.

Resource budget (defaults): 0.5 vCPU, 512MB RAM. Override via TEMPO_LIMITS_CPUS / TEMPO_LIMITS_MEMORY in compose/.env.

Source

apps/api/src/config/otel/otel.ts: SDK init.
apps/api/src/instrument.ts: bootstrap side-effect import (run before everything else).
apps/api/src/lib/tracing/: withQueueSpan + withDbSpan helpers.
compose/tempo/tempo.yml: Tempo backend config.
compose/grafana/provisioning/datasources/datasources.yml: Tempo datasource + Loki↔Tempo click-through wiring.

Observability: metrics + logs that traces pivot to.
Error tracking: trace_id flows through GlitchTip events too, so an error in GlitchTip has a clickable trace context.
Alerts: when an alert fires, the trace timeline for the same minute often shows what went wrong before the metric did.