Skip to content
Available for work
Adam
←  All posts

Observability: Logs, Metrics, and Traces - Debugging Things I Can't See

Once my system went event-driven, print-debugging died. Here's how I learned the three pillars of observability and instrumented my services with OpenTelemetry.

4 min readobservability · monitoring · distributed-systems

TL;DR

  • The three pillars: logs (what happened), metrics (how much/how often), traces (the path of one request).
  • Structured logging (JSON) turns logs from noise into queryable data.
  • Prometheus scrapes metrics; Grafana visualizes them.
  • Distributed tracing with OpenTelemetry was the only way to follow a request across Kafka consumers.

The problem async created

After moving to Kafka, a "task created" action touched the API, a producer, three consumers, Redis, and Postgres. When a notification silently failed to send, my old workflow - add a console.log, reproduce, read the terminal - completely fell apart. The failure was in a different process than the request, happening milliseconds later. I literally could not see it. I needed observability, not print statements.

Pillar 1: Structured logs

Plain-text logs are fine for one process. Across many, they're unsearchable. The fix was structured logging - emit JSON with consistent fields:

logger.info({
  event: "task.created",
  taskId: task.id,
  projectId: task.projectId,
  traceId: ctx.traceId,        // the magic field - links logs across services
  durationMs: 42,
});

Now I can query "show me every log line for traceId=abc123" and watch a single request's journey across every service. The traceId is what stitches it together.

Pillar 2: Metrics

Logs tell you about one event. Metrics tell you about aggregate behavior - and they're cheap because they're just numbers over time. I learned two standard frameworks for deciding what to measure:

  • RED (for request-driven services): Rate, Errors, Duration.
  • USE (for resources): Utilization, Saturation, Errors.

Prometheus uses a pull model - it scrapes a /metrics endpoint your app exposes:

const httpDuration = new Histogram({
  name: "http_request_duration_seconds",
  help: "Request duration",
  labelNames: ["method", "route", "status"],
});

Then PromQL lets me ask real questions, like the p95 latency per route:

histogram_quantile(0.95,
  rate(http_request_duration_seconds_bucket[5m]))

Wired into Grafana, I get dashboards and alerts: "page me when error rate > 1% for 5 minutes." That's the difference between finding out from a graph vs. finding out from an angry user.

Pillar 3: Distributed tracing

This is the one that actually solved my original problem. A trace follows a single request end-to-end; each unit of work is a span, and spans nest to form a tree. A trace shows you exactly where time went and where it broke - across process and network boundaries.

OpenTelemetry (the vendor-neutral standard) propagates a trace context across service calls and Kafka messages, so the span created in the API connects to the span in the consumer:

const tracer = trace.getTracer("task-service");
await tracer.startActiveSpan("publishTaskCreated", async (span) => {
  span.setAttribute("task.id", task.id);
  // trace context is injected into Kafka message headers here,
  // so the consumer's span becomes a child of this one
  await producer.send(/* ... */);
  span.end();
});

When I opened the trace for that failing notification, the picture was immediate: the API span was green, the Kafka publish was green, the notifier's consume span was red with an exception attached - a malformed template. Five seconds to find what print-debugging couldn't show me in an hour.

Logs vs. metrics vs. traces - when to reach for each

  • Metrics: "Is something wrong, and how bad?" (alerting, trends)
  • Traces: "Where in the request is it wrong?" (latency, failure location)
  • Logs: "What exactly happened at that point?" (the detail, the stack trace)

You triage with metrics, localize with traces, and confirm with logs. They're a workflow, not three separate tools.

Takeaways

  • Add a traceId to everything early. It's the thread that ties the whole story together.
  • Instrument the RED metrics on every service from day one - it's a tiny amount of code.
  • Distributed tracing is mandatory the moment work crosses a process boundary.

Next

I can now see how my system behaves under load. The natural next question: how do I design it to handle that load on purpose? Time to study system design and scaling fundamentals.