K8S Observability Platform

The question that started it

Running HubStream across regions produced a recurring, un-debuggable complaint: 'the video froze for a second.' Node CPU looked fine. Pods were healthy. Nothing in the platform's default metrics could tell me whether the stall came from a publisher's uplink, the inter-region pipe, the subscriber's downlink, or the media server itself.

So this stack is not generic Kubernetes monitoring with a WebRTC label on it. It exists to attribute a specific user-visible symptom to a specific layer.

The stack

• Prometheus scrapes cluster metrics plus a custom exporter that surfaces mediasoup transport stats — bitrate, packet loss, RTT, jitter, per producer and consumer
• Loki + Promtail collect signalling and media-server logs, keyed by the same room and peer ids the metrics carry
• Grafana joins the two: one dashboard, one time range, metric spike next to the log line that explains it
• Alert rules on sustained loss and RTT, sized so a single bad client does not page
• Everything packaged as Helm charts, so the stack redeploys with the cluster

Correlation is the whole feature

The design decision that made the stack useful was propagating room id and peer id as labels on both metric series and log lines. Without that, a dashboard is a wall of pretty graphs and a log search is a separate investigation. With it, clicking a spike gives you the peer, and the peer gives you the log stream — the attribution question answers itself.

The cost is label cardinality, which is a real Prometheus constraint. Peer-level series are kept short-lived and aggregated up to room level for anything retained long term.

What I would do next

• Recording rules for the expensive room-level aggregations
• Synthetic probe clients per region for a continuous baseline that does not depend on real traffic
• SLO-based alerting on call quality rather than raw thresholds

Stack