Observability och driftövervakning

Structured logging is the spine. structlog binds request_id, tenant_id, user_id and trace_id into the context of every log line, masks a frozen set of sensitive field names (passwords, tokens, secrets) before serialisation, and switches between a JSON renderer in production and a console renderer in dev. ObservabilityMiddleware logs every request with method, path, status, duration and client IP, plus a startup info event so a deploy is unmissable in the log stream.

Prometheus metrics layer two registries together. craft-easy MetricsMiddleware exposes generic HTTP metrics (request count, duration, in-flight, request/response size); petanque-specific gauges and counters cover business signals — petanque_license_operations_total, petanque_auth_operations_total, petanque_tenant_api_requests_total, petanque_match_scores_submitted_total, petanque_audit_writes_total. The /metrics endpoint scrapes into the same Prometheus that Grafana dashboards read.

OpenTelemetry distributed tracing is initialised through setup_telemetry() with an OTLP gRPC exporter, auto-instrumenting FastAPI, httpx and pymongo. trace_id and span_id are injected into every structlog event so logs and traces correlate by ID; an X-Trace-Id response header lets the apps echo the trace into client telemetry. When OTel packages are absent the integration becomes a no-op; configuration is environment-driven (OTEL_ENABLED, OTEL_EXPORTER_OTLP_ENDPOINT).

Health checks come at four levels: /health (liveness — process up), /ready (readiness with MongoDB ping for traffic admission), /startup (startup probe for Container Apps cold-start), /health/detailed (per-dependency status with latency for MongoDB and Redis). /observability/config exposes non-sensitive runtime config for ops verification.

Grafana dashboards are JSON-provisioned under infrastructure/grafana/ — an API overview with twelve panels (request rate, latency p50/p95/p99, in-flight, error rate, health status, license ops, audit writes), plus per-domain views. Alerts route per-tenant through AlertRoutingConfig with first-class support for PagerDuty Events API v2, Opsgenie Alerts API v2 and generic webhooks; cooldown-based dedup prevents alert storms and AlertHistory captures the audit trail. Per-tenant SLO measurement runs through TenantSLOConfig (defaults: 99.9% availability, p95 ≤ 300 ms, p99 ≤ 800 ms) with SLOMeasurement documents for history. Error-budget burndown computes burnrate per hour and estimated time to exhaustion, surfacing status as ok / warning / critical / exhausted with a Prometheus gauge. Sentry handles frontend errors through DSN config plus an envelope tunnel that satisfies the apps' CSP. The public status page hits /public/status and /public/status/incidents, backed by ServiceHealthSample (90-day TTL) and StatusIncident with auto-incident creation on status change driven by a 60-second background job.