Observability Guide

LuckyPlans includes a full observability stack covering metrics, logs, and traces. This guide covers how to use it day-to-day for development and debugging.

Quick Reference

Production host routing:

• Grafana is exposed at https://admin.luckyplans.xyz/grafana
• App/API traffic is split across app.luckyplans.xyz and api.luckyplans.xyz
• Legacy API remains available at https://v0.api.luckyplans.xyz through k3d ingress passthrough

Getting Started

Local development

The observability stack starts automatically with docker compose up -d:

docker compose up -d
pnpm dev

Open http://localhost:3002  — Grafana is pre-configured with anonymous admin access, datasources, and dashboards. No login required.

K8s (local k3d or prod)

The observability stack is deployed via Helm:

# Included automatically in full deploy:
pnpm deploy:local
 
# Or deploy observability only:
helm upgrade --install luckyplans-observability infrastructure/helm/observability \
  --namespace monitoring --create-namespace \
  -f infrastructure/helm/observability/values.yaml
 
# Access Grafana:
kubectl -n monitoring port-forward svc/grafana 3002:3000

Grafana Dashboards

RED Metrics Dashboard

Open Grafana → LuckyPlans folder → RED Metrics.

This dashboard shows the RED method  for each service:

Use the service dropdown at the top to filter by api-gateway or service-core.

Infrastructure Dashboard

Open Grafana → LuckyPlans folder → Infrastructure.

Viewing Traces

Traces show the full journey of a request across services.

Find traces for a request

1. Open Grafana → Explore (compass icon in sidebar)
2. Select Tempo datasource (top-left dropdown)
3. Use Search tab:
   • Service Name: api-gateway or service-core
   • Span Name: e.g., GET /graphql, HTTP POST
   • Min Duration: e.g., 100ms to find slow requests
4. Click Run query
5. Click any trace row to see the full span waterfall

Reading a trace

A typical GraphQL request trace looks like:

HTTP POST /graphql                          [api-gateway, 45ms]
  ├── graphql.resolve getItems              [api-gateway, 38ms]
  │   ├── ioredis: PUBLISH                  [api-gateway, 2ms]
  │   └── microservice.getItems             [service-core, 30ms]
  │       └── ioredis: GET/SET              [service-core, 1ms]
  └── graphql.serialize                     [api-gateway, 1ms]

• Parent span: The HTTP request hitting the gateway
• GraphQL resolver span: Auto-instrumented by OTel
• Redis spans: Auto-instrumented by ioredis instrumentation
• Microservice span: Created by TraceContextExtractor from the propagated W3C trace context

Trace-to-log correlation

Click the Logs for this span button on any span to jump to Loki with the trace ID pre-filtered. This shows all log lines emitted during that span’s execution.

Querying Logs

Basic log queries

1. Open Grafana → Explore
2. Select Loki datasource
3. Use LogQL queries:

# All logs from api-gateway
{service_name="api-gateway"}

# Error-level logs only
{service_name="api-gateway"} | json | level="error"

# Logs containing a specific trace ID
{service_name=~"api-gateway|service-core"} |= "abc123def456"

# Logs from a specific route
{service_name="api-gateway"} | json | req_url=~"/graphql.*"

# Count errors per minute
count_over_time({service_name="api-gateway"} | json | level="error" [1m])

Log format

Every log line is structured JSON with these fields:

Log-to-trace correlation

When viewing a log line that contains a traceId, click the TraceID link to jump to the full trace in Tempo.

Observing Edge Connectivity and Upgrades

Edge fleet operations are visible through worker state and upgrade status transitions.

What to watch

1. lastSeenAt freshness for each worker (connectivity heartbeat health)
2. version vs targetVersion drift (pending upgrade intent)
3. upgradeStatus transitions:
   • UPGRADE_PENDING
   • DOWNLOADING
   • VERIFYING
   • RESTARTING
   • SUCCEEDED / FAILED

Useful log queries for edge flows

# Registration and connectivity endpoint logs
{service_name="api-gateway"} |= "/internal/edges/register"
{service_name="api-gateway"} |= "/internal/edges/connectivity"

# Upgrade lifecycle status logs
{service_name="api-gateway"} |= "upgradeStatus"

# Credential audit events
{service_name="api-gateway"} |= "audit credential."

Common edge operational signals

1. Stale lastSeenAt: worker likely offline, blocked, or unable to reach gateway.
2. targetVersion set but status never leaves UPGRADE_PENDING: worker is likely busy (idle-only upgrade policy) or connectivity is unstable.
3. Frequent FAILED upgrades: verify release URLs, checksum/signature metadata, and edge host permissions.

Querying Metrics

Using Prometheus directly

Open http://localhost:9090  (or port-forward in K8s) for the Prometheus UI.

Useful PromQL queries

# Request rate per service (last 5 minutes)
sum(rate(http_server_request_duration_seconds_count[5m])) by (service_name)

# Error rate (5xx) as percentage
sum(rate(http_server_request_duration_seconds_count{http_status_code=~"5.."}[5m])) by (service_name)
/
sum(rate(http_server_request_duration_seconds_count[5m])) by (service_name)

# p95 latency per service
histogram_quantile(0.95,
  sum(rate(http_server_request_duration_seconds_bucket[5m])) by (le, service_name)
)

# p99 latency for a specific route
histogram_quantile(0.99,
  sum(rate(http_server_request_duration_seconds_bucket{http_route="/graphql"}[5m])) by (le)
)

# Redis connected clients
redis_connected_clients

# Redis memory usage
redis_memory_used_bytes

# OTel Collector: exported spans per second
rate(otelcol_exporter_sent_spans_total[5m])

Checking scrape targets

Open http://localhost:9090/targets  to verify all targets are UP:

Alerts

Prometheus evaluates these alert rules (defined in the Prometheus configmap):

View active alerts: http://localhost:9090/alerts 

Note: Alert notifications (email, Slack, PagerDuty) are not yet configured. Alerts currently only show in the Prometheus UI.

Debugging Common Issues

”No data” in Grafana dashboards

1. Check OTel Collector: docker compose logs otel-collector (local) or kubectl -n monitoring logs deployment/otel-collector (K8s)
2. Check Prometheus targets: http://localhost:9090/targets  — all should be UP
3. Check NestJS apps are sending telemetry: Look for OTEL_EXPORTER_OTLP_ENDPOINT in the app’s env. Default is http://localhost:4317 (local dev)
4. Make some requests: The dashboards need traffic. Hit http://localhost:3000/graphql  with a query

Traces missing the service-core span

The Redis trace propagation requires injectTraceContext() in the gateway resolver and TraceContextExtractor in service-core. Check:

1. Gateway resolver uses injectTraceContext() when calling ClientProxy.send()
2. service-core AppModule has { provide: APP_INTERCEPTOR, useClass: TraceContextExtractor }

Logs not showing traceId

The Pino mixin() function reads the active OTel span. If traceId is missing:

1. Verify instrument.ts is imported as the first line of main.ts
2. Check that OTEL_EXPORTER_OTLP_ENDPOINT is set (the SDK starts even without a collector, but auto-instrumentation needs the SDK initialized)

OTel Collector dropping data

Check the Infrastructure dashboard — “Dropped Telemetry” panel. If data is being dropped:

1. Increase memory_limiter.limit_mib in the collector config
2. Increase the collector’s memory limits in values.yaml

High memory usage

The observability stack uses ~900Mi total. If the k3s node is constrained:

1. Use --no-observability flag: ./deploy-local.sh --no-observability
2. Or reduce retention: edit values.yaml → prometheus.retention, loki.retention, tempo.retention

Architecture Reference

┌─────────────────────────────────────────────────────┐
│                  luckyplans namespace                │
│                                                     │
│  api-gateway ──OTLP──┐   service-core ──OTLP──┐    │
│                       │                        │    │
└───────────────────────┼────────────────────────┼────┘
                        │                        │
┌───────────────────────┼────────────────────────┼────┐
│                  monitoring namespace                │
│                       ▼                        ▼    │
│               ┌──────────────┐                      │
│               │ OTel Collector│                      │
│               └──┬───┬───┬───┘                      │
│                  │   │   │                          │
│          metrics │   │   │ traces                   │
│                  ▼   │   ▼                          │
│            Prometheus│  Tempo                       │
│                  │   │   │                          │
│                  │   │logs│                         │
│                  │   ▼   │                          │
│                  │  Loki │                          │
│                  │   │   │                          │
│                  ▼   ▼   ▼                          │
│               ┌──────────────┐                      │
│               │   Grafana    │ ← dashboards         │
│               └──────────────┘                      │
│                                                     │
│  Promtail (DaemonSet) ────────▶ Loki (pod logs)    │
│  Redis Exporter ──────────────▶ Prometheus          │
│                                                     │
└─────────────────────────────────────────────────────┘

See ADR: Full-Stack Observability for the architectural decision and alternatives considered.