How to Deploy

All environments use the same Helm chart at infrastructure/helm/luckyplans/ with per-environment values files. See Helm Deployment for full design decisions.

Continuous delivery is handled by ArgoCD (pull-based GitOps). See ArgoCD for the operational guide.

Architecture Overview

                     Traefik Ingress (k3d built-in)
                      │              │            │           │
                      │ /            │ /graphql   │ /auth/*   │ /uploads/*
                      ▼              ▼            ▼           ▼
                    web:3000    api-gateway:3001 ─────────────┘
                                     │         │         │
                                     │ Redis   │ OIDC    │ S3 API
                                     ▼         ▼         ▼
                                service-core  Keycloak  MinIO:9000
                                     │           │         │
                                     ▼           ▼         ▼
                                  Redis:6379  PostgreSQL  /data (PVC)
                                                  ▲
                                  prisma-migrate ─┘ (Helm pre-upgrade Job)

    monitoring namespace:
      OTel Collector ← api-gateway, service-core (OTLP)
        ├── Prometheus (metrics)
        ├── Loki (logs via Promtail)
        └── Tempo (traces)
              └── Grafana (dashboards)

App services run in the luckyplans namespace. Observability services run in the monitoring namespace.

Prerequisites

Install k3d

# Linux/Mac
curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
 
# Windows (via Chocolatey)
choco install k3d

Install cert-manager

Required for prod deployment (automatic TLS via Let’s Encrypt). Not needed for local development.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.1/cert-manager.yaml
kubectl -n cert-manager rollout status deploy/cert-manager
kubectl -n cert-manager rollout status deploy/cert-manager-webhook
kubectl -n cert-manager rollout status deploy/cert-manager-cainjector

See TLS Certificates for full documentation.

Environments

Production domains

• app.luckyplans.xyz → web frontend
• api.luckyplans.xyz → api-gateway (/graphql, /auth, /uploads, /health)
• admin.luckyplans.xyz → Keycloak, ArgoCD (/argocd), Grafana (/grafana)
• v0.api.luckyplans.xyz → legacy API routed by k3d ingress to host.k3d.internal:9000

Local Deployment (laptop)

Single command (full deploy)

pnpm deploy:local

This handles everything: cluster creation, image builds (app + observability), k3d import, and helm upgrade --install for both the app and observability charts.

After completion:

• Frontend: http://localhost 
• GraphQL Playground: http://localhost/graphql 
• Grafana: kubectl -n monitoring port-forward svc/grafana 3002:3000 → http://localhost:3002 
• Prometheus: kubectl -n monitoring port-forward svc/prometheus 9090:9090 → http://localhost:9090 

Targeted deploy (rebuild one or more services)

# Rebuild and redeploy only the web frontend:
./infrastructure/scripts/deploy-local.sh web
 
# Rebuild multiple services:
./infrastructure/scripts/deploy-local.sh api-gateway web
 
# Helm upgrade only — for config/secret changes (no image builds):
./infrastructure/scripts/deploy-local.sh --helm-only
 
# Skip observability stack (faster if you don't need monitoring):
./infrastructure/scripts/deploy-local.sh --no-observability

Targeted deploy builds only the specified service images, imports them into k3d, and does a kubectl rollout restart — much faster than a full deploy.

Teardown

pnpm deploy:teardown

Status

pnpm deploy:status

Manual step-by-step

# 1. Create the k3d cluster
k3d cluster create luckyplans-local \
  --port "80:80@loadbalancer" \
  --port "443:443@loadbalancer" \
  --agents 1
 
kubectl config use-context k3d-luckyplans-local
 
# 2. Build Docker images
docker build \
  --build-arg NEXT_PUBLIC_GRAPHQL_URL="/graphql" \
  -t luckyplans/web:latest -f apps/web/Dockerfile .
docker build -t luckyplans/api-gateway:latest  -f apps/api-gateway/Dockerfile .
docker build -t luckyplans/service-core:latest  -f apps/service-core/Dockerfile .
docker build -t luckyplans/prisma-migrate:latest -f packages/prisma/Dockerfile .
 
# 3. Import images into k3d
docker pull redis:7-alpine
docker pull postgres:17-alpine
k3d image import redis:7-alpine              -c luckyplans-local
k3d image import postgres:17-alpine          -c luckyplans-local
k3d image import luckyplans/web:latest       -c luckyplans-local
k3d image import luckyplans/api-gateway:latest  -c luckyplans-local
k3d image import luckyplans/service-core:latest -c luckyplans-local
k3d image import luckyplans/prisma-migrate:latest -c luckyplans-local
 
# 4. Deploy with Helm
helm upgrade --install luckyplans infrastructure/helm/luckyplans \
  --namespace luckyplans \
  --create-namespace \
  --rollback-on-failure --timeout 3m

CI/CD with ArgoCD (recommended)

Prod deployments are handled by ArgoCD GitOps. See ArgoCD for the full operational guide.

How it works

Push/merge to main → CI → Docker Build & Push → Update Tags → ArgoCD auto-sync → smoke tests

Manual tag deployment

1. Go to Actions → Update Image Tags → Run workflow
2. Enter the image tag (e.g. sha-abc1234)
3. Click Run workflow

ArgoCD will auto-sync the new tag.

Prod Deployment — first-time setup

Prerequisites

• DNS A records:
  • app.luckyplans.xyz → <your-server-ip>
  • api.luckyplans.xyz → <your-server-ip>
  • admin.luckyplans.xyz → <your-server-ip>
  • v0.api.luckyplans.xyz → <your-server-ip>
• Ports open on firewall: 22 (SSH), 80 (HTTP/ACME), 443 (HTTPS)
• A GitHub PAT with read:packages and repo read access
• CD_PUSH_TOKEN: fine-grained PAT with Contents: read+write scope (required with branch protection)
• kubeseal CLI installed locally

Setup steps

# SSH into the prod server
 
# 1. Create the k3d cluster
k3d cluster create luckyplans-prod \
  --port "80:80@loadbalancer" \
  --port "443:443@loadbalancer" \
  --agents 1
 
# 2. Install cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.17.1/cert-manager.yaml
kubectl -n cert-manager rollout status deploy/cert-manager
 
# 3. Install Sealed Secrets controller
./infrastructure/scripts/install-sealed-secrets.sh
 
# 4. Install ArgoCD
git clone https://github.com/takeshi-su57/luckyplans.git
cd luckyplans
./infrastructure/scripts/install-argocd.sh --github-token <your-github-pat>
 
# 5. Generate and seal production secrets
./infrastructure/scripts/seal-secrets.sh
# IMPORTANT: Save the plain-text output! You'll need KEYCLOAK_CLIENT_SECRET
# and KEYCLOAK_ADMIN_PASSWORD for step 7.
 
# 6. Paste the sealedSecrets block into values.prod.yaml
#    Commit and push — ArgoCD will auto-sync and deploy all services
 
# 7. Sync Keycloak client secret (after Keycloak is running)
#    a. Log into Keycloak Admin at https://admin.luckyplans.xyz/admin
#       Username: admin | Password: the KEYCLOAK_ADMIN_PASSWORD from step 5
#    b. Select realm: luckyplans (top-left dropdown)
#    c. Go to: Clients → luckyplans-frontend → Credentials tab
#    d. Set the Client Secret to the KEYCLOAK_CLIENT_SECRET from step 5
#       (paste the plain-text value the script printed, then click Save)
#    e. Both the gateway and Keycloak now use the same secret — auth works.
#
# Alternative: if you prefer to use Keycloak's auto-generated secret:
#    a. Copy the secret from Keycloak Admin → Clients → Credentials
#    b. Re-seal it: ./seal-secrets.sh --seal-only KEYCLOAK_CLIENT_SECRET=<copied-value>
#    c. Update values.prod.yaml with the new sealed value, commit and push

Secrets Management (Sealed Secrets)

Production secrets are managed via Bitnami Sealed Secrets. Encrypted values are committed to git in values.prod.yaml — only the cluster controller can decrypt them. See ADR: Bitnami Sealed Secrets.

How it works

1. The sealed-secrets-controller runs in the prod cluster (installed once via install-sealed-secrets.sh)
2. seal-secrets.sh generates secrets, encrypts each with the controller’s public key
3. Encrypted values are stored in values.prod.yaml under sealedSecrets.encryptedData
4. The Helm chart renders a SealedSecret CRD (not a plain Secret) in production
5. The controller decrypts the SealedSecret into a standard luckyplans-secrets K8s Secret

Required secrets

Rotating secrets

# Re-generate and re-seal all secrets
./infrastructure/scripts/seal-secrets.sh
 
# Paste the new encryptedData into values.prod.yaml
# Commit and push — ArgoCD auto-syncs, pods restart with new secrets

After rotating KEYCLOAK_CLIENT_SECRET, update Keycloak Admin Console: Clients → luckyplans-frontend → Credentials → set the new value.

After rotating WORKER_CREDENTIAL_PEPPER, previously issued worker credentials become invalid unless your rollout includes a dual-pepper migration strategy. Plan this as a maintenance window and re-issue credentials where needed.

Important: All secrets in the table above must be present in values.prod.yaml under sealedSecrets.encryptedData. If any are missing, the corresponding Pod will fail to start because its secretKeyRef cannot resolve — and any PVCs it mounts will stay in WaitForFirstConsumer indefinitely.

Local dev secrets

Local development uses plain-text dev defaults in values.yaml — no sealed secrets needed. The realm export pre-configures dev-client-secret for Keycloak.

Kubernetes Security Baseline (Helm + ArgoCD)

Use this as the minimum production baseline:

1. Keep sealedSecrets.enabled: true in values.prod.yaml; never commit plain-text secrets.
2. Rotate SESSION_SECRET, WORKER_CREDENTIAL_PEPPER, JWT_SECRET, KEYCLOAK_CLIENT_SECRET, and MinIO secrets on a fixed schedule (for example, every 90 days) and after incidents.
3. Restrict access to values.prod.yaml, Sealed Secrets private key backups, and CI secrets to least privilege.
4. Enforce HTTPS-only ingress (websecure + HSTS middleware) and keep certManager.enabled: true.
5. Keep image tags immutable (sha-<commit>), avoid mutable prod tags.
6. Use non-root containers and dropped Linux capabilities (already defined in gateway Deployment); do not relax these defaults.
7. Keep ArgoCD auto-sync with Git as source of truth; avoid manual kubectl edit drift in production.
8. Scope observability access: expose Grafana/Prometheus via authenticated channels (VPN, SSO, or restricted ingress), never public anonymous endpoints.

Required update for current error

If you are seeing Missing required environment variable: WORKER_CREDENTIAL_PEPPER, run:

./infrastructure/scripts/seal-secrets.sh --seal-only WORKER_CREDENTIAL_PEPPER

Then add the output key under sealedSecrets.encryptedData in infrastructure/helm/luckyplans/values.prod.yaml, commit, and let ArgoCD sync.

If edge registration fails due to invalid enrollment token, create a new token in the Edges UI and revoke the compromised token.

Key backup

The controller’s signing key is backed up during installation to .sealed-secrets-backup/. Store this backup securely offline — if lost, all sealed secrets must be re-created.

Scaling

# values.prod.yaml
apiGateway:
  replicas: 3
web:
  replicas: 2

Commit and push — ArgoCD auto-syncs.

Warning: Do not use kubectl scale on ArgoCD-managed clusters — ArgoCD self-heal will revert the change immediately.

Observability (K8s)

The observability stack is deployed as a separate Helm chart (infrastructure/helm/observability/) in the monitoring namespace. In production, it’s managed by its own ArgoCD Application (infrastructure/argocd/apps/observability-prod.yaml).

Components: OTel Collector, Prometheus, Grafana, Loki, Tempo, Promtail, Redis Exporter.

# Check observability pods:
kubectl -n monitoring get pods
 
# Port-forward Grafana:
kubectl -n monitoring port-forward svc/grafana 3002:3000
 
# Port-forward Prometheus:
kubectl -n monitoring port-forward svc/prometheus 9090:9090

NestJS services push traces and metrics via OTLP to the OTel Collector. Promtail ships pod logs to Loki. Grafana provides unified dashboards with trace↔log correlation.

Edge Agent Release and Rollout

The edge lifecycle now includes release metadata and target-version orchestration.

Release registration flow

1. Build and publish edge artifacts to GitHub Releases (Windows + Linux assets).
2. Register the release in LuckyPlans with:
   • version
   • windowsUrl
   • linuxUrl
   • checksum
   • signature
3. Set targetVersion for selected workers or start an upgrade campaign.

Edge rollout behavior

1. Edge sends connectivity heartbeats to the gateway.
2. Gateway returns upgrade intent when targetVersion differs from version.
3. Edge upgrades only when idle (no active task).
4. Edge reports upgrade statuses (DOWNLOADING, VERIFYING, RESTARTING, SUCCEEDED, FAILED).

Registration token requirement

Edge registration requires an active enrollment token created in the Edges UI. Tokens are one-time reveal secrets and can be revoked without affecting already registered edges.

Recommended operator flow:

1. Open /edges in the app.
2. Create an enrollment token (set optional maxUses / expiresAt).
3. Copy the revealed token immediately (shown once).
4. Use it only at edge onboarding prompt (Enrollment token (from Edge UI)).
5. Revoke the token when onboarding is complete.

Viewing Logs

kubectl -n luckyplans logs -f deployment/api-gateway
kubectl -n luckyplans logs -f deployment/web
kubectl -n luckyplans logs <pod-name> --previous  # after crash
 
# Observability stack logs:
kubectl -n monitoring logs -f deployment/otel-collector
kubectl -n monitoring logs -f deployment/grafana

Rollback

ArgoCD-managed (prod)

Git revert (recommended):

git log --oneline -5
git revert <commit-sha>
git push origin main

Local (no ArgoCD)

helm -n luckyplans rollback luckyplans