Refinement Phase 9 — Observability, ops hardening, docs honesty & scale-later

Mission: make the running platform diagnosable and honest, and record the explicitly-deferred scale work so nobody mistakes it for missing MVP scope. This is the "finish before launch, then keep for later" bucket.

Track: backend (observability/docs) + explicit deferrals · Depends on: nothing hard (start anytime; finish the observability items before launch) · Unlocks: production diagnosability Before you start, read ../../phases/_shared/agent-operating-rules.md.

1. Context

This phase is the server audit's post-phase-7 (observability & ops hardening) + post-phase-8 (scale & later). Full evidence & fixes: ../server/post-phase-backend-plan.md § post-phase-7/8.

2. Required reading

../server/post-phase-backend-plan.md § post-phase-7 (7.1–7.6) and § post-phase-8 (8.1–8.5).
../server/runtime-services.md § 4 (Prometheus), § 17 (Elasticsearch — deliberately later).

3. Scope

Observability & ops hardening (post-phase-7 — finish before launch)

9.1 — add tracing + consolidate the two metric stacks. OTel is metrics-only (no WithTracing/OTLP); two overlapping Prometheus stacks run. Add WithTracing (AspNetCore + EF) exporting OTLP; pick one metrics stack; wire trace-id into ApiResult.requestId for support correlation.
9.2 — broaden health checks + split readiness/liveness. The single check is app-DB only; add log-DB, object-storage (write probe), Redis (when Phase 7 lands); expose /healthz/live vs /healthz/ready; remove the dead currentUrl line.
9.3 — revisit prod log levels + notification channels. Deployed envs write Warning+ only (every Information-level audit trail dropped); raise to Information+ with retention (or a file/OTLP sink); ensure no PII is logged (the SMS OTP log disappears with Phase 8 5.1); delete the dead Elasticsearch sink block + package or revive it deliberately.
9.4 — audit-log growth & archival. audit_logs is append-only with no retention (and Phase 6 §6.3 grows it); add a retention/archival policy as a Phase 7 job; define legal retention for money/verification rows first.
9.5 — decide TicketMessage.Body encryption + the gRPC plugin's fate. Ticket bodies are the refund/dispute paper trail (users type phone numbers, addresses, clinical detail) and are plaintext with no documented decision — recommend encrypting via the existing converter pattern. The gRPC plugin duplicates only the OTP/token flow, forces the HTTP/2 posture, and enables reflection unconditionally — remove it or give it a dedicated HTTP/2 endpoint + disable reflection outside Development.
9.6 — keep the docs honest. Prune the stale mocks-registry duplicate rows, rename the IJobScheduler row to "recurring jobs", mark delivered/answered REQs, note IPaymentCaptureSimulator's removal. (Root CLAUDE.md rule 7 — stale instructions are worse than none.)

Scale & later (post-phase-8 — explicitly NOT MVP; record, don't build)

9.7 — Elasticsearch read backend + outbox feeder. SqlNurseSearch is real and correct; Search:Backend fails fast on any non-sql value. Build ElasticNurseSearch + the outbox/CDC feeder only when SQL search shows strain. Not now.
9.8 — analytics pipeline. IAnalyticsSink writes ops.SystemEvents fire-and-forget; pipe to a warehouse/stream when product needs it.
9.9 — holiday-calendar feed. The table is manually maintained; a yearly ops-checklist item is an acceptable alternative to a lunar-Hijri drift feed.
9.10 — push/SMS notification channels. InAppNotificationDispatcher drops non-InApp channels; add fan-out (SMS via Phase 8 5.1's sender, FCM push) when the UX demands.
9.11 — deferred product tables (organizations, organization_nurses, fraud_flags, recurring_booking_schedules, bnpl_settlement_entries, availability slots, customer national-ID KYC, geo bulk import) — all verified absent and all pure additive migrations when product pulls them. Leave documented.

4. Mocks & seams

None new. 9.7/9.10 are the make-it-real steps for the INurseSearch (Elastic backend) and INotificationDispatcher (SMS/push channels) rows — deferred by design.

5. Critical rules

No PII in logs — verify after 9.3 (the OTP log must be gone; clinical text and IBANs are never logged).
Deferrals are documented, not silent — 9.7–9.11 stay explicitly out of MVP with a written "when to pull it" trigger, so a future reader doesn't mistake them for gaps.
Don't build Elasticsearch/analytics/push "because" — each has a concrete trigger (search strain, product need, UX demand). Until then, SQL search / in-app notifications are the real, correct MVP.

6. Definition of Done

Tracing exports OTLP; one metrics stack; requestId carries the trace id.
Health checks cover the real dependencies with /healthz/live vs /healthz/ready.
Prod logs Information+ with retention and no PII; the dead ES sink is resolved.
Audit retention policy exists; ticket-body encryption + gRPC decisions are made and documented.
The mocks-registry / REQ tracker / architecture maps are honest and current.
9.7–9.11 are recorded as deferred with explicit pull-triggers. dotnet build/dotnet test green.

7. How to test

A request produces a trace with a requestId that matches the ApiResult.requestId in the response.
/healthz/ready fails when a dependency (object storage / Redis) is down; /healthz/live stays up.
Grep the logs after an OTP + a booking + a payout → no phone number, OTP, IBAN, or clinical text appears.

8. Hand off & document

Update server/CLAUDE.md (observability wiring, ticket encryption, gRPC decision), the mocks-registry, and the REQ tracker. Update ../server/runtime-services.md for any new observability service. Save a memory note capturing the encryption/gRPC decisions and the deferral triggers.

6.3 KiB Raw Blame History Unescape Escape