add refinement phases

2026-07-10 20:59:47 +03:30
parent a87b47bedb
commit 850cdf3414
11 changed files with 1428 additions and 0 deletions
@@ -0,0 +1,94 @@
+# Refinement Phase 9 — Observability, ops hardening, docs honesty & scale-later
+
+> **Mission:** make the running platform diagnosable and honest, and record the explicitly-deferred scale work
+> so nobody mistakes it for missing MVP scope. This is the "finish before launch, then keep for later" bucket.
+>
+> **Track:** backend (observability/docs) + explicit deferrals · **Depends on:** nothing hard (start anytime;
+> finish the observability items before launch) · **Unlocks:** production diagnosability
+> **Before you start, read [../../phases/_shared/agent-operating-rules.md](../../phases/_shared/agent-operating-rules.md).**
+
+## 1. Context
+
+This phase **is** the server audit's **post-phase-7 (observability & ops hardening)** + **post-phase-8 (scale &
+later)**. Full evidence & fixes:
+**[../server/post-phase-backend-plan.md](../server/post-phase-backend-plan.md) § post-phase-7/8**.
+
+## 2. Required reading
+
+- **[../server/post-phase-backend-plan.md](../server/post-phase-backend-plan.md)** § post-phase-7 (7.1–7.6) and
+  § post-phase-8 (8.1–8.5).
+- **[../server/runtime-services.md](../server/runtime-services.md)** § 4 (Prometheus), § 17 (Elasticsearch —
+  deliberately later).
+
+## 3. Scope
+
+### Observability & ops hardening (post-phase-7 — finish before launch)
+- **9.1 — add tracing + consolidate the two metric stacks.** OTel is metrics-only (no `WithTracing`/OTLP); two
+  overlapping Prometheus stacks run. Add `WithTracing` (AspNetCore + EF) exporting OTLP; pick **one** metrics
+  stack; wire trace-id into `ApiResult.requestId` for support correlation.
+- **9.2 — broaden health checks + split readiness/liveness.** The single check is app-DB only; add log-DB,
+  object-storage (write probe), Redis (when [Phase 7](refinement-phase-7-unattended-ops.md) lands); expose
+  `/healthz/live` vs `/healthz/ready`; remove the dead `currentUrl` line.
+- **9.3 — revisit prod log levels + notification channels.** Deployed envs write **Warning+ only** (every
+  Information-level audit trail dropped); raise to Information+ with retention (or a file/OTLP sink); ensure
+  **no PII is logged** (the SMS OTP log disappears with [Phase 8](refinement-phase-8-external-rails.md) 5.1);
+  delete the dead Elasticsearch sink block + package or revive it deliberately.
+- **9.4 — audit-log growth & archival.** `audit_logs` is append-only with no retention (and Phase 6 §6.3 grows
+  it); add a retention/archival policy as a [Phase 7](refinement-phase-7-unattended-ops.md) job; define legal
+  retention for money/verification rows first.
+- **9.5 — decide `TicketMessage.Body` encryption + the gRPC plugin's fate.** Ticket bodies are the refund/dispute
+  paper trail (users type phone numbers, addresses, clinical detail) and are plaintext with no documented
+  decision — recommend encrypting via the existing converter pattern. The gRPC plugin duplicates only the
+  OTP/token flow, forces the HTTP/2 posture, and enables reflection unconditionally — remove it or give it a
+  dedicated HTTP/2 endpoint + disable reflection outside Development.
+- **9.6 — keep the docs honest.** Prune the stale mocks-registry duplicate rows, rename the `IJobScheduler` row
+  to "recurring jobs", mark delivered/answered REQs, note `IPaymentCaptureSimulator`'s removal. (Root `CLAUDE.md`
+  rule 7 — stale instructions are worse than none.)
+
+### Scale & later (post-phase-8 — explicitly NOT MVP; record, don't build)
+- **9.7 — Elasticsearch read backend + outbox feeder.** `SqlNurseSearch` is real and correct; `Search:Backend`
+  fails fast on any non-`sql` value. Build `ElasticNurseSearch` + the outbox/CDC feeder **only when SQL search
+  shows strain**. Not now.
+- **9.8 — analytics pipeline.** `IAnalyticsSink` writes `ops.SystemEvents` fire-and-forget; pipe to a
+  warehouse/stream when product needs it.
+- **9.9 — holiday-calendar feed.** The table is manually maintained; a yearly ops-checklist item is an
+  acceptable alternative to a lunar-Hijri drift feed.
+- **9.10 — push/SMS notification channels.** `InAppNotificationDispatcher` drops non-InApp channels; add
+  fan-out (SMS via [Phase 8](refinement-phase-8-external-rails.md) 5.1's sender, FCM push) when the UX demands.
+- **9.11 — deferred product tables** (`organizations`, `organization_nurses`, `fraud_flags`,
+  `recurring_booking_schedules`, `bnpl_settlement_entries`, availability slots, customer national-ID KYC, geo
+  bulk import) — all verified absent and all pure additive migrations when product pulls them. Leave documented.
+
+## 4. Mocks & seams
+
+None new. 9.7/9.10 are the make-it-real steps for the `INurseSearch` (Elastic backend) and
+`INotificationDispatcher` (SMS/push channels) rows — deferred by design.
+
+## 5. Critical rules
+
+- **No PII in logs** — verify after 9.3 (the OTP log must be gone; clinical text and IBANs are never logged).
+- **Deferrals are documented, not silent** — 9.7–9.11 stay explicitly out of MVP with a written "when to pull
+  it" trigger, so a future reader doesn't mistake them for gaps.
+- **Don't build Elasticsearch/analytics/push "because"** — each has a concrete trigger (search strain, product
+  need, UX demand). Until then, SQL search / in-app notifications are the real, correct MVP.
+
+## 6. Definition of Done
+
+- [ ] Tracing exports OTLP; one metrics stack; `requestId` carries the trace id.
+- [ ] Health checks cover the real dependencies with `/healthz/live` vs `/healthz/ready`.
+- [ ] Prod logs Information+ with retention and **no PII**; the dead ES sink is resolved.
+- [ ] Audit retention policy exists; ticket-body encryption + gRPC decisions are made and documented.
+- [ ] The mocks-registry / REQ tracker / architecture maps are honest and current.
+- [ ] 9.7–9.11 are recorded as deferred with explicit pull-triggers. `dotnet build`/`dotnet test` green.
+
+## 7. How to test
+
+- A request produces a trace with a `requestId` that matches the `ApiResult.requestId` in the response.
+- `/healthz/ready` fails when a dependency (object storage / Redis) is down; `/healthz/live` stays up.
+- Grep the logs after an OTP + a booking + a payout → no phone number, OTP, IBAN, or clinical text appears.
+
+## 8. Hand off & document
+
+- Update `server/CLAUDE.md` (observability wiring, ticket encryption, gRPC decision), the mocks-registry, and
+  the REQ tracker. Update [../server/runtime-services.md](../server/runtime-services.md) for any new
+  observability service. Save a memory note capturing the encryption/gRPC decisions and the deferral triggers.