# Refinement Phase 9 — Observability, ops hardening, docs honesty & scale-later

> **Mission:** make the running platform diagnosable and honest, and record the explicitly-deferred scale work
> so nobody mistakes it for missing MVP scope. This is the "finish before launch, then keep for later" bucket.
>
> **Track:** backend (observability/docs) + explicit deferrals · **Depends on:** nothing hard (start anytime;
> finish the observability items before launch) · **Unlocks:** production diagnosability
> **Before you start, read [../../phases/_shared/agent-operating-rules.md](../../phases/_shared/agent-operating-rules.md).**

## 1. Context

This phase **is** the server audit's **post-phase-7 (observability & ops hardening)** + **post-phase-8 (scale &
later)**. Full evidence & fixes:
**[../server/post-phase-backend-plan.md](../server/post-phase-backend-plan.md) § post-phase-7/8**.

## 2. Required reading

- **[../server/post-phase-backend-plan.md](../server/post-phase-backend-plan.md)** § post-phase-7 (7.1–7.6) and
  § post-phase-8 (8.1–8.5).
- **[../server/runtime-services.md](../server/runtime-services.md)** § 4 (Prometheus), § 17 (Elasticsearch —
  deliberately later).

## 3. Scope

### Observability & ops hardening (post-phase-7 — finish before launch)
- **9.1 — add tracing + consolidate the two metric stacks.** OTel is metrics-only (no `WithTracing`/OTLP); two
  overlapping Prometheus stacks run. Add `WithTracing` (AspNetCore + EF) exporting OTLP; pick **one** metrics
  stack; wire trace-id into `ApiResult.requestId` for support correlation.
- **9.2 — broaden health checks + split readiness/liveness.** The single check is app-DB only; add log-DB,
  object-storage (write probe), Redis (when [Phase 7](refinement-phase-7-unattended-ops.md) lands); expose
  `/healthz/live` vs `/healthz/ready`; remove the dead `currentUrl` line.
- **9.3 — revisit prod log levels + notification channels.** Deployed envs write **Warning+ only** (every
  Information-level audit trail dropped); raise to Information+ with retention (or a file/OTLP sink); ensure
  **no PII is logged** (the SMS OTP log disappears with [Phase 8](refinement-phase-8-external-rails.md) 5.1);
  delete the dead Elasticsearch sink block + package or revive it deliberately.
- **9.4 — audit-log growth & archival.** `audit_logs` is append-only with no retention (and Phase 6 §6.3 grows
  it); add a retention/archival policy as a [Phase 7](refinement-phase-7-unattended-ops.md) job; define legal
  retention for money/verification rows first.
- **9.5 — decide `TicketMessage.Body` encryption + the gRPC plugin's fate.** Ticket bodies are the refund/dispute
  paper trail (users type phone numbers, addresses, clinical detail) and are plaintext with no documented
  decision — recommend encrypting via the existing converter pattern. The gRPC plugin duplicates only the
  OTP/token flow, forces the HTTP/2 posture, and enables reflection unconditionally — remove it or give it a
  dedicated HTTP/2 endpoint + disable reflection outside Development.
- **9.6 — keep the docs honest.** Prune the stale mocks-registry duplicate rows, rename the `IJobScheduler` row
  to "recurring jobs", mark delivered/answered REQs, note `IPaymentCaptureSimulator`'s removal. (Root `CLAUDE.md`
  rule 7 — stale instructions are worse than none.)

### Scale & later (post-phase-8 — explicitly NOT MVP; record, don't build)
- **9.7 — Elasticsearch read backend + outbox feeder.** `SqlNurseSearch` is real and correct; `Search:Backend`
  fails fast on any non-`sql` value. Build `ElasticNurseSearch` + the outbox/CDC feeder **only when SQL search
  shows strain**. Not now.
- **9.8 — analytics pipeline.** `IAnalyticsSink` writes `ops.SystemEvents` fire-and-forget; pipe to a
  warehouse/stream when product needs it.
- **9.9 — holiday-calendar feed.** The table is manually maintained; a yearly ops-checklist item is an
  acceptable alternative to a lunar-Hijri drift feed.
- **9.10 — push/SMS notification channels.** `InAppNotificationDispatcher` drops non-InApp channels; add
  fan-out (SMS via [Phase 8](refinement-phase-8-external-rails.md) 5.1's sender, FCM push) when the UX demands.
- **9.11 — deferred product tables** (`organizations`, `organization_nurses`, `fraud_flags`,
  `recurring_booking_schedules`, `bnpl_settlement_entries`, availability slots, customer national-ID KYC, geo
  bulk import) — all verified absent and all pure additive migrations when product pulls them. Leave documented.

## 4. Mocks & seams

None new. 9.7/9.10 are the make-it-real steps for the `INurseSearch` (Elastic backend) and
`INotificationDispatcher` (SMS/push channels) rows — deferred by design.

## 5. Critical rules

- **No PII in logs** — verify after 9.3 (the OTP log must be gone; clinical text and IBANs are never logged).
- **Deferrals are documented, not silent** — 9.7–9.11 stay explicitly out of MVP with a written "when to pull
  it" trigger, so a future reader doesn't mistake them for gaps.
- **Don't build Elasticsearch/analytics/push "because"** — each has a concrete trigger (search strain, product
  need, UX demand). Until then, SQL search / in-app notifications are the real, correct MVP.

## 6. Definition of Done

- [ ] Tracing exports OTLP; one metrics stack; `requestId` carries the trace id.
- [ ] Health checks cover the real dependencies with `/healthz/live` vs `/healthz/ready`.
- [ ] Prod logs Information+ with retention and **no PII**; the dead ES sink is resolved.
- [ ] Audit retention policy exists; ticket-body encryption + gRPC decisions are made and documented.
- [ ] The mocks-registry / REQ tracker / architecture maps are honest and current.
- [ ] 9.7–9.11 are recorded as deferred with explicit pull-triggers. `dotnet build`/`dotnet test` green.

## 7. How to test

- A request produces a trace with a `requestId` that matches the `ApiResult.requestId` in the response.
- `/healthz/ready` fails when a dependency (object storage / Redis) is down; `/healthz/live` stays up.
- Grep the logs after an OTP + a booking + a payout → no phone number, OTP, IBAN, or clinical text appears.

## 8. Hand off & document

- Update `server/CLAUDE.md` (observability wiring, ticket encryption, gRPC decision), the mocks-registry, and
  the REQ tracker. Update [../server/runtime-services.md](../server/runtime-services.md) for any new
  observability service. Save a memory note capturing the encryption/gRPC decisions and the deferral triggers.