refinement phase 9

dev/shared-working-context/backend/STATUS.md

@@ -12,6 +12,25 @@ One block per completed backend phase. Newest at the top. Backend lane writes he
 - **Notes for frontend:** <anything load-bearing>
 -->
 
+## refinement-phase-9 — Observability, ops hardening, docs honesty & scale-later — 2026-07-13
+- **Shipped:** **one OpenTelemetry stack** (metrics scraped at `/metrics` + **tracing** ASP.NET Core/EF, opt-in
+  OTLP via `OpenTelemetry:Otlp:Endpoint`; prometheus-net removed) — `ApiResult.requestId` = W3C trace id.
+  **Health split** `/healthz/live` vs `/healthz/ready` (app DB + logDb[deployed] + object-storage write probe);
+  `/HealthCheck` aggregate kept. **Prod logs Information+** with **no PII** (OTP code no longer logged) + dead
+  Elasticsearch sink/package removed. **`AuditLogRetentionJob`** (`IRecurringJob`, two-tier legal retention).
+  **`TicketMessage.Body` encrypted at rest** (`IFieldEncryptor`; column → nvarchar(max)). **gRPC reflection
+  Development-only.** Docs reconciled (mocks-registry stale rows pruned; deferrals 9.7–9.11 recorded with pull-
+  triggers). Migration `RefinementPhase9TicketBodyEncryptionAndAuditRetention` (Body widen + 3 config seed rows).
+- **Contracts:** none changed — no wire/shape change (observability + docs + at-rest encryption only).
+- **Mocked:** none new. Deferred (recorded, not gaps): Elasticsearch `INurseSearch`, SMS/push
+  `INotificationDispatcher`, analytics pipeline, holiday feed, 8 product tables — each with a written pull-trigger.
+- **Gate:** build clean (0 new warnings) / **407 tests pass** (402 prior + 5 new: audit-retention, ticket-body
+  encryption, liveness).
+- **Handoff:** backend/handoff/after-refinement-phase-9.md
+- **Notes for frontend:** no client-facing change. Ticket message bodies are now encrypted at rest server-side
+  (the thread read still returns plaintext — unchanged wire). A request's `requestId` is a real trace id useful
+  for support correlation.
+
 ## refinement-phase-7 — Unattended ops: scheduler, locking & multi-instance readiness — 2026-07-13
 - **Shipped:** one in-process **`RecurringJobSchedulerHostedService`** + the **`IRecurringJob`** seam
   (`Persistence/Services/Scheduling/`) replacing the two `PeriodicTimer` hosted services and scheduling the

dev/shared-working-context/backend/handoff/after-refinement-phase-9.md

@@ -0,0 +1,43 @@
+# After refinement-phase-9 — Observability, ops hardening, docs honesty & scale-later
+
+**For the frontend / next backend phase. Backend-owned; frontend reads.**
+
+## What changed for a client
+
+**Nothing user-facing.** No route, envelope, shape, or enum changed. Two things worth knowing:
+
+- **`ApiResult.requestId` is now a real W3C trace id** (the request's OpenTelemetry trace). It's the id to quote in
+  a support ticket — it maps 1:1 to the server-side trace once an OTLP collector is wired.
+- **Ticket message bodies are encrypted at rest** server-side. The thread read still returns **plaintext** (the wire
+  is unchanged); the change is purely storage-side (the refund/dispute paper trail is no longer plaintext in the DB).
+
+## What the platform now does / exposes (ops)
+
+- **One OpenTelemetry stack.** Metrics scrape at `/metrics`; distributed **tracing** (ASP.NET Core + EF Core) is
+  wired. **OTLP export is opt-in** — set `OpenTelemetry:Otlp:Endpoint` (Grafana Tempo / Jaeger / OTEL Collector) to
+  turn trace + metric export on. Prometheus-scrape-only is an acceptable MVP; prometheus-net was removed.
+- **Health endpoints:** `/healthz/live` (process only — safe liveness), `/healthz/ready` (app DB + log DB [deployed]
+  + an object-storage write probe — pull an instance out of rotation when a dependency is down), `/HealthCheck`
+  (aggregate, kept for compat). Point the orchestrator's liveness probe at `/healthz/live`, readiness at
+  `/healthz/ready`.
+- **Prod logs are Information+ with no PII/secrets.** The OTP code is no longer logged in any environment. Log-table
+  retention on `Baya_Logs` is an ops/DBA task (or ship logs to the OTLP collector).
+- **Audit-log retention** runs as a scheduled `IRecurringJob` — two-tier (financial/verification rows kept ~7 yr,
+  everyday rows ~2 yr) via the `audit_retention_*` config keys.
+- **gRPC reflection is Development-only** (the plugin itself is unchanged; it shares the mixed-protocol listener).
+
+## For the next backend phase / deploy
+
+- **Turn tracing on in deployed envs** by provisioning an OTLP collector and setting `OpenTelemetry:Otlp:Endpoint`.
+- **When Redis lands (>1 instance)**, add a `redis` readiness check (tagged `ready`) to `ConfigureHealthChecks`.
+- **Register any new retention/cron via `IRecurringJob`** (unchanged from phase 7).
+
+## Deferred — recorded, NOT gaps (each has a written pull-trigger; see the phase report)
+
+- **Elasticsearch `INurseSearch` backend + outbox feeder** — pull when SQL search shows strain. SQL search is the
+  real MVP (`Search:Backend=sql`; any other value fails fast).
+- **SMS/push channels of `INotificationDispatcher`** — pull when the notification UX demands out-of-app reach.
+  In-app notifications are real now.
+- **Analytics warehouse/stream, holiday-calendar feed, 8 deferred product tables** (`organizations`,
+  `organization_nurses`, `fraud_flags`, `recurring_booking_schedules`, `bnpl_settlement_entries`, availability
+  slots, customer national-ID KYC, geo bulk import) — each a pure additive step when product pulls it.