hamid
95 lines
6.3 KiB
Markdown
95 lines
6.3 KiB
Markdown
# Refinement Phase 9 — Observability, ops hardening, docs honesty & scale-later
|
||
|
||
> **Mission:** make the running platform diagnosable and honest, and record the explicitly-deferred scale work
|
||
> so nobody mistakes it for missing MVP scope. This is the "finish before launch, then keep for later" bucket.
|
||
>
|
||
> **Track:** backend (observability/docs) + explicit deferrals · **Depends on:** nothing hard (start anytime;
|
||
> finish the observability items before launch) · **Unlocks:** production diagnosability
|
||
> **Before you start, read [../../phases/_shared/agent-operating-rules.md](../../phases/_shared/agent-operating-rules.md).**
|
||
|
||
## 1. Context
|
||
|
||
This phase **is** the server audit's **post-phase-7 (observability & ops hardening)** + **post-phase-8 (scale &
|
||
later)**. Full evidence & fixes:
|
||
**[../server/post-phase-backend-plan.md](../server/post-phase-backend-plan.md) § post-phase-7/8**.
|
||
|
||
## 2. Required reading
|
||
|
||
- **[../server/post-phase-backend-plan.md](../server/post-phase-backend-plan.md)** § post-phase-7 (7.1–7.6) and
|
||
§ post-phase-8 (8.1–8.5).
|
||
- **[../server/runtime-services.md](../server/runtime-services.md)** § 4 (Prometheus), § 17 (Elasticsearch —
|
||
deliberately later).
|
||
|
||
## 3. Scope
|
||
|
||
### Observability & ops hardening (post-phase-7 — finish before launch)
|
||
- **9.1 — add tracing + consolidate the two metric stacks.** OTel is metrics-only (no `WithTracing`/OTLP); two
|
||
overlapping Prometheus stacks run. Add `WithTracing` (AspNetCore + EF) exporting OTLP; pick **one** metrics
|
||
stack; wire trace-id into `ApiResult.requestId` for support correlation.
|
||
- **9.2 — broaden health checks + split readiness/liveness.** The single check is app-DB only; add log-DB,
|
||
object-storage (write probe), Redis (when [Phase 7](refinement-phase-7-unattended-ops.md) lands); expose
|
||
`/healthz/live` vs `/healthz/ready`; remove the dead `currentUrl` line.
|
||
- **9.3 — revisit prod log levels + notification channels.** Deployed envs write **Warning+ only** (every
|
||
Information-level audit trail dropped); raise to Information+ with retention (or a file/OTLP sink); ensure
|
||
**no PII is logged** (the SMS OTP log disappears with [Phase 8](refinement-phase-8-external-rails.md) 5.1);
|
||
delete the dead Elasticsearch sink block + package or revive it deliberately.
|
||
- **9.4 — audit-log growth & archival.** `audit_logs` is append-only with no retention (and Phase 6 §6.3 grows
|
||
it); add a retention/archival policy as a [Phase 7](refinement-phase-7-unattended-ops.md) job; define legal
|
||
retention for money/verification rows first.
|
||
- **9.5 — decide `TicketMessage.Body` encryption + the gRPC plugin's fate.** Ticket bodies are the refund/dispute
|
||
paper trail (users type phone numbers, addresses, clinical detail) and are plaintext with no documented
|
||
decision — recommend encrypting via the existing converter pattern. The gRPC plugin duplicates only the
|
||
OTP/token flow, forces the HTTP/2 posture, and enables reflection unconditionally — remove it or give it a
|
||
dedicated HTTP/2 endpoint + disable reflection outside Development.
|
||
- **9.6 — keep the docs honest.** Prune the stale mocks-registry duplicate rows, rename the `IJobScheduler` row
|
||
to "recurring jobs", mark delivered/answered REQs, note `IPaymentCaptureSimulator`'s removal. (Root `CLAUDE.md`
|
||
rule 7 — stale instructions are worse than none.)
|
||
|
||
### Scale & later (post-phase-8 — explicitly NOT MVP; record, don't build)
|
||
- **9.7 — Elasticsearch read backend + outbox feeder.** `SqlNurseSearch` is real and correct; `Search:Backend`
|
||
fails fast on any non-`sql` value. Build `ElasticNurseSearch` + the outbox/CDC feeder **only when SQL search
|
||
shows strain**. Not now.
|
||
- **9.8 — analytics pipeline.** `IAnalyticsSink` writes `ops.SystemEvents` fire-and-forget; pipe to a
|
||
warehouse/stream when product needs it.
|
||
- **9.9 — holiday-calendar feed.** The table is manually maintained; a yearly ops-checklist item is an
|
||
acceptable alternative to a lunar-Hijri drift feed.
|
||
- **9.10 — push/SMS notification channels.** `InAppNotificationDispatcher` drops non-InApp channels; add
|
||
fan-out (SMS via [Phase 8](refinement-phase-8-external-rails.md) 5.1's sender, FCM push) when the UX demands.
|
||
- **9.11 — deferred product tables** (`organizations`, `organization_nurses`, `fraud_flags`,
|
||
`recurring_booking_schedules`, `bnpl_settlement_entries`, availability slots, customer national-ID KYC, geo
|
||
bulk import) — all verified absent and all pure additive migrations when product pulls them. Leave documented.
|
||
|
||
## 4. Mocks & seams
|
||
|
||
None new. 9.7/9.10 are the make-it-real steps for the `INurseSearch` (Elastic backend) and
|
||
`INotificationDispatcher` (SMS/push channels) rows — deferred by design.
|
||
|
||
## 5. Critical rules
|
||
|
||
- **No PII in logs** — verify after 9.3 (the OTP log must be gone; clinical text and IBANs are never logged).
|
||
- **Deferrals are documented, not silent** — 9.7–9.11 stay explicitly out of MVP with a written "when to pull
|
||
it" trigger, so a future reader doesn't mistake them for gaps.
|
||
- **Don't build Elasticsearch/analytics/push "because"** — each has a concrete trigger (search strain, product
|
||
need, UX demand). Until then, SQL search / in-app notifications are the real, correct MVP.
|
||
|
||
## 6. Definition of Done
|
||
|
||
- [ ] Tracing exports OTLP; one metrics stack; `requestId` carries the trace id.
|
||
- [ ] Health checks cover the real dependencies with `/healthz/live` vs `/healthz/ready`.
|
||
- [ ] Prod logs Information+ with retention and **no PII**; the dead ES sink is resolved.
|
||
- [ ] Audit retention policy exists; ticket-body encryption + gRPC decisions are made and documented.
|
||
- [ ] The mocks-registry / REQ tracker / architecture maps are honest and current.
|
||
- [ ] 9.7–9.11 are recorded as deferred with explicit pull-triggers. `dotnet build`/`dotnet test` green.
|
||
|
||
## 7. How to test
|
||
|
||
- A request produces a trace with a `requestId` that matches the `ApiResult.requestId` in the response.
|
||
- `/healthz/ready` fails when a dependency (object storage / Redis) is down; `/healthz/live` stays up.
|
||
- Grep the logs after an OTP + a booking + a payout → no phone number, OTP, IBAN, or clinical text appears.
|
||
|
||
## 8. Hand off & document
|
||
|
||
- Update `server/CLAUDE.md` (observability wiring, ticket encryption, gRPC decision), the mocks-registry, and
|
||
the REQ tracker. Update [../server/runtime-services.md](../server/runtime-services.md) for any new
|
||
observability service. Save a memory note capturing the encryption/gRPC decisions and the deferral triggers.
|