refinement phase 9

dev/post-phase/server/runtime-services.md

@@ -15,6 +15,15 @@ is invented; "not needed" claims are backed by the absence of the package/code.
 > and flipping the selector turns each on, no code change. What is still genuinely absent (no adapter): **Redis**
 > (5), **Elasticsearch** (17), and the LLM **review-moderation classifier** (15, optional). MoH/INO/eNamad (16)
 > stay **manual by design**. See the mocks-registry refinement-phase-8 banner for the provider tokens per rail.
+>
+> **Refinement-phase-9 update (2026-07-13) — observability (service 4):** the two overlapping metric stacks were
+> consolidated onto **one OpenTelemetry stack** — prometheus-net was removed; metrics are scraped at `/metrics` via
+> the OTel Prometheus exporter, and **distributed tracing** (ASP.NET Core + EF Core) was added, exporting **OTLP
+> only when `OpenTelemetry:Otlp:Endpoint` is configured** (an MVP with Prometheus alone is unchanged). A request's
+> `ApiResult.requestId` is its W3C trace id (support ↔ trace 1:1). Health checks split into `/healthz/live`
+> (process) vs `/healthz/ready` (app DB + log DB [deployed] + an object-storage write probe); `/HealthCheck` stays
+> as the aggregate. An **OTLP collector** (Grafana Tempo / Jaeger / OTEL Collector) becomes the optional new
+> observability service when trace export is turned on. Elasticsearch (17) is still deferred.
 
 ## Service inventory
 
@@ -23,7 +32,7 @@ is invented; "not needed" claims are backed by the absence of the package/code.
 | 1 | **SQL Server** (app DB `Baya`) | System of record — 12 schemas (`usr ops geo catalog verif search booking payments payouts reviews messaging partner`) | EF Core; `ConnectionStrings:SqlServer` | **Required now** | — |
 | 2 | **SQL Server** (log DB `Baya_Logs`) | Serilog sink in deployed envs (Warning+, auto-created `log.LogEvents`) | `ConnectionStrings:logDb` | **Required now** (deployed) | — |
 | 3 | **Reverse proxy / TLS** (nginx·caddy·traefik) | TLS termination, HTTP/1.1+2, forwarded headers | Kestrel config; JWE bearer | **Required now** | — |
-| 4 | **Prometheus** (+ Grafana) | Scrapes `/metrics`; health forwarded to gauges | `UseMetricServer` + OTel exporter | **Recommended now** | — |
+| 4 | **Prometheus** (+ Grafana) · optional **OTLP collector** | Scrapes `/metrics` (one OTel stack, refinement-phase-9); traces export OTLP when configured | OTel metrics + tracing; `OpenTelemetry:Otlp:Endpoint` | **Recommended now** (collector optional) | — |
 | 5 | **Redis** | `ICacheService` + `IDistributedLock` (money-path mutex) | `Seams:*` (keys TBD; none today) | Before >1 API instance | rows 14, 42 |
 | 6 | **MinIO / S3 / ArvanCloud** | `IObjectStorage` — verification docs, avatars (REQ-006), invoice PDFs | `Seams:ObjectStorage:*` | Before real verification | row 13 |
 | 7 | **Job scheduler** — in-process, SQL only (refinement-phase-7 **done**) | The recurring crons: booking-expiry, notification-retention, credential-expiry scan, no-show sweep, weekly payout-batch generation (Moadian/refund-settlement poll = Phase 8) | `RecurringJobSchedulerHostedService` + `IRecurringJob`s | **Done for single instance** (no new infra) | row 26 |
@@ -133,8 +142,9 @@ flowchart LR
   Both DBs fit one instance; `Baya_Logs` can move later.
 - **Config:** `ConnectionStrings:SqlServer`, `ConnectionStrings:logDb` — **rotate + externalize first**
   (plan §1.1; live `sa` credentials are committed today).
-- **Health/readiness:** the app's only health check (`/HealthCheck`,
-  `Monitoring/Configurations/HealthCheckConfigurations.cs:17`); `logDb` has none (plan §7.2). Boot runs
+- **Health/readiness (refinement-phase-9 — §7.2 landed):** split `/healthz/live` (process) vs `/healthz/ready`
+  (app DB + `logDb` [deployed] + object-storage write probe); `/HealthCheck` stays as the aggregate
+  (`Monitoring/Configurations/HealthCheckConfigurations.cs`). Boot runs
   `MigrateAsync` + 3 seeders (`Program.cs:99-104`) → the login needs DDL rights and concurrent multi-node
   boot races (plan §4.3).
 
@@ -149,13 +159,16 @@ flowchart LR
 - **Default:** caddy 2 / nginx 1.27; terminate TLS, h2 to clients, HTTP/1.1 (or h2c) upstream once §1.5
   lands.
 
-### 4 · Prometheus (+ Grafana)
+### 4 · Prometheus (+ Grafana) · optional OTLP collector
 
-- **Evidence:** `/metrics` via prometheus-net `UseMetricServer` + OTel `AddPrometheusExporter` (two stacks —
-  consolidate, plan §7.1) at `Monitoring/Configurations/PrometheusMetricsConfigurations.cs:11` and
-  `OpenTelemetryConfigurations.cs:21`; health forwarded (`HealthCheckConfigurations.cs:18`).
-- **Default:** `prom/prometheus:v2.53` + `grafana/grafana:11`. No tracing backend exists yet (metrics-only);
-  an OTLP collector becomes relevant with plan §7.1.
+- **Evidence (refinement-phase-9 — §7.1 landed):** **one** OpenTelemetry stack — metrics scraped at `/metrics` via
+  `UseOpenTelemetryPrometheusScrapingEndpoint()` (`Monitoring/Configurations/PrometheusMetricsConfigurations.cs`),
+  plus `WithTracing` (ASP.NET Core + EF Core) exporting **OTLP only when `OpenTelemetry:Otlp:Endpoint` is set**
+  (`OpenTelemetryConfigurations.cs`). The duplicate prometheus-net stack (`UseMetricServer`/`UseHttpMetrics`/
+  `ForwardToPrometheus` + packages) was removed. A request's `ApiResult.requestId` is its W3C trace id.
+- **Default:** `prom/prometheus:v2.53` + `grafana/grafana:11`. A **tracing backend / OTLP collector** (Grafana
+  Tempo · Jaeger · OpenTelemetry Collector) is the optional new service — set `OpenTelemetry:Otlp:Endpoint` at it to
+  turn trace + metric export on; Prometheus-scrape-only is an acceptable MVP.
 
 ### 5 · Redis
 
@@ -278,14 +291,16 @@ flowchart LR
 2. **Environment files:** `appsettings.json` ≡ `appsettings.Development.json` (byte-identical); **no
    Production/Staging file exists.** All non-secret env differences ride on ~14 `Seams:*` groups whose
    defaults live in code (`SeamOptions.cs`), not in config files.
-3. **HTTP posture:** HTTP/2-only Kestrel default (plan §1.5), gRPC plugin + reflection always on
-   (plan §7.5), TLS required for JWE sanity.
+3. **HTTP posture:** mixed `Http1AndHttp2` Kestrel (refinement-phase-5), so gRPC shares the listener via ALPN;
+   gRPC **reflection is Development-only** (refinement-phase-9 §7.5); TLS required for JWE sanity.
 4. **Single-instance constraints today:** in-memory cache, in-proc money lock, the in-proc recurring-job
    scheduler (refinement-phase-7 — its per-tick lock is that same in-proc seam), per-instance rate-limit buckets.
    §4.3 (migrations split from boot) **landed**; scaling past one instance still requires §4.2 (Redis for the
    shared cache + the cross-instance lock the scheduler/money path use) first — the DB uniques keep money
    *correct* either way, but locks/cache/limits/scheduler-de-dup silently degrade.
-5. **Logs:** deployed envs write Warning+ to `Baya_Logs` only (Information dropped — plan §7.3); dev writes
-   console + `logs/log.json`.
+5. **Logs (refinement-phase-9 — §7.3 landed):** deployed envs write **Information+** to `Baya_Logs` (framework
+   categories held at Warning); **no PII/secrets** (the OTP code is no longer logged in any env). The dead
+   Elasticsearch sink + package were removed; log-table retention is an ops/DBA task (or ship logs to the OTLP
+   collector). Dev writes console + `logs/log.json`.
 6. **Client:** the Next.js app needs `NEXT_PUBLIC_API_URL` pointing at the proxy; wire casing camelCase;
    snake_case routes.

dev/shared-working-context/backend/STATUS.md

@@ -12,6 +12,25 @@ One block per completed backend phase. Newest at the top. Backend lane writes he
 - **Notes for frontend:** <anything load-bearing>
 -->
 
+## refinement-phase-9 — Observability, ops hardening, docs honesty & scale-later — 2026-07-13
+- **Shipped:** **one OpenTelemetry stack** (metrics scraped at `/metrics` + **tracing** ASP.NET Core/EF, opt-in
+  OTLP via `OpenTelemetry:Otlp:Endpoint`; prometheus-net removed) — `ApiResult.requestId` = W3C trace id.
+  **Health split** `/healthz/live` vs `/healthz/ready` (app DB + logDb[deployed] + object-storage write probe);
+  `/HealthCheck` aggregate kept. **Prod logs Information+** with **no PII** (OTP code no longer logged) + dead
+  Elasticsearch sink/package removed. **`AuditLogRetentionJob`** (`IRecurringJob`, two-tier legal retention).
+  **`TicketMessage.Body` encrypted at rest** (`IFieldEncryptor`; column → nvarchar(max)). **gRPC reflection
+  Development-only.** Docs reconciled (mocks-registry stale rows pruned; deferrals 9.7–9.11 recorded with pull-
+  triggers). Migration `RefinementPhase9TicketBodyEncryptionAndAuditRetention` (Body widen + 3 config seed rows).
+- **Contracts:** none changed — no wire/shape change (observability + docs + at-rest encryption only).
+- **Mocked:** none new. Deferred (recorded, not gaps): Elasticsearch `INurseSearch`, SMS/push
+  `INotificationDispatcher`, analytics pipeline, holiday feed, 8 product tables — each with a written pull-trigger.
+- **Gate:** build clean (0 new warnings) / **407 tests pass** (402 prior + 5 new: audit-retention, ticket-body
+  encryption, liveness).
+- **Handoff:** backend/handoff/after-refinement-phase-9.md
+- **Notes for frontend:** no client-facing change. Ticket message bodies are now encrypted at rest server-side
+  (the thread read still returns plaintext — unchanged wire). A request's `requestId` is a real trace id useful
+  for support correlation.
+
 ## refinement-phase-7 — Unattended ops: scheduler, locking & multi-instance readiness — 2026-07-13
 - **Shipped:** one in-process **`RecurringJobSchedulerHostedService`** + the **`IRecurringJob`** seam
   (`Persistence/Services/Scheduling/`) replacing the two `PeriodicTimer` hosted services and scheduling the

dev/shared-working-context/backend/handoff/after-refinement-phase-9.md

@@ -0,0 +1,43 @@
+# After refinement-phase-9 — Observability, ops hardening, docs honesty & scale-later
+
+**For the frontend / next backend phase. Backend-owned; frontend reads.**
+
+## What changed for a client
+
+**Nothing user-facing.** No route, envelope, shape, or enum changed. Two things worth knowing:
+
+- **`ApiResult.requestId` is now a real W3C trace id** (the request's OpenTelemetry trace). It's the id to quote in
+  a support ticket — it maps 1:1 to the server-side trace once an OTLP collector is wired.
+- **Ticket message bodies are encrypted at rest** server-side. The thread read still returns **plaintext** (the wire
+  is unchanged); the change is purely storage-side (the refund/dispute paper trail is no longer plaintext in the DB).
+
+## What the platform now does / exposes (ops)
+
+- **One OpenTelemetry stack.** Metrics scrape at `/metrics`; distributed **tracing** (ASP.NET Core + EF Core) is
+  wired. **OTLP export is opt-in** — set `OpenTelemetry:Otlp:Endpoint` (Grafana Tempo / Jaeger / OTEL Collector) to
+  turn trace + metric export on. Prometheus-scrape-only is an acceptable MVP; prometheus-net was removed.
+- **Health endpoints:** `/healthz/live` (process only — safe liveness), `/healthz/ready` (app DB + log DB [deployed]
+  + an object-storage write probe — pull an instance out of rotation when a dependency is down), `/HealthCheck`
+  (aggregate, kept for compat). Point the orchestrator's liveness probe at `/healthz/live`, readiness at
+  `/healthz/ready`.
+- **Prod logs are Information+ with no PII/secrets.** The OTP code is no longer logged in any environment. Log-table
+  retention on `Baya_Logs` is an ops/DBA task (or ship logs to the OTLP collector).
+- **Audit-log retention** runs as a scheduled `IRecurringJob` — two-tier (financial/verification rows kept ~7 yr,
+  everyday rows ~2 yr) via the `audit_retention_*` config keys.
+- **gRPC reflection is Development-only** (the plugin itself is unchanged; it shares the mixed-protocol listener).
+
+## For the next backend phase / deploy
+
+- **Turn tracing on in deployed envs** by provisioning an OTLP collector and setting `OpenTelemetry:Otlp:Endpoint`.
+- **When Redis lands (>1 instance)**, add a `redis` readiness check (tagged `ready`) to `ConfigureHealthChecks`.
+- **Register any new retention/cron via `IRecurringJob`** (unchanged from phase 7).
+
+## Deferred — recorded, NOT gaps (each has a written pull-trigger; see the phase report)
+
+- **Elasticsearch `INurseSearch` backend + outbox feeder** — pull when SQL search shows strain. SQL search is the
+  real MVP (`Search:Backend=sql`; any other value fails fast).
+- **SMS/push channels of `INotificationDispatcher`** — pull when the notification UX demands out-of-app reach.
+  In-app notifications are real now.
+- **Analytics warehouse/stream, holiday-calendar feed, 8 deferred product tables** (`organizations`,
+  `organization_nurses`, `fraud_flags`, `recurring_booking_schedules`, `bnpl_settlement_entries`, availability
+  slots, customer national-ID KYC, geo bulk import) — each a pure additive step when product pulls it.

dev/shared-working-context/reports/frontend-phase-9-report.md

@@ -146,7 +146,7 @@ A 26-agent review/verify pass over the diff confirmed and fixed, pre-merge:
 - **i18n/UX:** inline initiate errors always use the localized copy (never raw `ApiError.message`); en
   `cta_pay` arrow points → (fa keeps ←); fa `error_body` matches the app's «بارگذاری … ممکن نشد» pattern;
   the C6 service-cost row carries the quantity (`row_service_cost_with_count`); the invoice issuer line
-  uses the product spelling «بالین‌یار» (note: fa `common.brand` reads «بلینیار» — a pre-existing
+  uses the product spelling «بالین‌یار» (note: fa `common.brand` reads «بالین یار» — a pre-existing
   wordmark/product-spelling divergence worth a product decision).
 - **Dark scheme:** EscrowNotice text/border use `--bal-primary` (the info token is an alert *background*
   and is illegible as dark-mode text); the print button temporarily flips `data-mui-color-scheme` to

dev/shared-working-context/reports/mocks-registry.md

@@ -8,6 +8,16 @@ seam updates its row. This is the checklist the team works through to go from "M
 Status legend: 🔴 not built · 🟡 mocked (seam + fake impl in place) · 🟢 real integration live · 🟢◐ **real
 adapter shipped, config-selected (mock remains the default/fallback)** — the refinement-phase-8 state.
 
+> **Refinement-phase-9 — docs honesty + observability (2026-07-13).** §7.6: pruned the **stale duplicate 🔴 rows**
+> (`IDistributedLock`/`INurseSearch`/`IPaymentProvider`/`ISettlementSplitProvider`/`IWebhookVerifier`/`IMoadianClient`/
+> `ILicenseVerificationService`) that the detailed rows below already correct; the recurring-jobs row is the real
+> in-process scheduler; `IPaymentCaptureSimulator` now reflects its 6.4 prod removal. **No seam was un-mocked** — the
+> deferred **Elasticsearch `INurseSearch` backend** (row) and the **SMS/push channels of `INotificationDispatcher`**
+> (row) stay explicitly out of MVP with the pull-triggers in their "Make it real →" columns (SQL search / in-app
+> notifications are the real MVP). Also removed the unused `Serilog.Sinks.Elasticsearch` package (dead sink block);
+> tracing/metrics unified on OpenTelemetry; audit-log retention is now a scheduled `IRecurringJob`; `TicketMessage.Body`
+> is encrypted at rest (§9.5). See `backend-phase-9-report.md` (refinement) for the full observability + deferral list.
+
 > **Refinement-phase-8 — external rails go real (2026-07-13).** Every vendor rail below now has a **real HTTP
 > adapter behind the same seam**, config-selected via a per-rail **`Seams:*:Provider`** selector (default = the
 > mock, so an unconfigured environment is unchanged; a typo falls closed to the mock). Selecting a real provider
@@ -39,11 +49,6 @@ adapter shipped, config-selected (mock remains the default/fallback)** — the r
 | `ISmsSender` | backend-phase-2 | OTP/SMS delivery — `LoggingSmsSender` (`Baya.Infrastructure.CrossCutting/Seams/`) logs the OTP code (phone shown as last-4 only) and returns success; registered singleton in `AddCrossCuttingSeams`. **refinement-phase-0:** in **Development only**, `DevCapturingSmsSender` decorates it (via `AddDevelopmentOtpCapture`, called from `Program.cs` inside `IsDevelopment()`) to also capture the code in `DevOtpStore` for the `GET /api/v1/dev/last_otp/{phone}` bring-up helper — not wired / 404 outside Development | none today; real client will need `Seams:Sms:ApiKey` + `Seams:Sms:SenderLine` (+ gateway base URL) | 1) pick a gateway (Kavenegar/Ghasedak/SMS.ir), add its client package to `Directory.Packages.props`; 2) implement `ISmsSender.SendOtpAsync`/`SendAsync` against it (template/pattern-based OTP send); 3) bind the new `Seams:Sms` options; 4) swap the registration in `AddCrossCuttingSeams` (config-selected) — handlers unchanged; 5) keep the per-phone resend window + `otp` rate-limit policy exactly as-is; test with a real SIM | 🟡 |
 | `IObjectStorage` | backend-phase-0/6 | File storage — local-disk store under a scratch root (`LocalDiskObjectStorage`, `Baya.Infrastructure.CrossCutting/Seams/`) | `Seams:ObjectStorage:RootPath` (default: temp dir) | Point at MinIO/S3/ArvanCloud; presigned upload/download; bucket + creds | 🟡 |
 | `ICacheService` | backend-phase-0 | Caching — in-memory `IMemoryCache` (`MemoryCacheService`, `Baya.Infrastructure.CrossCutting/Seams/`) | _none_ | Swap to Redis (`StackExchange.Redis`); keep key/TTL scheme. **refinement-phase-7: this is the >1-instance scale-out gate** — a single-instance MVP intentionally keeps the in-proc cache (its generation-token invalidation is process-local); add Redis only when a second API instance runs | 🟡 (in-proc is correct single-instance) |
-| `IDistributedLock` | backend-phase-10 | Money-path locks — no-op/in-proc | _tbd_ | Redis lock (RedLock); DB constraint remains the backstop | 🔴 |
-| `INurseSearch` | backend-phase-7 | Search — SQL over `nurse_search_index` | _tbd_ | Elasticsearch index + feeder; reimplement the interface | 🔴 |
-| `IPaymentProvider` | backend-phase-10 | Card PSP/IPG — deterministic success | _tbd_ | ZarinPal/Sadad/Vandar/Jibit + Shaparak; merchant/terminal/تسهیم | 🔴 |
-| `ISettlementSplitProvider` | backend-phase-10 | تسهیم split — accepts any balanced legs | _tbd_ | Provider split-by-ratio to registered Shebas | 🔴 |
-| `IWebhookVerifier` | backend-phase-10 | Callback auth — always valid | _tbd_ | Per-provider HMAC/signature + server-side re-verify | 🔴 |
 | `IBnplProvider` | backend-phase-12 | BNPL — `MockBnplProvider` drives the full state machine (eligible→settled→reverted), settle returns `order − commission%` | `Seams:Bnpl:{CommissionRate,SettlementInstant,CreditCeilingIrr,NotEligibleMobile,ForceFailure,ReverseProviderCommission}` | SnappPay/Digipay OAuth + verb set; encrypted creds in `payment_gateways.config_json` | 🟡 |
 | `IBnplProviderResolver` | backend-phase-12 | Per-`provider_code` selection — maps every known code to the one mock | _none_ | One concrete adapter per code; resolver returns the right one | 🟡 |
 | `ICurrencyNormalizer` | backend-phase-12 | Toman↔IRR — ×10 at the boundary | `Seams:Currency:TomanToIrrMultiplier` (default `10`) | Config-driven per provider boundary | 🟡 |
@@ -56,12 +61,10 @@ adapter shipped, config-selected (mock remains the default/fallback)** — the r
 | `ICredentialVerifier` | backend-phase-6 | MoH پروانه صلاحیت حرفه‌ای / INO / عدم سوء پیشینه verification — `MockCredentialVerifier` (`.../Seams/`) **is** the manual-admin default: every call returns `RequiresManualReview` with `verification_method=manual` (an admin verifies the uploaded document against the official portal in `AdminReviewStep`). No portal call; registered singleton. There is **no public B2B API** for MoH/INO, so this stays manual until one appears | _none_ | 1) when an MoH/INO portal or API becomes available, implement `VerifyAsync(credentialType, credentialNumber)` to return `Verified`/`Failed` with `verification_method=portal|api` (+ `external_response_json`); 2) swap the registration (config-selected) for those credential types — the manual path stays the fallback; 3) the structured `nurse_credentials` registry already stores number/authority/expiry so cross-check + renewal survive the swap. **MoH/INO have no public B2B API today** | 🟡 |
 | `IBankAccountOwnershipVerifier` | backend-phase-3 | استعلام شبا IBAN-owner ↔ national-id inquiry — `MockBankAccountOwnershipVerifier` (`Baya.Infrastructure.CrossCutting/Seams/`) returns a deterministic fake: every IBAN matches (`matched_national_id=true`, echoes a holder name + `MOCK-SHEBA-{sha}` vendor ref) except the configured mismatch IBAN which returns `false`; registered singleton in `AddCrossCuttingSeams`. No real bank/KYC call, no money moves | `Seams:BankOwnership:MismatchIban` (default `IR000000000000000000000000`), `Seams:BankOwnership:MatchedHolderName`, `Seams:BankOwnership:MismatchHolderName` | 1) pick a Finnotech / banking-bridge استعلام شبا provider, add its client package to `Directory.Packages.props`; 2) add `Seams:BankOwnership:{ApiKey,BaseUrl}` options; 3) implement `VerifyOwnershipAsync(iban, nurseNationalId)` against the real Sheba-owner inquiry, mapping to `OwnershipInquiryResult`; 4) persist the real `ownership_vendor_ref` (+ raw response if a column is added); 5) swap the registration in `AddCrossCuttingSeams` (config-selected) — handlers unchanged; 6) test match/mismatch + that the b13 first-payout gate honours `matched_national_id=true` | 🟡 |
 | `IGeocoder` | backend-phase-4 | Address→lat/lng — `MockGeocoder` (`Baya.Infrastructure.CrossCutting/Seams/`) returns deterministic `decimal` coordinates jittered (FNV-1a, ~±5 km) around the known city centroid (unknown city → Iran centroid) plus `formatted_address` + `confidence`; **no network call**. A global switch or a per-address marker forces the null-coordinate ("no map pin") path; registered singleton in `AddCrossCuttingSeams` | `Seams:Geocoding:ReturnNullCoordinates` (default `false`), `Seams:Geocoding:LowConfidenceMarker` (default `NO_GEO`), `Seams:Geocoding:ResolvedConfidence` (default `0.9`) | 1) pick Neshan (or Google) geocoding, add its client package to `Directory.Packages.props`; 2) add `Seams:Geocoding:{ApiKey,BaseUrl}` options; 3) implement `IGeocoder.GeocodeAsync(addressText, cityName, districtName?)` against it, mapping to `(lat, lng, formatted_address, confidence)` with `decimal` coords; 4) add rate-limit/retry; 5) swap the registration in `AddCrossCuttingSeams` (config-selected) — handlers unchanged; 6) test a known Tehran address resolves within expected bounds | 🟡 |
-| `IMoadianClient` | backend-phase-11 | سامانه مودیان e-invoice — leaves ref pending | _tbd_ | Real مودیان submission → 22-digit ref | 🔴 |
 | `IReviewModerationService` | backend-phase-14 | AI review pre-screen — `MockReviewModerationService` (`Baya.Infrastructure.CrossCutting/Seams/`), **no external call**: `ScreenAsync(reviewText)` returns a `ModerationVerdict(Decision, Reason)` — a banned-word substring hit → `Reject` (`banned_word:{w}`); otherwise clean text → a human-review `Flag` by default (so the publish gate holds), or `Approve` when `AutoApproveClean` is set. The `SubmitReview` handler maps the verdict to the initial status (`Approve`→published, `Reject`→hidden, else pending) — **decision authority stays with `ModerateReviewCommand` (human override)**. Registered singleton in `AddCrossCuttingSeams` | `Seams:ReviewModeration:AutoApproveClean` (default `false`), `Seams:ReviewModeration:BannedWords` (default `scam,fraud,کلاهبردار`) | 1) pick a text classifier / LLM moderation endpoint, add its client package to `Directory.Packages.props`; 2) add `Seams:ReviewModeration:{ApiKey,BaseUrl}` options; 3) implement `ScreenAsync(reviewText)` → map the provider's toxicity/spam scores to `Approve`/`Flag`/`Reject` + a reason; 4) swap the registration in `AddCrossCuttingSeams` (config-selected) — `SubmitReviewCommand`/`ModerateReviewCommand` unchanged, and the human moderation path always overrides; 5) test clean/flagged/rejected dispositions + that the publish gate still holds for a `Flag` | 🟡 |
 | `IFieldEncryptor` | backend-phase-0 | PII encryption — AES-256-CBC + HMAC hash from a local symmetric key (`SymmetricFieldEncryptor`, `Baya.Infrastructure.CrossCutting/Seams/`) | `Seams:FieldEncryption:Key`, `Seams:FieldEncryption:HashKey` | KMS / column encryption / Key Vault / HSM | 🟡 |
 | `INotificationDispatcher` | backend-phase-0/**1** | Notification channels — **in-app write is now real** (`InAppNotificationDispatcher`, `Persistence/Services/Notifications/`, writes an `ops.Notifications` row); b0 log stub removed. SMS/push channels still deferred (no-op) behind the same seam | _none_ | Add SMS (`ISmsSender`) / push (FCM) channels; polling → Redis pub/sub or SignalR later | 🟡 |
-| `ILicenseVerificationService` | backend-phase-15 | eNamad / MoH establishment-permit — manual approve | _tbd_ | Real registry/API | 🔴 |
-| `IPaymentCaptureSimulator` | backend-phase-9 | The **temporary conversion trigger** standing in for b10's real card capture. `MockPaymentCaptureSimulator` (`Baya.Infrastructure.CrossCutting/Seams/`) returns a deterministic *succeeded* capture (a fake `gateway_reference` + a configurable `psp_fee_amount`) so `ConvertRequestToBookingCommand` is exercisable now; a config switch forces a *failed* capture (→ no booking is created). **This is the trigger, not a parallel money path** — registered singleton in `AddCrossCuttingSeams` | `Seams:PaymentCapture:ForceFailure` (default `false`), `Seams:PaymentCapture:PspFeeAmount` (default unset) | In b10: 1) build the real card capture (`payment_transactions`, PSP/IPG client, webhook verify); 2) on a real `payment_transactions.succeeded`, call `ConvertRequestToBooking` **directly** (the same conversion command that computes the three-amount split + generates sessions) instead of this seam; 3) remove the `IPaymentCaptureSimulator` registration + `MockPaymentCaptureSimulator`; the conversion/idempotency logic is unchanged | 🟡 |
+| `IPaymentCaptureSimulator` | backend-phase-9 → **removed from prod refinement-phase-8 §6.4** | The **temporary conversion trigger** that stood in for b10's real card capture. **Prod now gets the fail-closed `DisabledPaymentCaptureSimulator`**; only Dev/Testing re-register the succeeding `MockPaymentCaptureSimulator` (the `bookings/convert` path is a Dev/Testing affordance — prod converts via the b10 webhook confirm calling `ConvertRequestToBooking` directly). Registered in `AddCrossCuttingSeams` (prod) + re-registered in Dev/Testing from `Program.cs` | `Seams:PaymentCapture:ForceFailure` (default `false`), `Seams:PaymentCapture:PspFeeAmount` (default unset) | Nothing further for prod — the real conversion trigger is the b10 webhook confirm. The Dev/Testing mock stays as a test affordance; drop it only if/when `bookings/convert` is retired | 🟢 (prod fail-closed; Dev/Testing mock is a test affordance) |
 | `INurseSearch` | backend-phase-7 | The search-service seam (read side). **The MVP impl `SqlNurseSearch` (`Persistence/Services/Search/`) is REAL, not a mock** — it reads the maintained `nurse_search_index WHERE is_searchable=1`, applies the category/city/district (NULL=whole-city)/gender/price filters + rating sort + pagination, projected & `AsNoTracking`. Registered by `AddPersistenceServices`, config-selected. Only the DEFERRED Elasticsearch backend is unbuilt | `Search:Backend` (default `sql`; any other value throws until Elastic ships) | 1) add an Elasticsearch client package (`Elastic.Clients.Elasticsearch`) to `Directory.Packages.props`; 2) define the index mapping (the `NurseSearchResultDto` fields + `is_searchable`); 3) implement `ElasticNurseSearch : INurseSearch` (same filters/sort/paging) reading the ES index; 4) build the feeder that consumes the `ISearchIndexMaintainer` change events via an **outbox/CDC** stream into ES (see the next row); 5) point `Search:Backend=elastic` in config — **callers unchanged**; 6) keep the SQL index as the projection/fallback + the reconciliation source (`RebuildAsync`); 7) test filter/sort/paging parity vs `SqlNurseSearch` | 🟢 SQL real; Elastic 🟡 |
 | `IPaymentProvider` | backend-phase-10 | Card PSP acquirer — `MockPaymentProvider` (`Baya.Infrastructure.CrossCutting/Seams/`), **no external call**: `InitPaymentAsync` → a deterministic `gatewayReferenceCode` (`mock-ref-{requestId}-{key}`) + a fake redirect URL; `VerifyAsync` → instant `Succeeded` echoing the expected amount (the server-side re-check); `RefundAsync(ref, amount, idempotencyKey, ct)` → always `Succeeded`, echoes a deterministic refund ref (b11 refunds carry the booking+refund idempotency key so a retry never double-refunds). Registered singleton in `AddCrossCuttingSeams` | none today; real client needs merchant id + terminal/IBAN registration + sandbox flag from `payment_gateways.config_json` (encrypted), not appsettings | 1) pick ZarinPal/Sadad/Vandar/Jibit as an acquirer-with-تسهیم, add its client package to `Directory.Packages.props`; 2) implement `InitPaymentAsync` (open the IPG session, return the Shaparak-routed redirect + reference), `VerifyAsync` (the mandatory server-side `verify` re-check of amount + reference — **never trust the callback alone**), `RefundAsync`; 3) read merchant id/terminal from the encrypted `payment_gateways.config_json`; 4) a config-driven `IProviderRegistry`/factory selects the concrete provider per gateway so a cut-off provider swaps without code change; 5) persist the full gateway response into `gateway_response_json`; 6) swap the registration (config-selected) — handlers unchanged | 🟡 |
 | `ISettlementSplitProvider` | backend-phase-10 | تسهیم settlement-sharing — `MockSettlementSplitProvider` (`.../Seams/`) records the split intent and returns `Settled` for any legs whose sum is positive; the platform never moves money. Registered singleton in `AddCrossCuttingSeams` | none today; real client needs each beneficiary's registered SHEBA + split-by-ratio config | 1) pick the acquirer's تسهیم API, implement `RegisterSplitAsync(bookingId, legs)` to register the split-by-ratio to each beneficiary's **registered IBAN** (nurse payout + platform commission), honouring the ~100,000 IRR min-amount caveat; 2) resolve each nurse's SHEBA from `nurse_bank_accounts` (the b3 `matched_national_id` gate) and the platform SHEBA from config; 3) `GetSplitStatusAsync` polls the provider; the provider credits IBANs directly — the ledger only mirrors it; 4) swap the registration (config-selected) | 🟡 |

dev/shared-working-context/reports/refinement-phase-9-report.md

@@ -0,0 +1,110 @@
+# Refinement Phase 9 — Observability, ops hardening, docs honesty & scale-later — Report (2026-07-13)
+
+**Track:** backend (observability/docs) + explicit deferrals · **Depends on:** nothing hard ·
+**Gate:** `dotnet build` **0 new warnings** · `dotnet test` **407 pass** (402 prior + 5 new: audit-retention ×2,
+ticket-body encryption ×2, liveness ×1; the existing messaging suite now also exercises the encrypted body).
+
+This phase makes the running platform **diagnosable and honest**, and records the explicitly-deferred scale work so
+nobody mistakes it for missing MVP scope. No feature behaviour changed; the money/trust rules are untouched.
+
+## Observability & ops hardening (finish before launch)
+
+### 9.1 — Tracing added; one metrics stack; `requestId` = trace id
+- **One metrics stack.** Removed the duplicate **prometheus-net** stack (`UseMetricServer`/`UseHttpMetrics`/
+  `ForwardToPrometheus` + the three `prometheus-net*` packages). **OpenTelemetry is now the only metrics source**,
+  scraped at `/metrics` via `UseOpenTelemetryPrometheusScrapingEndpoint()`. HTTP request metrics now come from the
+  OTel ASP.NET Core instrumentation; the `mediator_meter` request-duration histogram (`MetricsBehaviour`) is now
+  actually exported (added to the meter list — the old prometheus-net stack never captured it).
+- **Tracing.** Added `WithTracing` (ASP.NET Core + **EF Core** instrumentation) sharing one resource
+  (`service.name = Baya.Web.Api`), so a cross-service money flow (webhook → confirm → ledger) is one trace.
+- **OTLP export is opt-in.** Traces + metrics export to an OTLP collector **only when `OpenTelemetry:Otlp:Endpoint`
+  is set** — an MVP with Prometheus alone runs unchanged and no exporter spams an absent collector.
+- **`requestId` already carries the trace id** (`ApiResult.RequestId = Activity.Current.TraceId`) with
+  `Activity.DefaultIdFormat = W3C` — a support ticket maps 1:1 to a trace with no extra wiring.
+- **New packages** (all cached, no restore risk): `OpenTelemetry.Exporter.OpenTelemetryProtocol` (1.15.3),
+  `OpenTelemetry.Instrumentation.EntityFrameworkCore` (1.15.1-beta.1).
+
+### 9.2 — Health checks broadened; liveness/readiness split
+- `/healthz/live` — process only (a dependency-free `self` check), so a dependency outage never restart-loops.
+- `/healthz/ready` — the app DB, the log DB (**deployed only** — its conn string is a placeholder in Dev/Testing),
+  and a real **object-storage write round-trip** (`ObjectStorageWriteHealthCheck`: put → get → delete a probe blob).
+- `/HealthCheck` — the aggregate, retained for backward compatibility. The dead `currentUrl` line is gone.
+- **Redis** is noted as the next readiness check to add *when* it becomes a real dependency (>1 instance); not now.
+- `Baya.Infrastructure.Monitoring` now references `Baya.Application` (for the `IObjectStorage` probe) — a legitimate
+  Infrastructure→Application edge, noted in the server Project map.
+
+### 9.3 — Prod log level raised to Information+; no PII; dead ES sink removed
+- Deployed envs now log **Information+** (was Warning+, which dropped every Information-level audit trail), with
+  framework categories held at Warning so the floor raise doesn't flood the sink.
+- **No secrets/PII in logs.** `LoggingSmsSender` **no longer logs the OTP code** (a login secret) in any
+  environment — a developer gets it from the Development-only `GET /api/v1/dev/last_otp`. Clinical text / IBANs /
+  phone numbers are already encrypted or masked before any handler logs. The `columnOptions` (previously built but
+  never applied) are now wired to the SQL sink.
+- **Dead Elasticsearch sink resolved by deletion:** removed the commented ES sink block *and* the unused
+  `Serilog.Sinks.Elasticsearch` package (this also revealed `Serilog.Sinks.File` was only a transitive of the ES
+  package — added it explicitly). Log-table retention is documented as an ops/DBA responsibility (or ship logs to
+  the OTLP collector).
+
+### 9.4 — Audit-log retention as a scheduled job
+- New `AuditLogRetentionJob` (`IRecurringJob`, registered like the others) runs a **two-tier** retention sweep over
+  the append-only `ops.AuditLogs`: **financial/verification** entity types (`Refund`, `NurseClawback`,
+  `NursePayout`, `NursePayoutBatch`, `NurseVerification`, `PlatformConfig`, `PartnerCenter`) keep a long legal window
+  (`audit_retention_financial_days`, default **2555** ≈ 7 years); everyday rows a shorter one
+  (`audit_retention_general_days`, default **730** ≈ 2 years). Cadence key `audit_retention_scan_cadence_hours` (24).
+- `IAuditLogger.PurgeExpiredAsync(...)` does the delete: oldest-first (Id is monotonic with `OccurredAt`), capped at
+  20 000 rows/run so a backlog drains across runs; age compared in memory (SQLite can't translate a `DateTimeOffset`
+  predicate), delete is a single id-keyed `ExecuteDeleteAsync`. Idempotent.
+- Migration `RefinementPhase9TicketBodyEncryptionAndAuditRetention` seeds the three config keys (ids 24–26).
+
+### 9.5 — `TicketMessage.Body` encrypted; gRPC reflection gated to Development
+- **Ticket bodies are the refund/dispute paper trail** (users type phone numbers, addresses, clinical detail) — now
+  **encrypted at rest** through the existing `IFieldEncryptor` converter (wired in `ApplicationDbContext`, like every
+  other PII column). The stored column is widened to `nvarchar(max)` (ciphertext is longer than plaintext); the 4000-
+  char plaintext limit stays a boundary-validation rule (Open/PostMessage validators). Body is never a SQL search/
+  filter predicate (the admin thread read decrypts per row), so losing SQL-searchability is an accepted trade-off.
+- **gRPC decision — keep the plugin, gate reflection to Development.** The plugin exposes only the User service and
+  the client is HTTP/JSON, but removing it is more invasive than the risk warrants. gRPC **reflection** (which
+  advertises the full schema) is now registered/mapped **only in Development**. The HTTP/2-posture concern is already
+  mitigated (refinement-phase-5 set Kestrel `Http1AndHttp2`), so the plugin shares the mixed-protocol listener (ALPN
+  negotiates h2 for gRPC clients) — no dedicated port needed.
+
+### 9.6 — Docs made honest
+- **Mocks-registry:** pruned the 7 **stale duplicate 🔴 rows** (`IDistributedLock`/`INurseSearch`/`IPaymentProvider`/
+  `ISettlementSplitProvider`/`IWebhookVerifier`/`IMoadianClient`/`ILicenseVerificationService`) the detailed rows
+  already correct; the recurring-jobs row is the real in-process scheduler; the `IPaymentCaptureSimulator` row now
+  reflects its 6.4 prod removal (fail-closed in prod; Dev/Testing mock is a test affordance). Added a phase-9 banner.
+- **REQ tracker:** already honest — refinement-phase-3 marked every delivered/deferred/resolved REQ; this phase ships
+  no new contract, so no REQ status changed. (The pre-phase-3 "all 15 open" state the audit flagged is long fixed.)
+- **Architecture maps:** `server/CLAUDE.md` updated (observability wiring, the audit-retention cron, ticket-body
+  encryption, the gRPC decision, the new Monitoring→Application edge); `runtime-services.md` updated (OTel
+  consolidation, tracing, health split).
+
+## Scale & later — explicitly NOT MVP (recorded, not built). Each has a written pull-trigger.
+
+| # | Deferred item | Where it lives today (the real MVP) | **Pull it when…** |
+| --- | --- | --- | --- |
+| 9.7 | **Elasticsearch read backend + outbox feeder** — `ElasticNurseSearch` + the CDC/outbox stream | `SqlNurseSearch` is real & correct; `Search:Backend` fails fast on any non-`sql` value | **SQL search shows strain** (latency/throughput on `nurse_search_index`). Build `ElasticNurseSearch` (same filters/sort/paging) + the outbox feeder off `ISearchIndexMaintainer`; keep SQL as the reconciliation source (`RebuildAsync`). |
+| 9.8 | **Analytics pipeline** — warehouse/stream sink | `IAnalyticsSink` writes `ops.SystemEvents` fire-and-forget (real, queryable) | **Product needs cross-event analytics** beyond SQL queries. Pipe `SystemEvents` to a warehouse/stream (e.g. Kafka→ClickHouse), keeping fire-and-forget semantics. |
+| 9.9 | **Holiday-calendar feed** — automated lunar-Hijri drift feed | `IHolidayCalendar` reads the seeded, manually-maintained `ops.IranianHolidays` table (real) | **The manual yearly refresh becomes a burden.** A **yearly ops-checklist item to top up the table is an acceptable MVP alternative** to a feed — the read interface stays. |
+| 9.10 | **Push/SMS notification channels** — SMS/FCM fan-out | `InAppNotificationDispatcher` writes real in-app `ops.Notifications`; non-InApp channels are dropped by design | **The notification UX demands** out-of-app reach. Fan out to SMS (via the phase-8 `ISmsSender`) and FCM push behind the same `INotificationDispatcher`. |
+| 9.11 | **Deferred product tables** — `organizations`, `organization_nurses`, `fraud_flags`, `recurring_booking_schedules`, `bnpl_settlement_entries`, availability slots, customer national-ID KYC, geo bulk import | All **verified absent**; each is a **pure additive migration** when product pulls it | **Product pulls the feature.** No structural blocker — additive migration + feature slice; nothing in the current schema needs to change first. |
+
+**These are decisions, not gaps.** SQL search, in-app notifications, and the manual holiday table are the real,
+correct MVP; Elasticsearch/analytics/push each has a concrete trigger above and stays out until then.
+
+## How it was verified
+- **Build:** `dotnet build Baya.sln` — 0 new warnings (the pre-existing NU1510 + NU1903 transitive-dependency audit
+  warnings are unrelated to this phase).
+- **Tests:** `dotnet test Baya.sln` — all green. New: `AuditLogRetentionTests` (two-tier purge + idempotency),
+  `TicketMessageEncryptionTests` (encrypted at rest + round-trips on read), `HealthCheckApiTests` (liveness healthy
+  without dependencies).
+- **Trace/requestId:** a request's `ApiResult.requestId` is `Activity.Current.TraceId` (W3C) — the same id a
+  configured OTLP collector records.
+- **No PII in logs:** the OTP code is no longer logged in any environment; clinical text/IBANs are encrypted/masked.
+
+## Follow-ups for later phases
+- Wire an OTLP collector (Grafana Tempo / Jaeger / OTEL Collector) in the deploy topology and set
+  `OpenTelemetry:Otlp:Endpoint` to turn tracing export on.
+- When the first `redis` dependency lands (>1 instance), add its readiness check to `/healthz/ready`.
+- The NU1903 transitive-dependency vulnerability warnings (`Microsoft.OpenApi`, `SQLitePCLRaw`) are a separate
+  dependency-bump task, out of this phase's scope.