refinement phase 7

2026-07-13 17:48:50 +03:30
parent 70268ecc06
commit 7edadadea1
30 changed files with 6971 additions and 154 deletions
@@ -0,0 +1,92 @@
+# Refinement Phase 7 — Unattended operation: scheduler, locking & multi-instance readiness — Report (2026-07-13)
+
+**Track:** backend (infra) · **Depends on:** phase 6 (its settlement reconciliation is a future job here) ·
+**Gate:** `dotnet build` 0 new warnings · `dotnet test` **402 pass** (396 prior + 6 new scheduling tests).
+
+## The headline (7.1) — the platform now runs itself
+
+Before this phase only two hand-written `PeriodicTimer` hosted services existed; the credential-expiry scan, EVV
+no-show sweep, and **weekly payout-batch generation** were admin-click-only while their seeded cadence keys sat
+unread — so **nurses were paid only when an operator clicked**. Now a single in-process scheduler drives every job
+on its own cadence.
+
+**`RecurringJobSchedulerHostedService`** (`Persistence/Services/Scheduling/`) + the **`IRecurringJob`** seam:
+- The scheduler owns one independent loop per job (their cadences don't couple; a crash in one never stops the
+  others), the per-tick DI scope, error isolation (a throwing tick logs and the next tick retries on schedule),
+  and a per-tick `IDistributedLock("scheduler:{name}")`. A job says only *how often* (usually a `platform_configs`
+  cadence key, re-read each tick so an admin change applies without a restart) and *what one idempotent run does*.
+- **No new infrastructure.** SQL Server stays the only external dependency — a single-instance MVP needs neither
+  Hangfire/Quartz (durable/cross-restart scheduling is the only thing they add for idempotent periodic sweeps) nor
+  Redis. Adding a cron = implement `IRecurringJob` + one `AddSingleton<IRecurringJob, …>()`.
+
+Jobs registered (`Services/Scheduling/Jobs/`), each dispatching the **same idempotent command the admin trigger
+sends** (the admin endpoints are unchanged and remain overrides):
+
+| Job (`Name`) | Cadence source | Re-homed / new |
+| --- | --- | --- |
+| `booking_request_expiry` | 1 min const | re-homed from `BookingRequestExpiryHostedService` (deleted) |
+| `notification_retention` | 24 h const | re-homed from `NotificationRetentionHostedService` (deleted) |
+| `verification_expiry_scan` | `verification_expiry_scan_cadence_hours` (24) | **new** → `ScanExpiringCredentialsCommand` |
+| `no_show_sweep` | `no_show_scan_cadence_hours` (1) | **new** → `DetectNoShowSessionsCommand` |
+| `weekly_payout_generation` | `nurse_payout_interval_days` (7) | **new** → `GeneratePayoutBatchCommand` |
+
+## Money movement stays human-approved (critical rule)
+
+The payout job schedules **generation only** — it opens a `draft` batch over the trailing window; the irreversible
+`process` (money-moving) step remains an explicit admin action until trust is earned. To let an unattended run
+record a batch with no human initiator, `NursePayoutBatch.InitiatedByAdminId` is now **nullable** (`null` =
+system-initiated) — migration `RefinementPhase7SystemPayoutBatch` (alters the column + FK to nullable; the FK, the
+`PayoutBatchDto` projection, and `swagger.v1.json` were updated to match). The command's `SystemInitiated` flag is
+**scheduler-only**: `AdminPayoutsController.Generate` neutralizes any request-supplied value (`command with {
+SystemInitiated = false }`), so an API caller can never bypass the authenticated-admin requirement. A quiet week
+(no eligible bookings) is a benign no-op; a re-run over an overlapping window is safe — the
+`nurse_payout_booking_links.booking_id` UNIQUE prevents re-selecting an already-paid booking.
+
+## 7.2 — Redis is the scale-out gate, NOT added
+
+Per the phase's "don't add Redis because", the in-process `ICacheService`/`IDistributedLock` stay. They are the
+documented **>1-instance scale-out gate**: the moment a second API instance runs, swap the lock seam to Redis and
+the scheduler's per-tick lock serializes ticks across nodes (idempotency + the DB uniques cover a double-run
+either way). Nothing speaks Redis today; no package added. (Registry rows `ICacheService`/`IDistributedLock`
+updated with the framing.)
+
+## 7.3 — Migrations split from boot
+
+`dotnet run -- migrate` is a deploy-time one-shot: it applies EF migrations + the idempotent seeders, then exits —
+so concurrent multi-instance start-ups never race on DDL and the runtime login needs no permanent DDL rights.
+**Development** boot still migrates + seeds (incl. the Development-only sandbox gateway + demo world) for
+convenience; **deployed** boot only *checks* the schema is current (`EnsureSchemaUpToDateAsync` — fail-fast on a
+pending migration) and seeds roles/break-glass admin. Env-gated in `Program.cs`.
+
+## What is now testable and exactly how
+
+- **6 new Foundation tests** (`Tests/Baya.Test.Foundation/Scheduling/`): each cadence job reads the right config
+  key and dispatches the right command (incl. the payout job asserting `SystemInitiated=true` + the trailing
+  window); the scheduler runs a job at startup under `scheduler:{name}`, keeps siblings alive when one throws, and
+  stays **dormant under the `Testing` environment**.
+- **Live cadence check:** set a short `no_show_scan_cadence_hours` / `verification_expiry_scan_cadence_hours` (or
+  a short interval) in `platform_configs`, run the API (Development), and watch the job fire on schedule in the
+  logs, producing the same result as the admin manual trigger.
+- **Migration path:** `dotnet run -- migrate` applies + seeds and exits; a deployed-env boot with a pending
+  migration fails fast with the list of pending migrations.
+
+## What is mocked / deferred (follow-ups)
+
+- **Moadian reconciliation + refund-settlement poll** are **Phase 8's** jobs — they have no command/cadence key
+  today and Phase 8 explicitly owns registering them. They slot in as new `IRecurringJob`s with one `AddSingleton`
+  — no scheduler change. Documented in the mocks-registry row.
+- **Redis** — the scale-out gate above (only when >1 instance).
+
+## Contracts produced/consumed
+
+- `PayoutBatchDto.initiatedByAdminId` is now nullable (`null` = system/scheduled batch). Updated
+  `dev/contracts/domains/payouts.md` + `dev/contracts/openapi/swagger.v1.json`. No other wire change.
+
+## Files
+
+New: `Services/Scheduling/{IRecurringJob, RecurringJobSchedulerHostedService}.cs` +
+`Services/Scheduling/Jobs/{BookingRequestExpiry, NotificationRetention, CredentialExpiryScan, NoShowSweep,
+WeeklyPayoutGeneration}Job.cs`; migration `RefinementPhase7SystemPayoutBatch`; 2 test files. Deleted: the two old
+hosted services. Changed: `AddPersistenceServices` (registration) + `EnsureSchemaUpToDateAsync`; `Program.cs`
+(migrate one-shot + env-gated boot); `NursePayoutBatch`/`NursePayoutBatchConfig`/`PayoutBatchDto` (nullable
+initiator); `GeneratePayoutBatchCommand`(+Handler)/`AdminPayoutsController` (system-initiated path).