Refinement Phase 7 — Unattended operation: scheduler, locking & multi-instance readiness — Report (2026-07-13)

Track: backend (infra) · Depends on: phase 6 (its settlement reconciliation is a future job here) · Gate: dotnet build 0 new warnings · dotnet test 402 pass (396 prior + 6 new scheduling tests).

The headline (7.1) — the platform now runs itself

Before this phase only two hand-written PeriodicTimer hosted services existed; the credential-expiry scan, EVV no-show sweep, and weekly payout-batch generation were admin-click-only while their seeded cadence keys sat unread — so nurses were paid only when an operator clicked. Now a single in-process scheduler drives every job on its own cadence.

RecurringJobSchedulerHostedService (Persistence/Services/Scheduling/) + the IRecurringJob seam:

The scheduler owns one independent loop per job (their cadences don't couple; a crash in one never stops the others), the per-tick DI scope, error isolation (a throwing tick logs and the next tick retries on schedule), and a per-tick IDistributedLock("scheduler:{name}"). A job says only how often (usually a platform_configs cadence key, re-read each tick so an admin change applies without a restart) and what one idempotent run does.
No new infrastructure. SQL Server stays the only external dependency — a single-instance MVP needs neither Hangfire/Quartz (durable/cross-restart scheduling is the only thing they add for idempotent periodic sweeps) nor Redis. Adding a cron = implement IRecurringJob + one AddSingleton<IRecurringJob, …>().

Jobs registered (Services/Scheduling/Jobs/), each dispatching the same idempotent command the admin trigger sends (the admin endpoints are unchanged and remain overrides):

Job (`Name`)	Cadence source	Re-homed / new
`booking_request_expiry`	1 min const	re-homed from `BookingRequestExpiryHostedService` (deleted)
`notification_retention`	24 h const	re-homed from `NotificationRetentionHostedService` (deleted)
`verification_expiry_scan`	`verification_expiry_scan_cadence_hours` (24)	new → `ScanExpiringCredentialsCommand`
`no_show_sweep`	`no_show_scan_cadence_hours` (1)	new → `DetectNoShowSessionsCommand`
`weekly_payout_generation`	`nurse_payout_interval_days` (7)	new → `GeneratePayoutBatchCommand`

Money movement stays human-approved (critical rule)

The payout job schedules generation only — it opens a draft batch over the trailing window; the irreversible process (money-moving) step remains an explicit admin action until trust is earned. To let an unattended run record a batch with no human initiator, NursePayoutBatch.InitiatedByAdminId is now nullable (null = system-initiated) — migration RefinementPhase7SystemPayoutBatch (alters the column + FK to nullable; the FK, the PayoutBatchDto projection, and swagger.v1.json were updated to match). The command's SystemInitiated flag is scheduler-only: AdminPayoutsController.Generate neutralizes any request-supplied value (command with { SystemInitiated = false }), so an API caller can never bypass the authenticated-admin requirement. A quiet week (no eligible bookings) is a benign no-op; a re-run over an overlapping window is safe — the nurse_payout_booking_links.booking_id UNIQUE prevents re-selecting an already-paid booking.

7.2 — Redis is the scale-out gate, NOT added

Per the phase's "don't add Redis because", the in-process ICacheService/IDistributedLock stay. They are the documented >1-instance scale-out gate: the moment a second API instance runs, swap the lock seam to Redis and the scheduler's per-tick lock serializes ticks across nodes (idempotency + the DB uniques cover a double-run either way). Nothing speaks Redis today; no package added. (Registry rows ICacheService/IDistributedLock updated with the framing.)

7.3 — Migrations split from boot

dotnet run -- migrate is a deploy-time one-shot: it applies EF migrations + the idempotent seeders, then exits — so concurrent multi-instance start-ups never race on DDL and the runtime login needs no permanent DDL rights. Development boot still migrates + seeds (incl. the Development-only sandbox gateway + demo world) for convenience; deployed boot only checks the schema is current (EnsureSchemaUpToDateAsync — fail-fast on a pending migration) and seeds roles/break-glass admin. Env-gated in Program.cs.

What is now testable and exactly how

6 new Foundation tests (Tests/Baya.Test.Foundation/Scheduling/): each cadence job reads the right config key and dispatches the right command (incl. the payout job asserting SystemInitiated=true + the trailing window); the scheduler runs a job at startup under scheduler:{name}, keeps siblings alive when one throws, and stays dormant under the Testing environment.
Live cadence check: set a short no_show_scan_cadence_hours / verification_expiry_scan_cadence_hours (or a short interval) in platform_configs, run the API (Development), and watch the job fire on schedule in the logs, producing the same result as the admin manual trigger.
Migration path: dotnet run -- migrate applies + seeds and exits; a deployed-env boot with a pending migration fails fast with the list of pending migrations.

What is mocked / deferred (follow-ups)

Moadian reconciliation + refund-settlement poll are Phase 8's jobs — they have no command/cadence key today and Phase 8 explicitly owns registering them. They slot in as new IRecurringJobs with one AddSingleton — no scheduler change. Documented in the mocks-registry row.
Redis — the scale-out gate above (only when >1 instance).

Contracts produced/consumed

PayoutBatchDto.initiatedByAdminId is now nullable (null = system/scheduled batch). Updated dev/contracts/domains/payouts.md + dev/contracts/openapi/swagger.v1.json. No other wire change.

Files

New: Services/Scheduling/{IRecurringJob, RecurringJobSchedulerHostedService}.cs + Services/Scheduling/Jobs/{BookingRequestExpiry, NotificationRetention, CredentialExpiryScan, NoShowSweep, WeeklyPayoutGeneration}Job.cs; migration RefinementPhase7SystemPayoutBatch; 2 test files. Deleted: the two old hosted services. Changed: AddPersistenceServices (registration) + EnsureSchemaUpToDateAsync; Program.cs (migrate one-shot + env-gated boot); NursePayoutBatch/NursePayoutBatchConfig/PayoutBatchDto (nullable initiator); GeneratePayoutBatchCommand(+Handler)/AdminPayoutsController (system-initiated path).

6.6 KiB Raw Blame History