Refinement Phase 7 — Unattended operation: scheduler, locking & multi-instance readiness

Mission: make the platform run itself. Today only two recurring jobs exist; the weekly payout batch, credential-expiry scan, EVV no-show sweep, and Moadian reconciliation are admin-click-only while their cadence config keys sit unread — so nurses are not paid unless an operator clicks. This phase adds a real scheduler, and (only when you need more than one API instance) shared cache + lock, and splits migrations from boot.

This is where "do I need Redis / Elasticsearch?" gets answered: no, not for a single-instance MVP. Redis becomes necessary the moment you run a second API instance (shared cache invalidation + cross-instance money lock). Elasticsearch is never MVP — SQL search is real and correct (Phase 9 covers the deferred ES path).

Track: backend (infra) · Depends on: Phase 6 (6.1 settlement is a reconciliation job here) · Unlocks: unattended ops; >1 API instance Before you start, read ../../phases/_shared/agent-operating-rules.md.

1. Context

This phase is the server audit's post-phase-4. Full evidence & fixes: ../server/post-phase-backend-plan.md § post-phase-4 and the topology in ../server/runtime-services.md § 5, 7.

Verified state today:

Two in-process PeriodicTimer hosted services only: booking-request expiry (1 min) + notification retention (24 h). No IJobScheduler interface actually exists (the registry name is aspirational); no Hangfire/Quartz package is referenced.
The verification credential-expiry scan, EVV no-show sweep, weekly payout batch generation, and Moadian reconciliation are admin-manual endpoints whose seeded cadence keys (verification_expiry_scan_cadence_hours, no_show_scan_cadence_hours, nurse_payout_interval_days) are never read on a schedule.
ICacheService = in-process MemoryCache; IDistributedLock = per-key SemaphoreSlim (one process only). The DB uniques/state-machines are the money-correctness backstop either way, but cache + lock silently degrade the moment a second instance runs.
Migrations run on boot (MigrateAsync) → concurrent multi-node start-up races on DDL; the app login needs permanent DDL rights.

2. Required reading

../server/post-phase-backend-plan.md § post-phase-4 (items 4.1–4.3).
../server/runtime-services.md § 5 (Redis), § 7 (job scheduler), Deployment notes 1 & 4.
The mocks-registry rows for IJobScheduler, ICacheService, IDistributedLock.

3. Scope — deliver plan items 4.1–4.3

7.1 — real job scheduler + register the deferred crons. Adopt Hangfire (SQL Server storage — no new infra) or Quartz; re-home the two existing sweeps and add recurring jobs for: credential-expiry scan, EVV no-show sweep, weekly payout-batch generation, Phase 6's refund settlement reconciliation, and Phase 8's Moadian poll — each reading its seeded cadence key. Keep the admin manual triggers as overrides. Money-moving payout processing can stay human-approved — schedule generation, keep process manual until trust is earned. Jobs are already idempotent by design.
7.2 — Redis for ICacheService + IDistributedLock (only when scaling past one instance). Add StackExchange.Redis; RedisCacheService (same key/TTL scheme) + RedisDistributedLock (SET NX PX + token-checked release, lease ≥ the longest money handler), config-selected registration. Single-instance MVP does not need this — document it as the gate for horizontal scaling; keep DB uniques authoritative.
7.3 — separate migrations from boot (multi-instance + least privilege). A deploy-time migration step (dotnet ef database update in CI, or a --migrate one-shot mode) + a boot-time schema check instead of an apply; seeders become idempotent deploy steps. (Local dev can keep boot-migrate for convenience — gate the behavior by environment.)

4. Mocks & seams

The two hosted services move behind the real scheduler; the IJobScheduler registry row becomes "recurring jobs (Hangfire/Quartz)".
Redis is introduced only if 7.2 is in scope for your deployment — otherwise the in-process cache/lock stay and this is documented as the scale-out gate.

5. Critical rules

Jobs must stay idempotent (they are) — a scheduler retry must never double-pay or double-post.
Keep payout processing human-approved at first — schedule generation, not money movement, until trust is earned.
Don't add Redis "because." It's required only for >1 instance; adding it single-instance is complexity without payoff. Same for ES (never MVP).
The DB uniques remain the correctness backstop even with a real distributed lock.

6. Definition of Done

A real scheduler runs the two existing sweeps + the four deferred crons (each reading its cadence key); admin manual triggers still work as overrides.
Payout generation is scheduled; processing remains an explicit admin action.
(If scaling) Redis-backed cache + lock behind a config switch, with the DB uniques still authoritative; otherwise the single-instance limitation is documented as the scale-out gate.
Migration-on-boot is environment-gated; a deploy-time migration path exists. dotnet build/dotnet test green.

7. How to test

Set a short cadence in config → observe the payout-batch-generation / expiry-scan / no-show jobs fire on schedule (logs) and produce the same result as the admin manual trigger.
(If Redis) run two API instances against one Redis → a cache invalidation on instance A is seen by B, and the money lock is held across both.
A one-shot --migrate (or CI step) applies migrations; a normal boot only checks schema.

8. Hand off & document

Update server/CLAUDE.md startup wiring (scheduler) + the mocks-registry rows. Update ../server/runtime-services.md if the deployment topology changes (scheduler storage, optional Redis). Save a memory note: Redis = scale-out gate, ES = never MVP, scheduler now runs the deferred crons.

6.5 KiB Raw Blame History Unescape Escape