baya-monorepo/archive/post-phase/refinement/refinement-phase-7-unattended-ops.md

# Refinement Phase 7 — Unattended operation: scheduler, locking & multi-instance readiness

> **Mission:** make the platform run itself. Today only two recurring jobs exist; the **weekly payout batch,
> credential-expiry scan, EVV no-show sweep, and Moadian reconciliation are admin-click-only** while their
> cadence config keys sit unread — so **nurses are not paid unless an operator clicks**. This phase adds a real
> scheduler, and (only when you need more than one API instance) shared cache + lock, and splits migrations from
> boot.
>
> **This is where "do I need Redis / Elasticsearch?" gets answered: no, not for a single-instance MVP.** Redis
> becomes necessary the moment you run a **second** API instance (shared cache invalidation + cross-instance
> money lock). Elasticsearch is **never** MVP — SQL search is real and correct
> ([Phase 9](refinement-phase-9-observability-and-scale.md) covers the deferred ES path).
>
> **Track:** backend (infra) · **Depends on:** [Phase 6](refinement-phase-6-money-correctness.md) (6.1
> settlement is a reconciliation job here) · **Unlocks:** unattended ops; >1 API instance
> **Before you start, read [../../phases/_shared/agent-operating-rules.md](../../phases/_shared/agent-operating-rules.md).**

## 1. Context

This phase **is** the server audit's **post-phase-4**. Full evidence & fixes:
**[../server/post-phase-backend-plan.md](../server/post-phase-backend-plan.md) § post-phase-4** and the topology
in **[../server/runtime-services.md](../server/runtime-services.md)** § 5, 7.

Verified state today:
- Two in-process `PeriodicTimer` hosted services only: booking-request expiry (1 min) + notification retention
  (24 h). **No `IJobScheduler` interface actually exists** (the registry name is aspirational); **no
  Hangfire/Quartz** package is referenced.
- The verification credential-expiry scan, EVV no-show sweep, weekly **payout batch generation**, and Moadian
  reconciliation are **admin-manual endpoints** whose seeded cadence keys
  (`verification_expiry_scan_cadence_hours`, `no_show_scan_cadence_hours`, `nurse_payout_interval_days`) are
  never read on a schedule.
- `ICacheService` = in-process `MemoryCache`; `IDistributedLock` = per-key `SemaphoreSlim` (one process only).
  The DB uniques/state-machines are the money-correctness backstop either way, but cache + lock **silently
  degrade** the moment a second instance runs.
- Migrations run on boot (`MigrateAsync`) → concurrent multi-node start-up races on DDL; the app login needs
  permanent DDL rights.

## 2. Required reading

- **[../server/post-phase-backend-plan.md](../server/post-phase-backend-plan.md)** § post-phase-4 (items
  4.1–4.3).
- **[../server/runtime-services.md](../server/runtime-services.md)** § 5 (Redis), § 7 (job scheduler),
  Deployment notes 1 & 4.
- The mocks-registry rows for `IJobScheduler`, `ICacheService`, `IDistributedLock`.

## 3. Scope — deliver plan items 4.1–4.3

- **7.1 — real job scheduler + register the deferred crons.** Adopt Hangfire (SQL Server storage — **no new
  infra**) or Quartz; re-home the two existing sweeps and add recurring jobs for: credential-expiry scan, EVV
  no-show sweep, **weekly payout-batch generation**, [Phase 6](refinement-phase-6-money-correctness.md)'s refund
  settlement reconciliation, and [Phase 8](refinement-phase-8-external-rails.md)'s Moadian poll — each reading
  its seeded cadence key. **Keep the admin manual triggers as overrides.** Money-*moving* payout processing can
  stay human-approved — schedule *generation*, keep `process` manual until trust is earned. Jobs are already
  idempotent by design.
- **7.2 — Redis for `ICacheService` + `IDistributedLock` (only when scaling past one instance).** Add
  `StackExchange.Redis`; `RedisCacheService` (same key/TTL scheme) + `RedisDistributedLock` (SET NX PX +
  token-checked release, lease ≥ the longest money handler), config-selected registration. **Single-instance
  MVP does not need this** — document it as the gate for horizontal scaling; keep DB uniques authoritative.
- **7.3 — separate migrations from boot** (multi-instance + least privilege). A deploy-time migration step
  (`dotnet ef database update` in CI, or a `--migrate` one-shot mode) + a boot-time schema *check* instead of an
  apply; seeders become idempotent deploy steps. (Local dev can keep boot-migrate for convenience — gate the
  behavior by environment.)

## 4. Mocks & seams

- The two hosted services move behind the real scheduler; the `IJobScheduler` registry row becomes "recurring
  jobs (Hangfire/Quartz)".
- Redis is introduced **only if 7.2 is in scope for your deployment** — otherwise the in-process cache/lock
  stay and this is documented as the scale-out gate.

## 5. Critical rules

- **Jobs must stay idempotent** (they are) — a scheduler retry must never double-pay or double-post.
- **Keep payout *processing* human-approved at first** — schedule generation, not money movement, until trust
  is earned.
- **Don't add Redis "because."** It's required only for >1 instance; adding it single-instance is complexity
  without payoff. Same for ES (never MVP).
- **The DB uniques remain the correctness backstop** even with a real distributed lock.

## 6. Definition of Done

- [ ] A real scheduler runs the two existing sweeps + the four deferred crons (each reading its cadence key);
      admin manual triggers still work as overrides.
- [ ] Payout *generation* is scheduled; *processing* remains an explicit admin action.
- [ ] (If scaling) Redis-backed cache + lock behind a config switch, with the DB uniques still authoritative;
      otherwise the single-instance limitation is documented as the scale-out gate.
- [ ] Migration-on-boot is environment-gated; a deploy-time migration path exists. `dotnet build`/`dotnet test`
      green.

## 7. How to test

- Set a short cadence in config → observe the payout-batch-generation / expiry-scan / no-show jobs fire on
  schedule (logs) and produce the same result as the admin manual trigger.
- (If Redis) run two API instances against one Redis → a cache invalidation on instance A is seen by B, and the
  money lock is held across both.
- A one-shot `--migrate` (or CI step) applies migrations; a normal boot only *checks* schema.

## 8. Hand off & document

- Update `server/CLAUDE.md` startup wiring (scheduler) + the mocks-registry rows. Update
  [../server/runtime-services.md](../server/runtime-services.md) if the deployment topology changes (scheduler
  storage, optional Redis). Save a memory note: Redis = scale-out gate, ES = never MVP, scheduler now runs the
  deferred crons.