add refinement phases
This commit is contained in:
hamid
@@ -0,0 +1,103 @@
|
||||
# Refinement Phase 7 — Unattended operation: scheduler, locking & multi-instance readiness
|
||||
|
||||
> **Mission:** make the platform run itself. Today only two recurring jobs exist; the **weekly payout batch,
|
||||
> credential-expiry scan, EVV no-show sweep, and Moadian reconciliation are admin-click-only** while their
|
||||
> cadence config keys sit unread — so **nurses are not paid unless an operator clicks**. This phase adds a real
|
||||
> scheduler, and (only when you need more than one API instance) shared cache + lock, and splits migrations from
|
||||
> boot.
|
||||
>
|
||||
> **This is where "do I need Redis / Elasticsearch?" gets answered: no, not for a single-instance MVP.** Redis
|
||||
> becomes necessary the moment you run a **second** API instance (shared cache invalidation + cross-instance
|
||||
> money lock). Elasticsearch is **never** MVP — SQL search is real and correct
|
||||
> ([Phase 9](refinement-phase-9-observability-and-scale.md) covers the deferred ES path).
|
||||
>
|
||||
> **Track:** backend (infra) · **Depends on:** [Phase 6](refinement-phase-6-money-correctness.md) (6.1
|
||||
> settlement is a reconciliation job here) · **Unlocks:** unattended ops; >1 API instance
|
||||
> **Before you start, read [../../phases/_shared/agent-operating-rules.md](../../phases/_shared/agent-operating-rules.md).**
|
||||
|
||||
## 1. Context
|
||||
|
||||
This phase **is** the server audit's **post-phase-4**. Full evidence & fixes:
|
||||
**[../server/post-phase-backend-plan.md](../server/post-phase-backend-plan.md) § post-phase-4** and the topology
|
||||
in **[../server/runtime-services.md](../server/runtime-services.md)** § 5, 7.
|
||||
|
||||
Verified state today:
|
||||
- Two in-process `PeriodicTimer` hosted services only: booking-request expiry (1 min) + notification retention
|
||||
(24 h). **No `IJobScheduler` interface actually exists** (the registry name is aspirational); **no
|
||||
Hangfire/Quartz** package is referenced.
|
||||
- The verification credential-expiry scan, EVV no-show sweep, weekly **payout batch generation**, and Moadian
|
||||
reconciliation are **admin-manual endpoints** whose seeded cadence keys
|
||||
(`verification_expiry_scan_cadence_hours`, `no_show_scan_cadence_hours`, `nurse_payout_interval_days`) are
|
||||
never read on a schedule.
|
||||
- `ICacheService` = in-process `MemoryCache`; `IDistributedLock` = per-key `SemaphoreSlim` (one process only).
|
||||
The DB uniques/state-machines are the money-correctness backstop either way, but cache + lock **silently
|
||||
degrade** the moment a second instance runs.
|
||||
- Migrations run on boot (`MigrateAsync`) → concurrent multi-node start-up races on DDL; the app login needs
|
||||
permanent DDL rights.
|
||||
|
||||
## 2. Required reading
|
||||
|
||||
- **[../server/post-phase-backend-plan.md](../server/post-phase-backend-plan.md)** § post-phase-4 (items
|
||||
4.1–4.3).
|
||||
- **[../server/runtime-services.md](../server/runtime-services.md)** § 5 (Redis), § 7 (job scheduler),
|
||||
Deployment notes 1 & 4.
|
||||
- The mocks-registry rows for `IJobScheduler`, `ICacheService`, `IDistributedLock`.
|
||||
|
||||
## 3. Scope — deliver plan items 4.1–4.3
|
||||
|
||||
- **7.1 — real job scheduler + register the deferred crons.** Adopt Hangfire (SQL Server storage — **no new
|
||||
infra**) or Quartz; re-home the two existing sweeps and add recurring jobs for: credential-expiry scan, EVV
|
||||
no-show sweep, **weekly payout-batch generation**, [Phase 6](refinement-phase-6-money-correctness.md)'s refund
|
||||
settlement reconciliation, and [Phase 8](refinement-phase-8-external-rails.md)'s Moadian poll — each reading
|
||||
its seeded cadence key. **Keep the admin manual triggers as overrides.** Money-*moving* payout processing can
|
||||
stay human-approved — schedule *generation*, keep `process` manual until trust is earned. Jobs are already
|
||||
idempotent by design.
|
||||
- **7.2 — Redis for `ICacheService` + `IDistributedLock` (only when scaling past one instance).** Add
|
||||
`StackExchange.Redis`; `RedisCacheService` (same key/TTL scheme) + `RedisDistributedLock` (SET NX PX +
|
||||
token-checked release, lease ≥ the longest money handler), config-selected registration. **Single-instance
|
||||
MVP does not need this** — document it as the gate for horizontal scaling; keep DB uniques authoritative.
|
||||
- **7.3 — separate migrations from boot** (multi-instance + least privilege). A deploy-time migration step
|
||||
(`dotnet ef database update` in CI, or a `--migrate` one-shot mode) + a boot-time schema *check* instead of an
|
||||
apply; seeders become idempotent deploy steps. (Local dev can keep boot-migrate for convenience — gate the
|
||||
behavior by environment.)
|
||||
|
||||
## 4. Mocks & seams
|
||||
|
||||
- The two hosted services move behind the real scheduler; the `IJobScheduler` registry row becomes "recurring
|
||||
jobs (Hangfire/Quartz)".
|
||||
- Redis is introduced **only if 7.2 is in scope for your deployment** — otherwise the in-process cache/lock
|
||||
stay and this is documented as the scale-out gate.
|
||||
|
||||
## 5. Critical rules
|
||||
|
||||
- **Jobs must stay idempotent** (they are) — a scheduler retry must never double-pay or double-post.
|
||||
- **Keep payout *processing* human-approved at first** — schedule generation, not money movement, until trust
|
||||
is earned.
|
||||
- **Don't add Redis "because."** It's required only for >1 instance; adding it single-instance is complexity
|
||||
without payoff. Same for ES (never MVP).
|
||||
- **The DB uniques remain the correctness backstop** even with a real distributed lock.
|
||||
|
||||
## 6. Definition of Done
|
||||
|
||||
- [ ] A real scheduler runs the two existing sweeps + the four deferred crons (each reading its cadence key);
|
||||
admin manual triggers still work as overrides.
|
||||
- [ ] Payout *generation* is scheduled; *processing* remains an explicit admin action.
|
||||
- [ ] (If scaling) Redis-backed cache + lock behind a config switch, with the DB uniques still authoritative;
|
||||
otherwise the single-instance limitation is documented as the scale-out gate.
|
||||
- [ ] Migration-on-boot is environment-gated; a deploy-time migration path exists. `dotnet build`/`dotnet test`
|
||||
green.
|
||||
|
||||
## 7. How to test
|
||||
|
||||
- Set a short cadence in config → observe the payout-batch-generation / expiry-scan / no-show jobs fire on
|
||||
schedule (logs) and produce the same result as the admin manual trigger.
|
||||
- (If Redis) run two API instances against one Redis → a cache invalidation on instance A is seen by B, and the
|
||||
money lock is held across both.
|
||||
- A one-shot `--migrate` (or CI step) applies migrations; a normal boot only *checks* schema.
|
||||
|
||||
## 8. Hand off & document
|
||||
|
||||
- Update `server/CLAUDE.md` startup wiring (scheduler) + the mocks-registry rows. Update
|
||||
[../server/runtime-services.md](../server/runtime-services.md) if the deployment topology changes (scheduler
|
||||
storage, optional Redis). Save a memory note: Redis = scale-out gate, ES = never MVP, scheduler now runs the
|
||||
deferred crons.
|
||||
Reference in New Issue
Block a user