Commit Graph
2 Commits
Author SHA1 Message Date
ilkerandClaude Opus 5 e8d90cf375 Treat a zombie as dead, and test fairness where it is deterministic
CI / test (push) Successful in 52s
CI went red on Linux with two failures that never appear on macOS. Both were
worth the trip.

The first is a real bug. processAlive used kill(pid, 0), which succeeds on a
zombie — a process that has exited and is only waiting to be reaped. That
happens for real whenever an engine's parent dies before it does and nothing
reaps the orphan, which in a container is the normal case: PID 1 is usually the
workload rather than an init. The supervisor would then report a dead adopted
engine as running forever, and hold its GPUs against every future placement.
Zombies are now detected through procfs on Linux, where the answer exists, and
the check degrades to its old behaviour elsewhere rather than to something worse.

The second was a bad test. TestTenantFairnessWithinBand asserted on the order
goroutines sent to a channel after being admitted, which is the order the Go
scheduler woke them, not the order the dispatcher chose. Under load it measured
the runtime. The rotation itself lives in bandQueue.pop, which is pure, so it is
tested there — including what happens when a tenant runs out mid-rotation and
when a cancelled waiter is removed. The controller-level test now asserts the
thing that is actually stable: nobody is starved.

Verified with three consecutive `go test -race ./...` runs in a Linux container.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 06:34:08 +02:00
ilkerandClaude Opus 5 dc38dd6096 Phase 1.5: admission control
CI / test (push) Successful in 48s
Inserted between quota checks and scheduling: admission answers when a request
may proceed, the scheduler still answers where it goes. Keeping them apart is
the point — an engine queue is FIFO and knows nothing about tenants, so once a
batch job is in it, no endpoint selection rescues the request behind it.

- saturation detector: one number per pool, the worse of queue depth and KV
  occupancy, since either alone misleads.
- priority bands with per-band saturation ceilings, so the cluster degrades from
  the bottom up.
- tenant round-robin within a band: one caller sending a thousand requests takes
  one slot in the rotation, not a thousand.
- bounded queues with capacity (503) and TTL (504); cancelled requests leave the
  queue instead of holding capacity.
- metrics for queue depth, wait time, rejections and pool saturation.

Exit criterion, measured in test/e2e: with 500 concurrent bronze requests
running continuously, interactive p99 in the platinum band stays at 1.04-1.24x
idle against a 1.5x budget. With admission off the same load pushes it to
55-60x. Both directions are asserted, so the test fails if admission regresses
and also if the load generator stops being a real load.

Two lags had to be closed, and they needed different answers. Engine metrics are
scraped and lag an interval; router inflight does not, but it counts running
requests, and treating those as queue makes a healthy engine look overloaded
every interval. What is actually queued is inflight minus batch capacity, and
capacity is not in the metric contract, so it is learned as the high-water mark
of observed running. Separately, between admission granting a request and the
router dispatching it the request is invisible everywhere, so a simultaneous
burst passed the check en masse; the controller now counts its own outstanding
leases against dispatched inflight.

One plan correction: a saturation ceiling of 1.0 was documented as "never held
back". It is not — it means "flow up to the healthy limit". A band that must
never wait needs a ceiling above 1. Plan §7.2.2 updated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:53:38 +02:00