Commit Graph
3 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 b661bfead8 Wait for what tests check instead of sleeping and hoping
CI / test (push) Successful in 3m23s
CI's test job has failed intermittently — two of three runs on some days —
while every run here passed, including six repetitions on one CPU under the
race detector with eight packages at once. The difference is a shared,
loaded Linux runner, and the tests most exposed to it are the ones that
sleep a fixed time and then assert:

- admission: idle admission bounded at 50ms of wall clock against a 1ms
  tick, and a queued request checked for after a 50ms sleep;
- scrape: a 10ms scrape timeout, which on a slow runner reports the engine
  unreachable and then the violation a second time, in a test that counts
  one;
- supervisor: log lines, three lines of output and a removed state file
  each checked after a fixed sleep, and an exited worker bounded at 1.5s;
- agent: a heartbeat checked for after four intervals.

Each now waits for its condition up to a deadline of seconds, and the two
wall-clock bounds are set against what they exist to distinguish — the
admission queue's tick raised to ten seconds, the exited worker bounded
below the five-second grace it must not wait out — rather than against how
fast this laptop is.

Without the CI logs this cannot be proven to be the failure; these are the
tests that fail when the machine is slow, and they no longer do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 01:36:55 +02:00
Ilker ManapandClaude Opus 5 f19db242ae Stop naming a metric llama.cpp does not have, and say so when it happens
Deployed against real llama.cpp, every scrape failed. The mapping named
llamacpp:kv_cache_usage_ratio; build b10991 serves fifteen metric families
and none is a cache ratio. Because KV usage was mandatory, that one absent
name threw away the two metrics the engine does publish, and the router
scheduled to all three replicas blind while the cluster looked healthy.

A mapping that leaves a metric unnamed now means the engine does not publish
it, and the field stays zero. Naming one the engine does not serve is still a
violation, so this does not become a way to tolerate a typo. Running and
waiting can never be omitted: an engine that reports neither what it is doing
nor what is queued cannot be scheduled against at all. For llama.cpp this
costs the KV half of the saturation signal, which is a real loss softened by
its fixed slot count — queue depth already says when it is full.

The second half is why it took a deployment to notice. A contract violation
was logged per scrape at debug level, so at the default level nothing said
anything. It is now a warning, once, with a matching line when it clears.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:18:27 +02:00
Ilker ManapandClaude Opus 5 501db3d3e8 Add the llama.cpp engine, and let a profile ask for no GPU
llama.cpp is what makes a cluster with no cards useful: it serves the same
OpenAI API on CPU, so the whole path — placement, supervision, scraping,
scheduling, admission — can be exercised end to end on ordinary machines.

Three things had to give way for it.

A profile may now omit its GPU class, and that is how it says it runs on the
CPU. Only engines that can actually serve without a card may omit it; for
vLLM an omitted class stays an error, because otherwise a typo becomes a
replica placed on a machine with no hardware and diagnosed minutes later
during model loading. Placement spreads CPU replicas instead of bin packing
them, since unlike GPUs there is no allocation to stop two of them sharing a
machine's cores.

Metric mappings are now per engine rather than per cluster. llama.cpp calls
the same three numbers requests_processing, requests_deferred and
kv_cache_usage_ratio, and a single global mapping would leave half a mixed
fleet unmeasurable. An empty mapping still defaults to vLLM's names; one that
names its own Running metric is taken as a complete statement, so the fields
it leaves blank stay blank rather than being filled with names that engine
cannot have.

The adapter refuses multi-node and tensor-parallel profiles. llama.cpp splits
across local GPUs by a different mechanism than this adapter drives, and
accepting either would produce a replica that starts and then serves from one
machine while the catalog believes it has several.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:14:05 +02:00