llama.cpp is what makes a cluster with no cards useful: it serves the same
OpenAI API on CPU, so the whole path — placement, supervision, scraping,
scheduling, admission — can be exercised end to end on ordinary machines.
Three things had to give way for it.
A profile may now omit its GPU class, and that is how it says it runs on the
CPU. Only engines that can actually serve without a card may omit it; for
vLLM an omitted class stays an error, because otherwise a typo becomes a
replica placed on a machine with no hardware and diagnosed minutes later
during model loading. Placement spreads CPU replicas instead of bin packing
them, since unlike GPUs there is no allocation to stop two of them sharing a
machine's cores.
Metric mappings are now per engine rather than per cluster. llama.cpp calls
the same three numbers requests_processing, requests_deferred and
kv_cache_usage_ratio, and a single global mapping would leave half a mixed
fleet unmeasurable. An empty mapping still defaults to vLLM's names; one that
names its own Running metric is taken as a complete statement, so the fields
it leaves blank stay blank rather than being filled with names that engine
cannot have.
The adapter refuses multi-node and tensor-parallel profiles. llama.cpp splits
across local GPUs by a different mechanism than this adapter drives, and
accepting either would produce a replica that starts and then serves from one
machine while the catalog believes it has several.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two hard constraints from plan §1.1 are enforced structurally rather than
checked after the fact: there is no code path that can combine nodes from two
fabrics, and only GPUs of the profile's class are ever considered. Getting
either wrong produces a replica that starts, passes health checks, and then
hangs or crawls under load — which looks like success until traffic arrives.
- inventory: nodes, hardware, fabric attachments, allocations. A re-registering
agent cannot clear an operator's cordon, since a cordon is usually in place
because something is wrong with the node. A GPU with an ECC fault or another
process on it is not free in any useful sense. Replicas are released whole,
because they fail whole.
- placement: highest-priority profile that fits, with every rejected candidate
explained. "Unschedulable" on its own tells an operator nothing about which
of six constraints bit.
- bin packing fills partly used nodes first so whole nodes stay free for large
models that cannot be split; multi-node replicas get the emptiest machines
instead, being the least movable thing in the cluster.
- NVLink-aware GPU selection keeps a tensor-parallel group inside one domain
where the topology is known, falling back to lowest free indices otherwise.
Property-based tests generate 400 random clusters and assert the invariants on
every placement produced: one fabric per multi-node replica, only that fabric's
HCAs per shard, one GPU class, no unhealthy or already-allocated GPU. The
table-driven tests cover the cases someone thought of; these cover the rest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ilkerandClaude Opus 5