notify.webhooks sends signed JSON or Slack-compatible messages, queued,
retried, de-duplicated hourly; ctl notify test checks each webhook.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
env is an allowlist of tuning families less path, plugin, library, address
and credential names (NCCL_*_PLUGIN, CUDA_INJECTION64_PATH and the like
got through the old denylist); llama.cpp's LLAMA_ARG_* are checked as
flags. args refuse --trust-remote-code, --log-file, --slot-save-path,
--tool-parser-plugin, --middleware, --allowed-local-media-path and others
that read, write or load, and what the agent sets itself. Checked in the
catalog and again on the node before an engine starts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Monitor draws CPU, memory, GPUs, requests, tokens, latency, network and disk
as Braille charts over 5m/1h/24h/7d, then per node, card and model
sparklines; node and model pages have their own. History is read from
monitoring.prometheus through POST /api/v1/metrics/range, or without it
from what the console saw. The control plane serves the cluster's machines
for Prometheus http_sd at /prometheus/targets on its metrics port, and node
add installs prometheus-node-exporter.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The console starts a serve or update job over the admin API and stays,
instead of closing to follow the download. Jobs lists them with their
progress and steps; the sidebar counts those running. ctl model serve and
update take --detach; ctl jobs lists jobs and follows one.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ranking what was free first made every file come out on the CPU on a cluster
whose one card was in use. The plan now ranks by hardware alone, free now
only breaking a tie; a model served onto cards that are busy is added and
waits for them, the job ending as waiting rather than failed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The control plane reads the model's shape from one GGUF header or its
config.json and lists, for every file, the best way it runs here: one card,
several of a node's (by layer for llama.cpp, tensor-parallel for vLLM),
several nodes' over one fabric (vLLM, pipeline-parallel between them), or a
CPU node; what each card holds and whether it is free now. ctl model plan
prints it; model serve takes its file and way unless told, gives a GGUF on
GPUs a GPU profile, and with --fallbacks adds the next best ways. The
console shows it per file and asks for the way and the fallback.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
model serve takes --projector, fetched beside the weights and loaded by
llama.cpp with --mmproj, and --format safetensors with --gpu-class and --tp,
which fetches the weights with their config and tokenizer and runs them on
vLLM. The console offers both, F switching between GGUF and original
weights. The mirror view is one row per version with its files on Enter.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The mirror keeps where it fetched a file beside it, and reports that and the
checksum after it restarts. A model whose files the mirror cannot trace is
found on the registry: a repository holding a file of that name that has the
commit the mirror's directory is named after, with every file at the same
size, the most downloaded first. ctl model origin and the console set one by
hand, checked the same way. Commits compare shortened.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model served from the registry records its origin: repository, commit and
each file's checksum; one served before is worked out from the mirror's
record of where it fetched the file. Every mirror.update_check (6h) the
control plane asks whether those files changed there. ctl model updates
lists it, ctl model update fetches beside the old files, checks, swaps and
replaces replicas one at a time, ctl model rollback goes back. The console
shows it on Models and offers the same.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
server add gives another machine this llmgw, a certificate, the
configuration and store address, lets it into the store through Patroni, and
starts it; --vip keeps a floating address on the leader with keepalived,
following a new /leader endpoint. Nodes learn every control plane from the
leader and notice a dead one by HTTP/2 ping instead of after the kernel's
fifteen minutes. The setup, console, node and init screens are drawn full
screen in one frame: title bar, sidebar, bordered panel, key bar, dialogs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
init --patroni user@host ... installs PostgreSQL, Patroni and etcd on the
given machines over ssh, with this one as the third etcd vote, makes the role
and database, and moves the store in use into the cluster. Each step checks
first, so it changes nothing on a cluster that is up. For a store of several
hosts the control plane publishes each member's role, state and lag from
Patroni, a stopped member as missing, prints it in ctl status, and alerts on
it. Making an administrator on PostgreSQL sent a bool to an integer column;
it sends an integer now.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A card with something else on it when the agent started stayed out of
placement after that process let go of it, until the agent restarted. The
agent now sends what processes other than its own engines hold on each card,
and the control plane replaces the registration verdict with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every minute the agent reads nvidia-smi's full report, its NVLink error
counters and the kernel's XID lines. They are published per card,
shown on the node page and the GPU map, charted in Grafana, and five
alerts watch them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cluster page draws every card coloured by utilisation, memory or
temperature, and lists replicas whose every card is under 5% busy.
llmgw_gpu_assigned says which replica, model and owner hold each card.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
They were read once, at agent start, and published unchanged since.
Each sample now carries utilisation, memory and bandwidth in use, the SM
clock and uncorrected ECC errors. The node page charts each card, the
cluster page the fleet by class, Grafana has a GPU row, and three GPU
alerts are added.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tp on a llama.cpp GPU profile is read as gpus_per_node rather than
refused, and a Hugging Face repository page as a vLLM source as the
repository id, so models written that way are placed after the upgrade
instead of dropped from the catalog. Engines adopted after an upgrade
are checked for computing on their GPUs too.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A node that cannot run the CUDA 13 build installs the same version's
CUDA 12.9 build from vLLM's GitHub release, with PyTorch's cu129 torch;
the install computes on the GPU before it counts. The build can be
chosen per node — auto, cu13, cu129, none — on the cluster page, with
ctl engine-build, or over the admin API; the control plane keeps the
choice and tells the node at once.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The explorer fetches a safetensors set with its config and tokenizer;
the mirror gives out a version's manifest only once nothing is
downloading and every shard its index names is there, with each file's
sha256, which nodes check; mirror add takes several URLs; a repository
page is refused as a source.
Refs #2 (item 9).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A restart with no room beside it waited the drain grace period before
stopping the old replica, and stopped it even when the replacement
would fit nowhere. It now stops at once when the replica alone blocks
its replacement, gives the restart up (saying why) when nothing would
fit, and ctl restart --in-place stops first on request.
Refs #2 (item 3).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
engine_overrides give nodes picked by GPU class, id or labels their own
engine versions; node add --disable-engine keeps one off a node; what
vLLM cannot do on a node is a warning rather than a refused node; the
CUDA requirement follows the version; a hand-named binary is no longer
replaced by an installed one.
Refs #2 (items 4 and 7).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tp on a llama.cpp profile was a width nothing placed. gpus_per_node is
now the count: placement reserves that many cards on one node and the
engine splits layers across them. tp, pp and nodes are refused with
what to write instead.
Refs #2 (item 8).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw ctl model adapters lists a model's adapters and adds or removes
them through POST /api/v1/models/{name}/adapters, which reads each new
file first and refuses one that is not an adapter for the model's
architecture, as the mirror page's card does.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An adapters view in the explorer, filtered by the model a held model was
made from; GGUF files told apart as adapter or model by their first bytes;
held adapter files marked; and a model's card attaches and detaches held
adapters of its architecture, refusing models and mismatched architectures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's adapters are fetched by each node with its weights and loaded
unapplied at start; a request naming an adapter reaches the model's
replicas with llama.cpp's per-request lora field. Adapter names share the
model namespace, are listed, and follow allowlists on their own. The
router's config seed and pushed catalog now build the name map in one place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Candidates tried on the way to finding no room were listed as whose replicas
a placement would stop.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The quotas page gains a capacity section over the same API methods the CLI
uses; a model's card sets its class, owner, memory and cores, applying only
what changed. Headroom counts the replicas a model could get by preempting
what its class outranks, and the card and ctl model target use it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every agent samples, at most every ten seconds: nvidia-smi's temperature,
HBM temperature, power draw, enforced limit and throttle reasons per GPU;
hwmon's CPU package and board inlet temperatures; and CPU package power
from RAPL's energy counters. Sent with heartbeats, published as metrics
and shown on the node page; unmeasured sensors are absent, never zero.
Observe-only, step 0 of docs/facility-aware-placement.md, which now
describes the agent source and what these readings can and cannot say.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It said "stopping a drained replica without a replacement after=5m0s" when
it had been given the 30-second preemption grace.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Idle times are unknown after a control plane restart, which left the choice
between a 1-core and a 4-core replica to map order.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Candidates were added in order until the model fitted, and every one added
was stopped, including those tried first that freed room on the wrong node.
A set that works is now pruned to the replicas it needs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stored ctl bench results reach the reconciler: requests per replica past
the measured SLO concurrency scale a model up, a replica is taken back only
if the rest stay under 70% of theirs, and among profiles of equal priority
the most measured tokens per GPU or core is tried first. ctl model target
sizes a model for a throughput against the room there is.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model has an owner (org, team or project). Quotas guarantee a scope
GPUs, cores and memory and may limit it; what a scope holds is summed from
its models' replicas and those beneath it. A replica within its owner's
guarantee is not taken for another scope; what is held beyond a guarantee
is borrowed and taken back for a scope within its own. A limit stops a
model before it starts, and headroom stops there too. ctl quota and ctl
model owner set them; metrics and a CapacityAtLimit alert report them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's class (critical, standard, opportunistic; unset follows its
scaling) decides what may be stopped to make room for it: a lower class
for a higher, and within a class only what another grew into under load,
for a model below its minimum. Victims are tried on a copy of the
inventory and stopped only if they make the room. Preempted replicas are
drained with a short grace and their model is held back until the room is
used. ctl model resources and class move under ctl model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
