45e3f52da3d9fa58c2d8ad5ee394701d542e16f6
75
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ilker ManapandClaude Opus 5.5
|
20479a2d5f |
Find multi-node NVLink domains, place across them, and show them as racks
A GB200 NVL72 rack or an NVLink Switch system is a scale-up island: its GPUs reach each other over NVLink whatever node they are in. Each GPU now reports its fabric from nvidia-smi -q - state, status, ClusterUUID, CliqueId, bandwidth - at registration and once a minute after, since a fabric manager restarting drops every card out at once; and each node reports nvidia-imex and its channels, without which the domain's GPUs cannot share memory across nodes. A single machine's NVSwitch, which reports a cluster of zeros, is no domain. A replica spanning nodes is placed inside one domain first, on nodes whose every card is joined and whose IMEX is up, and its shards are told NCCL_MNNVL_ENABLE=1; InfiniBand is the fallback. /admin/nvlink draws each domain as a rack - the NVLink Switch band on top, light running down a backplane, trays of GPU chips that glow when joined, pulse red when not, turn amber on degraded bandwidth - and lists what keeps a tray out. A node's page shows its GPU-by-GPU NVLink matrix, each card's NUMA node and nearest adapter, and its domain and IMEX. doctor checks joining, bandwidth, IMEX and its channel; metrics llmgw_gpu_nvlink_joined and llmgw_imex_up; /api/v1/nvlink. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
5bbc619a31 |
Map every InfiniBand fabric, draw it, and place replicas by it
The control plane asks one node on each physical fabric to run
ibnetdiscover every five minutes and parses the text itself, so a
format the parser has not met is a control plane upgrade, not one on
every node. From the map it works out the switch tiers (leaf, spine,
core, by distance from the adapters), which adapter is which node's,
which cables trained below the rate the rest of their kind run at, and
whether the rails line up: every node's Nth adapter on its own leaf.
A page per fabric (/admin/fabrics) draws it: switches in their tiers,
adapters fanned out under their leaves, cables coloured by rate and
bundled, degraded ones marching red with a marker, and the cluster's
own adapters live - down ports glowing red, and particles running each
cable as fast as its adapter is sending and receiving, from the
counters. It zooms, pans, fades what a hovered switch or adapter does
not touch, opens a panel of detail on a click, colours by rail, finds a
node, and refreshes every fifteen seconds. The data is also at
/api/v1/fabrics/{id}/topology.
Placement reads the map: a replica spanning nodes takes nodes of one
leaf together when it can, and nodes whose cable trained short last.
Static assets are named by their content, as the stylesheet already
was.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
||
|
Ilker ManapandClaude Opus 5.5
|
1b1ca67e35 |
Add llmgw doctor: what a node's GPUs, NVLink and fabric need, checked
For machines nobody here can log into. It reads the GPUs and their nvidia-smi topology, NVSwitch and the fabric manager, every InfiniBand and RoCE port with its counters and RoCE v2 GID, the RDMA and GPUDirect kernel modules, the agent's declared fabrics against the adapters, the management interface NCCL will bootstrap over, the agent unit's memlock limit and the engines installed - and says what is wrong and how to fix it, most serious first, with the NCCL environment a shard on each fabric would be given. --json writes every fact it concluded from, to send back. It exits non-zero when something fails. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
4d9284ce9f |
Find each node's management interface, and lift the agent's memlock limit
The agent's defaults said eth0 before normalize looked for the interface holding the management address, so it never looked: a node on ens3 or bond0 had NCCL and Gloo bootstrap over an interface it does not have. It is now found, with eth0 only as the last resort. The agent unit sets LimitMEMLOCK=infinity. RDMA pins what it registers, and under systemd's 8 MB default an engine without the capability that lets root ignore the limit fails its first collective over InfiniBand. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
daed835d88 |
Measure each node's CPU and memory, and give every node a page
The nodes are CPU machines, and nothing measured how busy one was: the heartbeat carried GPU memory and disk, and llmgw_node_memory_bytes is the total. The agent now reads /proc between heartbeats - CPU across all cores, load averages, memory used and available, and each engine's process group's cores and resident memory - and sends it with the heartbeat. Nothing new runs on the nodes and Prometheus needs no new target: the control plane exports it with the rest of the fleet, as llmgw_node_cpu_utilization_percent, llmgw_node_cpus, llmgw_node_load, llmgw_node_memory_used_bytes, llmgw_node_memory_available_bytes, llmgw_shard_cpu_cores and llmgw_shard_memory_bytes. A measurement more than a minute old is left out rather than repeated. The cluster page's node table shows CPU and RAM as bars, and each node links to a page of its own: current CPU, RAM, load and disk, the engines on it with their cores and memory, and an hour of CPU, RAM, load, per-model cores and memory and queue depth from Prometheus. It refreshes in place. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
f13b18149a |
Refuse weights the nodes' llama.cpp cannot load, before fetching them
kev-0.5b declared architecture "ggmlc", which llama.cpp does not know. It was fetched, catalogued and placed, and failed on every node — while the answer was in the first kilobytes of the file. internal/gguf reads a GGUF header (v1-v3) up to general.architecture, over HTTP with a Range request, and judges it against the architectures llama.cpp b10991 lists in LLM_ARCH_NAMES (152, copied from the source at that tag); a vision projector is not a model on its own. An unknown build is not refused, only not vouched for, and a header that cannot be fetched does not refuse anything. Checked in three places: the registry listing (from the architecture Hugging Face reports with expand[]=gguf, so one request per repository) shows it and withholds the fetch button; a fetch reads the header and refuses unless "fetch anyway"; and the model service refuses to add such a model, which covers the serve form and ctl model add. Tried on real files: the mirror's qwen2.5-1.5b (qwen2) and bge-m3 (bert), and Qwen2.5-0.5B on Hugging Face through its redirect. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
e493b2f53d |
Give an organization's managers a page of their own
A member of an organization can be its manager (users join --org X --role manager, or the star on the people page). /org shows a manager their organization: usage over time by team, project or person, its teams and projects with limits and 30-day spend, and its members, each once with every place they belong. Read-only; administrators see every organization. Only what was spent in the organization's projects is shown — UsageBucketsIn filters by where a request was spent. Found on the way: - Usage by org, team and project, and the quotas page's spend, joined through api_keys and so dropped every chat-page request, which has no key row. They now read usage.project_id; keys no longer move between projects, so the two agree. - /org was not exempt from API-key auth and answered browsers with a JSON error. A router test now drives the session pages through the middleware; it fails without the fix. Membership queries, org-scoped usage and the reworked joins pass the PostgreSQL store tests on 16. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
098886e0e4 |
Let a person belong to many places, or to none
A person was in exactly one project, chosen at approval, and could not use the gateway before that. People belong to several teams and to none, so: - memberships: a person is a member of any number of organizations, teams and projects, or of none; a member of a team or organization may use every project in it. users join/leave, and the people page adds and takes out from either side. Approval no longer needs a project. - personal keys: a key may have an owner and no project. /profile asks what a new key spends against; keys create --for does the same. auth.personal sets default limits for personal use or turns it off. - limits for people: --scope user holds a person across every key, project and the chat page; --per-member on an org, team or project gives each member their own share there. Resolution folds the tree, each per-member limit for the key's owner, and the owner's own, each in its own pool; the quotas page shows and sets both. - the chat page asks what to spend against and checks it per message. - usage records who spent it, so a person's budget survives a restart, personal use and chat included; restore no longer drops rows with no project. Leaving a project revokes the keys made there unless another membership still reaches it. A personal key keeps no threads or notes, which are per project, and is refused rather than dropped into the anonymous pool. Migration 0010 turns each person's project into a membership, adds usage.user_id and backfills it, and makes api_keys.project_id nullable. SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign keys off, which only works outside a transaction, so migrations marked for it run on one connection with foreign_key_check before commit. Tested on a populated SQLite store with claims, and on a copy of the cluster's PostgreSQL store: 5 people became members, 17 keys and 3 claims intact, 145 of 577 usage rows and all 69 chat rows attributed. The PostgreSQL store tests, including new membership cases, pass on 16. users move is gone; join and leave replace it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
42fa602247 |
Make a new control plane serve a model by itself
server init now also: - runs a model mirror beside the control plane (/srv/models, bytes on :8081 at an address nodes reach, management API on loopback so it needs no certificate) and adds it to server.yaml; - makes the machine a node of its own control plane: issues its agent certificate, writes agent.yaml labelled as the control plane, starts llmgw-agent, and waits for it to register and install llama.cpp; - checks for NVIDIA GPUs with nvidia-smi, and reports a card on the PCI bus whose driver does not answer rather than ignoring it; - fetches Qwen2.5 0.5B (Q4_K_M) into the mirror and adds it to the catalog pinned to this node: a llama.cpp profile per GPU class found, with a CPU profile as the fallback, then waits for it to serve. --no-mirror, --no-node and --no-model leave these out. Every step looks before it acts, so a rerun finishes what an interrupted one started. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
3104f4b706 |
Manage people and the tenancy tree, from the web and the CLI
Nothing in the tree could be renamed, moved or removed, and an approved person stayed in the project they were first put in — from the web or the CLI. Now: - /admin/users is the tree, each project opening onto its people and keys, with approve, move, suspend, reinstate and remove for people, revoke for keys, and add, rename, move and remove at every level, plus a search over every account. Forms post and get the page back, so it works with no script; htmx 2.0.11 (0BSD, embedded, checked against npm's integrity hash) swaps only the changed part, keeping open projects open. - llmgw org tree/add/rename/move/remove, users move/delete, and keys list --all. - Projects, teams and organizations are named by id, path (Acme/platform/gateway) or unique name; users by id or email. With a single project --project can be left out, instead of defaulting to "default", which a tree built from the web does not have. Moving a person moves the keys they made; operator keys stay with the project. Only empty things are removed, and never a project that had a key, since usage history refers to it. Also: keys init no longer reparents an existing team or project; reinstating an account with no project is refused (a rejected signup used to come back active and fail at its first key); rejecting a signup removes it; keys limits --budget takes whole units like the quotas page. Checked in headless Chrome: a move from an open project updates the page in place, keeps the project open, and the search still filters. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
a764cfacbc |
Find the PostgreSQL a machine has, rather than assume its names
server init --postgres failed on Rocky with postgresql-setup not found. The PostgreSQL project's packages (PGDG) provide "postgresql-server", so dnf can satisfy that name with postgresql16-server — whose unit is postgresql-16.service, whose tools are in /usr/pgsql-16/bin and whose setup script is postgresql-16-setup. The code assumed the distribution's names for all three. The server unit is now picked from systemd's postgresql* unit files, its data directory read from the unit's own PGDATA, and initdb run directly as postgres when that directory is empty, with password logins over TCP. psql, initdb and the restart use what was found. The unit names and PGDATA values in the tests are the ones Rocky's and PGDG's packages ship. Not yet handled: a server initialised with ident for TCP logins, one on a port other than 5432, and several PGDG versions where the highest is not the one on 5432. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
cba65f5be7 |
Move a control plane to PostgreSQL with server init --postgres
Installs PostgreSQL with the machine's own package manager when it has none (initdb with password auth on the RHEL family, whose default is ident), creates the llmgw role and database with a generated password kept in /etc/llmgw/secrets/store-dsn, and checks the address connects. On a machine already on SQLite it stops the gateway, copies the store so accounts and keys survive, points auth.store at the new address and starts it again. A move that fails starts the gateway on SQLite again. server.yaml is edited as text so its comments survive, and kept beside it. A PostgreSQL init did not install is not resized: it may serve something else. It refuses to move one PostgreSQL store into another. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
2a381ad669 |
Set up a control plane with one command
`llmgw server init` does what install.md's steps 2 and 3 asked of the operator: the llmgw user and directories, a certificate authority, the control plane's certificate for every address the machine has, a configuration, the default project, an administrator with a generated password, and the systemd unit — then starts it and waits for /health. Every step looks before it acts, so a second run changes nothing, and --host later reissues the certificate with the old names kept. It never overwrites server.yaml or an authority, and leaves alone certificates issued by an authority kept on another machine. The configuration install.md showed could not start: an empty catalog without control.enabled is refused as having nothing to serve. The one init writes sets it, and a test holds that. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
33e678ab84 |
Show a node's logs on the admin page
Seeing what a node did meant ssh through a jump host, to a machine whose address nobody remembers, to read a journal only root may open. When an engine install failed on a new node today, the reason was one line — and it was four steps away from the page that said the install had failed. Each node's row on the cluster page now links to its logs: the agent's own, and every engine running there, chosen from the list the node itself sends. The page can follow, refreshing every five seconds, and is on both the admin listener and /admin. `llmgw ctl logs NODE [--source SHARD] [-n N]` reads the same. They travel over the stream the agent already holds, so they need nothing the control plane does not already have: a request goes down, the lines come back, and a reply that does not come in ten seconds is an agent too old to answer. The agent keeps its last 2000 lines in memory as well as writing them wherever it did before; the engines' last few hundred lines were already kept and sent only when one crashed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
7d97c6b74f |
Install a node's engines on the node, instead of trusting what is there
A node ran whatever llama-server somebody had once unpacked on it. Onboarding checked an engine only when told its path, installed none, and a node with nothing on it registered as ready. The one time that went wrong it went wrong invisibly: the binary was there, executable, and could not start for want of libgomp1, while the node reported healthy. The control plane now says which engines every node has, at which version — `engines:` in server.yaml, defaulting to the llama.cpp build this release was checked against — and each agent converges on it. What only the node can know, the node works out: the build for its processor and driver (CUDA with its runtime when an NVIDIA driver is loaded, CPU otherwise), the OS packages that build needs from its own package sources, and whether the result runs. Every archive is checked against the sha256 GitHub publishes for it and unpacked refusing any path or link that leaves the install directory. ldd names a missing library outright; running `--version` proves the rest. Only then is a directory marked installed, and it is checked again every ten minutes, because an unrelated upgrade can remove a library. An engine is reported only once installed and seen to run, or when the agent's configuration names it on purpose — never because something happened to be on PATH. The agent now says when it has probed, so an empty list means "has none" rather than "too old to say": a node rebuilt from nothing is no longer a candidate for every model at the moment it can run none of them. Placement's refusal carries the node's own words, "installing: downloading llama-b10991-bin-ubuntu-x64.tar.gz (16 MB)", and so do `ctl nodes` and the cluster page, along with a warning for a CPU without AVX2 — the VM CPU type that makes everything five times slower and nothing look wrong. Agents also report their OS, architecture, vector extensions and NVIDIA driver. Onboarding installs llmgw itself from the CI release for the node's architecture, checked against the release's checksums, rather than copying whatever binary the operator's machine had. vLLM and SGLang are not installed yet: they are Python environments of several gigabytes, not an archive, and still have to be named in the agent's configuration. Also fixes a race on the weights fingerprint, which two heartbeat loops touch during a failover. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
40f5c930c4 |
Say when a replica is running an edited profile, and check a source from the web too
CI / test (push) Successful in 1m43s
Two defects, both of the same kind: something the CLI knew and the rest of the system did not. A profile can be edited while a replica is running against the old one. The reconciler deliberately leaves it alone — churning a replica that is serving costs more than the imperfection — but there was no way to see it, so the cluster could disagree with its own configuration indefinitely and read as healthy. Profile now carries a fingerprint, a replica records the one it was placed with, and both the API and `ctl replicas` say "stale profile" when they differ. An empty fingerprint is not stale: it means a replica adopted from an older agent, which is unknown rather than wrong. Adding a model from the admin page skipped the source check `ctl model add` has done all along, so a typo or a mirror that is down became a replica failing on a node minutes later, with an error naming the node. The check moves out of the CLI into a package both use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
dfbb7557dd |
Serve a model made of several files
CI / test (push) Successful in 1m42s
The mirror would happily hold a four-shard model that nothing could serve. A catalog entry names one source, the agent fetched one URL, and so a version of several files had no way to be described — which is every safetensors model, and every GGUF too large to ship whole. A source ending in a slash names a version rather than a file. The agent fetches every file in it into one directory and hands the engine the directory: vLLM reads the shards and the config out of the folder, and llama.cpp is given the first part and finds the rest beside it. The trailing slash is the whole signal. A flag beside the source would be a second place for the two to disagree, and a URL already distinguishes a directory from a file everywhere else. The mirror answers a version with a manifest instead of a file listing. A plain file server gives HTML there, and something that has to know what a version consists of would be left parsing it. It is on the byte listener, not the management API: fetching weights is the one thing every node does, and making it depend on the administrative port would undo the split that keeps that port closeable. Built from the directory rather than a file written beside it. A manifest on disk is a second record of what is there, and the two disagree the first time somebody copies a file in by hand — which the mirror explicitly allows. The whole set or nothing. A directory with three of four shards loads as a broken model rather than failing to load, so a completion marker is written last and an interrupted fetch starts the missing files again rather than being found and served. Files already there at the right size are not fetched twice. Progress is reported under the version's address, not each file's: the shards waiting on it asked for the version, and progress under a file's own URL reaches nobody. A filename from the manifest is refused if it contains a separator. The mirror is a place an operator copies files into by hand, so what it reports is not automatically safe to join onto a path. An HTML listing where a manifest was expected says so. "invalid character '<'" sends somebody looking at the wrong thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
9ef02dc9ec |
Give the charts to both copies of the dashboard
CI / test (push) Successful in 1m48s
The page is served twice — on the admin listener, and at /admin behind a session — and the history source reached only the second. The first version built it after the admin listener's copy was already registered and passed that one an empty NavExtras, so the page an operator reaches by SSH tunnel had no charts and no explanation. Built before either is registered, and handed to both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c6df0659af |
Put the cluster's own charts on its own admin page
CI / test (push) Successful in 1m50s
The dashboard charted what the cluster had been billed for — requests, tokens, spend, all from llmgw's own store — and nothing about whether it was healthy. Saturation, queue depth, cache occupancy and replica counts were numbers in a table at the instant the page loaded, with no way to see that one of them had been climbing for an hour. Four line charts now, read back from wherever the metrics were scraped to. Read back rather than kept twice. llmgw publishes these already; what a gauge cannot do is remember, and a ring buffer in this process would be a rival copy of the same numbers with its own retention, lost on every restart and disagreeing with the one an operator has on screen elsewhere. monitoring.prometheus is empty by default and the panels are then absent rather than empty. One axis per chart, always. Two measures of different scale belong in two charts: a second y-axis lets any two lines be made to cross wherever the author likes and the reader cannot tell it happened. Four colours, handed out in fixed order, never cycled. A fifth series is summed into "other" and drawn in grey — dropping it would make a chart that does not add up, and inventing a hue for it would put two lines on the page that nobody can tell apart. The fold is by timestamp, not by position: the tail's series do not share sample times, and adding them by index slides points sideways onto each other's clock. A legend for every chart with more than one line, because colour alone is not an encoding a reader can rely on. A flat zero is drawn along the bottom rather than left blank: nothing happening is a real answer, and an empty frame reads as a broken chart. A single sample is not a line and says so instead of putting a dot in the corner. One failed query fails all four panels. Three drawn and one blank invites reading the blank one as "nothing is happening there". NaN is dropped rather than plotted. Prometheus sends it as a string for a sample it has no value for, Go's ParseFloat accepts it happily, and one NaN coordinate is an SVG path that does not draw at all — taking the rest of the line with it. Axis maxima are rounded to a number a person would pick. An axis topping out at 0.8347 makes every gridline unreadable, and the shape is what a chart is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
2acd794022 |
Stop placing engines on nodes that do not have them
CI / test (push) Successful in 1m42s
A node's registry holds an adapter for every engine whether or not its binary exists, the agent never said which were installed, and placement carried an Engine field it had never once read. So a vLLM replica was as happily placed on a node with no vLLM: the node registered healthy, the shard failed minutes later with "executable file not found" in the engine's own words, and the reconciler tried again. That is how fifteen gigabytes of safetensors reached this cluster's mirror before anybody noticed it had no card and no vLLM to run them with. Agents now probe what is installed and report it — the protocol has carried engine_versions since it was written and nothing ever filled it in. Placement filters on the key set, `ctl nodes` has a column, and the mirror page's serve warning is a fact instead of a guess about cards, which would have waved through a fleet of eight-GPU nodes that never had vLLM. Probing resolves binaries and runs none of them. The first version asked each engine for --version, which is fine for an engine and not fine for the wrapper scripts people put in front of one: a script that shifts past flags it does not recognise and then execs the server will start an inference server, at agent startup, on every node. The agent's own test suite installs exactly such a script and caught it. So presence is a path lookup, and the version is left to `ctl engine-check`, which is run by somebody who meant to run the engine. SGLang is the exception and has to execute, because a module has no path to look up — but the command is llmgw's own one-line import rather than the engine's launcher. Its exit code is the answer: python3 is on almost every machine, and treating a failed import as "installed" reported SGLang on every node in a fleet that had none, which is the failure being fixed, inverted. Probed once, not per reconnect, and bounded by the context that registration waits on: an agent that cannot reach the control plane is worse than one reporting an incomplete list. A node that reports no engines at all is not excluded anywhere — by placement, or by the page. That is what every agent older than this reports, and emptying a fleet on a control plane upgrade is a worse failure than the one being fixed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
e034862b79 |
Serve a mirrored model from the list, instead of writing YAML for it
CI / test (push) Successful in 1m35s
Fetching a model put it on the mirror and left the operator to hand-write a catalogue entry and run `ctl model add` on the control plane — for a source the page was already displaying, with an engine decided by the file extension it had just shown them. A present row now offers to serve it. The form opens on that row with the name, the mirror's own URL and the engine filled in, and asks for the two things nobody can infer: the context pool and how many replicas. What it refuses is the point. A version of several files is refused with the reason. A catalogue entry names one source, so an entry for a sharded model is an entry for one shard, which loads as a broken model rather than failing to load. Nothing said so before — the mirror happily holds a four-shard model that no catalogue entry can describe. Safetensors on a fleet with no GPU is warned about before anything is added. This is the failure we hit today: fifteen gigabytes of shards arrived, and only then did looking at a node show no card, no vLLM and no SGLang. The control plane is not told which engines a node has installed, so the check is on cards, which it does know — a warning rather than a refusal, because a card can be added tomorrow. A name already declared in configuration is refused here too, matching what the model service would say anyway, but before the form is filled in rather than after. Without a store there is no serve action at all: a model added at runtime would not survive a restart, and offering it then refusing is worse than not offering. The mirror page's dependencies became a struct. Three are optional interfaces and two are the same shape, which at a call site is a swap nobody notices until a page shows the wrong thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ed530133b6 |
Find a model in the registry instead of pasting a URL into a box
CI / test (push) Successful in 1m41s
Mirroring anything meant knowing a repository existed, knowing which of its thirty files were the weights, and assembling a resolve URL by hand. The mirror page had a text box for that URL and no way to find one. It now searches Hugging Face and lists a repository's weight files with their sizes and published checksums, each with a button that fills the fetch form. Only weight files. A repository is mostly a README and a tokenizer, and listing those buries the four entries somebody came for. The size shown is the LFS size, not the plain one. For an LFS-stored file — every weights file — the plain field is the size of the pointer, so the other one reports a gigabyte model as 133 bytes, and that is the number somebody would then plan disk against. The source URL is pinned to the commit the listing came from rather than to a branch, so what the mirror fetches is what was on screen. A repository re-uploaded under the same branch otherwise leaves two different sets of weights with identical provenance and no way to tell them apart. The version defaults to that revision for the same reason. The registry's own failures are reported on their own line. It is somebody else's server on the other side of the internet, and it being down says nothing about the mirror — which is what the page is for and has to keep working regardless. With no registry configured there is no search box at all, rather than one that can only time out. Gating decodes from a bool or a string, because the field is documented as the first and is the second in practice: strict decoding fails the whole response, so one gated model would hide the other nineteen in a page of results. And a bug this made obvious: downloads were keyed by model and version, so the second shard of a sharded model was refused as "already fetching" until the first had finished. A fifteen-shard model had to be pulled one file at a time, each waiting on the last. Keyed by file now — which is what the single-flight was ever protecting, since the race is two writers on one .part. A version made of several files reports as the worst of them: it is not present until every shard is, and one shard failing makes the version unusable however many others arrived. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
a83493bddb |
Show what a group spent, and fix the reports that never ran on PostgreSQL
Quotas can be written on an organization, a team or a project, and nothing could show what any of them had used: usage came back per key, per user and per model only. A quota whose consumption cannot be seen is one nobody reviews until the day it refuses somebody — and since the previous commit made a group quota a shared pool, it is now a number several people are spending against without being able to look at it. `llmgw ctl usage --by team` sums every key beneath each group, with the budget that level sets beside it. Beside it, and only when that level sets one. A team bounded by its organization's budget has none of its own, and printing the organization's against each of its teams would read as though each had that much — the exact misreading this work exists to remove. Joined through api_keys rather than read from usage.project_id: that column records where the key was when the request was served, while the quota is written against the tree. They agree until a key is moved, and a report that disagrees with the limit shown beside it is the useless one. Three existing reports were fixed on the way, all failing on PostgreSQL and all passing in tests, because the tests run on SQLite: column "k.name" must appear in the GROUP BY clause (UsageSince) column "us.email" must appear in the GROUP BY clause (UsageByUser) column "usage.ts" must appear in the GROUP BY clause (UsageBuckets) SQLite picks an arbitrary row for an ungrouped column and carries on; PostgreSQL refuses. The first two were selecting names they had not grouped. The third grouped "ts / ?" while selecting "(ts / ?) * ?", which PostgreSQL matches as text and so saw a bare ts — it is grouped by ordinal now, the same expression by construction. The dashboard's usage charts have been empty on the cluster this whole time, with the reason only in the log. All four queries were run against the cluster's own PostgreSQL before this was committed, because passing on SQLite is what they were already doing. The grouping level picks from a fixed table of levels rather than being interpolated, so an unknown one is an error and not a fragment of SQL. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ebce41e22d |
Stop calling the PostgreSQL store SQLite, and stop guessing where it is
Two things that were only ever confusing, and one of them stopped a command working on the cluster. The store type was named SQLite and kept the name when PostgreSQL was added, so OpenPostgres returned a *SQLite and all seventy-odd methods on the production store were spelled after the database it was not running on. Nothing broke; people did. Earlier today I added three methods to *SQLite for a table that on this cluster only ever lives in PostgreSQL, and the name did not once look wrong. It is a field, which is all it ever was. `llmgw users` and `llmgw keys` defaulted --store to a SQLite path. On a deployment whose store is PostgreSQL every one of them failed with "unable to open database file" while the right answer sat in auth.store in the config file beside them; the conclusion an operator draws is that the tool is broken, and they are not far off. They now ask the server's own configuration before falling back, and the error names which of the three sources chose the address — the failure was never the path, it was which one picked it. The single-machine default still applies last, so a deployment with no config file keeps working with no flags. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
3aa250428a |
Keep what a benchmark found out, instead of leaving it in a terminal
`ctl bench` has always been able to work out what one replica can serve — it ramps concurrency until the SLO breaks and reports the last level that held. It printed that and nothing kept it, so the one number saying what a replica is worth lived in whoever ran it's scrollback. Everything downstream then had to guess: admission applies a single global default for healthy queue depth, and sizing a model against real demand had nothing to size against. `ctl bench --save` records it; `ctl capacity` reads it back. Keyed by model and profile. Capacity belongs to the hardware and the engine arguments as much as to the weights, so the same model on CPU and on an H100 are two numbers, and one key would overwrite the other on every run. The profile is looked up from the replica rather than asked for, because the control plane already knows which profile answers on which endpoint and a measurement filed under the wrong one is worse than none. A target that is not a replica — the gateway, usually — is refused with that said: benchmarking the whole path is worth doing and says nothing about the hardware. The conclusion is stored, not the ramp. Every level and its latencies are worth reading once while judging whether the run was sound; keeping them would grow without bound as a model is re-benchmarked. Re-measuring replaces. A second run is a better answer to the same question, not another data point, and keeping both would leave a reader choosing between numbers with no basis for the choice. The SLO and the date are stored with it. Capacity means nothing without the target it was measured against, and hardware and engine builds change — a measurement is an observation with a date on it, not a fact about the model. A failed save is reported and not fatal: the run has already cost the GPU time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
707d774c76 |
Change a model's replica count without editing a file on the control plane
The count is scaling.min, and until now the only way to change it was to edit server.yaml on the host and restart — or, for a model added at runtime, to remove it and add it back, which means having its whole definition to hand. `llmgw ctl scale <model> --min N` instead. Only the scaling block moves: everything else is read back from what was stored and written out untouched, so changing a number cannot quietly become a redefinition. The catalog is rebuilt and validated before anything is written, as in Add. A max below a min is refused with the stored definition untouched — persisting a rejected change would come back on the next restart as a catalog that does not build. The answer reports the scaling in force rather than the request. The catalog fills in a max that was left out, and a caller echoing its own request would print a number the cluster is not using. A model declared in configuration is refused, for the reason Remove already refuses one: the YAML is read on every start, so the new count would hold until the next restart and then silently revert. A cluster disagreeing with its own config file is worse than an error saying where to edit. The help says what --max does, because it is not what it looks like: a model with a minimum runs exactly that many and nothing scales it up under load, and an on-demand model loads one replica however high its max. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
cd7eedf09e |
Give the mirror a page, so managing it is not a curl command
CI / test (push) Successful in 1m49s
/admin/mirror, behind the same admin session as the other admin pages. What is held, what is arriving with its byte count, what failed and why; start a fetch, retry a failed one, remove a version. Not mounted at all without a mirror configured. The right page for a deployment with no mirror is no page, rather than an empty one pointing at nothing. It reloads itself only while something is downloading. A reload at any other moment takes with it whatever is half-typed into the fetch form, and there is nothing to watch when nothing is moving. A failed row offers Retry, not Remove. A fetch that failed left nothing on disk, so the remove could only answer "the mirror holds nothing for this" — a button whose single outcome is a confusing error. Starting the same download again is what somebody looking at that row wants, and the mirror allows it: only a download still running is refused. An unreachable mirror is said outright instead of rendering as an empty table. The two look nearly identical and mean opposite things: one is a mirror waiting to be filled, the other is a machine to go and look at. Dashboard takes NavExtras rather than a second bool. Two adjacent booleans at a call site get swapped silently and are discovered by somebody clicking the wrong link. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
16ec585411 |
Let the control plane ask the mirror for a model instead of pushing one at it
The mirror holds the weights, so the mirror does the downloading. A control plane that had to pull the bytes itself and push them here would carry every model twice over the network, through the one machine least able to spare the bandwidth. Two listeners, and the split is the point. Weights stay on plain HTTP where every node can reach them: that path is eight nodes pulling forty gigabytes, and putting a client certificate in front of it would mean re-certifying every agent's fetcher in order to change how models are administered. Management gets its own port, which can be given the fleet's CA — the same tlsconf the agent stream uses, so this is a configuration rather than a new idea. A fetch answers 202, not 200. The bytes are not here yet, and a caller that read it as "done" would point a catalog at a file still arriving, which every node would then copy. Progress is asked for rather than waited on, because no client between here and the operator will hold a connection open for an hour. A failed fetch is remembered with its reason. Forgetting it would leave the control plane waiting on a model that is never coming, with nothing to say why. Status falls back to reading the disk, so a restart does not report a model that is plainly there as unknown. Removal takes the whole version: a model split across files is useless with one of them missing, and half a version on disk looks present and will not load. It refuses while a download is running, because the fetch would carry on writing and the file would reappear, looking like the delete had silently failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b40bc481cf |
Manage the model mirror with a command instead of a runbook
CI / test (push) Successful in 1m36s
The mirror was a directory and a discipline written down in a README. This makes the three operations that matter into commands, on the mirror host, where `llmgw keys` already sets the precedent of operator tooling that works on files rather than through the API. The gateway still never writes to the mirror. That property was chosen deliberately — a mirror writable by the thing reading from it is a way to replace a model with one every node then fetches without complaint — and a person running a command does not take it away. `serve` states it rather than inheriting it, so a handler added later cannot quietly make the mirror writable. `add` downloads to a temporary name and renames, for the reason the agent's fetcher does: a mirror is copied to every node, so a partial file here is a partial file everywhere. It verifies the length the server promised, reports a checksum — the mirror is the one place a corrupt file reaches every node rather than being caught once — and refuses to replace an existing version, which is the whole point of the layout. `unused` refuses to answer when it cannot read the catalog. An empty catalog and an unreachable one produce the same list, everything, and the obvious next step after that list is deletion. It compares by path rather than by spelling, because a base typed with a different scheme or a trailing slash would otherwise report a file in use as unused. `remove` checks the catalog first and says to retire the entry before deleting the file, because a node that has to refetch a deleted file cannot, and that failure arrives later, on the node, looking like something else. Verified end to end: Range requests answer 206, which is what makes an interrupted fetch on a node resume; writes answer 405; a traversal attempt answers 404. Note: internal/agent/supervisor's TestSidecarRunsBeforeEngine fails under full suite load on main as well as here — it asserts that one shell's echo lands before another's, which the supervisor does not promise. Everything else passes. Left alone rather than fixed in a commit about mirrors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
a35e8796d2 |
Carry a download's progress to where somebody is looking at it
CI / test (push) Successful in 1m28s
The agent has always logged a percentage as it fetches weights, with a comment saying a replica pulling a hundred gigabytes and one that is wedged look identical from the control plane. They still did: the log is on the node, and the node is not where anyone is looking when a replica has been starting for ten minutes. Progress now rides on the shard status and appears under the replica in `ctl replicas` — a line rather than a column, because most replicas are not downloading anything and a column would be empty for all of them. It is deliberately not a state transition. Treating each chunk as one would re-mark the shard as starting on every report and reset the timer that decides when a slow start has become a failed one, so an hour-long download would keep a doomed replica alive indefinitely. It arrives on its own method for that reason, and a real transition clears it — otherwise a finished download sits in the listing at ninety-nine percent under a shard that has been serving for an hour. Watchers are keyed by source rather than by shard, because a fetch is single-flight: eight shards starting together produce one download and all eight are waiting on it. No percentage is invented when the server sent no length. One that stops moving is worse than a byte count that keeps rising, so the field is absent and the listing says "so far". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
8e57171c4f |
Let ctl ask a question instead of printing the whole fleet
CI / test (push) Successful in 1m42s
Five nodes fit in a table. Fifty do not, and this is built for people who already have racks of GPUs — so the flags are how a question gets asked: which nodes on this fabric, which of this GPU class, which are not ready, which replicas have a shard on this machine. Filtered here rather than in the API, so --json is narrowed the same way the table is. A server-side filter would be two implementations of one predicate, and the JSON one is the one nobody would notice had drifted. An empty result says which case it is. A filter that matched nothing and an empty fleet call for opposite next steps — one is a cluster to look at, the other is a word to correct — and printing "no nodes" for both makes a mistyped fabric name read as an outage. A state matches on prefix because states are few and distinct and somebody scanning a fleet types quickly. A pool matches either half, because it is fabric/class and both are worth narrowing by. A replica is found from any of its nodes, since one spanning two machines is on both. --label takes a bare key as "carries this at all", which is how you ask for every node with a role without caring which. The flag's help said so before the parser did; the test caught them disagreeing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
4ba84fa90e |
Let an agent find the leader instead of being told one address
CI / test (push) Successful in 1m32s
The second half of leadership. A standby that refuses to place but still holds an agent's stream is worse than useless: commands travel down the stream the agent opened, so the leader's placements fail with "no agent connected" for a node that is plainly connected. So a standby refuses the stream, before reading the opening message, and says why. And an agent takes a list rather than an address, trying each in turn — a standby refusing, a stopped control plane refusing the connection and a crashed one never answering are the same thing from a node, and the answer to all three is somewhere else. `server` and `servers` are both accepted, with `server` first when both are given: every agent configuration written before there could be a second control plane names one address, and an operator adding a standby expects the one already working to stay preferred. With a single address the behaviour is exactly what it was, including the backoff — which is skipped only while there is another address that has not just been tried, because the delay exists to stop hammering one control plane and the next one has not been hammered. Removed a check that asked whether one field was empty, which is how the first run of these tests failed: the list was populated and `server` was not, and an agent configured entirely correctly was refused by its own validation. The tests run a real agent against a real service over h2c. The first attempt used a plain HTTP server, and the refusal arrived as an HTTP/2 framing error that said nothing about why — which is exactly what an operator would have seen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
5b5e61cb97 |
Stop a second control plane from placing work behind the first's back
CI / test (push) Successful in 1m35s
The first thing HA needs that a shared store does not give. Two control planes against one store both reconcile: both see a model wanting two replicas and none running, and both place. The model ends up with twice what it asked for, each replica holding GPUs the other's inventory believes are free — and nothing crashes to say so. A lease in a table rather than a lock held by a connection. PostgreSQL has advisory locks and SQLite has nothing like them, and this store keeps one set of queries for both; a row with an expiry works everywhere, and it survives a leader being powered off rather than asked to stop, which is the case that matters. Taking and renewing are the same statement, so the two cannot disagree about what a held lease looks like, and it is one statement, so two processes racing cannot both believe they won. The owner id is random rather than the hostname. A hostname would let a restarted process match its own stale lease and believe it had been leading all along — including through the gap that made it restart, during which somebody else may have taken over and started placing. A store that cannot be reached is not evidence that somebody else is leading, so an unreachable store keeps the claim. Two leaders is worse than one, but no leader means nothing is placed and nothing is evicted, and a brief database blip should not stop a cluster healing itself. Unset means leader. Every deployment that existed before this, every single-control-plane install and every test acts unconditionally, with nothing to configure. The same seven assertions run against both backends, which is how the test harness was caught inheriting a lease between subtests — it passed on SQLite, where each subtest gets a fresh file, and failed only on PostgreSQL. What this does not yet do is move agents. A standby holding an agent's stream is one the leader cannot command, so failover is degraded rather than clean until agents know more than one address. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b6de469351 |
Move a store to another one without reissuing every key
CI / test (push) Has been cancelled
Keys are stored hashed, so a migration that drops them cannot recreate them: every person and every script needs a new one issued, and the old ones cannot even be looked up to say which they were. `llmgw keys move-store` copies instead. Found by running it against the real cluster, twice, after the gateway had already been stopped. The first order came from the order the migrations create tables in, and users is created in a later migration than api_keys even though api_keys references it — so the copy died three tables in. The order is now checked by a test that derives the foreign keys from the migrations rather than restating them, so a new reference fails there instead of during somebody's migration. Checked by putting the wrong order back and watching it fail. Then key_claims. Its sealed column is emptied the moment a claim is used, so every claimed row holds an empty blob in a NOT NULL column. An empty blob scans as a typed nil slice and a genuine NULL as an untyped one — a distinction this side can see and pgx cannot, since it encodes any nil slice as NULL. Restored before the insert, which is the difference between a copy that completes and one that stops four tables in. One transaction per table rather than one for the whole copy: a single transaction holding every row of a large usage table is how a migration runs out of memory on the machine least able to spare it. A failure therefore leaves a partial copy, which is why the destination must have been empty — starting again is dropping and recreating it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
a76ebdb79a |
Run the store on PostgreSQL, chosen by what the address looks like
CI / test (push) Successful in 3m11s
A path is still SQLite and a postgres:// URL is PostgreSQL, so every existing configuration keeps working untouched and moving is one line changed rather than a new setting to find. An address of the form @/path/to/file reads the real one out of that file, which is how a DSN carrying a password stays out of a configuration that gets copied, diffed and pasted into issues. The driver is pgx, which is pure Go. That is not a preference: the whole binary is built with CGO_ENABLED=0 so it runs on a node whose libc is older than the builder's, and a cgo driver would take that away from every deployment, including the ones still on SQLite. Tested against a real PostgreSQL, which found two things nothing else could. The schema_migrations table is created in Go rather than in a migration file, so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL, and a millisecond timestamp does not fit, so the very first migration failed to record that it had run. And the memory migration uses an FTS5 virtual table with three triggers keeping it in step, which has no counterpart: PostgreSQL derives a tsvector from the row, so there is nothing to keep in step and no trigger to forget. That leaves exactly one query that is not the same sentence in both dialects — the note search, MATCH against @@ — and it is written out twice rather than hidden behind a shared spelling that would mean something subtly different in each. Everything else is one set of queries. The text search uses the 'simple' configuration rather than English. These are an agent's own notes, in whatever language it chose, and stemming English rules over Turkish would quietly stop finding things. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 deniz
|
2eb79f3400 |
agenta memory eklendi
CI / test (pull_request) Successful in 2m31s
|
||
|
Ilker ManapandClaude Opus 5
|
eaf3382a6d |
Give an operator a way to turn conversation storage on
CI / test (push) Successful in 1m44s
SetThreadRetention existed and nothing called it, so the feature was unreachable: a project could never opt in and every request carrying a thread was refused. A command rather than a configuration key, because it is a decision about data rather than about deployment. Someone has to choose to keep what a team types, and choosing it should look like an act rather than like a default nobody read. --retain-hours is required. Without a number the command would create exactly the thing the design avoids: a conversation store with no expiry, holding everything a team ever typed because nobody chose one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
e6bc7859dc |
Carry the delete capability through the indirection that hid it
CI / test (push) Successful in 1m36s
`ctl weights --collect` answered "this control plane cannot ask nodes to delete files" against a real cluster, with the whole suite green. The controller is handed a dispatcherRef rather than the agent service — an indirection that exists to break a construction cycle, and one that satisfies the dispatcher interface by forwarding each method by hand. The controller reaches past that interface by type assertion to find capabilities beyond scheduling, and an assertion against dispatcherRef fails for anything nobody remembered to forward. Nothing could have caught it below the wiring. The dispatchers the tests use do not implement the capability either, so its absence there is the expected case and looks identical to this bug. The test therefore asserts on the production types themselves: that the forwarding dispatcher carries the capability, and that the service it forwards to has one to carry. Checked by deleting the method and watching it fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
774e134ed1 |
Refuse a node with no room, and take back only what llmgw put there
CI / test (push) Successful in 1m39s
Two halves of the same problem: nothing stopped a model being placed on a node that could not hold it, and nothing ever removed the weights of a model that had been deleted from the catalog. Placement now has a disk check, shaped exactly like the memory one it sits beside, with one addition: it is skipped on a node that already holds the weights. A node holding the file needs no room for it, and without that the node best placed to run a model — because it already has it — is the one most likely to be rejected, since holding the file is what used the space. Reclaiming is the dangerous half, so the whole design is about what it will not do. llmgw deletes only files it fetched itself, and that is a calculation rather than a heuristic: the cache name is a hash of the source, so the set of wanted files can be computed from the catalog exactly. Anything without that name was put there by an operator — scp, a shared mount, a copy by hand — and is none of this system's business however unreferenced it looks. A model that took a week to copy must not be removable by a command someone ran to tidy up. The node checks every path again before unlinking it, and that duplication is the point. The control plane knows the catalog; the node knows its disk. A control plane that is wrong, newer than the agent, or tampered with must not be able to talk a node into deleting something, so the agent refuses anything outside its own cache directory or without the cache name — including a path that tries to traverse out of it. Listing is the default and deleting takes --collect. The summary says "asked to remove", because each node decides, and one that refuses a path it cannot prove it fetched is this working rather than failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
753f4e3265 |
Let a node say how much room it has left
CI / test (push) Failing after 1m31s
Disk was the one resource nothing reported. `statfs` was already being called on every model directory, but only to ask whether the filesystem was a network one — the free space it returns in the same struct was thrown away. So the only way to find out why a node was filling up was to ssh to it, which is what this system exists to avoid. Reported per filesystem rather than per configured directory. Free space belongs to the disk, not the path: two model directories on one disk are one pool of bytes, and counting them separately would let the control plane add 24 GB to 24 GB and place a 40 GB model on a node that has 24. Shared filesystems are summed separately from local ones and left out of a node's own total. Space on a filer belongs to everyone mounting it, so twenty nodes would otherwise each report the same free terabyte, and a scheduler would place twenty models into room for one. Nothing reported means nothing is known, everywhere: an agent too old to send storage and a node with no model directories both send nothing, and both must read as unknown rather than as a full disk. `ctl nodes` and the dashboard show a dash, the metric is absent rather than zero, and a missing directory — which is what an unmounted filer looks like — is skipped rather than reported as having no space. Shown as free out of total, because the first is useless without the second: 8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root partition, and only the second stops a model being placed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b5177b70f6 |
Stop tool-check blaming the engine for what the model did
CI / test (push) Successful in 1m37s
It had one verdict for a failed attempt — "This is the engine, not the model. Choosing a bigger model will not help" — and that verdict was wrong, in a way that sends someone off rebuilding llama.cpp when the fix is a one-line catalog change. Measured on the cluster. The engine renders the tool prompt correctly: the <tools> block and the instruction to wrap the call in <tool_call> are both in the prompt, which I confirmed through the engine's own /apply-template. The coder models simply do not use that wrapper — qwen2.5-coder-3b puts the call in a markdown fence, qwen2.5-coder-7b invents <function-call> — so the parser, which looks for <tool_call>, has nothing to find and leaves it in the content. qwen2.5-3b-instruct, same engine, same build, same flags, returns a populated tool_calls array. So it was the model all along, and the advice against trying another one was the most expensive sentence in the output. A failed attempt is now two verdicts, told apart by whether the reply used the wrapper its template asked for. Wrapped and unparsed is genuinely the engine's parser. Unwrapped is the model, and the fix is one trained for the format rather than a bigger one from the same family. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1ea067d08e |
Give metrics a door that does not also open onto pprof
CI / test (push) Successful in 1m38s
The metrics existed and nothing could record them. They shared a listener with pprof, the management API and the dashboard, and pprof can dump the process heap where API keys sit in plaintext — so a fleet could be monitored or keep its heap private, and not both. That is not a choice anyone should have to make, and it is why every previous answer to "how do I scrape this" was a tunnel. metrics_listen serves metrics and health and nothing else, so it can be reached from wherever Prometheus lives. Metrics stay on the admin listener too: adding a door is not the same as moving one, and anything already scraping keeps working. While there: admin_listen defaulted to :9090, every interface, with a comment beside it saying not to expose it. A default that contradicts its own documentation is a footgun, so it is loopback now and someone who wants otherwise says so. The handler is nil until the registry is built, and says so rather than serving an empty page. Ordering has already caught this exact area once — a collector registered after AdminHandler published nothing at all, silently — and an empty scrape target looks like a working one. deploy/monitoring/prometheus.yml has a starting configuration and the alerts worth having: a node that has stopped reporting, a model that should be serving and is not, a card kept out of placement by a fault, and a fleet holding models nobody is asking for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
7fe59bd5fb |
Load a model when someone asks for it, and stop it when nobody does
CI / test (push) Successful in 1m29s
The feature the idle tracking was for. A cluster with more models than GPUs can now hold every model in its catalog and only the wanted ones in its cards: min 0 with a max above it means load on demand, and once nobody has asked for idle_timeout the replicas stop and the cards go back. No new field, because min 0 max 1 already reads correctly — keep none running, allow up to one. Which made min 0 max 0 meaningful for the first time: configured and deliberately not running. normalize() used to fill a zero max in, on the reasonable ground that a model with no maximum cannot run; that would now silently turn every switched-off model into an on-demand one, including the ones already written that way in this cluster's own configuration. A model with no scaling stanza stays off for the same reason — defaulting it to on-demand would let any client request start a 70B nobody intended to run. The one real decision is what happens to the request that triggers a load. Refusing makes the feature useless, since the first request for every model fails; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds holds a connection for ten minutes. So it waits briefly and then says what is happening — 503 with Retry-After and "the model is loading", which is a different answer from "no endpoint serves this" and produces different client behaviour. An unknown model still gets the unknown-model error, or on-demand would turn every typo into "loading" and the useful message would disappear. Use keeps a model alive rather than only the request that started it, without which the timer measures time since the load and a busy model is evicted on schedule however hard it is working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
3a4cc0c719 |
Register the fleet collector before the registry is built
CI / test (push) Successful in 1m31s
|
||
|
Ilker ManapandClaude Opus 5
|
42fd0aa31e |
Export what the cluster is made of, not only what it is serving
CI / test (push) Successful in 1m28s
The router published twenty-two metric families about requests, endpoints, admission and scoring — everything about traffic and nothing about the machines carrying it. There was no way to graph GPU usage, watch a node go unhealthy, see a fabric fill up, or answer "was the cluster full at the time", which is the first question anyone asks after an incident. The agents exported nothing at all. Exported by the control plane rather than by each agent. It already holds every number: hardware arrives at registration and state on each heartbeat, so this is a map read away. One scrape target for a fleet of hundreds, no Prometheus configuration per node, no firewall rule per machine, and nothing new running on the nodes. On the same /metrics as the traffic figures, because correlating "the cluster was full" with "requests were queueing" should not need two systems. A collector rather than gauges kept current by a ticker: the numbers are derived from state that already exists, so computing them when asked is simpler than maintaining a copy that can drift, and nothing runs between scrapes. GPU utilisation is new, and is the number people actually watch. Memory used says a model is loaded; this says whether it is doing anything, and a fleet at ninety percent occupancy and two percent utilisation is a fleet holding models nobody is asking for. Adding it caught two things. utilization.gpu goes last in the nvidia-smi query and is optional, because a driver too old to know it returns a shorter row — requiring it made an llmgw upgrade silently stop discovering GPUs on an older node, which is far worse than not knowing how busy a card is. And I gave the new protobuf field number 14, which ecc_errors already had; buf lint is why that was a moment rather than a corrupted wire format. Absent is kept distinct from zero throughout, for the same reason as everywhere else here: an unreported utilisation shown as a flat zero reads as an idle fleet rather than a missing measurement, and a replica nothing has reported would be the most idle thing in the cluster. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c13c567cc5 |
Know where the weights are, without owning the storage
CI / test (push) Successful in 1m32s
Storage will be different at every installation — Lustre, NFS over RDMA, an image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes it rather than managing it. An agent told which directories to look in reports what it can see; how the files got there is nobody's business here, and a node told nothing reports nothing and behaves exactly as every node did before this existed. That fact buys two things. Placement prefers a node that already holds the model over one that must first move tens of gigabytes, as a preference and never a filter: fetching them is a valid answer, and refusing to place would turn a slow start into no start at all. And ctl status shows what each fabric holds. Per fabric because fabrics are not connected to one another. Weights in ib-a are not weights in ib-b — the only path between them is the management network — so a model seeded into one fabric leaves the rest of the cluster unable to serve it without a slow copy, and nothing said so before now. One shared filer counts the same as twenty local copies, because "can this fabric serve it" is the question being asked. Three distinctions the tests hold to, because each is a way to get this quietly wrong. A node that was never asked to look must rank with one that looked and has nothing, or enabling the feature on a single node would drain every other. An empty directory is not weights, because a half-finished download looks exactly like one and would send placement to a node whose engine then fails to load. And local storage is never mistaken for shared: a filer read as local costs an unnecessary copy, while local read as shared would have the cluster believe a model is everywhere when it is on one node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
bd134a7ab9 |
Make the planned entry use an engine that can serve the precision
CI / test (push) Successful in 1m39s
Reading the tool's own output against the real fleet: it chose q8_0 as the best fit and emitted a vLLM profile with no quantisation. q8_0 is a GGUF quantisation that only llama.cpp serves, and from a file rather than a repository — so following that entry loads full-precision weights, which the same output had just listed as not fitting one card. The planner would have caused the out-of-memory it exists to prevent, from a recommendation that looked right. Choosing a precision and choosing an engine turn out to be one decision, so a Precision now carries the engine that serves it, the quantisation flag where one is needed, and whether it lives in a file. A GGUF fit prints a path and says to download it; a vLLM fit prints the repository and passes --quantization so the engine does not quietly load bf16. The arithmetic was right and the recommendation was wrong, which is the failure mode worth having tests for: everything reported a number, and the numbers agreed with each other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
300e298053 |
Work out whether a model fits the cluster, and how to split it
CI / test (push) Has been cancelled
llmgw ctl model plan Qwen/Qwen2.5-Coder-32B-Instruct The arithmetic this replaces is the last thing between having GPUs and serving a model on them. Choosing tp, gpu_class and a context that will actually fit means knowing the layer count, the KV head count, what the weights weigh at a chosen quantisation and how much of a card the runtime keeps — and getting it wrong is not an error message, it is an out-of-memory twenty minutes into a load. I computed those by hand for every model on this cluster. Both halves are read rather than retyped: the architecture from the model's own config.json and parameter count, the hardware from the running fleet. Attention heads in place of KV heads is the classic slip and is worth a factor of five on a model with grouped-query attention, so it is not a number anyone should be entering. Every candidate is listed, not one answer. With cards to spare you want the full-precision fit; with one card free you want to know a quantised one exists. Choosing between those is not the planner's job. The tests are calibrated against files actually downloaded here — llama.cpp reported 14.62 GiB and 8.37 GiB for the 14B at q8_0 and q4_K_M, and the model reproduces both — and they caught two errors in my own arithmetic. The overhead was charged per rank from the whole model's weights rather than each rank's share, and it reserved a tenth of a card that usableFraction had already reserved. Between them, Llama-3.3-70B at bf16 across two 80 GB cards came out as not fitting, which is a configuration people run every day. Slack reserved in two places, each unaware of the other, reads as caution and behaves as a bug. Pools are grouped by class and fabric together, because a replica may never span two fabrics: eight free cards split across two of them are not eight cards for a model that needs eight, and merging them would plan something that starts, passes its health checks and hangs on its first all-reduce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
d07dddb1bb |
Say plainly whether a model is serving, and why not
CI / test (push) Successful in 1m28s
Reported after watching a model appear in ctl models before it could answer anything. Checked: clients are not told early — /v1/models listed it only once a replica was ready, and the chat picker reads the same source. The operator view showing it is deliberate, because a model coming up is exactly what someone running the cluster wants to see. But the report pointed at something real. That view distinguished "a replica is coming up" from "nothing could be placed" by a parenthetical — "0/1 ready (1 total)" against "0/1 ready" — and those need opposite responses: one means wait, the other means look at explain-placement. On four models the difference is findable. On forty it is invisible, and a model that will never start reads exactly like one that is thirty seconds away. SERVING now answers the question that was being asked of the replica counts — can a client use this right now — and STATUS says which of ready, starting, not placed or stopped is the case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
331de9b495 |
Add and remove models from the command line
CI / test (push) Successful in 1m30s
The last piece of not restarting a control plane to try a model: llmgw ctl model add -f qwen-coder-32b.yaml llmgw ctl model remove qwen-coder-32b The file is the same YAML a model takes in server.yaml, read by the same koanf parser through the same tags, so an entry moves between the two by being copied and a field that works in one cannot silently mean nothing in the other. Wired only when there is a store. A control plane without one says so rather than accepting a model that vanishes on the next restart, which is the kind of success that wastes an afternoon. At startup the catalog is rebuilt from configuration plus what was stored, so models added before a restart are still there after it — without that the whole feature would be a slower way to lose work. Every failure adding a model is the definition being wrong — a collision, a missing fallback, a profile that cannot be placed — so they are 400s. A 500 would send someone to the server's logs for a mistake in their own file. The actor comes from the caller and is kept with the row, so a catalog that has grown over months can say where each entry came from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |