baaaa4e019fcdbe721483efc3510c867c7f2d92f
518
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3febb6ef63 |
Count a group's quota against the group, not against each of its keys
The tenancy tree handed limits down and never pooled what was spent against them. A limit written on a team was inherited by every key beneath it, and then enforced with the key as the counter — so a project given a budget of 1000 and ten keys permitted ten thousand, and a team given 60 RPM got 60 per member. That is "each of you may spend this much", which is a useful thing to be able to say. It is not what somebody writing a budget on a team means, and nothing in the system said which one you were getting. Resolution now records which scope set each value that won, and the counter is keyed by that scope. The value and the counter come from the same place, so a project tightening its team's limit is enforced against the project's own pool rather than eating into a quota another project never agreed to. Each limit is owned separately, because they resolve separately: a team budget with a per-key rate limit — the group's money, the individual's throughput — is the ordinary arrangement and has to stay expressible. Cluster defaults are owned by the key. Read as a shared pool, one default would become a single cluster-wide rate limit that every key in every organization competes for: not a default, an outage. Restoring after a restart had to change with it. Per-key totals rebuilt a counter named after the key while the check read one named after the team, so the control plane came back believing the team had spent nothing — a budget that reset on every restart, for exactly the scopes where it matters most. SpendSinceByScope returns each key's place in the tree and the accountant folds it into every scope that might own a budget. Counters are namespaced by level throughout. A bare key id in one path and a "key:"-prefixed one in another are two counters for the same key, which is how the restore was silently doing nothing. Still missing, and worth naming: no usage report rolls up by project, team or org, so a shared quota can now be enforced but not yet shown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
3aa250428a |
Keep what a benchmark found out, instead of leaving it in a terminal
`ctl bench` has always been able to work out what one replica can serve — it ramps concurrency until the SLO breaks and reports the last level that held. It printed that and nothing kept it, so the one number saying what a replica is worth lived in whoever ran it's scrollback. Everything downstream then had to guess: admission applies a single global default for healthy queue depth, and sizing a model against real demand had nothing to size against. `ctl bench --save` records it; `ctl capacity` reads it back. Keyed by model and profile. Capacity belongs to the hardware and the engine arguments as much as to the weights, so the same model on CPU and on an H100 are two numbers, and one key would overwrite the other on every run. The profile is looked up from the replica rather than asked for, because the control plane already knows which profile answers on which endpoint and a measurement filed under the wrong one is worse than none. A target that is not a replica — the gateway, usually — is refused with that said: benchmarking the whole path is worth doing and says nothing about the hardware. The conclusion is stored, not the ramp. Every level and its latencies are worth reading once while judging whether the run was sound; keeping them would grow without bound as a model is re-benchmarked. Re-measuring replaces. A second run is a better answer to the same question, not another data point, and keeping both would leave a reader choosing between numbers with no basis for the choice. The SLO and the date are stored with it. Capacity means nothing without the target it was measured against, and hardware and engine builds change — a measurement is an observation with a date on it, not a fact about the model. A failed save is reported and not fatal: the run has already cost the GPU time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
707d774c76 |
Change a model's replica count without editing a file on the control plane
The count is scaling.min, and until now the only way to change it was to edit server.yaml on the host and restart — or, for a model added at runtime, to remove it and add it back, which means having its whole definition to hand. `llmgw ctl scale <model> --min N` instead. Only the scaling block moves: everything else is read back from what was stored and written out untouched, so changing a number cannot quietly become a redefinition. The catalog is rebuilt and validated before anything is written, as in Add. A max below a min is refused with the stored definition untouched — persisting a rejected change would come back on the next restart as a catalog that does not build. The answer reports the scaling in force rather than the request. The catalog fills in a max that was left out, and a caller echoing its own request would print a number the cluster is not using. A model declared in configuration is refused, for the reason Remove already refuses one: the YAML is read on every start, so the new count would hold until the next restart and then silently revert. A cluster disagreeing with its own config file is worse than an error saying where to edit. The help says what --max does, because it is not what it looks like: a model with a minimum runs exactly that many and nothing scales it up under load, and an on-demand model loads one replica however high its max. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
10e323132c |
Let a control plane whose models all live in the database start
Control plane mode was inferred from the config file alone: a model with profiles meant "control plane", nothing meant "router only". So a deployment that manages its catalog at runtime could not start at all — the file declares no models, the process refuses for want of anything to serve, and the stored models it would have loaded are never read. A cluster unmanageable at runtime because it is managed at runtime. control.enabled says so outright. AgentListen cannot carry the signal: it has a default, so every deployment would silently become a control plane. The inference stays, so nothing already working has to add a flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
cd7eedf09e |
Give the mirror a page, so managing it is not a curl command
CI / test (push) Successful in 1m49s
/admin/mirror, behind the same admin session as the other admin pages. What is held, what is arriving with its byte count, what failed and why; start a fetch, retry a failed one, remove a version. Not mounted at all without a mirror configured. The right page for a deployment with no mirror is no page, rather than an empty one pointing at nothing. It reloads itself only while something is downloading. A reload at any other moment takes with it whatever is half-typed into the fetch form, and there is nothing to watch when nothing is moving. A failed row offers Retry, not Remove. A fetch that failed left nothing on disk, so the remove could only answer "the mirror holds nothing for this" — a button whose single outcome is a confusing error. Starting the same download again is what somebody looking at that row wants, and the mirror allows it: only a download still running is refused. An unreachable mirror is said outright instead of rendering as an empty table. The two look nearly identical and mean opposite things: one is a mirror waiting to be filled, the other is a machine to go and look at. Dashboard takes NavExtras rather than a second bool. Two adjacent booleans at a call site get swapped silently and are discovered by somebody clicking the wrong link. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
744a283cf1 |
Keep the checksum that finishing the download used to take away
All() preferred the disk entry wholesale, and a directory listing has no checksum in it. So the sha256 a fetch computed was reported while the bytes were still arriving and vanished the moment the file landed — it disappeared exactly when it became verifiable, which is the one moment it is for. It is the field an operator compares against what upstream published. A mirror is the single place in this system where a corrupted file is copied to every node instead of being caught once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
7e6fb0059a |
Let the control plane call the mirror instead of somebody SSHing to it
The mirror has had a management API since it was built, and the only thing driving it was curl on the mirror's own machine. This is the other half: a client, so the operations belong to whatever has a reason to perform them. The mirror's refusals travel as their own sentence rather than a status code. Putting the reason in the body was deliberate on the server side, and a client that kept only the number would undo it — "the mirror said 400" sends somebody to the wrong machine, "the source URL answered 404" sends them to the right one. NextProtos is cleared for the same reason the mirror's server clears it. That configuration is shaped for the agent stream, which pins h2; a transport carrying a pinned protocol it is not set up to speak negotiates h2 and then talks HTTP/1.1 over it, and fails with an error mentioning neither ALPN nor the mirror. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
dde2bcab19 |
Stop asserting an ordering the supervisor does not promise
The test started a sidecar and an engine, each a shell appending its name to a file, and demanded the sidecar's line come first. The supervisor spawns the sidecar first, but spawning is not sequencing: without WaitForLeaderPort nothing holds the engine back, so two shells race and the scheduler picks the winner. It failed under full-suite load, and lately on its own. The wait that is a real guarantee already has its own test, so the fix is to check what this one is actually for — that both processes run under one shard, that the shard's pid is the engine's rather than the sidecar's, and that their output is distinguishable in a single log tail. Renamed to say so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
16ec585411 |
Let the control plane ask the mirror for a model instead of pushing one at it
The mirror holds the weights, so the mirror does the downloading. A control plane that had to pull the bytes itself and push them here would carry every model twice over the network, through the one machine least able to spare the bandwidth. Two listeners, and the split is the point. Weights stay on plain HTTP where every node can reach them: that path is eight nodes pulling forty gigabytes, and putting a client certificate in front of it would mean re-certifying every agent's fetcher in order to change how models are administered. Management gets its own port, which can be given the fleet's CA — the same tlsconf the agent stream uses, so this is a configuration rather than a new idea. A fetch answers 202, not 200. The bytes are not here yet, and a caller that read it as "done" would point a catalog at a file still arriving, which every node would then copy. Progress is asked for rather than waited on, because no client between here and the operator will hold a connection open for an hour. A failed fetch is remembered with its reason. Forgetting it would leave the control plane waiting on a model that is never coming, with nothing to say why. Status falls back to reading the disk, so a restart does not report a model that is plainly there as unknown. Removal takes the whole version: a model split across files is useless with one of them missing, and half a version on disk looks present and will not load. It refuses while a download is running, because the fetch would carry on writing and the file would reappear, looking like the delete had silently failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
b40bc481cf |
Manage the model mirror with a command instead of a runbook
CI / test (push) Successful in 1m36s
The mirror was a directory and a discipline written down in a README. This makes the three operations that matter into commands, on the mirror host, where `llmgw keys` already sets the precedent of operator tooling that works on files rather than through the API. The gateway still never writes to the mirror. That property was chosen deliberately — a mirror writable by the thing reading from it is a way to replace a model with one every node then fetches without complaint — and a person running a command does not take it away. `serve` states it rather than inheriting it, so a handler added later cannot quietly make the mirror writable. `add` downloads to a temporary name and renames, for the reason the agent's fetcher does: a mirror is copied to every node, so a partial file here is a partial file everywhere. It verifies the length the server promised, reports a checksum — the mirror is the one place a corrupt file reaches every node rather than being caught once — and refuses to replace an existing version, which is the whole point of the layout. `unused` refuses to answer when it cannot read the catalog. An empty catalog and an unreachable one produce the same list, everything, and the obvious next step after that list is deletion. It compares by path rather than by spelling, because a base typed with a different scheme or a trailing slash would otherwise report a file in use as unused. `remove` checks the catalog first and says to retire the entry before deleting the file, because a node that has to refetch a deleted file cannot, and that failure arrives later, on the node, looking like something else. Verified end to end: Range requests answer 206, which is what makes an interrupted fetch on a node resume; writes answer 405; a traversal attempt answers 404. Note: internal/agent/supervisor's TestSidecarRunsBeforeEngine fails under full suite load on main as well as here — it asserts that one shell's echo lands before another's, which the supervisor does not promise. Everything else passes. Left alone rather than fixed in a commit about mirrors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
f2f375d824 |
Stop the text-search comment naming one language
CI / test (push) Successful in 1m41s
The choice is unchanged and now better argued: a deployment does not announce which language its notes are in, so any stemmer picked here is the wrong one somewhere — and applying the wrong one does not fail loudly, it quietly stops finding things. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
2bf7de1a43 |
Give two control planes a shared address, and a probe that means something
CI / test (push) Successful in 1m36s
The last piece of HA, and it is a deployment pattern rather than code: llmgw picks one leader and agents follow it, but client traffic arrives in front of both and llmgw is not in front of anything. Writing the pattern found a bug in the pattern. The first draft said to health-check `/health` and claimed a standby still routes to replicas the leader placed, so sending it traffic was fine. It does not. A standby's reconciler places nothing and it holds no agent streams to adopt from, so it publishes an empty endpoint list — a load balancer following that advice would have sent half the traffic to something that answers every request with 503. So there is now a `/ready` that means "this process has somewhere to send a request": 503 on a standby, 503 on a leader whose replicas have all failed, 200 when there is something to serve. `/health` keeps meaning the process is up, which is a different and less useful question. Both are open, because a load balancer has no key. There is a gap during failover where neither passes — three seconds when the leader stops cleanly, up to the lease TTL when it does not — and the document says so rather than implying otherwise. Traffic cannot go to a control plane that has nothing to route to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
a35e8796d2 |
Carry a download's progress to where somebody is looking at it
CI / test (push) Successful in 1m28s
The agent has always logged a percentage as it fetches weights, with a comment saying a replica pulling a hundred gigabytes and one that is wedged look identical from the control plane. They still did: the log is on the node, and the node is not where anyone is looking when a replica has been starting for ten minutes. Progress now rides on the shard status and appears under the replica in `ctl replicas` — a line rather than a column, because most replicas are not downloading anything and a column would be empty for all of them. It is deliberately not a state transition. Treating each chunk as one would re-mark the shard as starting on every report and reset the timer that decides when a slow start has become a failed one, so an hour-long download would keep a doomed replica alive indefinitely. It arrives on its own method for that reason, and a real transition clears it — otherwise a finished download sits in the listing at ninety-nine percent under a shard that has been serving for an hour. Watchers are keyed by source rather than by shard, because a fetch is single-flight: eight shards starting together produce one download and all eight are waiting on it. No percentage is invented when the server sent no length. One that stops moving is worse than a byte count that keeps rising, so the field is absent and the listing says "so far". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
100f10ac17 |
Generate the configuration reference, and fail when it goes stale
CI / test (push) Successful in 1m41s
There was no reference. Four narrative documents explain how to install and operate llmgw, and the only way to learn that `lease_ttl` exists was to read the one that happened to mention it — or the struct. Written by hand it would have been wrong within a fortnight. Six settings arrived in the last two weeks alone: store, servers, lease_ttl, the two retention windows, min_disk_gb. A reference nobody maintains is worse than none, because it is the same list with the setting you came to look up quietly missing from it. So it is derived from the structs, with the field comments as the descriptions — which means the only way for it to be wrong is for the code to be wrong, and it makes the comments worth writing twice over. 1163 lines, 24 blocks, 87 keys. Two tests keep that true after everyone has forgotten this exists. One regenerates and compares, so adding a setting without regenerating is a failure rather than an omission — checked by adding one and watching it fail. The other walks the tree for koanf tags and fails on a configuration file that is in neither the source list nor the short list of structs an operator never writes, because a list is a thing to forget. The generator finds the repository from its own path rather than the working directory, so `go generate ./...` produces the same document from anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
8e57171c4f |
Let ctl ask a question instead of printing the whole fleet
CI / test (push) Successful in 1m42s
Five nodes fit in a table. Fifty do not, and this is built for people who already have racks of GPUs — so the flags are how a question gets asked: which nodes on this fabric, which of this GPU class, which are not ready, which replicas have a shard on this machine. Filtered here rather than in the API, so --json is narrowed the same way the table is. A server-side filter would be two implementations of one predicate, and the JSON one is the one nobody would notice had drifted. An empty result says which case it is. A filter that matched nothing and an empty fleet call for opposite next steps — one is a cluster to look at, the other is a word to correct — and printing "no nodes" for both makes a mistyped fabric name read as an outage. A state matches on prefix because states are few and distinct and somebody scanning a fleet types quickly. A pool matches either half, because it is fabric/class and both are worth narrowing by. A replica is found from any of its nodes, since one spanning two machines is on both. --label takes a bare key as "carries this at all", which is how you ask for every node with a role without caring which. The flag's help said so before the parser did; the test caught them disagreeing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
4ba84fa90e |
Let an agent find the leader instead of being told one address
CI / test (push) Successful in 1m32s
The second half of leadership. A standby that refuses to place but still holds an agent's stream is worse than useless: commands travel down the stream the agent opened, so the leader's placements fail with "no agent connected" for a node that is plainly connected. So a standby refuses the stream, before reading the opening message, and says why. And an agent takes a list rather than an address, trying each in turn — a standby refusing, a stopped control plane refusing the connection and a crashed one never answering are the same thing from a node, and the answer to all three is somewhere else. `server` and `servers` are both accepted, with `server` first when both are given: every agent configuration written before there could be a second control plane names one address, and an operator adding a standby expects the one already working to stay preferred. With a single address the behaviour is exactly what it was, including the backoff — which is skipped only while there is another address that has not just been tried, because the delay exists to stop hammering one control plane and the next one has not been hammered. Removed a check that asked whether one field was empty, which is how the first run of these tests failed: the list was populated and `server` was not, and an agent configured entirely correctly was refused by its own validation. The tests run a real agent against a real service over h2c. The first attempt used a plain HTTP server, and the refusal arrived as an HTTP/2 framing error that said nothing about why — which is exactly what an operator would have seen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
ade689fc8e |
Refuse to run the destructive store tests against a real database
CI / test (push) Successful in 1m47s
These TRUNCATE every table before each test. Pointed at a deployment that is a silent and complete data loss — every key, every user, every conversation. Not hypothetical. I did it: an SSH tunnel to the production database had been opened minutes earlier for a perfectly good reason, and the same DSN was then handed to the test suite. Twenty-one users, nine API keys, eighteen sessions, four hundred and sixty-one usage records and three runtime-added models, gone between one command and the next. Recovered only because the pre-migration SQLite file was still there. The guard is the database name. A test database has to say so, because a check that could be satisfied by accident would not have stopped this one — the DSN was entirely correct for what it had been opened for, and catastrophic for what it was then used for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
5b5e61cb97 |
Stop a second control plane from placing work behind the first's back
CI / test (push) Successful in 1m35s
The first thing HA needs that a shared store does not give. Two control planes against one store both reconcile: both see a model wanting two replicas and none running, and both place. The model ends up with twice what it asked for, each replica holding GPUs the other's inventory believes are free — and nothing crashes to say so. A lease in a table rather than a lock held by a connection. PostgreSQL has advisory locks and SQLite has nothing like them, and this store keeps one set of queries for both; a row with an expiry works everywhere, and it survives a leader being powered off rather than asked to stop, which is the case that matters. Taking and renewing are the same statement, so the two cannot disagree about what a held lease looks like, and it is one statement, so two processes racing cannot both believe they won. The owner id is random rather than the hostname. A hostname would let a restarted process match its own stale lease and believe it had been leading all along — including through the gap that made it restart, during which somebody else may have taken over and started placing. A store that cannot be reached is not evidence that somebody else is leading, so an unreachable store keeps the claim. Two leaders is worse than one, but no leader means nothing is placed and nothing is evicted, and a brief database blip should not stop a cluster healing itself. Unset means leader. Every deployment that existed before this, every single-control-plane install and every test acts unconditionally, with nothing to configure. The same seven assertions run against both backends, which is how the test harness was caught inheriting a lease between subtests — it passed on SQLite, where each subtest gets a fresh file, and failed only on PostgreSQL. What this does not yet do is move agents. A standby holding an agent's stream is one the leader cannot command, so failover is degraded rather than clean until agents know more than one address. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
b6de469351 |
Move a store to another one without reissuing every key
CI / test (push) Has been cancelled
Keys are stored hashed, so a migration that drops them cannot recreate them: every person and every script needs a new one issued, and the old ones cannot even be looked up to say which they were. `llmgw keys move-store` copies instead. Found by running it against the real cluster, twice, after the gateway had already been stopped. The first order came from the order the migrations create tables in, and users is created in a later migration than api_keys even though api_keys references it — so the copy died three tables in. The order is now checked by a test that derives the foreign keys from the migrations rather than restating them, so a new reference fails there instead of during somebody's migration. Checked by putting the wrong order back and watching it fail. Then key_claims. Its sealed column is emptied the moment a claim is used, so every claimed row holds an empty blob in a NOT NULL column. An empty blob scans as a typed nil slice and a genuine NULL as an untyped one — a distinction this side can see and pgx cannot, since it encodes any nil slice as NULL. Restored before the insert, which is the difference between a copy that completes and one that stops four tables in. One transaction per table rather than one for the whole copy: a single transaction holding every row of a large usage table is how a migration runs out of memory on the machine least able to spare it. A failure therefore leaves a partial copy, which is why the destination must have been empty — starting again is dropping and recreating it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
a76ebdb79a |
Run the store on PostgreSQL, chosen by what the address looks like
CI / test (push) Successful in 3m11s
A path is still SQLite and a postgres:// URL is PostgreSQL, so every existing configuration keeps working untouched and moving is one line changed rather than a new setting to find. An address of the form @/path/to/file reads the real one out of that file, which is how a DSN carrying a password stays out of a configuration that gets copied, diffed and pasted into issues. The driver is pgx, which is pure Go. That is not a preference: the whole binary is built with CGO_ENABLED=0 so it runs on a node whose libc is older than the builder's, and a cgo driver would take that away from every deployment, including the ones still on SQLite. Tested against a real PostgreSQL, which found two things nothing else could. The schema_migrations table is created in Go rather than in a migration file, so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL, and a millisecond timestamp does not fit, so the very first migration failed to record that it had run. And the memory migration uses an FTS5 virtual table with three triggers keeping it in step, which has no counterpart: PostgreSQL derives a tsvector from the row, so there is nothing to keep in step and no trigger to forget. That leaves exactly one query that is not the same sentence in both dialects — the note search, MATCH against @@ — and it is written out twice rather than hidden behind a shared spelling that would mean something subtly different in each. Everything else is one set of queries. The text search uses the 'simple' configuration rather than English. These are an agent's own notes, in whatever language it chose, and stemming English rules over Turkish would quietly stop finding things. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
ccc9aa8b76 |
Route every query through a dialect, before there is a second one
CI / test (push) Successful in 2m30s
Groundwork for PostgreSQL, with no PostgreSQL in it yet: the store still opens SQLite and behaves exactly as it did. One set of queries rather than one per database. The SQL here is almost entirely portable — ON CONFLICT upserts work in both, and the only construct that genuinely differs is the placeholder — so two copies of every query would buy nothing and drift, one query at a time, discovered in production. The rewrite is a wrapper around the handle rather than a change at each call site. There are sixty of those and sixty chances to forget one; this way the only route to the driver goes through the rewrite. Making it a wrapper found six places that reached past it, including all three manual transactions and two prepared statements — every one of which would have worked on SQLite and failed on PostgreSQL, on a path nobody exercises before deploying. The rewriter skips quoted sections. No query here contains a question mark in a literal today; this is so that adding one is a test failure rather than a statement that silently binds the wrong argument. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
2eb79f3400 |
agenta memory eklendi
CI / test (pull_request) Successful in 2m31s
|
||
|
|
45e913bd10 |
Refresh disk and weights while the agent runs, not once when it starts
CI / test (push) Successful in 1m46s
Found by answering "where can we see this live". `ctl weights` was offering a file for deletion that I had deleted two days earlier, and the node's disk figure was the one it had when its agent started. Discover() is called at registration and nowhere else, so both were frozen. Disk is a fact about a moment, not about a machine. Sampled once, the figure is wrong on exactly the node worth watching — the one busy filling up — and it stays wrong until that agent restarts. The heartbeat already refreshes GPU memory for the same reason, with a comment about noticing a foreign process "after registration"; disk belonged beside it and I did not put it there. The stale list cut both ways. A model fetched after registration was invisible until a restart, so placement could not prefer the node that had just spent ten minutes downloading it, and the disk check could not see the space it used. Storage goes in every heartbeat, being a few numbers per filesystem. The file list goes only when a fingerprint of paths and sizes changes, with a flag saying the list is meant — without that, an empty list would be ambiguous between "this node now holds nothing" and "nothing to report", and the second reading would empty a node's inventory five seconds after it registered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
eaf3382a6d |
Give an operator a way to turn conversation storage on
CI / test (push) Successful in 1m44s
SetThreadRetention existed and nothing called it, so the feature was unreachable: a project could never opt in and every request carrying a thread was refused. A command rather than a configuration key, because it is a decision about data rather than about deployment. Someone has to choose to keep what a team types, and choosing it should look like an act rather than like a default nobody read. --retain-hours is required. Without a number the command would create exactly the thing the design avoids: a conversation store with no expiry, holding everything a team ever typed because nobody chose one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
d26a30471e |
Let people forget a conversation, and make retention actually expire things
CI / test (push) Successful in 1m38s
Two halves of keeping the promise that storing makes.
DELETE /v1/threads/{id} is a client endpoint, not an admin one: the person who
said the thing should be able to unsay it, and making them ask an operator
would mean the operator reads it first. Deleting a thread in another project
answers exactly as deleting one that never existed does — two answers would
let a caller map which ids are real in projects they cannot read, and this one
would do it by trying to destroy them.
And something now runs the pruner. Without it "we keep conversations for 24
hours" was a comment rather than a promise: the rows would accumulate and the
number a project chose would describe nothing. Hourly, and once at startup,
because a gateway that has been down for a week should not wait another hour
before honouring a window that expired while it was off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
9eb321ae3e |
Let a conversation continue on a different model than it started on
CI / test (push) Has been cancelled
The request path half. A caller sets x-llmgw-thread and sends only the new turn; llmgw puts the stored history in front of it, sends that to whichever model is serving, and stores the exchange afterwards. A header rather than a body field because some SDKs drop fields they do not know and all of them let a caller set a header. Capturing the reply is opt-in and bounded. This is the hottest path in the system, so a request without a thread accumulates nothing at all, and one with a thread stops accumulating at a cap rather than growing to it — a gateway that kept every reply in memory would be one long answer away from trouble. A reply is truncated on a rune boundary rather than dropped: a shortened turn replays as a shortened turn, while nothing at all replays as the assistant having said nothing, and invalid UTF-8 would break the next turn's body for good rather than for one turn. The streamed path sets the captured reply on every exit, not only the one that looks like the end. A stream can also finish with the client gone or the engine stalled, and what the assistant said up to that point is still what it said. A thread on a project that keeps no conversations is refused. The e2e test asserted that and got a cheerful 200: the request was served, nothing was stored, and the caller would have discovered it only when the model had no idea what they were referring to — which reads as the model being broken. It now asks whether storage is on before doing anything else. History is spliced after the model is finally settled, because a budget fallback may have changed it and trimming is measured against the context of the model that will actually serve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
cebf9906d2 |
Splice a stored conversation into a request, trimming only what must go
CI / test (push) Successful in 1m39s
The part that makes a thread work across models: the history goes in front of what the caller just said, because a client using a thread sends the new turn rather than the conversation again. Trimming has three rules, and each is there because the alternative fails quietly. The oldest stored turns go first, so a conversation that outgrew the model now serving it stays usable — after a budget fallback that model was not the caller's choice, and a permanently unusable thread would be a strange way to punish them for running out of budget. The caller's own messages are never dropped: answering something they did not ask is worse than refusing. And a system message is kept wherever it sits, because losing one silently changes what the model is for while it carries on answering. A model that declares no context length has nothing trimmed. Guessing a limit for it would drop turns for no reason and the caller would never learn why the model forgot. The body is rebuilt through a map so every field the client sent survives. Forwarding fields llmgw does not understand is how engine-specific options reach an engine at all, and a thread must not be what starts dropping them. An empty history returns the body byte for byte, so a request without a thread takes exactly the path it took before this existed. Not yet wired to the request path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
b0592670a4 |
Give projects somewhere to keep a conversation, off unless they ask
CI / test (push) Successful in 1m39s
The storage half of continuing a thread on a different model than it started on. Nothing uses it yet. Until now llmgw stored no message content anywhere: usage rows carry token counts, costs and latencies, never text, and even its own chat page keeps the conversation in a JavaScript variable that a refresh discards. That property is worth more than the feature, so it is the default and stays the default. A project stores nothing until someone turns it on for that project. Retention is a number of hours rather than a boolean, because a thread store with no expiry is a transcript of everything a team ever typed, kept forever because nobody chose a number. Turning it on means saying for how long. The window measures idleness rather than age — expiring a conversation someone is still having would be a strange way to keep that promise. Threads are scoped to a project and the scope is part of every lookup, not a check afterwards. Reading a thread from another project and reading one that does not exist return the same error: two errors would let a caller learn which ids are real in projects they cannot read. Writing across projects is refused for the same reason in the other direction — append, then read your own reply back. Deleting a thread takes its messages with it, which the test checks by counting rows rather than by trusting the cascade. Someone has to be able to say "forget that", and a retention window is no answer to a message sent by mistake. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c9c841387a |
Check the prompt fits the model that will actually serve it
CI / test (push) Successful in 1m40s
The context check ran against the model the caller asked for. The budget fallback then swapped in a different one and nothing looked again, so a prompt sized for a 32k model could be sent to a 512-token one. Measured before the fix: 10002 prompt tokens delivered to a model declaring 512. The caller sees that as a truncated reply or an engine error naming a model they never chose, which is a worse outcome than the refusal the fallback exists to avoid — it fails later and explains less. Candidates that cannot hold the prompt are now skipped, and a fallback further down the list that can hold it is used instead. When none fit, the request is refused as it was before the fallback existed. The history is deliberately not trimmed to make it fit. Dropping a caller's messages to stay under a limit they never asked about would be a silent loss, inside a request that was already being quietly moved to another model. The tests drive the real request path, because a unit test on the fallback would have asserted the same wrong thing the code did. Both were checked against the unfixed code and fail there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e6bc7859dc |
Carry the delete capability through the indirection that hid it
CI / test (push) Successful in 1m36s
`ctl weights --collect` answered "this control plane cannot ask nodes to delete files" against a real cluster, with the whole suite green. The controller is handed a dispatcherRef rather than the agent service — an indirection that exists to break a construction cycle, and one that satisfies the dispatcher interface by forwarding each method by hand. The controller reaches past that interface by type assertion to find capabilities beyond scheduling, and an assertion against dispatcherRef fails for anything nobody remembered to forward. Nothing could have caught it below the wiring. The dispatchers the tests use do not implement the capability either, so its absence there is the expected case and looks identical to this bug. The test therefore asserts on the production types themselves: that the forwarding dispatcher carries the capability, and that the service it forwards to has one to carry. Checked by deleting the method and watching it fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
774e134ed1 |
Refuse a node with no room, and take back only what llmgw put there
CI / test (push) Successful in 1m39s
Two halves of the same problem: nothing stopped a model being placed on a node that could not hold it, and nothing ever removed the weights of a model that had been deleted from the catalog. Placement now has a disk check, shaped exactly like the memory one it sits beside, with one addition: it is skipped on a node that already holds the weights. A node holding the file needs no room for it, and without that the node best placed to run a model — because it already has it — is the one most likely to be rejected, since holding the file is what used the space. Reclaiming is the dangerous half, so the whole design is about what it will not do. llmgw deletes only files it fetched itself, and that is a calculation rather than a heuristic: the cache name is a hash of the source, so the set of wanted files can be computed from the catalog exactly. Anything without that name was put there by an operator — scp, a shared mount, a copy by hand — and is none of this system's business however unreferenced it looks. A model that took a week to copy must not be removable by a command someone ran to tidy up. The node checks every path again before unlinking it, and that duplication is the point. The control plane knows the catalog; the node knows its disk. A control plane that is wrong, newer than the agent, or tampered with must not be able to talk a node into deleting something, so the agent refuses anything outside its own cache directory or without the cache name — including a path that tries to traverse out of it. Listing is the default and deleting takes --collect. The summary says "asked to remove", because each node decides, and one that refuses a path it cannot prove it fetched is this working rather than failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
753f4e3265 |
Let a node say how much room it has left
CI / test (push) Failing after 1m31s
Disk was the one resource nothing reported. `statfs` was already being called on every model directory, but only to ask whether the filesystem was a network one — the free space it returns in the same struct was thrown away. So the only way to find out why a node was filling up was to ssh to it, which is what this system exists to avoid. Reported per filesystem rather than per configured directory. Free space belongs to the disk, not the path: two model directories on one disk are one pool of bytes, and counting them separately would let the control plane add 24 GB to 24 GB and place a 40 GB model on a node that has 24. Shared filesystems are summed separately from local ones and left out of a node's own total. Space on a filer belongs to everyone mounting it, so twenty nodes would otherwise each report the same free terabyte, and a scheduler would place twenty models into room for one. Nothing reported means nothing is known, everywhere: an agent too old to send storage and a node with no model directories both send nothing, and both must read as unknown rather than as a full disk. `ctl nodes` and the dashboard show a dash, the metric is absent rather than zero, and a missing directory — which is what an unmounted filer looks like — is skipped rather than reported as having no space. Shown as free out of total, because the first is useless without the second: 8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root partition, and only the second stops a model being placed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
b5177b70f6 |
Stop tool-check blaming the engine for what the model did
CI / test (push) Successful in 1m37s
It had one verdict for a failed attempt — "This is the engine, not the model. Choosing a bigger model will not help" — and that verdict was wrong, in a way that sends someone off rebuilding llama.cpp when the fix is a one-line catalog change. Measured on the cluster. The engine renders the tool prompt correctly: the <tools> block and the instruction to wrap the call in <tool_call> are both in the prompt, which I confirmed through the engine's own /apply-template. The coder models simply do not use that wrapper — qwen2.5-coder-3b puts the call in a markdown fence, qwen2.5-coder-7b invents <function-call> — so the parser, which looks for <tool_call>, has nothing to find and leaves it in the content. qwen2.5-3b-instruct, same engine, same build, same flags, returns a populated tool_calls array. So it was the model all along, and the advice against trying another one was the most expensive sentence in the output. A failed attempt is now two verdicts, told apart by whether the reply used the wrapper its template asked for. Wrapped and unparsed is genuinely the engine's parser. Unwrapped is the model, and the fix is one trained for the format rather than a bigger one from the same family. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
0cac9d6986 |
Make the landing page look like something worth having a key for
CI / test (push) Successful in 1m30s
It is the only page a stranger sees, and until they have a key it is the whole product: a stack of headings and shell snippets said "internal tool you probably should not be here". Now it opens with what this is and a diagram of why it exists — one address, many GPUs, fabrics that cannot see each other — then the base URL as a headline rather than a line of prose with a colon in it, the route to a key as three cards, and the client examples as tabs instead of a pile. The tabs are radio buttons and a stylesheet. No JavaScript decides what is on the page, so someone who arrives with scripting off still gets every example; the only script is the copy buttons, which exist only where the clipboard API does and so cannot leave a dead control behind. The page says exactly what it said before, including that it lists no models and why. The diagram names no model, no node and no count: fabrics are drawn because llmgw keeps them apart, not because this install has two. Tests cover the parts that are easy to break later: every example lives in the page itself rather than being fetched per tab, the reflected base URL — now in five places rather than one — still cannot write markup, and the diagram carries a label for anyone not looking at it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
f8bb78103f |
Stop the ranked charts from spilling over the tables below them
CI / test (push) Successful in 1m32s
The bars were wrapped in <div class="chart">, and .chart was already the fixed-height box the time-series charts are drawn in: 120px, with a border and a background. A chart that grows a row at a time does not fit in a cage, so the SVG overflowed it and landed on the table underneath. Renamed to .ranked, which is a wrapper rather than the mark itself, and capped at the design width so a wide window scales the box instead of inflating the 12px labels. The test reads the class back out of the markup and checks the stylesheet does not cap it, because the collision was invisible in both files on their own — each rule was reasonable, and only the shared name was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
eb9e9b348c |
Show who is using the cluster and what they are running
CI / test (push) Successful in 1m28s
The dashboard could say what the cluster was made of and not who was spending it. Usage was per key, which is the right unit for revoking something and the wrong one for the question actually being asked: one person holds a laptop key, a CI key and a notebook key, and adding three rows up by eye is a chore that gets skipped. Per person now, joined through the keys they own, and per model — counted by the model that answered rather than the one requested, since those differ whenever a fallback serves and what actually ran is the question. Headline figures are stat tiles with a sparkline: a single current value is a number, and plotting one is how a chart ends up with a single bar in it. The sparkline says only rising, falling or flat, which is the one thing the number cannot. Ranked horizontal bars, because the labels are email addresses and model names and the label is the point — the question is which person, not what shape the distribution has. Rotated or truncated axis labels say the chart matters more than reading it. Trimmed where drawn and complete on hover, so the plot keeps its geometry and a reader can still find out whose row it is. The palette is validated rather than chosen: every adjacent pair clears the colour-vision separation threshold, and the dark values are the same hues re-stepped for the dark surface instead of an automatic flip. Run through a validator, because that is the only way anyone can honestly claim a palette is colourblind-safe. Which turned up a bug in the existing chart code. cssVar is an allowlist of four names that silently returns --accent for anything else, and callers wrap it in var() themselves — so every new series would have rendered in one colour with accent-coloured text. Caught by a test written from the rule that chart text wears text tokens, never the series colour. A key with no owner is labelled "operator-issued keys" rather than unknown: it predates accounts and is doing its job, and "unknown" would imply something is wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
28967edab3 |
Serve a spent budget something it can afford
CI / test (push) Successful in 1m38s
The accounting layer has always exempted a model that costs nothing, with a comment saying it is "what lets a key with no budget left still be served a cheap fallback". That described a caller nobody had written: the budget was checked once, against the requested model, and a refusal was the end of it, so the exemption could never fire. A caller that has run out is now offered the first model in its fallbacks it can still afford. The allowlist still applies — running out of money is not a way to reach a model a key was never allowed — and a model with no fallback is refused exactly as before, or a budget would stop meaning anything. Only a spent budget falls back: a rate limit is answered by waiting a second, and moving the request elsewhere would spend someone else's capacity to avoid a delay nobody asked to avoid. Which uncovered a header that has never worked. x-llmgw-served-model was set in finish, and finish runs after the proxy has written the response — so the one mechanism for telling a client that something else answered has been announcing it to nobody since it was written. Nothing failed, because the only thing asserting it was a comment. It is now set before dispatch, for the capacity fallback as well as this one. Finding that took bisecting a test I had written wrong: it polled for a 429 that never came, because the budget had run out and the fallback had served the request — the feature working, reported as the feature failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c232f41a66 |
Put the fleet view in the dashboard, so nothing else is needed to see it
CI / test (push) Successful in 1m32s
The metrics were exported and the only way to look at them was to stand up Prometheus beside the binary. For a project whose whole shape is one static binary with no dependencies, that is the wrong default: the common case is somebody wanting to know how full their cluster is, and that should not cost a monitoring stack. /admin now shows capacity per GPU class with a usage bar, free whole nodes per fabric — the number that decides whether a replica spanning nodes can land, since free cards on the wrong fabric are not capacity for a model that needs this one — and which replicas are holding hardware nobody is asking for. Nodes gain their memory and how many model files they can see. Taken from the same Summary that ctl status uses rather than derived in the page, because two implementations of the same arithmetic drift apart and the one on the screen is the one people would believe. A faulty card counts as neither used nor free, matching the scheduler. Counting it free is how a cluster looks like it has room it does not, and a page that disagreed with placement about that would be worse than no page. Prometheus stays supported and is now the answer to a narrower question: history, alerting, several clusters at once. The dashboard shows what is true now and keeps nothing, which is the honest division between them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
1ea067d08e |
Give metrics a door that does not also open onto pprof
CI / test (push) Successful in 1m38s
The metrics existed and nothing could record them. They shared a listener with pprof, the management API and the dashboard, and pprof can dump the process heap where API keys sit in plaintext — so a fleet could be monitored or keep its heap private, and not both. That is not a choice anyone should have to make, and it is why every previous answer to "how do I scrape this" was a tunnel. metrics_listen serves metrics and health and nothing else, so it can be reached from wherever Prometheus lives. Metrics stay on the admin listener too: adding a door is not the same as moving one, and anything already scraping keeps working. While there: admin_listen defaulted to :9090, every interface, with a comment beside it saying not to expose it. A default that contradicts its own documentation is a footgun, so it is loopback now and someone who wants otherwise says so. The handler is nil until the registry is built, and says so rather than serving an empty page. Ordering has already caught this exact area once — a collector registered after AdminHandler published nothing at all, silently — and an empty scrape target looks like a working one. deploy/monitoring/prometheus.yml has a starting configuration and the alerts worth having: a node that has stopped reporting, a model that should be serving and is not, a card kept out of placement by a fault, and a fleet holding models nobody is asking for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
9b32531ce3 |
Stop an idle model to make room for one somebody asked for
CI / test (push) Successful in 1m29s
On-demand loading worked right up to the moment the cluster was full and then stopped meaning anything: the model somebody had just asked for could not be placed, and the cards were held by a model nobody had asked for in an hour. That is the only situation in which any of it matters. Deliberately narrow. Only idle replicas of other on-demand models — a model at min 1 is a promise that it stays up, and breaking it to serve a different request would make every guarantee in the catalog conditional on load. Only single-node replicas, since evicting a multi-node one frees several machines and costs tens of minutes to rebuild. One at a time, because freeing everything that qualifies would empty a cluster for a request that needed one node. And never a replica with no reported usage: after a router restart that is all of them, and unknown is not idle. Writing the test found a gap the feature had rather than a bug in the test. Two on-demand models on a cluster with room for one would take turns evicting each other, at a model load per cycle — worse for both than one of them waiting. A replica is now safe from eviction for a few minutes after it starts serving, which bounds that. Which in turn found a latent bug: deriveState stamped ReadyAt with time.Now() inside a package that has an injectable clock, so the new lifetime rule could not be tested at all — the fake clock said three minutes and the wall clock said eight months. Invisible while nothing compared ReadyAt to anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
7fe59bd5fb |
Load a model when someone asks for it, and stop it when nobody does
CI / test (push) Successful in 1m29s
The feature the idle tracking was for. A cluster with more models than GPUs can now hold every model in its catalog and only the wanted ones in its cards: min 0 with a max above it means load on demand, and once nobody has asked for idle_timeout the replicas stop and the cards go back. No new field, because min 0 max 1 already reads correctly — keep none running, allow up to one. Which made min 0 max 0 meaningful for the first time: configured and deliberately not running. normalize() used to fill a zero max in, on the reasonable ground that a model with no maximum cannot run; that would now silently turn every switched-off model into an on-demand one, including the ones already written that way in this cluster's own configuration. A model with no scaling stanza stays off for the same reason — defaulting it to on-demand would let any client request start a 70B nobody intended to run. The one real decision is what happens to the request that triggers a load. Refusing makes the feature useless, since the first request for every model fails; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds holds a connection for ten minutes. So it waits briefly and then says what is happening — 503 with Retry-After and "the model is loading", which is a different answer from "no endpoint serves this" and produces different client behaviour. An unknown model still gets the unknown-model error, or on-demand would turn every typo into "loading" and the useful message would disappear. Use keeps a model alive rather than only the request that started it, without which the timer measures time since the load and a busy model is evicted on schedule however hard it is working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
b619546516 |
Stop trusting hardware facts that were never checked
CI / test (push) Successful in 1m32s
Three places where the system took the infrastructure's word for something and would have failed silently on real hardware. A declared fabric adapter that is not on the node. Fabric membership has to be configuration — OpenSM's default subnet prefix is identical on every fabric, so a matching prefix proves nothing — but taking it entirely on trust was a different mistake. A typo, or a card that enumerated differently after a reboot, produced NCCL_IB_HCA naming an adapter that does not exist. An adapter with no ACTIVE port. Physically linked with no subnet manager sits in INIT: it looks healthy to anything counting adapters, carries no data, survives a reboot, and looks like a software problem. Discovery had recorded port state all along and placement only ever asked whether a fabric had any adapters listed. Both end the same way — the replica starts, passes its health checks, and hangs on its first all-reduce, which is the failure the fabric invariant exists to prevent arriving through a door nobody was watching. The agent now compares declared against discovered, where both are visible for the only time, and does not advertise a fabric that cannot carry traffic. The reason is logged on the machine that knows it, because the operator who typed the adapter name is the only person who can fix it. The fabric's rate is the slowest active link, since a collective runs at the speed of its slowest. And NVLink topology, which no published binary could see. NVML is its only source, the bindings need cgo, and CGO_ENABLED=0 is deliberate so the portable binary runs on an older libc. The cost was silent: tensor groups fall back to the lowest free indices, which is right on an SXM board with NVSwitch and wrong on a PCIe box where bridges pair specific cards — a group straddling two of them runs several times slower and nothing says so. Releases now also publish an nvml build for those nodes, and llmgw_gpu_nvlink_peers makes the difference visible: a flat zero across a fleet means the topology is unknown rather than absent. A test that declared mlx5_0 and mlx5_1 on a node whose sysfs held neither had been passing, because nothing compared the two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
3a4cc0c719 |
Register the fleet collector before the registry is built
CI / test (push) Successful in 1m31s
|
||
|
|
42fd0aa31e |
Export what the cluster is made of, not only what it is serving
CI / test (push) Successful in 1m28s
The router published twenty-two metric families about requests, endpoints, admission and scoring — everything about traffic and nothing about the machines carrying it. There was no way to graph GPU usage, watch a node go unhealthy, see a fabric fill up, or answer "was the cluster full at the time", which is the first question anyone asks after an incident. The agents exported nothing at all. Exported by the control plane rather than by each agent. It already holds every number: hardware arrives at registration and state on each heartbeat, so this is a map read away. One scrape target for a fleet of hundreds, no Prometheus configuration per node, no firewall rule per machine, and nothing new running on the nodes. On the same /metrics as the traffic figures, because correlating "the cluster was full" with "requests were queueing" should not need two systems. A collector rather than gauges kept current by a ticker: the numbers are derived from state that already exists, so computing them when asked is simpler than maintaining a copy that can drift, and nothing runs between scrapes. GPU utilisation is new, and is the number people actually watch. Memory used says a model is loaded; this says whether it is doing anything, and a fleet at ninety percent occupancy and two percent utilisation is a fleet holding models nobody is asking for. Adding it caught two things. utilization.gpu goes last in the nvidia-smi query and is optional, because a driver too old to know it returns a shorter row — requiring it made an llmgw upgrade silently stop discovering GPUs on an older node, which is far worse than not knowing how busy a card is. And I gave the new protobuf field number 14, which ecc_errors already had; buf lint is why that was a moment rather than a corrupted wire format. Absent is kept distinct from zero throughout, for the same reason as everywhere else here: an unreported utilisation shown as a flat zero reads as an idle fleet rather than a missing measurement, and a replica nothing has reported would be the most idle thing in the cluster. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
943c23a412 |
Fetch a model's weights when the simple arrangement needs it
CI / test (push) Successful in 1m26s
The default arrangement is the simple one: every node keeps its own copy. So a source naming a remote file is now fetched before the engine is launched, rather than left to fail as a missing path minutes into a load. Eight shards of a tensor-parallel replica start at the same instant and want the same file; they produce one download. An interrupted fetch resumes instead of starting over, which matters at forty gigabytes, and the download is written beside its target and renamed only when complete — a partial file under the model's name would be found by the next scan, reported as present, and handed to an engine that fails on it. The exotic arrangements need no code, which is the point. Weights on a Lustre mount or an image exported over the fabric are simply present, and present means nothing is fetched. The escape hatch from a distribution mechanism is a distribution mechanism that checks first. Wiring it up caught a regression in my own design. A bare model id — "Qwen/Qwen2.5-7B-Instruct" — is a documented source that vLLM and SGLang resolve themselves, and treating every non-remote source as a local path made every existing vLLM profile fail. Only an absolute or explicitly relative path is now checked as a file; anything else is passed through untouched. Three agent tests failed on the same change for a better reason: they used "/models/x", and the new check is exactly the one that catches that. Whole repositories of shards are deliberately not fetched here. vLLM and SGLang do that well already, and a second snapshot downloader would be work spent arriving back where we started. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c13c567cc5 |
Know where the weights are, without owning the storage
CI / test (push) Successful in 1m32s
Storage will be different at every installation — Lustre, NFS over RDMA, an image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes it rather than managing it. An agent told which directories to look in reports what it can see; how the files got there is nobody's business here, and a node told nothing reports nothing and behaves exactly as every node did before this existed. That fact buys two things. Placement prefers a node that already holds the model over one that must first move tens of gigabytes, as a preference and never a filter: fetching them is a valid answer, and refusing to place would turn a slow start into no start at all. And ctl status shows what each fabric holds. Per fabric because fabrics are not connected to one another. Weights in ib-a are not weights in ib-b — the only path between them is the management network — so a model seeded into one fabric leaves the rest of the cluster unable to serve it without a slow copy, and nothing said so before now. One shared filer counts the same as twenty local copies, because "can this fabric serve it" is the question being asked. Three distinctions the tests hold to, because each is a way to get this quietly wrong. A node that was never asked to look must rank with one that looked and has nothing, or enabling the feature on a single node would drain every other. An empty directory is not weights, because a half-finished download looks exactly like one and would send placement to a node whose engine then fails to load. And local storage is never mistaken for shared: a filer read as local costs an unnecessary copy, while local read as shared would have the cluster believe a model is everywhere when it is on one node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
bd134a7ab9 |
Make the planned entry use an engine that can serve the precision
CI / test (push) Successful in 1m39s
Reading the tool's own output against the real fleet: it chose q8_0 as the best fit and emitted a vLLM profile with no quantisation. q8_0 is a GGUF quantisation that only llama.cpp serves, and from a file rather than a repository — so following that entry loads full-precision weights, which the same output had just listed as not fitting one card. The planner would have caused the out-of-memory it exists to prevent, from a recommendation that looked right. Choosing a precision and choosing an engine turn out to be one decision, so a Precision now carries the engine that serves it, the quantisation flag where one is needed, and whether it lives in a file. A GGUF fit prints a path and says to download it; a vLLM fit prints the repository and passes --quantization so the engine does not quietly load bf16. The arithmetic was right and the recommendation was wrong, which is the failure mode worth having tests for: everything reported a number, and the numbers agreed with each other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
88a5949f7c |
Stop offering the same fit once per fabric
CI / test (push) Has been cancelled
Running the planner against the real fleet produced twenty-six rows for a 32B, of which thirteen were duplicates: two fabrics of identical cards, and every single-node fit listed once for each. A replica on one node does not span anything, so which fabric its node happens to be attached to changes nothing about whether the model fits. A multi-node fit is still listed per fabric, because there the fabric is the constraint and which one has room is the thing the operator needs told. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
300e298053 |
Work out whether a model fits the cluster, and how to split it
CI / test (push) Has been cancelled
llmgw ctl model plan Qwen/Qwen2.5-Coder-32B-Instruct The arithmetic this replaces is the last thing between having GPUs and serving a model on them. Choosing tp, gpu_class and a context that will actually fit means knowing the layer count, the KV head count, what the weights weigh at a chosen quantisation and how much of a card the runtime keeps — and getting it wrong is not an error message, it is an out-of-memory twenty minutes into a load. I computed those by hand for every model on this cluster. Both halves are read rather than retyped: the architecture from the model's own config.json and parameter count, the hardware from the running fleet. Attention heads in place of KV heads is the classic slip and is worth a factor of five on a model with grouped-query attention, so it is not a number anyone should be entering. Every candidate is listed, not one answer. With cards to spare you want the full-precision fit; with one card free you want to know a quantised one exists. Choosing between those is not the planner's job. The tests are calibrated against files actually downloaded here — llama.cpp reported 14.62 GiB and 8.37 GiB for the 14B at q8_0 and q4_K_M, and the model reproduces both — and they caught two errors in my own arithmetic. The overhead was charged per rank from the whole model's weights rather than each rank's share, and it reserved a tenth of a card that usableFraction had already reserved. Between them, Llama-3.3-70B at bf16 across two 80 GB cards came out as not fitting, which is a configuration people run every day. Slack reserved in two places, each unaware of the other, reads as caution and behaves as a bug. Pools are grouped by class and fabric together, because a replica may never span two fabrics: eight free cards split across two of them are not eight cards for a model that needs eight, and merging them would plan something that starts, passes its health checks and hangs on its first all-reduce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
d07dddb1bb |
Say plainly whether a model is serving, and why not
CI / test (push) Successful in 1m28s
Reported after watching a model appear in ctl models before it could answer anything. Checked: clients are not told early — /v1/models listed it only once a replica was ready, and the chat picker reads the same source. The operator view showing it is deliberate, because a model coming up is exactly what someone running the cluster wants to see. But the report pointed at something real. That view distinguished "a replica is coming up" from "nothing could be placed" by a parenthetical — "0/1 ready (1 total)" against "0/1 ready" — and those need opposite responses: one means wait, the other means look at explain-placement. On four models the difference is findable. On forty it is invisible, and a model that will never start reads exactly like one that is thirty seconds away. SERVING now answers the question that was being asked of the replica counts — can a client use this right now — and STATUS says which of ready, starting, not placed or stopped is the case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |