4ba84fa90e02aa7e470edeb43da0ad55ad3dcd99
127
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ilker ManapandClaude Opus 5
|
4ba84fa90e |
Let an agent find the leader instead of being told one address
CI / test (push) Successful in 1m32s
The second half of leadership. A standby that refuses to place but still holds an agent's stream is worse than useless: commands travel down the stream the agent opened, so the leader's placements fail with "no agent connected" for a node that is plainly connected. So a standby refuses the stream, before reading the opening message, and says why. And an agent takes a list rather than an address, trying each in turn — a standby refusing, a stopped control plane refusing the connection and a crashed one never answering are the same thing from a node, and the answer to all three is somewhere else. `server` and `servers` are both accepted, with `server` first when both are given: every agent configuration written before there could be a second control plane names one address, and an operator adding a standby expects the one already working to stay preferred. With a single address the behaviour is exactly what it was, including the backoff — which is skipped only while there is another address that has not just been tried, because the delay exists to stop hammering one control plane and the next one has not been hammered. Removed a check that asked whether one field was empty, which is how the first run of these tests failed: the list was populated and `server` was not, and an agent configured entirely correctly was refused by its own validation. The tests run a real agent against a real service over h2c. The first attempt used a plain HTTP server, and the refusal arrived as an HTTP/2 framing error that said nothing about why — which is exactly what an operator would have seen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ade689fc8e |
Refuse to run the destructive store tests against a real database
CI / test (push) Successful in 1m47s
These TRUNCATE every table before each test. Pointed at a deployment that is a silent and complete data loss — every key, every user, every conversation. Not hypothetical. I did it: an SSH tunnel to the production database had been opened minutes earlier for a perfectly good reason, and the same DSN was then handed to the test suite. Twenty-one users, nine API keys, eighteen sessions, four hundred and sixty-one usage records and three runtime-added models, gone between one command and the next. Recovered only because the pre-migration SQLite file was still there. The guard is the database name. A test database has to say so, because a check that could be satisfied by accident would not have stopped this one — the DSN was entirely correct for what it had been opened for, and catastrophic for what it was then used for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
5b5e61cb97 |
Stop a second control plane from placing work behind the first's back
CI / test (push) Successful in 1m35s
The first thing HA needs that a shared store does not give. Two control planes against one store both reconcile: both see a model wanting two replicas and none running, and both place. The model ends up with twice what it asked for, each replica holding GPUs the other's inventory believes are free — and nothing crashes to say so. A lease in a table rather than a lock held by a connection. PostgreSQL has advisory locks and SQLite has nothing like them, and this store keeps one set of queries for both; a row with an expiry works everywhere, and it survives a leader being powered off rather than asked to stop, which is the case that matters. Taking and renewing are the same statement, so the two cannot disagree about what a held lease looks like, and it is one statement, so two processes racing cannot both believe they won. The owner id is random rather than the hostname. A hostname would let a restarted process match its own stale lease and believe it had been leading all along — including through the gap that made it restart, during which somebody else may have taken over and started placing. A store that cannot be reached is not evidence that somebody else is leading, so an unreachable store keeps the claim. Two leaders is worse than one, but no leader means nothing is placed and nothing is evicted, and a brief database blip should not stop a cluster healing itself. Unset means leader. Every deployment that existed before this, every single-control-plane install and every test acts unconditionally, with nothing to configure. The same seven assertions run against both backends, which is how the test harness was caught inheriting a lease between subtests — it passed on SQLite, where each subtest gets a fresh file, and failed only on PostgreSQL. What this does not yet do is move agents. A standby holding an agent's stream is one the leader cannot command, so failover is degraded rather than clean until agents know more than one address. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
fa60b81f37 |
Document moving to PostgreSQL, and when it is worth it
CI / test (push) Successful in 1m36s
The section leads with why SQLite is still the default rather than with how to leave it: one file, no server, and a request path that barely touches it. Two things change that, and both are recent — conversations and notes read and write on every turn, and a second control plane is impossible while the store is a file on one machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b6de469351 |
Move a store to another one without reissuing every key
CI / test (push) Has been cancelled
Keys are stored hashed, so a migration that drops them cannot recreate them: every person and every script needs a new one issued, and the old ones cannot even be looked up to say which they were. `llmgw keys move-store` copies instead. Found by running it against the real cluster, twice, after the gateway had already been stopped. The first order came from the order the migrations create tables in, and users is created in a later migration than api_keys even though api_keys references it — so the copy died three tables in. The order is now checked by a test that derives the foreign keys from the migrations rather than restating them, so a new reference fails there instead of during somebody's migration. Checked by putting the wrong order back and watching it fail. Then key_claims. Its sealed column is emptied the moment a claim is used, so every claimed row holds an empty blob in a NOT NULL column. An empty blob scans as a typed nil slice and a genuine NULL as an untyped one — a distinction this side can see and pgx cannot, since it encodes any nil slice as NULL. Restored before the insert, which is the difference between a copy that completes and one that stops four tables in. One transaction per table rather than one for the whole copy: a single transaction holding every row of a large usage table is how a migration runs out of memory on the machine least able to spare it. A failure therefore leaves a partial copy, which is why the destination must have been empty — starting again is dropping and recreating it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
a76ebdb79a |
Run the store on PostgreSQL, chosen by what the address looks like
CI / test (push) Successful in 3m11s
A path is still SQLite and a postgres:// URL is PostgreSQL, so every existing configuration keeps working untouched and moving is one line changed rather than a new setting to find. An address of the form @/path/to/file reads the real one out of that file, which is how a DSN carrying a password stays out of a configuration that gets copied, diffed and pasted into issues. The driver is pgx, which is pure Go. That is not a preference: the whole binary is built with CGO_ENABLED=0 so it runs on a node whose libc is older than the builder's, and a cgo driver would take that away from every deployment, including the ones still on SQLite. Tested against a real PostgreSQL, which found two things nothing else could. The schema_migrations table is created in Go rather than in a migration file, so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL, and a millisecond timestamp does not fit, so the very first migration failed to record that it had run. And the memory migration uses an FTS5 virtual table with three triggers keeping it in step, which has no counterpart: PostgreSQL derives a tsvector from the row, so there is nothing to keep in step and no trigger to forget. That leaves exactly one query that is not the same sentence in both dialects — the note search, MATCH against @@ — and it is written out twice rather than hidden behind a shared spelling that would mean something subtly different in each. Everything else is one set of queries. The text search uses the 'simple' configuration rather than English. These are an agent's own notes, in whatever language it chose, and stemming English rules over Turkish would quietly stop finding things. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ccc9aa8b76 |
Route every query through a dialect, before there is a second one
CI / test (push) Successful in 2m30s
Groundwork for PostgreSQL, with no PostgreSQL in it yet: the store still opens SQLite and behaves exactly as it did. One set of queries rather than one per database. The SQL here is almost entirely portable — ON CONFLICT upserts work in both, and the only construct that genuinely differs is the placeholder — so two copies of every query would buy nothing and drift, one query at a time, discovered in production. The rewrite is a wrapper around the handle rather than a change at each call site. There are sixty of those and sixty chances to forget one; this way the only route to the driver goes through the rewrite. Making it a wrapper found six places that reached past it, including all three manual transactions and two prepared statements — every one of which would have worked on SQLite and failed on PostgreSQL, on a path nobody exercises before deploying. The rewriter skips quoted sections. No query here contains a question mark in a literal today; this is so that adding one is a test failure rather than a statement that silently binds the wrong argument. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
539bdac152 |
Say where the fetch progress actually goes
CI / test (push) Successful in 2m38s
The gap list said a fetch "reports no progress". It does — the agent logs a percentage as it downloads, with a comment saying a replica pulling a hundred gigabytes and one that is wedged look identical from the control plane. That comment describes the real gap, which is narrower and more useful than the one written down: the progress exists, it just never leaves the node, so the operator looking at ctl or the dashboard still cannot tell those two apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
7dd1de6537 |
Cut v0.7.0
Seventeen commits since v0.6.0, and not a phase of the roadmap — which is why the table leaves the phase column empty rather than inventing one. What landed is what a cluster needs to be lived with rather than demonstrated: somewhere to keep a conversation, somewhere to keep a note, and an honest answer about the disk underneath both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>v0.7.0 |
||
 ilker
|
c815b629e2 |
Merge pull request 'agenta memory eklendi' (#1) from deniz/llmgw:memory into main
CI / test (push) Successful in 2m30s
Reviewed-on: #1 |
||
 deniz
|
2eb79f3400 |
agenta memory eklendi
CI / test (pull_request) Successful in 2m31s
|
||
|
Ilker ManapandClaude Opus 5
|
45e913bd10 |
Refresh disk and weights while the agent runs, not once when it starts
CI / test (push) Successful in 1m46s
Found by answering "where can we see this live". `ctl weights` was offering a file for deletion that I had deleted two days earlier, and the node's disk figure was the one it had when its agent started. Discover() is called at registration and nowhere else, so both were frozen. Disk is a fact about a moment, not about a machine. Sampled once, the figure is wrong on exactly the node worth watching — the one busy filling up — and it stays wrong until that agent restarts. The heartbeat already refreshes GPU memory for the same reason, with a comment about noticing a foreign process "after registration"; disk belonged beside it and I did not put it there. The stale list cut both ways. A model fetched after registration was invisible until a restart, so placement could not prefer the node that had just spent ten minutes downloading it, and the disk check could not see the space it used. Storage goes in every heartbeat, being a few numbers per filesystem. The file list goes only when a fingerprint of paths and sizes changes, with a flag saying the list is meant — without that, an empty list would be ambiguous between "this node now holds nothing" and "nothing to report", and the second reading would empty a node's inventory five seconds after it registered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
cf20570ff1 |
Correct what the status section says is missing
CI / test (push) Successful in 1m39s
Two of the four items were already built. A model falls through to its fallbacks on no capacity as well as on a spent budget, and the per-request routing profile header exists — it is ignored under authentication on purpose, because a header that picks a routing profile is a header that promotes a client into the top priority band. Checked against the code rather than against memory, which is how both got onto the list in the first place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
21a502035f |
Document conversations, starting with what is not kept
CI / test (push) Successful in 1m35s
The section opens on the default rather than on the feature, because the default is the part someone deploying this needs to be sure of: nothing anyone types is stored until a project asks, and the command to ask is an act rather than a setting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
eaf3382a6d |
Give an operator a way to turn conversation storage on
CI / test (push) Successful in 1m44s
SetThreadRetention existed and nothing called it, so the feature was unreachable: a project could never opt in and every request carrying a thread was refused. A command rather than a configuration key, because it is a decision about data rather than about deployment. Someone has to choose to keep what a team types, and choosing it should look like an act rather than like a default nobody read. --retain-hours is required. Without a number the command would create exactly the thing the design avoids: a conversation store with no expiry, holding everything a team ever typed because nobody chose one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
d26a30471e |
Let people forget a conversation, and make retention actually expire things
CI / test (push) Successful in 1m38s
Two halves of keeping the promise that storing makes.
DELETE /v1/threads/{id} is a client endpoint, not an admin one: the person who
said the thing should be able to unsay it, and making them ask an operator
would mean the operator reads it first. Deleting a thread in another project
answers exactly as deleting one that never existed does — two answers would
let a caller map which ids are real in projects they cannot read, and this one
would do it by trying to destroy them.
And something now runs the pruner. Without it "we keep conversations for 24
hours" was a comment rather than a promise: the rows would accumulate and the
number a project chose would describe nothing. Hourly, and once at startup,
because a gateway that has been down for a week should not wait another hour
before honouring a window that expired while it was off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Ilker ManapandClaude Opus 5
|
9eb321ae3e |
Let a conversation continue on a different model than it started on
CI / test (push) Has been cancelled
The request path half. A caller sets x-llmgw-thread and sends only the new turn; llmgw puts the stored history in front of it, sends that to whichever model is serving, and stores the exchange afterwards. A header rather than a body field because some SDKs drop fields they do not know and all of them let a caller set a header. Capturing the reply is opt-in and bounded. This is the hottest path in the system, so a request without a thread accumulates nothing at all, and one with a thread stops accumulating at a cap rather than growing to it — a gateway that kept every reply in memory would be one long answer away from trouble. A reply is truncated on a rune boundary rather than dropped: a shortened turn replays as a shortened turn, while nothing at all replays as the assistant having said nothing, and invalid UTF-8 would break the next turn's body for good rather than for one turn. The streamed path sets the captured reply on every exit, not only the one that looks like the end. A stream can also finish with the client gone or the engine stalled, and what the assistant said up to that point is still what it said. A thread on a project that keeps no conversations is refused. The e2e test asserted that and got a cheerful 200: the request was served, nothing was stored, and the caller would have discovered it only when the model had no idea what they were referring to — which reads as the model being broken. It now asks whether storage is on before doing anything else. History is spliced after the model is finally settled, because a budget fallback may have changed it and trimming is measured against the context of the model that will actually serve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
cebf9906d2 |
Splice a stored conversation into a request, trimming only what must go
CI / test (push) Successful in 1m39s
The part that makes a thread work across models: the history goes in front of what the caller just said, because a client using a thread sends the new turn rather than the conversation again. Trimming has three rules, and each is there because the alternative fails quietly. The oldest stored turns go first, so a conversation that outgrew the model now serving it stays usable — after a budget fallback that model was not the caller's choice, and a permanently unusable thread would be a strange way to punish them for running out of budget. The caller's own messages are never dropped: answering something they did not ask is worse than refusing. And a system message is kept wherever it sits, because losing one silently changes what the model is for while it carries on answering. A model that declares no context length has nothing trimmed. Guessing a limit for it would drop turns for no reason and the caller would never learn why the model forgot. The body is rebuilt through a map so every field the client sent survives. Forwarding fields llmgw does not understand is how engine-specific options reach an engine at all, and a thread must not be what starts dropping them. An empty history returns the body byte for byte, so a request without a thread takes exactly the path it took before this existed. Not yet wired to the request path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b0592670a4 |
Give projects somewhere to keep a conversation, off unless they ask
CI / test (push) Successful in 1m39s
The storage half of continuing a thread on a different model than it started on. Nothing uses it yet. Until now llmgw stored no message content anywhere: usage rows carry token counts, costs and latencies, never text, and even its own chat page keeps the conversation in a JavaScript variable that a refresh discards. That property is worth more than the feature, so it is the default and stays the default. A project stores nothing until someone turns it on for that project. Retention is a number of hours rather than a boolean, because a thread store with no expiry is a transcript of everything a team ever typed, kept forever because nobody chose a number. Turning it on means saying for how long. The window measures idleness rather than age — expiring a conversation someone is still having would be a strange way to keep that promise. Threads are scoped to a project and the scope is part of every lookup, not a check afterwards. Reading a thread from another project and reading one that does not exist return the same error: two errors would let a caller learn which ids are real in projects they cannot read. Writing across projects is refused for the same reason in the other direction — append, then read your own reply back. Deleting a thread takes its messages with it, which the test checks by counting rows rather than by trusting the cascade. Someone has to be able to say "forget that", and a retention window is no answer to a message sent by mistake. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c9c841387a |
Check the prompt fits the model that will actually serve it
CI / test (push) Successful in 1m40s
The context check ran against the model the caller asked for. The budget fallback then swapped in a different one and nothing looked again, so a prompt sized for a 32k model could be sent to a 512-token one. Measured before the fix: 10002 prompt tokens delivered to a model declaring 512. The caller sees that as a truncated reply or an engine error naming a model they never chose, which is a worse outcome than the refusal the fallback exists to avoid — it fails later and explains less. Candidates that cannot hold the prompt are now skipped, and a fallback further down the list that can hold it is used instead. When none fit, the request is refused as it was before the fallback existed. The history is deliberately not trimmed to make it fit. Dropping a caller's messages to stay under a limit they never asked about would be a silent loss, inside a request that was already being quietly moved to another model. The tests drive the real request path, because a unit test on the fallback would have asserted the same wrong thing the code did. Both were checked against the unfixed code and fail there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1b7e11b91e |
Document reclaiming disk, and what it deliberately will not reclaim
CI / test (push) Successful in 1m49s
A command that deletes data needs its limits written down more than its syntax. The section says plainly that llmgw only takes back files it fetched itself, and then says the uncomfortable half out loud: that this means it will not reclaim the storage most likely to be filling your disk, because weights you placed yourself are yours to remove. Also records the dash in the DISK column as "not asked" rather than "no space", which is the reading that matters when a node will not schedule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
e6bc7859dc |
Carry the delete capability through the indirection that hid it
CI / test (push) Successful in 1m36s
`ctl weights --collect` answered "this control plane cannot ask nodes to delete files" against a real cluster, with the whole suite green. The controller is handed a dispatcherRef rather than the agent service — an indirection that exists to break a construction cycle, and one that satisfies the dispatcher interface by forwarding each method by hand. The controller reaches past that interface by type assertion to find capabilities beyond scheduling, and an assertion against dispatcherRef fails for anything nobody remembered to forward. Nothing could have caught it below the wiring. The dispatchers the tests use do not implement the capability either, so its absence there is the expected case and looks identical to this bug. The test therefore asserts on the production types themselves: that the forwarding dispatcher carries the capability, and that the service it forwards to has one to carry. Checked by deleting the method and watching it fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
774e134ed1 |
Refuse a node with no room, and take back only what llmgw put there
CI / test (push) Successful in 1m39s
Two halves of the same problem: nothing stopped a model being placed on a node that could not hold it, and nothing ever removed the weights of a model that had been deleted from the catalog. Placement now has a disk check, shaped exactly like the memory one it sits beside, with one addition: it is skipped on a node that already holds the weights. A node holding the file needs no room for it, and without that the node best placed to run a model — because it already has it — is the one most likely to be rejected, since holding the file is what used the space. Reclaiming is the dangerous half, so the whole design is about what it will not do. llmgw deletes only files it fetched itself, and that is a calculation rather than a heuristic: the cache name is a hash of the source, so the set of wanted files can be computed from the catalog exactly. Anything without that name was put there by an operator — scp, a shared mount, a copy by hand — and is none of this system's business however unreferenced it looks. A model that took a week to copy must not be removable by a command someone ran to tidy up. The node checks every path again before unlinking it, and that duplication is the point. The control plane knows the catalog; the node knows its disk. A control plane that is wrong, newer than the agent, or tampered with must not be able to talk a node into deleting something, so the agent refuses anything outside its own cache directory or without the cache name — including a path that tries to traverse out of it. Listing is the default and deleting takes --collect. The summary says "asked to remove", because each node decides, and one that refuses a path it cannot prove it fetched is this working rather than failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
753f4e3265 |
Let a node say how much room it has left
CI / test (push) Failing after 1m31s
Disk was the one resource nothing reported. `statfs` was already being called on every model directory, but only to ask whether the filesystem was a network one — the free space it returns in the same struct was thrown away. So the only way to find out why a node was filling up was to ssh to it, which is what this system exists to avoid. Reported per filesystem rather than per configured directory. Free space belongs to the disk, not the path: two model directories on one disk are one pool of bytes, and counting them separately would let the control plane add 24 GB to 24 GB and place a 40 GB model on a node that has 24. Shared filesystems are summed separately from local ones and left out of a node's own total. Space on a filer belongs to everyone mounting it, so twenty nodes would otherwise each report the same free terabyte, and a scheduler would place twenty models into room for one. Nothing reported means nothing is known, everywhere: an agent too old to send storage and a node with no model directories both send nothing, and both must read as unknown rather than as a full disk. `ctl nodes` and the dashboard show a dash, the metric is absent rather than zero, and a missing directory — which is what an unmounted filer looks like — is skipped rather than reported as having no space. Shown as free out of total, because the first is useless without the second: 8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root partition, and only the second stops a model being placed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b5177b70f6 |
Stop tool-check blaming the engine for what the model did
CI / test (push) Successful in 1m37s
It had one verdict for a failed attempt — "This is the engine, not the model. Choosing a bigger model will not help" — and that verdict was wrong, in a way that sends someone off rebuilding llama.cpp when the fix is a one-line catalog change. Measured on the cluster. The engine renders the tool prompt correctly: the <tools> block and the instruction to wrap the call in <tool_call> are both in the prompt, which I confirmed through the engine's own /apply-template. The coder models simply do not use that wrapper — qwen2.5-coder-3b puts the call in a markdown fence, qwen2.5-coder-7b invents <function-call> — so the parser, which looks for <tool_call>, has nothing to find and leaves it in the content. qwen2.5-3b-instruct, same engine, same build, same flags, returns a populated tool_calls array. So it was the model all along, and the advice against trying another one was the most expensive sentence in the output. A failed attempt is now two verdicts, told apart by whether the reply used the wrapper its template asked for. Wrapped and unparsed is genuinely the engine's parser. Unwrapped is the model, and the fix is one trained for the format rather than a bigger one from the same family. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b58d67602d |
Stop the install guide contradicting itself about weights
CI / test (push) Successful in 1m41s
It said "there is no distribution mechanism yet" and then, two paragraphs later, documented the distribution mechanism. The known-gaps list — the part someone reads at 2am — still said "no cache, no fetch, no progress". Fetching landed in this cycle; both sentences were written before it and left standing. node-onboarding.md carried the same claim. What is actually still missing is narrower and worth naming precisely: a fetch reports no progress, so a first start on a large model looks like a hang, and every node fetches its own copy, so eight nodes on a new model are eight downloads from one origin. Also: the dashboard now shows who is using the cluster and what they are running, which the sentence describing it did not mention; and one sentence about idle timeouts had lost the words that made it a sentence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
7de8bacfc0 |
Stop reading JSON with a greedy regex, and stop set -e hiding the empty case
My first fix was wrong about why the release could not be created. Gitea materialises a release row for every pushed tag, so the lookup should have found one; the create that followed hit a release that already existed. The lookup failed because of how it read the id. `sed 's/.*"id":\([0-9]*\).*/\1/'` is greedy, so `.*"id":` matches the last occurrence in the document, not the first — "id" appears again inside the nested author object — and it does not match `"id": 13` with a space after the colon at all. Checked against this instance: for v0.5.0, whose id is 12, the old pattern returns 1. It now greps the first "id", tolerates the whitespace, and re-reads the release after creating rather than parsing the create response, so a 409 resolves to the release that was already there instead of to an error. `set -e` was hiding the empty case: a command substitution in an assignment fails the script, and grep exits non-zero when it matches nothing, so the "no release yet" branch could never be reached — the job would simply stop. Both extractions are guarded, and both paths are exercised. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>v0.6.0 |
||
|
Ilker ManapandClaude Opus 5
|
4815a50c94 |
Ask for the scope creating a release actually needs, and say when it is refused
CI / test (push) Has been cancelled
The first v0.6.0 run built every binary, including the NVML one, and then stopped at "could not find or create the release". Two things were wrong. `releases: write` is not enough to create one. Creating a release can create its tag, which is a code-unit operation; `contents` is the scope that covers code and releases together. And the step threw away the answer. It piped the response through sed looking for an id and, finding none, reported only that it had failed — so a refusal, a malformed request and an unreachable server all produced the same line. It now prints the status code and the body, which is how the above was diagnosed at all: by reading a log, not by knowing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
a2b3f63a0d |
Say what v0.6.0 is, and let the release actually publish it
The changelog had two "Unreleased" headings. The second sat below v0.5.0 and was the detail behind it — phases 2 and 3, written while they were in progress and never folded in when the release was cut — so it claimed as unreleased work that had been shipped for two days. It is now a section of v0.5.0. The status table said v0.5.0 was planned, which its own release notes disprove. The first "Unreleased" becomes v0.6.0: eighty commits, of which phase 4 is the headline and the rest is what a cluster needs before anyone but its operator is let near it. Two things would have stopped the release workflow publishing anything: Gitea gives every job a built-in GITEA_TOKEN, but what it may do depends on the instance's default mode, and under the restricted default it is read-only on releases. The job now asks for `releases: write`, which is correct under either default. An upload that is refused now fails the job — without `-f`, curl treats a 403 body as a successful transfer, so the release would end up with some of its files, or none, and a green tick either way. `workflow_dispatch` is offered but the upload assumed a tag. Fired from a branch it would have created a release named "main". README's status paragraph still said authentication, budgets and the control plane were missing. It now says what is actually left, including the part worth saying out loud: the InfiniBand, NVML and NCCL paths have never run on a machine that has any of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
0cac9d6986 |
Make the landing page look like something worth having a key for
CI / test (push) Successful in 1m30s
It is the only page a stranger sees, and until they have a key it is the whole product: a stack of headings and shell snippets said "internal tool you probably should not be here". Now it opens with what this is and a diagram of why it exists — one address, many GPUs, fabrics that cannot see each other — then the base URL as a headline rather than a line of prose with a colon in it, the route to a key as three cards, and the client examples as tabs instead of a pile. The tabs are radio buttons and a stylesheet. No JavaScript decides what is on the page, so someone who arrives with scripting off still gets every example; the only script is the copy buttons, which exist only where the clipboard API does and so cannot leave a dead control behind. The page says exactly what it said before, including that it lists no models and why. The diagram names no model, no node and no count: fabrics are drawn because llmgw keeps them apart, not because this install has two. Tests cover the parts that are easy to break later: every example lives in the page itself rather than being fetched per tab, the reflected base URL — now in five places rather than one — still cannot write markup, and the diagram carries a label for anyone not looking at it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
f8bb78103f |
Stop the ranked charts from spilling over the tables below them
CI / test (push) Successful in 1m32s
The bars were wrapped in <div class="chart">, and .chart was already the fixed-height box the time-series charts are drawn in: 120px, with a border and a background. A chart that grows a row at a time does not fit in a cage, so the SVG overflowed it and landed on the table underneath. Renamed to .ranked, which is a wrapper rather than the mark itself, and capped at the design width so a wide window scales the box instead of inflating the 12px labels. The test reads the class back out of the markup and checks the stylesheet does not cap it, because the collision was invisible in both files on their own — each rule was reasonable, and only the shared name was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
eb9e9b348c |
Show who is using the cluster and what they are running
CI / test (push) Successful in 1m28s
The dashboard could say what the cluster was made of and not who was spending it. Usage was per key, which is the right unit for revoking something and the wrong one for the question actually being asked: one person holds a laptop key, a CI key and a notebook key, and adding three rows up by eye is a chore that gets skipped. Per person now, joined through the keys they own, and per model — counted by the model that answered rather than the one requested, since those differ whenever a fallback serves and what actually ran is the question. Headline figures are stat tiles with a sparkline: a single current value is a number, and plotting one is how a chart ends up with a single bar in it. The sparkline says only rising, falling or flat, which is the one thing the number cannot. Ranked horizontal bars, because the labels are email addresses and model names and the label is the point — the question is which person, not what shape the distribution has. Rotated or truncated axis labels say the chart matters more than reading it. Trimmed where drawn and complete on hover, so the plot keeps its geometry and a reader can still find out whose row it is. The palette is validated rather than chosen: every adjacent pair clears the colour-vision separation threshold, and the dark values are the same hues re-stepped for the dark surface instead of an automatic flip. Run through a validator, because that is the only way anyone can honestly claim a palette is colourblind-safe. Which turned up a bug in the existing chart code. cssVar is an allowlist of four names that silently returns --accent for anything else, and callers wrap it in var() themselves — so every new series would have rendered in one colour with accent-coloured text. Caught by a test written from the rule that chart text wears text tokens, never the series colour. A key with no owner is labelled "operator-issued keys" rather than unknown: it predates accounts and is doing its job, and "unknown" would imply something is wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
28967edab3 |
Serve a spent budget something it can afford
CI / test (push) Successful in 1m38s
The accounting layer has always exempted a model that costs nothing, with a comment saying it is "what lets a key with no budget left still be served a cheap fallback". That described a caller nobody had written: the budget was checked once, against the requested model, and a refusal was the end of it, so the exemption could never fire. A caller that has run out is now offered the first model in its fallbacks it can still afford. The allowlist still applies — running out of money is not a way to reach a model a key was never allowed — and a model with no fallback is refused exactly as before, or a budget would stop meaning anything. Only a spent budget falls back: a rate limit is answered by waiting a second, and moving the request elsewhere would spend someone else's capacity to avoid a delay nobody asked to avoid. Which uncovered a header that has never worked. x-llmgw-served-model was set in finish, and finish runs after the proxy has written the response — so the one mechanism for telling a client that something else answered has been announcing it to nobody since it was written. Nothing failed, because the only thing asserting it was a comment. It is now set before dispatch, for the capacity fallback as well as this one. Finding that took bisecting a test I had written wrong: it polled for a 429 that never came, because the budget had run out and the fallback had served the request — the feature working, reported as the feature failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c232f41a66 |
Put the fleet view in the dashboard, so nothing else is needed to see it
CI / test (push) Successful in 1m32s
The metrics were exported and the only way to look at them was to stand up Prometheus beside the binary. For a project whose whole shape is one static binary with no dependencies, that is the wrong default: the common case is somebody wanting to know how full their cluster is, and that should not cost a monitoring stack. /admin now shows capacity per GPU class with a usage bar, free whole nodes per fabric — the number that decides whether a replica spanning nodes can land, since free cards on the wrong fabric are not capacity for a model that needs this one — and which replicas are holding hardware nobody is asking for. Nodes gain their memory and how many model files they can see. Taken from the same Summary that ctl status uses rather than derived in the page, because two implementations of the same arithmetic drift apart and the one on the screen is the one people would believe. A faulty card counts as neither used nor free, matching the scheduler. Counting it free is how a cluster looks like it has room it does not, and a page that disagreed with placement about that would be worse than no page. Prometheus stays supported and is now the answer to a narrower question: history, alerting, several clusters at once. The dashboard shows what is true now and keeps nothing, which is the honest division between them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1ea067d08e |
Give metrics a door that does not also open onto pprof
CI / test (push) Successful in 1m38s
The metrics existed and nothing could record them. They shared a listener with pprof, the management API and the dashboard, and pprof can dump the process heap where API keys sit in plaintext — so a fleet could be monitored or keep its heap private, and not both. That is not a choice anyone should have to make, and it is why every previous answer to "how do I scrape this" was a tunnel. metrics_listen serves metrics and health and nothing else, so it can be reached from wherever Prometheus lives. Metrics stay on the admin listener too: adding a door is not the same as moving one, and anything already scraping keeps working. While there: admin_listen defaulted to :9090, every interface, with a comment beside it saying not to expose it. A default that contradicts its own documentation is a footgun, so it is loopback now and someone who wants otherwise says so. The handler is nil until the registry is built, and says so rather than serving an empty page. Ordering has already caught this exact area once — a collector registered after AdminHandler published nothing at all, silently — and an empty scrape target looks like a working one. deploy/monitoring/prometheus.yml has a starting configuration and the alerts worth having: a node that has stopped reporting, a model that should be serving and is not, a card kept out of placement by a fault, and a fleet holding models nobody is asking for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ad37bee081 |
Drive on-demand loading end to end, not just per component
CI / test (push) Successful in 1m30s
On-demand loading is four components agreeing: the router notices nothing is serving, the control plane decides the model may be started, the reconciler places it, and the endpoint list carries it back. Each half was tested alone, and the wiring between them is where this session's real bugs have been — a metrics collector registered after the registry was built, an adoption path that never health-checked — neither of which failed a unit test. Three tests through the real request path: one request loads a model that was not running and is answered by it, an idle one is stopped with the node still heartbeating so it is eviction rather than node loss, and an unknown model still gets the unknown-model error, because on-demand must not turn every typo into "loading". Writing them was itself the check. The first version heartbeated once and then slept through an UnreachableAfter of 300 milliseconds, so the node went quiet and nothing was placed — which looks exactly like the feature failing. A wrong assertion too: with an on-demand-only cluster and nothing loaded, naming what is served means naming nothing, and "no model is currently being served" is the honest answer rather than a lost error message. This matters more now that there is no cluster to deploy to. Live verification found several things today that no unit test would have, and these tests are what has to replace it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
9b32531ce3 |
Stop an idle model to make room for one somebody asked for
CI / test (push) Successful in 1m29s
On-demand loading worked right up to the moment the cluster was full and then stopped meaning anything: the model somebody had just asked for could not be placed, and the cards were held by a model nobody had asked for in an hour. That is the only situation in which any of it matters. Deliberately narrow. Only idle replicas of other on-demand models — a model at min 1 is a promise that it stays up, and breaking it to serve a different request would make every guarantee in the catalog conditional on load. Only single-node replicas, since evicting a multi-node one frees several machines and costs tens of minutes to rebuild. One at a time, because freeing everything that qualifies would empty a cluster for a request that needed one node. And never a replica with no reported usage: after a router restart that is all of them, and unknown is not idle. Writing the test found a gap the feature had rather than a bug in the test. Two on-demand models on a cluster with room for one would take turns evicting each other, at a model load per cycle — worse for both than one of them waiting. A replica is now safe from eviction for a few minutes after it starts serving, which bounds that. Which in turn found a latent bug: deriveState stamped ReadyAt with time.Now() inside a package that has an injectable clock, so the new lifetime rule could not be tested at all — the fake clock said three minutes and the wall clock said eight months. Invisible while nothing compared ReadyAt to anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
7fe59bd5fb |
Load a model when someone asks for it, and stop it when nobody does
CI / test (push) Successful in 1m29s
The feature the idle tracking was for. A cluster with more models than GPUs can now hold every model in its catalog and only the wanted ones in its cards: min 0 with a max above it means load on demand, and once nobody has asked for idle_timeout the replicas stop and the cards go back. No new field, because min 0 max 1 already reads correctly — keep none running, allow up to one. Which made min 0 max 0 meaningful for the first time: configured and deliberately not running. normalize() used to fill a zero max in, on the reasonable ground that a model with no maximum cannot run; that would now silently turn every switched-off model into an on-demand one, including the ones already written that way in this cluster's own configuration. A model with no scaling stanza stays off for the same reason — defaulting it to on-demand would let any client request start a 70B nobody intended to run. The one real decision is what happens to the request that triggers a load. Refusing makes the feature useless, since the first request for every model fails; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds holds a connection for ten minutes. So it waits briefly and then says what is happening — 503 with Retry-After and "the model is loading", which is a different answer from "no endpoint serves this" and produces different client behaviour. An unknown model still gets the unknown-model error, or on-demand would turn every typo into "loading" and the useful message would disappear. Use keeps a model alive rather than only the request that started it, without which the timer measures time since the load and a busy model is evicted on schedule however hard it is working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b619546516 |
Stop trusting hardware facts that were never checked
CI / test (push) Successful in 1m32s
Three places where the system took the infrastructure's word for something and would have failed silently on real hardware. A declared fabric adapter that is not on the node. Fabric membership has to be configuration — OpenSM's default subnet prefix is identical on every fabric, so a matching prefix proves nothing — but taking it entirely on trust was a different mistake. A typo, or a card that enumerated differently after a reboot, produced NCCL_IB_HCA naming an adapter that does not exist. An adapter with no ACTIVE port. Physically linked with no subnet manager sits in INIT: it looks healthy to anything counting adapters, carries no data, survives a reboot, and looks like a software problem. Discovery had recorded port state all along and placement only ever asked whether a fabric had any adapters listed. Both end the same way — the replica starts, passes its health checks, and hangs on its first all-reduce, which is the failure the fabric invariant exists to prevent arriving through a door nobody was watching. The agent now compares declared against discovered, where both are visible for the only time, and does not advertise a fabric that cannot carry traffic. The reason is logged on the machine that knows it, because the operator who typed the adapter name is the only person who can fix it. The fabric's rate is the slowest active link, since a collective runs at the speed of its slowest. And NVLink topology, which no published binary could see. NVML is its only source, the bindings need cgo, and CGO_ENABLED=0 is deliberate so the portable binary runs on an older libc. The cost was silent: tensor groups fall back to the lowest free indices, which is right on an SXM board with NVSwitch and wrong on a PCIe box where bridges pair specific cards — a group straddling two of them runs several times slower and nothing says so. Releases now also publish an nvml build for those nodes, and llmgw_gpu_nvlink_peers makes the difference visible: a flat zero across a fleet means the topology is unknown rather than absent. A test that declared mlx5_0 and mlx5_1 on a node whose sysfs held neither had been passing, because nothing compared the two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
3a4cc0c719 |
Register the fleet collector before the registry is built
CI / test (push) Successful in 1m31s
|
||
|
Ilker ManapandClaude Opus 5
|
42fd0aa31e |
Export what the cluster is made of, not only what it is serving
CI / test (push) Successful in 1m28s
The router published twenty-two metric families about requests, endpoints, admission and scoring — everything about traffic and nothing about the machines carrying it. There was no way to graph GPU usage, watch a node go unhealthy, see a fabric fill up, or answer "was the cluster full at the time", which is the first question anyone asks after an incident. The agents exported nothing at all. Exported by the control plane rather than by each agent. It already holds every number: hardware arrives at registration and state on each heartbeat, so this is a map read away. One scrape target for a fleet of hundreds, no Prometheus configuration per node, no firewall rule per machine, and nothing new running on the nodes. On the same /metrics as the traffic figures, because correlating "the cluster was full" with "requests were queueing" should not need two systems. A collector rather than gauges kept current by a ticker: the numbers are derived from state that already exists, so computing them when asked is simpler than maintaining a copy that can drift, and nothing runs between scrapes. GPU utilisation is new, and is the number people actually watch. Memory used says a model is loaded; this says whether it is doing anything, and a fleet at ninety percent occupancy and two percent utilisation is a fleet holding models nobody is asking for. Adding it caught two things. utilization.gpu goes last in the nvidia-smi query and is optional, because a driver too old to know it returns a shorter row — requiring it made an llmgw upgrade silently stop discovering GPUs on an older node, which is far worse than not knowing how busy a card is. And I gave the new protobuf field number 14, which ecc_errors already had; buf lint is why that was a moment rather than a corrupted wire format. Absent is kept distinct from zero throughout, for the same reason as everywhere else here: an unreported utilisation shown as a flat zero reads as an idle fleet rather than a missing measurement, and a replica nothing has reported would be the most idle thing in the cluster. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
943c23a412 |
Fetch a model's weights when the simple arrangement needs it
CI / test (push) Successful in 1m26s
The default arrangement is the simple one: every node keeps its own copy. So a source naming a remote file is now fetched before the engine is launched, rather than left to fail as a missing path minutes into a load. Eight shards of a tensor-parallel replica start at the same instant and want the same file; they produce one download. An interrupted fetch resumes instead of starting over, which matters at forty gigabytes, and the download is written beside its target and renamed only when complete — a partial file under the model's name would be found by the next scan, reported as present, and handed to an engine that fails on it. The exotic arrangements need no code, which is the point. Weights on a Lustre mount or an image exported over the fabric are simply present, and present means nothing is fetched. The escape hatch from a distribution mechanism is a distribution mechanism that checks first. Wiring it up caught a regression in my own design. A bare model id — "Qwen/Qwen2.5-7B-Instruct" — is a documented source that vLLM and SGLang resolve themselves, and treating every non-remote source as a local path made every existing vLLM profile fail. Only an absolute or explicitly relative path is now checked as a file; anything else is passed through untouched. Three agent tests failed on the same change for a better reason: they used "/models/x", and the new check is exactly the one that catches that. Whole repositories of shards are deliberately not fetched here. vLLM and SGLang do that well already, and a second snapshot downloader would be work spent arriving back where we started. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c13c567cc5 |
Know where the weights are, without owning the storage
CI / test (push) Successful in 1m32s
Storage will be different at every installation — Lustre, NFS over RDMA, an image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes it rather than managing it. An agent told which directories to look in reports what it can see; how the files got there is nobody's business here, and a node told nothing reports nothing and behaves exactly as every node did before this existed. That fact buys two things. Placement prefers a node that already holds the model over one that must first move tens of gigabytes, as a preference and never a filter: fetching them is a valid answer, and refusing to place would turn a slow start into no start at all. And ctl status shows what each fabric holds. Per fabric because fabrics are not connected to one another. Weights in ib-a are not weights in ib-b — the only path between them is the management network — so a model seeded into one fabric leaves the rest of the cluster unable to serve it without a slow copy, and nothing said so before now. One shared filer counts the same as twenty local copies, because "can this fabric serve it" is the question being asked. Three distinctions the tests hold to, because each is a way to get this quietly wrong. A node that was never asked to look must rank with one that looked and has nothing, or enabling the feature on a single node would drain every other. An empty directory is not weights, because a half-finished download looks exactly like one and would send placement to a node whose engine then fails to load. And local storage is never mistaken for shared: a filer read as local costs an unnecessary copy, while local read as shared would have the cluster believe a model is everywhere when it is on one node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
bd134a7ab9 |
Make the planned entry use an engine that can serve the precision
CI / test (push) Successful in 1m39s
Reading the tool's own output against the real fleet: it chose q8_0 as the best fit and emitted a vLLM profile with no quantisation. q8_0 is a GGUF quantisation that only llama.cpp serves, and from a file rather than a repository — so following that entry loads full-precision weights, which the same output had just listed as not fitting one card. The planner would have caused the out-of-memory it exists to prevent, from a recommendation that looked right. Choosing a precision and choosing an engine turn out to be one decision, so a Precision now carries the engine that serves it, the quantisation flag where one is needed, and whether it lives in a file. A GGUF fit prints a path and says to download it; a vLLM fit prints the repository and passes --quantization so the engine does not quietly load bf16. The arithmetic was right and the recommendation was wrong, which is the failure mode worth having tests for: everything reported a number, and the numbers agreed with each other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
88a5949f7c |
Stop offering the same fit once per fabric
CI / test (push) Has been cancelled
Running the planner against the real fleet produced twenty-six rows for a 32B, of which thirteen were duplicates: two fabrics of identical cards, and every single-node fit listed once for each. A replica on one node does not span anything, so which fabric its node happens to be attached to changes nothing about whether the model fits. A multi-node fit is still listed per fabric, because there the fabric is the constraint and which one has room is the thing the operator needs told. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
300e298053 |
Work out whether a model fits the cluster, and how to split it
CI / test (push) Has been cancelled
llmgw ctl model plan Qwen/Qwen2.5-Coder-32B-Instruct The arithmetic this replaces is the last thing between having GPUs and serving a model on them. Choosing tp, gpu_class and a context that will actually fit means knowing the layer count, the KV head count, what the weights weigh at a chosen quantisation and how much of a card the runtime keeps — and getting it wrong is not an error message, it is an out-of-memory twenty minutes into a load. I computed those by hand for every model on this cluster. Both halves are read rather than retyped: the architecture from the model's own config.json and parameter count, the hardware from the running fleet. Attention heads in place of KV heads is the classic slip and is worth a factor of five on a model with grouped-query attention, so it is not a number anyone should be entering. Every candidate is listed, not one answer. With cards to spare you want the full-precision fit; with one card free you want to know a quantised one exists. Choosing between those is not the planner's job. The tests are calibrated against files actually downloaded here — llama.cpp reported 14.62 GiB and 8.37 GiB for the 14B at q8_0 and q4_K_M, and the model reproduces both — and they caught two errors in my own arithmetic. The overhead was charged per rank from the whole model's weights rather than each rank's share, and it reserved a tenth of a card that usableFraction had already reserved. Between them, Llama-3.3-70B at bf16 across two 80 GB cards came out as not fitting, which is a configuration people run every day. Slack reserved in two places, each unaware of the other, reads as caution and behaves as a bug. Pools are grouped by class and fabric together, because a replica may never span two fabrics: eight free cards split across two of them are not eight cards for a model that needs eight, and merging them would plan something that starts, passes its health checks and hangs on its first all-reduce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ea4dd37f80 |
Document installing from nothing
CI / test (push) Successful in 1m24s
Written for whoever is standing up the cluster with root on some servers and nothing installed. Control plane, certificate authority, first node, first model, first request. It leads with what llmgw does not install — drivers, CUDA, engines, weights — because a node missing any of them registers, reports ready, and fails every replica placed on it, and the reason is three logs away. The useful order is to make a node able to serve before onboarding it, which is the opposite of what the tooling's convenience suggests. The failures it warns about are ones this cluster actually hit: llama-server present and unable to start for want of libgomp1 while the node reported healthy; a Proxmox VM on the default kvm64 CPU, where inference is six to nine times slower and nothing looks broken because every number is simply small; requests dying at exactly sixty seconds on a liveness timeout that is a generation budget in disguise for non-streaming calls. Verified rather than recalled. Every command it prints was checked against the binary, every config key against the struct tags, and its own verification and troubleshooting commands were run against the live cluster. That caught two errors in my draft: the health endpoint is /health and returns an empty body, not /healthz, and the Go version needed is 1.26. The known-gaps section is there so a reader finds them on a Tuesday afternoon rather than at 2am: weights and engines are theirs to distribute, there is one control plane and no HA, and multi-node replicas have never run on real InfiniBand — only against simulated hardware. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
d07dddb1bb |
Say plainly whether a model is serving, and why not
CI / test (push) Successful in 1m28s
Reported after watching a model appear in ctl models before it could answer anything. Checked: clients are not told early — /v1/models listed it only once a replica was ready, and the chat picker reads the same source. The operator view showing it is deliberate, because a model coming up is exactly what someone running the cluster wants to see. But the report pointed at something real. That view distinguished "a replica is coming up" from "nothing could be placed" by a parenthetical — "0/1 ready (1 total)" against "0/1 ready" — and those need opposite responses: one means wait, the other means look at explain-placement. On four models the difference is findable. On forty it is invisible, and a model that will never start reads exactly like one that is thirty seconds away. SERVING now answers the question that was being asked of the replica counts — can a client use this right now — and STATUS says which of ready, starting, not placed or stopped is the case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
331de9b495 |
Add and remove models from the command line
CI / test (push) Successful in 1m30s
The last piece of not restarting a control plane to try a model: llmgw ctl model add -f qwen-coder-32b.yaml llmgw ctl model remove qwen-coder-32b The file is the same YAML a model takes in server.yaml, read by the same koanf parser through the same tags, so an entry moves between the two by being copied and a field that works in one cannot silently mean nothing in the other. Wired only when there is a store. A control plane without one says so rather than accepting a model that vanishes on the next restart, which is the kind of success that wastes an afternoon. At startup the catalog is rebuilt from configuration plus what was stored, so models added before a restart are still there after it — without that the whole feature would be a slower way to lose work. Every failure adding a model is the definition being wrong — a collision, a missing fallback, a profile that cannot be placed — so they are 400s. A 500 would send someone to the server's logs for a mistake in their own file. The actor comes from the caller and is kept with the row, so a catalog that has grown over months can say where each entry came from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
277add5ccb |
Keep runtime models, so an added model outlives a restart
CI / test (push) Successful in 1m25s
The catalog can change while the cluster runs; this is where a change goes so it is still there tomorrow. Models declared in configuration and models added at runtime are two sources on purpose: config-as-code stays declarative and under review, and the database holds what someone added to try it. Configuration wins every conflict, and the direction matters. A request for a reviewed production model must never be answered by something added from a laptop, so a stored model colliding with a declared name is refused when it is added. A declared model cannot be removed at runtime either — it would return on the next restart, leaving the cluster disagreeing with its own config file until then, which is a worse answer than refusing and saying where to edit. The whole catalog is built before anything is written. A model naming a missing fallback, or colliding, or malformed in any of the ways catalog.New refuses, is rejected rather than persisted and then skipped forever after — a model that looks added and never appears is the worst outcome available. Removal is checked the same way, because another model may fall back to the one being removed, and a fallback pointing at nothing is discovered during the outage it was meant to cover. At startup each stored model is validated on its own against what is already accepted, so one bad row is skipped and named rather than taking the cluster with it. The declared models are precisely the ones that must survive that. Catalog types now carry json tags mirroring their koanf ones, so a stored definition is the same shape as the YAML someone writes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |