f90daa1ee9005a0ceba5e893d43a56885fabb7db
177
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ilker ManapandClaude Opus 5.5
|
f90daa1ee9 |
Describe v0.8.1
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>v0.8.1 |
||
|
Ilker ManapandClaude Opus 5.5
|
20deb22dfa |
Let a node describe itself instead of being described by hand
CI / test (push) Successful in 1m47s
Every agent file on the cluster named its own node, its own address, the path to llama-server and two weight directories — different on every node, kept true by hand, and not true: node05 carried "ram: 32g" beside a measured 31 GB, looked for weights in an empty directory while fetching them into another, and the other four never looked in theirs at all, so four nodes reported no disk. Growing a node's RAM would have gone unnoticed, because memory was read once, at registration. The agent now finds these out. Its id is the hostname, which is also the certificate's common name. Its address is the one on the route to the control plane, which the kernel picks for a UDP dial without sending anything. llama-server is looked for on PATH and then where release archives are unpacked. A missing nvidia-smi is a node without NVIDIA cards rather than a failed discovery — which also cut discovery short before InfiniBand on every CPU node not told to skip GPUs. Fetched weights land in /var/lib/llmgw-weights by default, and wherever they land is always looked in. Memory is read on every heartbeat, and installed engines are probed again each minute and sent when they change, so a VM given more RAM or a node given vLLM becomes a candidate without anyone restarting its agent. The control plane logs both changes, since "the node now qualifies" is the answer to why a model placed an hour after it was asked for. What is written still wins. The agent file shrinks to the control plane's addresses and the TLS files, the same on every node; onboarding writes only the overrides it was given. What cannot be found stays configuration: which fabric an adapter is on, because OpenSM's default prefix is the same on all of them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5.5
|
2cae92d113 |
Offer a model that loads on demand in the chat picker
The picker listed what had an endpoint, and a model at min 0 has none until somebody asks for it. On this page a message is the asking, so an on-demand model could never be started from here: mistral-nemo-12b-instruct sat in the catalog, served by nothing, and absent from the one place someone went to try it. Such a model is now offered, marked "not loaded", with a note saying the first message fetches the weights and starts the engine, and that until it is up the reply is "the model is loading". A model stopped outright — max 0 — is still left out, because nothing would load it and offering it would only produce an error. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b56130ace3 |
Link the usage report from the cluster page
CI / test (push) Successful in 2m0s
A report nobody can reach from the page they land on is a report nobody reads. It sits beside accounts, on the client port only — the admin listener's copy of this page serves pprof and has no session to authorize anybody. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
07809490e3 |
Show usage over time, per organization, team, project and person
CI / test (push) Has been cancelled
Usage could be read as a total — per key, per user, per model, per group — and the cluster's own charts covered the fleet's hardware. What nobody could see was when a group used it. "Research doubled its traffic three weeks ago" is not a question a column of totals can be asked, and it is the question somebody asks before a quota conversation. /admin/usage draws requests, tokens and spend over a chosen window, grouped by any level of the tenancy tree or by person. Person is not a level of that tree — one person holds keys in several projects — so the grouping is its own type rather than store.Scope with a fourth value nothing else could use. Three things learned from the charts already here. Buckets are gap-filled per group, because a line drawn straight across an absence claims traffic that did not happen, and because the chart sums its tail by timestamp and can only do that if every series shares a clock. Series are ranked by total rather than by peak — a counter's question is "who used the most", and ranking by peak would colour the team that made one burst over the team that ran steadily all week. And the spend chart is replaced by a sentence when no model has a price, rather than drawn flat along the bottom. The bucket query groups by ordinal and was planned against the cluster's PostgreSQL before being believed, with a test alongside the others that run there: three reports have shipped working on SQLite and failing on it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
f80eef9fa6 |
Stop the chat page asking for more tokens than the model allows
CI / test (push) Successful in 1m49s
The page imposes its own ceiling on a turn, so a playground request cannot occupy a replica for ten minutes. It sent that ceiling regardless of the model, and a model declaring a smaller max_output then answered every turn with "max_tokens is 2048, but this model allows at most 1024" — the gateway refusing its own page, over a number the person never chose and has no way to change. The ceiling is the page's to impose on itself and the model's limit is not negotiable, so it sends the smaller of the two. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c848b2c73a |
Offer a model once in the chat picker, not once per name
CI / test (push) Successful in 1m46s
An endpoint advertises every name a client may address it by — the model's own and all of its aliases — because that is what the router matches a request against. The picker took those strings as they came, so one model appeared as several: "coder", "coder-7b" and "qwen2.5-coder-7b-instruct" were three choices, and choosing between them is a question with no answer. Each name is now resolved through the catalog and the model listed once, under its catalog name — the one that also appears in the model list, the usage table and the logs. /v1/models still lists the aliases separately, which is deliberate: there a client is discovering which ids it may send, and every alias is one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
4535bf362f |
Price a request by the model that serves it
CI / test (push) Successful in 1m38s
The price table is written against the catalog. The name reaching the accountant was the one the caller typed, so every request made by an alias missed the table and fell to the default price — the same request charged two different rates depending on which of a model's names the client happened to be given. With no default price that is zero, which is silent. With one it is a wrong number, which is worse: a budget spent at the rate of whatever the default says rather than at the model's own. The router already resolves the alias, two lines above, to decide whether the key is allowed the model at all. It now passes the resolved name to the accountant as well, in the ordinary path and in the budget fallback. The test charges a real request through an aliased name and compares the figure with arithmetic it states itself, because the first version of it asserted "more than the fallback" and passed with the bug in place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
d8fb198aa5 |
Make the install guide describe the system that exists
Read start to finish as somebody standing up a new cluster, it sent them down a path we no longer take. The mirror was absent from all 821 lines, and the page still said that distributing weights was "a known gap, not an oversight" — so the reader would have gone away to solve, by hand, the problem the binary in front of them solves. It is step 4 now, before the nodes, because where a node fetches from is the difference between minutes and a day, and on a customer network the nodes often have no route out at all. Two of the listed gaps had been closed and were still listed: a fetch does report its progress to the control plane, and there is an HA pattern. A gap somebody believes in is worse than one they discover. Added what this week made obvious was missing. Upgrading, including the zero-byte copy that ssh -n silently produces and the size check that is the only thing between that and a fleet on a truncated binary. Backing up, in the order that matters, and the fact that API keys are hashed and do not come back. Pricing in the first configuration, because a budget without a price is displayed, counted against nothing, and never binds. And scale, bench and the source check, none of which were written down anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
03e38e7dda |
Carry the profile fingerprint to the agent and back
CI / test (push) Successful in 1m39s
The stale-profile flag could not fire where it matters. The control plane keeps no record of a running replica across its own restart — it rebuilds them from what the agents report — and adoption left the fingerprint empty, so every replica read as unknown afterwards. That is precisely the moment the catalog is most likely to have been edited: change the configuration, restart the server, and nothing would ever say the fleet disagrees with it. The fingerprint now travels with StartShard, the agent keeps it beside the pool and fabric it already echoes back, and adoption takes the replica's version from the agent rather than from the catalog it is about to be compared against. An agent too old to report it sends nothing, which still reads as unknown rather than as matching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
40f5c930c4 |
Say when a replica is running an edited profile, and check a source from the web too
CI / test (push) Successful in 1m43s
Two defects, both of the same kind: something the CLI knew and the rest of the system did not. A profile can be edited while a replica is running against the old one. The reconciler deliberately leaves it alone — churning a replica that is serving costs more than the imperfection — but there was no way to see it, so the cluster could disagree with its own configuration indefinitely and read as healthy. Profile now carries a fingerprint, a replica records the one it was placed with, and both the API and `ctl replicas` say "stale profile" when they differ. An empty fingerprint is not stale: it means a replica adopted from an older agent, which is unknown rather than wrong. Adding a model from the admin page skipped the source check `ctl model add` has done all along, so a typo or a mirror that is down became a replica failing on a node minutes later, with an error naming the node. The check moves out of the CLI into a package both use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
832d0a652c |
Say when a budget cannot be spent against
CI / test (push) Successful in 1m45s
A team was given a budget of a million, the quota page showed it, and after sixty requests it read zero percent used. Not because the counting was wrong: cost comes from a price table, no model had a price, and every request therefore cost nothing. The number was a limit in appearance and bound on nothing. An empty price table is the ordinary case — self-hosted models have no per-token price, which the schema already says. Showing somebody a budget without mentioning it is not. The page says so, and says it precisely: rate and token limits are unaffected, because they count requests and tokens rather than money. A warning that read as "no quota works here" would be worse than the silence it replaces — those two were enforcing perfectly well, as twenty served and ten refused had just shown. AnythingPriced is on the accountant rather than the price table, because what a caller is asking is whether the budgets it is about to display mean anything, and the accountant is what would enforce them. A model priced at zero is not priced; a fallback price prices everything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
dfbb7557dd |
Serve a model made of several files
CI / test (push) Successful in 1m42s
The mirror would happily hold a four-shard model that nothing could serve. A catalog entry names one source, the agent fetched one URL, and so a version of several files had no way to be described — which is every safetensors model, and every GGUF too large to ship whole. A source ending in a slash names a version rather than a file. The agent fetches every file in it into one directory and hands the engine the directory: vLLM reads the shards and the config out of the folder, and llama.cpp is given the first part and finds the rest beside it. The trailing slash is the whole signal. A flag beside the source would be a second place for the two to disagree, and a URL already distinguishes a directory from a file everywhere else. The mirror answers a version with a manifest instead of a file listing. A plain file server gives HTML there, and something that has to know what a version consists of would be left parsing it. It is on the byte listener, not the management API: fetching weights is the one thing every node does, and making it depend on the administrative port would undo the split that keeps that port closeable. Built from the directory rather than a file written beside it. A manifest on disk is a second record of what is there, and the two disagree the first time somebody copies a file in by hand — which the mirror explicitly allows. The whole set or nothing. A directory with three of four shards loads as a broken model rather than failing to load, so a completion marker is written last and an interrupted fetch starts the missing files again rather than being found and served. Files already there at the right size are not fetched twice. Progress is reported under the version's address, not each file's: the shards waiting on it asked for the version, and progress under a file's own URL reaches nobody. A filename from the manifest is refused if it contains a separator. The mirror is a place an operator copies files into by hand, so what it reports is not automatically safe to join onto a path. An HTML listing where a manifest was expected says so. "invalid character '<'" sends somebody looking at the wrong thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c95cfbe426 |
Keep a year of metrics, and watch the disk they share with the models
CI / test (push) Successful in 1m48s
Where to put them turned out not to be a question about capacity. Measured on this cluster: 2,359 series at 41 samples a second, about 7 MB a day, so a year of history costs roughly 3 GB against 900 GB free. Retention was the package default of fifteen days, which answers none of the questions this data is for — what capacity looked like last quarter, whether it was always this slow, how much the fleet grew. A year, with a size cap as well. The time limit is the intent; the cap is what stops Prometheus becoming the thing that fills the disk if somebody adds a fleet and the series count grows tenfold. The part that matters is what shares that disk. The model mirror and the metrics database are on one filesystem, so a mirror that fills it stops Prometheus writing — monitoring goes quiet exactly when the disk is the problem, and the alert that would have said so is the one that died. Nothing was watching it. node_exporter arrives with the Prometheus package and was already running on that machine, unscraped, while llmgw's own disk metrics cover only its nodes — and the mirror is not one. It is a target now, and the alert fires at fifteen percent rather than five, because what is being protected is the ability to be told. No remote write and no second tier. At two thousand series that is infrastructure for a problem this cluster does not have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
9ef02dc9ec |
Give the charts to both copies of the dashboard
CI / test (push) Successful in 1m48s
The page is served twice — on the admin listener, and at /admin behind a session — and the history source reached only the second. The first version built it after the admin listener's copy was already registered and passed that one an empty NavExtras, so the page an operator reaches by SSH tunnel had no charts and no explanation. Built before either is registered, and handed to both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c6df0659af |
Put the cluster's own charts on its own admin page
CI / test (push) Successful in 1m50s
The dashboard charted what the cluster had been billed for — requests, tokens, spend, all from llmgw's own store — and nothing about whether it was healthy. Saturation, queue depth, cache occupancy and replica counts were numbers in a table at the instant the page loaded, with no way to see that one of them had been climbing for an hour. Four line charts now, read back from wherever the metrics were scraped to. Read back rather than kept twice. llmgw publishes these already; what a gauge cannot do is remember, and a ring buffer in this process would be a rival copy of the same numbers with its own retention, lost on every restart and disagreeing with the one an operator has on screen elsewhere. monitoring.prometheus is empty by default and the panels are then absent rather than empty. One axis per chart, always. Two measures of different scale belong in two charts: a second y-axis lets any two lines be made to cross wherever the author likes and the reader cannot tell it happened. Four colours, handed out in fixed order, never cycled. A fifth series is summed into "other" and drawn in grey — dropping it would make a chart that does not add up, and inventing a hue for it would put two lines on the page that nobody can tell apart. The fold is by timestamp, not by position: the tail's series do not share sample times, and adding them by index slides points sideways onto each other's clock. A legend for every chart with more than one line, because colour alone is not an encoding a reader can rely on. A flat zero is drawn along the bottom rather than left blank: nothing happening is a real answer, and an empty frame reads as a broken chart. A single sample is not a line and says so instead of putting a dot in the corner. One failed query fails all four panels. Three drawn and one blank invites reading the blank one as "nothing is happening there". NaN is dropped rather than plotted. Prometheus sends it as a string for a sample it has no value for, Go's ParseFloat accepts it happily, and one NaN coordinate is an SVG path that does not draw at all — taking the rest of the line with it. Axis maxima are rounded to a number a person would pick. An axis topping out at 0.8347 makes every gridline unreadable, and the shape is what a chart is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b921fa0d6b |
Watch the cluster, and prove the alerts fire
CI / test (push) Successful in 1m45s
Prometheus, Alertmanager and Grafana, with every rule written against a series this system actually publishes. Nothing here is aimed at a metric that might exist. One scrape target for the whole fleet. Agents publish nothing of their own — the control plane gathers what they report and republishes it — which is worth writing down before somebody goes looking for an exporter to install on each node. It scrapes metrics_listen, not the admin listener. That one is loopback because it also serves pprof, and pprof can dump the heap where API keys sit in plaintext. Metrics are the one thing on it something else genuinely needs, so they get a port of their own rather than becoming a reason to open the other. The README shows how to confirm the new port carries nothing else; it answers 404 to both pprof and the admin API. Nothing is sent anywhere. Alertmanager groups and holds and has no receiver: where an alert should go is a decision with credentials attached, and a monitoring system that quietly starts mailing people is worse than one that is visibly not wired up. critical already routes separately so it can be pointed somewhere else without touching the rest. The rules encode what the metrics mean rather than round numbers. A saturation of 1 is the healthy limit measured by ctl bench, not the point of failure, so the alert waits five minutes above it. A thirty-second p99 first token is the threshold because this is CPU inference and would be absurd on a GPU. A model that is deliberately stopped publishes no series at all, so "nothing serving" cannot fire for one nobody asked for — which is what lets an on-demand catalogue sit quiet. Inhibitions, because a node that is down explains every replica on it and reporting both buries the cause. The dashboard is provisioned from a file rather than clicked together, so it survives a reinstall and can be read in a diff. Checked by firing one: cordoning a node moved llmgw_node_up to 0, the rule went pending and then firing after its two minutes, the alert arrived at Alertmanager with the node in its labels, and uncordoning resolved it. An alerting system nobody has seen fire is an assumption. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
8fd4c1392d |
Stop handing the same memory to two models
CI / test (push) Successful in 1m46s
Placement compared a profile's memory requirement against the node's total, so a 31 GB machine accepted two models that each declared 15: each was asked about on its own, and nothing recorded that the first had claimed anything. GPUs were never handed out twice this way — they are tracked as claimed indices — and memory was, because the allocation carried no figure for it. An allocation records what its profile asked for, and eligibility subtracts what a node already owes before comparing. The rejection names both numbers, because "needs 15" beside a 31 GB machine reads as a bug in the scheduler rather than a full node. Declared, not measured. A profile that says 15 is treated as holding 15 whether the process has touched that much yet or not — the alternative is deciding against what a model has grown into so far, which is wrong every time one is still loading. Zero is nobody said, and claims nothing. Counting it would let one undeclared model make a whole node look full. Adoption after a restart re-reads the figure from the catalog rather than from the agent: what a profile asked for is not something a running process reports, and a control plane that forgot every claim on restart would put a second large model onto a node already full of the first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
d998ea7ef5 |
Let a model say what it is for
CI / test (push) Successful in 1m42s
A gateway serving nine models offered nine names and nothing to choose between them. /v1/models is the only view most clients ever have, and it listed ids. Models carry a description now: written when the model is added, shown on the dashboard and sent to clients in /v1/models. Written rather than derived. A registry offers a pipeline tag and a wall of README, and neither answers the question somebody picking from a list is asking — which of these should I send this request to. "General chat, strong Swedish; slower than the 8B" is not in any metadata field. It is optional, and absent means absent rather than an empty string in the response: a model nobody described should read as undescribed, not as described with nothing. On the dashboard it sits under its row rather than in a column. A sentence in a table cell either wraps every other column into a stripe or gets cut off. Co-Authored-By: Claude Opus 5 (1M context) <moreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
e0ea4f6be9 |
Fill in the memory a model needs, instead of leaving the field blank
CI / test (push) Successful in 1m41s
Serving a model from the mirror page left "Needs RAM (GB)" optional and empty. Empty writes no constraint, and a CPU profile has no card to be short of, so every ready node became a candidate — and a 4.68 GB model went onto a 7.9 GB machine and left it 3.5 GB for a KV pool. Placement was not wrong; it was never told what the model needed. The mirror knows the size of the weights, so the field arrives filled in: weights plus half again for the KV pool, plus two for the engine. The half is there because the cache grows with the same things the weights do, layers and heads. It is an estimate and the field is still editable — what it is not is blank. A 1.12 GB model asks for 3 GB, so the small nodes stay usable. Pinning every model to the one large machine would be a different way of getting placement wrong. An unknown size fills in nothing rather than the floor of two, which would exclude no node while reading as though somebody had measured it. And when the estimate is larger than any node in the fleet, the page says so, with the largest node's memory. Better read there than discovered by a node being OOM-killed twenty minutes into a load. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
2acd794022 |
Stop placing engines on nodes that do not have them
CI / test (push) Successful in 1m42s
A node's registry holds an adapter for every engine whether or not its binary exists, the agent never said which were installed, and placement carried an Engine field it had never once read. So a vLLM replica was as happily placed on a node with no vLLM: the node registered healthy, the shard failed minutes later with "executable file not found" in the engine's own words, and the reconciler tried again. That is how fifteen gigabytes of safetensors reached this cluster's mirror before anybody noticed it had no card and no vLLM to run them with. Agents now probe what is installed and report it — the protocol has carried engine_versions since it was written and nothing ever filled it in. Placement filters on the key set, `ctl nodes` has a column, and the mirror page's serve warning is a fact instead of a guess about cards, which would have waved through a fleet of eight-GPU nodes that never had vLLM. Probing resolves binaries and runs none of them. The first version asked each engine for --version, which is fine for an engine and not fine for the wrapper scripts people put in front of one: a script that shifts past flags it does not recognise and then execs the server will start an inference server, at agent startup, on every node. The agent's own test suite installs exactly such a script and caught it. So presence is a path lookup, and the version is left to `ctl engine-check`, which is run by somebody who meant to run the engine. SGLang is the exception and has to execute, because a module has no path to look up — but the command is llmgw's own one-line import rather than the engine's launcher. Its exit code is the answer: python3 is on almost every machine, and treating a failed import as "installed" reported SGLang on every node in a fleet that had none, which is the failure being fixed, inverted. Probed once, not per reconnect, and bounded by the context that registration waits on: an agent that cannot reach the control plane is worse than one reporting an incomplete list. A node that reports no engines at all is not excluded anywhere — by placement, or by the page. That is what every agent older than this reports, and emptying a fleet on a control plane upgrade is a worse failure than the one being fixed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
e034862b79 |
Serve a mirrored model from the list, instead of writing YAML for it
CI / test (push) Successful in 1m35s
Fetching a model put it on the mirror and left the operator to hand-write a catalogue entry and run `ctl model add` on the control plane — for a source the page was already displaying, with an engine decided by the file extension it had just shown them. A present row now offers to serve it. The form opens on that row with the name, the mirror's own URL and the engine filled in, and asks for the two things nobody can infer: the context pool and how many replicas. What it refuses is the point. A version of several files is refused with the reason. A catalogue entry names one source, so an entry for a sharded model is an entry for one shard, which loads as a broken model rather than failing to load. Nothing said so before — the mirror happily holds a four-shard model that no catalogue entry can describe. Safetensors on a fleet with no GPU is warned about before anything is added. This is the failure we hit today: fifteen gigabytes of shards arrived, and only then did looking at a node show no card, no vLLM and no SGLang. The control plane is not told which engines a node has installed, so the check is on cards, which it does know — a warning rather than a refusal, because a card can be added tomorrow. A name already declared in configuration is refused here too, matching what the model service would say anyway, but before the form is filled in rather than after. Without a store there is no serve action at all: a model added at runtime would not survive a restart, and offering it then refusing is worse than not offering. The mirror page's dependencies became a struct. Three are optional interfaces and two are the same shape, which at a call site is a swap nobody notices until a page shows the wrong thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
a588646b7c |
Fetch a sharded model in one action, not one shard at a time
CI / test (push) Successful in 1m39s
The explorer listed every weight file with its own button. For a repository of alternatives that is right — q4_k_m or q8_0, you want one — and for a sharded model it is fifteen clicks to fetch one thing, with nothing stopping you from losing count and leaving a version with fourteen of its parts. Files are grouped into sets now. A run of "-00002-of-00004" is one row and one button; everything else stays its own row. Not "fetch all the weights", which would be wrong half the time: two safetensors files are halves of one thing and useless apart, two GGUF files are usually alternatives and fetching both spends gigabytes on a copy nobody asked for. The set is the unit that is whole. A repository can hold both the split and the joined copy of the same quantisation — the real Qwen coder repos do — so a shard set keeps the -of- notation in its name rather than collapsing onto the name of the whole file beside it. A set publishes no single checksum. Several files have several, and showing one of them invites comparing it against the wrong thing. The form posts every source in the set, and a failure part way says how many already started: a set that fails on its fourth shard has three arriving, and reading that as "nothing happened" sends somebody to fetch them all again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ed530133b6 |
Find a model in the registry instead of pasting a URL into a box
CI / test (push) Successful in 1m41s
Mirroring anything meant knowing a repository existed, knowing which of its thirty files were the weights, and assembling a resolve URL by hand. The mirror page had a text box for that URL and no way to find one. It now searches Hugging Face and lists a repository's weight files with their sizes and published checksums, each with a button that fills the fetch form. Only weight files. A repository is mostly a README and a tokenizer, and listing those buries the four entries somebody came for. The size shown is the LFS size, not the plain one. For an LFS-stored file — every weights file — the plain field is the size of the pointer, so the other one reports a gigabyte model as 133 bytes, and that is the number somebody would then plan disk against. The source URL is pinned to the commit the listing came from rather than to a branch, so what the mirror fetches is what was on screen. A repository re-uploaded under the same branch otherwise leaves two different sets of weights with identical provenance and no way to tell them apart. The version defaults to that revision for the same reason. The registry's own failures are reported on their own line. It is somebody else's server on the other side of the internet, and it being down says nothing about the mirror — which is what the page is for and has to keep working regardless. With no registry configured there is no search box at all, rather than one that can only time out. Gating decodes from a bool or a string, because the field is documented as the first and is the second in practice: strict decoding fails the whole response, so one gated model would hide the other nineteen in a page of results. And a bug this made obvious: downloads were keyed by model and version, so the second shard of a sharded model was refused as "already fetching" until the first had finished. A fifteen-shard model had to be pulled one file at a time, each waiting on the last. Keyed by file now — which is what the single-flight was ever protecting, since the race is two writers on one .part. A version made of several files reports as the worst of them: it is not present until every shard is, and one shard failing makes the version unusable however many others arrived. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
f999ee2fb2 |
Make an organization's budget bound the sum of its teams
CI / test (push) Successful in 1m40s
Resolution took the most restrictive value along the chain and enforced only that one. For a single caller's ceiling that is right: a team cannot raise what its organization set. For a pool it is not, because two levels are two pools. An organization given 1000 with four teams given 500 each spent 2000. Each team filled its own counter, the organization's value lost to the tighter one and was never looked at again. The number was written, displayed, reported against — and enforced by nothing. Every level that sets a limit is now checked against its own counter, and settled onto all of them, so an organization's pool fills from its teams' spending rather than only from requests that somehow named it. A team still stops before its organization when its own pool runs out first, which is what a sub-allocation is for. Effective carries the chain as well as the collapsed values. The collapsed ones are still what a header or a profile page should show — they are the ceiling a single caller faces — they are just not enough to enforce with. Rate limits charge as they check, deliberately, so that two concurrent requests cannot both see room. With several levels that means a refusal at one has already taken from the levels before it, so those are handed back. Without the refund a team sitting at its organization's ceiling would quietly eat its own rate limit with requests that were never served. Budgets are checked before any of that, and only read, so a refusal on a budget charges nothing at all. Status reports the fullest pool rather than the smallest number. A team inside an organization with the same budget is bounded by whichever is nearer its ceiling, and that is the organization as soon as a second team spends anything; a header showing the team's would disagree with the refusal after it. A cluster default joins the chain as a key-level entry. Left out, it would be reported as the ceiling and then never checked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
5726252825 |
Open the quota form on the branch, instead of asking which branch again
CI / test (push) Successful in 1m44s
The tree was rendered and then, underneath it, a form with a dropdown asking which of those rows you meant. The tree already said. Picking a branch, reading its current numbers, scrolling past the tree and selecting the same branch from a list is three steps to do one thing, and two of them are a chance to pick the wrong row. A quota belongs to a branch, so the form opens on it: each row links to ?edit=scope:id, that row renders its form inline, pre-filled with what is set there now and naming the branch in full. By link rather than by disclosure on every row: a collapsed form per branch would put a form in the markup for every group on a page whose job is to be read as a tree. One is open at a time, and the rest keep their link. Clearing moved into that form as a second submit, so a row that is not being edited has one control and not two. Saving closes it. Re-rendering the form over the row it just changed hides the result and invites submitting the same thing twice. The note about a quota being shared came out of the footer: the open form says it, at the point where somebody is about to do it. The dropdown's parsing went with it rather than being left unreachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
55262418c9 |
Put quotas somewhere they can be seen and changed
CI / test (push) Successful in 1m35s
A quota could be enforced and reported, and set nowhere but a shell. Nothing in the browser showed one at all — not what a team may spend, not what it has spent, not that it had no limit. In practice that means quotas nobody sets. /admin/quotas lists the whole tree, whether or not a group has a limit: the one without a quota is the one most likely to need one, so it cannot be the row the page leaves out. Spend over the last thirty days sits beside the budget, with the fraction used. Setting a quota here is a change of position. It was kept to the shell because raising a budget is a financial act and a stolen admin session should not perform one. That argument holds for exactly one row on this page: resolution takes the most restrictive value along the chain, so raising a team's or a project's limit cannot exceed what the organization allows. Only an organization's own limit is bounded by nothing. Every change is logged with who made it and what it was before. Per-key limits stay on the CLI. A key belongs to a person and is already bounded by its project; letting an admin session reach individual keys would put every caller's throughput one click away, and the page refuses a key scope rather than rendering it and failing later. A limit an ancestor already caps lower is marked as written-but-unreachable. Resolution means a project given more than its team is not a bigger budget, it is a number nobody will reach — and without saying so the page would show it as though it applied, with the first sign being a refusal that disagrees with the screen. The group is chosen, not typed. Ids became random in the previous commit so two organizations could both have a "platform" team, which also made them impossible to remember and a typo impossible to notice: it would write a quota onto nothing and report success. A blank field is no limit of that kind, not zero — zero is a limit of nothing, which refuses every request. Clearing removes the row rather than writing nulls across it: both resolve the same, only one tells a reader nobody has set a limit here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
fc1b47976d |
Let an admin build the tenancy tree, instead of sending them to a shell
CI / test (push) Successful in 1m35s
An account is approved into a project, a project belongs to a team, a team to an organization — and until now the only way to make any of them was `llmgw keys init` on the control plane. So the first thing to do after installing was leave the interface you were already in, and an approval form with an empty dropdown gave no hint why. The accounts page now creates all three. Creating a container is not granting access: nobody can use one until an account is approved into it, which is the step already on this page. Quotas stay on the CLI. Making somewhere to put people is structural; raising what they may spend is financial, and a stolen admin session should be able to do the first and not the second. Ids are random now instead of slugs of the name, which fixes a bug that only appears when you have a second organization — which is exactly what this feature invites. Team ids were slugs in one flat namespace and PutTeam upserts on id, so creating "platform" under a second organization moved the first one's team into it. Nothing failed. The team changed owner. Create* is separate from Put* rather than replacing it. The upserts exist so `keys init` can be re-run while provisioning a machine; a person filling in a form means something else by a name that already exists, and gets told. A duplicate is refused among siblings only. Two companies both having a "platform" team is ordinary, and refusing the second would be the flat namespace again wearing a better error message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
a83493bddb |
Show what a group spent, and fix the reports that never ran on PostgreSQL
Quotas can be written on an organization, a team or a project, and nothing could show what any of them had used: usage came back per key, per user and per model only. A quota whose consumption cannot be seen is one nobody reviews until the day it refuses somebody — and since the previous commit made a group quota a shared pool, it is now a number several people are spending against without being able to look at it. `llmgw ctl usage --by team` sums every key beneath each group, with the budget that level sets beside it. Beside it, and only when that level sets one. A team bounded by its organization's budget has none of its own, and printing the organization's against each of its teams would read as though each had that much — the exact misreading this work exists to remove. Joined through api_keys rather than read from usage.project_id: that column records where the key was when the request was served, while the quota is written against the tree. They agree until a key is moved, and a report that disagrees with the limit shown beside it is the useless one. Three existing reports were fixed on the way, all failing on PostgreSQL and all passing in tests, because the tests run on SQLite: column "k.name" must appear in the GROUP BY clause (UsageSince) column "us.email" must appear in the GROUP BY clause (UsageByUser) column "usage.ts" must appear in the GROUP BY clause (UsageBuckets) SQLite picks an arbitrary row for an ungrouped column and carries on; PostgreSQL refuses. The first two were selecting names they had not grouped. The third grouped "ts / ?" while selecting "(ts / ?) * ?", which PostgreSQL matches as text and so saw a bare ts — it is grouped by ordinal now, the same expression by construction. The dashboard's usage charts have been empty on the cluster this whole time, with the reason only in the log. All four queries were run against the cluster's own PostgreSQL before this was committed, because passing on SQLite is what they were already doing. The grouping level picks from a fixed table of levels rather than being interpolated, so an unknown one is an error and not a fragment of SQL. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
ebce41e22d |
Stop calling the PostgreSQL store SQLite, and stop guessing where it is
Two things that were only ever confusing, and one of them stopped a command working on the cluster. The store type was named SQLite and kept the name when PostgreSQL was added, so OpenPostgres returned a *SQLite and all seventy-odd methods on the production store were spelled after the database it was not running on. Nothing broke; people did. Earlier today I added three methods to *SQLite for a table that on this cluster only ever lives in PostgreSQL, and the name did not once look wrong. It is a field, which is all it ever was. `llmgw users` and `llmgw keys` defaulted --store to a SQLite path. On a deployment whose store is PostgreSQL every one of them failed with "unable to open database file" while the right answer sat in auth.store in the config file beside them; the conclusion an operator draws is that the tool is broken, and they are not far off. They now ask the server's own configuration before falling back, and the error names which of the three sources chose the address — the failure was never the path, it was which one picked it. The single-machine default still applies last, so a deployment with no config file keeps working with no flags. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
3febb6ef63 |
Count a group's quota against the group, not against each of its keys
The tenancy tree handed limits down and never pooled what was spent against them. A limit written on a team was inherited by every key beneath it, and then enforced with the key as the counter — so a project given a budget of 1000 and ten keys permitted ten thousand, and a team given 60 RPM got 60 per member. That is "each of you may spend this much", which is a useful thing to be able to say. It is not what somebody writing a budget on a team means, and nothing in the system said which one you were getting. Resolution now records which scope set each value that won, and the counter is keyed by that scope. The value and the counter come from the same place, so a project tightening its team's limit is enforced against the project's own pool rather than eating into a quota another project never agreed to. Each limit is owned separately, because they resolve separately: a team budget with a per-key rate limit — the group's money, the individual's throughput — is the ordinary arrangement and has to stay expressible. Cluster defaults are owned by the key. Read as a shared pool, one default would become a single cluster-wide rate limit that every key in every organization competes for: not a default, an outage. Restoring after a restart had to change with it. Per-key totals rebuilt a counter named after the key while the check read one named after the team, so the control plane came back believing the team had spent nothing — a budget that reset on every restart, for exactly the scopes where it matters most. SpendSinceByScope returns each key's place in the tree and the accountant folds it into every scope that might own a budget. Counters are namespaced by level throughout. A bare key id in one path and a "key:"-prefixed one in another are two counters for the same key, which is how the restore was silently doing nothing. Still missing, and worth naming: no usage report rolls up by project, team or org, so a shared quota can now be enforced but not yet shown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
3aa250428a |
Keep what a benchmark found out, instead of leaving it in a terminal
`ctl bench` has always been able to work out what one replica can serve — it ramps concurrency until the SLO breaks and reports the last level that held. It printed that and nothing kept it, so the one number saying what a replica is worth lived in whoever ran it's scrollback. Everything downstream then had to guess: admission applies a single global default for healthy queue depth, and sizing a model against real demand had nothing to size against. `ctl bench --save` records it; `ctl capacity` reads it back. Keyed by model and profile. Capacity belongs to the hardware and the engine arguments as much as to the weights, so the same model on CPU and on an H100 are two numbers, and one key would overwrite the other on every run. The profile is looked up from the replica rather than asked for, because the control plane already knows which profile answers on which endpoint and a measurement filed under the wrong one is worse than none. A target that is not a replica — the gateway, usually — is refused with that said: benchmarking the whole path is worth doing and says nothing about the hardware. The conclusion is stored, not the ramp. Every level and its latencies are worth reading once while judging whether the run was sound; keeping them would grow without bound as a model is re-benchmarked. Re-measuring replaces. A second run is a better answer to the same question, not another data point, and keeping both would leave a reader choosing between numbers with no basis for the choice. The SLO and the date are stored with it. Capacity means nothing without the target it was measured against, and hardware and engine builds change — a measurement is an observation with a date on it, not a fact about the model. A failed save is reported and not fatal: the run has already cost the GPU time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
707d774c76 |
Change a model's replica count without editing a file on the control plane
The count is scaling.min, and until now the only way to change it was to edit server.yaml on the host and restart — or, for a model added at runtime, to remove it and add it back, which means having its whole definition to hand. `llmgw ctl scale <model> --min N` instead. Only the scaling block moves: everything else is read back from what was stored and written out untouched, so changing a number cannot quietly become a redefinition. The catalog is rebuilt and validated before anything is written, as in Add. A max below a min is refused with the stored definition untouched — persisting a rejected change would come back on the next restart as a catalog that does not build. The answer reports the scaling in force rather than the request. The catalog fills in a max that was left out, and a caller echoing its own request would print a number the cluster is not using. A model declared in configuration is refused, for the reason Remove already refuses one: the YAML is read on every start, so the new count would hold until the next restart and then silently revert. A cluster disagreeing with its own config file is worse than an error saying where to edit. The help says what --max does, because it is not what it looks like: a model with a minimum runs exactly that many and nothing scales it up under load, and an on-demand model loads one replica however high its max. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
10e323132c |
Let a control plane whose models all live in the database start
Control plane mode was inferred from the config file alone: a model with profiles meant "control plane", nothing meant "router only". So a deployment that manages its catalog at runtime could not start at all — the file declares no models, the process refuses for want of anything to serve, and the stored models it would have loaded are never read. A cluster unmanageable at runtime because it is managed at runtime. control.enabled says so outright. AgentListen cannot carry the signal: it has a default, so every deployment would silently become a control plane. The inference stays, so nothing already working has to add a flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
cd7eedf09e |
Give the mirror a page, so managing it is not a curl command
CI / test (push) Successful in 1m49s
/admin/mirror, behind the same admin session as the other admin pages. What is held, what is arriving with its byte count, what failed and why; start a fetch, retry a failed one, remove a version. Not mounted at all without a mirror configured. The right page for a deployment with no mirror is no page, rather than an empty one pointing at nothing. It reloads itself only while something is downloading. A reload at any other moment takes with it whatever is half-typed into the fetch form, and there is nothing to watch when nothing is moving. A failed row offers Retry, not Remove. A fetch that failed left nothing on disk, so the remove could only answer "the mirror holds nothing for this" — a button whose single outcome is a confusing error. Starting the same download again is what somebody looking at that row wants, and the mirror allows it: only a download still running is refused. An unreachable mirror is said outright instead of rendering as an empty table. The two look nearly identical and mean opposite things: one is a mirror waiting to be filled, the other is a machine to go and look at. Dashboard takes NavExtras rather than a second bool. Two adjacent booleans at a call site get swapped silently and are discovered by somebody clicking the wrong link. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
744a283cf1 |
Keep the checksum that finishing the download used to take away
All() preferred the disk entry wholesale, and a directory listing has no checksum in it. So the sha256 a fetch computed was reported while the bytes were still arriving and vanished the moment the file landed — it disappeared exactly when it became verifiable, which is the one moment it is for. It is the field an operator compares against what upstream published. A mirror is the single place in this system where a corrupted file is copied to every node instead of being caught once. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
7e6fb0059a |
Let the control plane call the mirror instead of somebody SSHing to it
The mirror has had a management API since it was built, and the only thing driving it was curl on the mirror's own machine. This is the other half: a client, so the operations belong to whatever has a reason to perform them. The mirror's refusals travel as their own sentence rather than a status code. Putting the reason in the body was deliberate on the server side, and a client that kept only the number would undo it — "the mirror said 400" sends somebody to the wrong machine, "the source URL answered 404" sends them to the right one. NextProtos is cleared for the same reason the mirror's server clears it. That configuration is shaped for the agent stream, which pins h2; a transport carrying a pinned protocol it is not set up to speak negotiates h2 and then talks HTTP/1.1 over it, and fails with an error mentioning neither ALPN nor the mirror. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
afb3797e57 |
Document the mirror that exists rather than the one you assemble
The page described nginx and a curl loop, which was accurate when the mirror was a directory somebody kept by hand. It is a subcommand now, so it says so: the two listeners and why they are two, the API and what each endpoint answers, and the certificates coming from the authority the fleet already has. nginx stays, further down. It serves a directory perfectly well and anyone already running it should not be told to install something to replace it. The systemd unit spells out --base in a comment because it is the flag that goes wrong quietly: it ends up in catalog entries, and a wrong one makes `mirror unused` call files unreferenced while they are in use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
dde2bcab19 |
Stop asserting an ordering the supervisor does not promise
The test started a sidecar and an engine, each a shell appending its name to a file, and demanded the sidecar's line come first. The supervisor spawns the sidecar first, but spawning is not sequencing: without WaitForLeaderPort nothing holds the engine back, so two shells race and the scheduler picks the winner. It failed under full-suite load, and lately on its own. The wait that is a real guarantee already has its own test, so the fix is to check what this one is actually for — that both processes run under one shard, that the shard's pid is the engine's rather than the sidecar's, and that their output is distinguishable in a single log tail. Renamed to say so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
16ec585411 |
Let the control plane ask the mirror for a model instead of pushing one at it
The mirror holds the weights, so the mirror does the downloading. A control plane that had to pull the bytes itself and push them here would carry every model twice over the network, through the one machine least able to spare the bandwidth. Two listeners, and the split is the point. Weights stay on plain HTTP where every node can reach them: that path is eight nodes pulling forty gigabytes, and putting a client certificate in front of it would mean re-certifying every agent's fetcher in order to change how models are administered. Management gets its own port, which can be given the fleet's CA — the same tlsconf the agent stream uses, so this is a configuration rather than a new idea. A fetch answers 202, not 200. The bytes are not here yet, and a caller that read it as "done" would point a catalog at a file still arriving, which every node would then copy. Progress is asked for rather than waited on, because no client between here and the operator will hold a connection open for an hour. A failed fetch is remembered with its reason. Forgetting it would leave the control plane waiting on a model that is never coming, with nothing to say why. Status falls back to reading the disk, so a restart does not report a model that is plainly there as unknown. Removal takes the whole version: a model split across files is useless with one of them missing, and half a version on disk looks present and will not load. It refuses while a download is running, because the fetch would carry on writing and the file would reappear, looking like the delete had silently failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b40bc481cf |
Manage the model mirror with a command instead of a runbook
CI / test (push) Successful in 1m36s
The mirror was a directory and a discipline written down in a README. This makes the three operations that matter into commands, on the mirror host, where `llmgw keys` already sets the precedent of operator tooling that works on files rather than through the API. The gateway still never writes to the mirror. That property was chosen deliberately — a mirror writable by the thing reading from it is a way to replace a model with one every node then fetches without complaint — and a person running a command does not take it away. `serve` states it rather than inheriting it, so a handler added later cannot quietly make the mirror writable. `add` downloads to a temporary name and renames, for the reason the agent's fetcher does: a mirror is copied to every node, so a partial file here is a partial file everywhere. It verifies the length the server promised, reports a checksum — the mirror is the one place a corrupt file reaches every node rather than being caught once — and refuses to replace an existing version, which is the whole point of the layout. `unused` refuses to answer when it cannot read the catalog. An empty catalog and an unreachable one produce the same list, everything, and the obvious next step after that list is deletion. It compares by path rather than by spelling, because a base typed with a different scheme or a trailing slash would otherwise report a file in use as unused. `remove` checks the catalog first and says to retire the entry before deleting the file, because a node that has to refetch a deleted file cannot, and that failure arrives later, on the node, looking like something else. Verified end to end: Range requests answer 206, which is what makes an interrupted fetch on a node resume; writes answer 405; a traversal attempt answers 404. Note: internal/agent/supervisor's TestSidecarRunsBeforeEngine fails under full suite load on main as well as here — it asserts that one shell's echo lands before another's, which the supervisor does not promise. Everything else passes. Left alone rather than fixed in a commit about mirrors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
e86457cf02 |
Make the model mirror a file you can copy, not an idea you have to have
CI / test (push) Successful in 1m39s
"Set up a local mirror" was advice. This is the nginx config, the directory layout and the reason the layout has a date in it — a repository that changes upstream changes what you are running, and the first sign is a prompt that worked last week answering differently this week. It says what the mirror is not, because both mistakes are easy. It is not a caching proxy: nothing populates it on demand, deliberately, since a mirror that reaches the internet at the least convenient moment fails in a way that looks like llmgw being broken. And it is not needed at all if you already run a shared filesystem — put the models there, use a path, and nothing is fetched. It also says when a mirror stops being enough, and that llmgw is positioned to distribute between nodes because it already knows which of them hold which weights. That is not built, and the page says so rather than implying it is coming. Every behaviour it relies on was checked against the source: an http:// source is fetched and a path is not, a killed download resumes, a partial file is never visible under the model's name, and a shared filesystem is not counted once per node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
b7a97468e0 |
Answer the two questions asked before a gateway is allowed near anything
CI / test (push) Successful in 1m34s
Neither had a page. "We have GPUs, now what" was left to the reader, and what llmgw stores was scattered across four documents and a changelog — which is the same as not being written down when somebody is doing a security review. docs/data.md describes the behaviour and leaves the legal mapping to whoever holds the obligation. Every claim in it was checked against the source, and it names where to look for each: usage rows carry no text field, keys are stored hashed, conversations are off until a named project turns them on and cannot be turned on without a retention window. It also states the egress exactly, which took reading every outbound call to say honestly: nothing on the serving path leaves the cluster, and the two things that can reach outside — fetching weights, and ctl plan reading a model's architecture — are both started by an operator. Weight fetching points wherever you tell it, including an internal mirror. docs/choosing-models.md is the opinion llmgw itself does not have. Three jobs rather than one model, with the third — embeddings — being the one that gets forgotten and the one that decides whether the deployment can answer questions about your company or only questions in general. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
f2f375d824 |
Stop the text-search comment naming one language
CI / test (push) Successful in 1m41s
The choice is unchanged and now better argued: a deployment does not announce which language its notes are in, so any stemmer picked here is the wrong one somewhere — and applying the wrong one does not fail loudly, it quietly stops finding things. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
2bf7de1a43 |
Give two control planes a shared address, and a probe that means something
CI / test (push) Successful in 1m36s
The last piece of HA, and it is a deployment pattern rather than code: llmgw picks one leader and agents follow it, but client traffic arrives in front of both and llmgw is not in front of anything. Writing the pattern found a bug in the pattern. The first draft said to health-check `/health` and claimed a standby still routes to replicas the leader placed, so sending it traffic was fine. It does not. A standby's reconciler places nothing and it holds no agent streams to adopt from, so it publishes an empty endpoint list — a load balancer following that advice would have sent half the traffic to something that answers every request with 503. So there is now a `/ready` that means "this process has somewhere to send a request": 503 on a standby, 503 on a leader whose replicas have all failed, 200 when there is something to serve. `/health` keeps meaning the process is up, which is a different and less useful question. Both are open, because a load balancer has no key. There is a gap during failover where neither passes — three seconds when the leader stops cleanly, up to the lease TTL when it does not — and the document says so rather than implying otherwise. Traffic cannot go to a control plane that has nothing to route to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
a35e8796d2 |
Carry a download's progress to where somebody is looking at it
CI / test (push) Successful in 1m28s
The agent has always logged a percentage as it fetches weights, with a comment saying a replica pulling a hundred gigabytes and one that is wedged look identical from the control plane. They still did: the log is on the node, and the node is not where anyone is looking when a replica has been starting for ten minutes. Progress now rides on the shard status and appears under the replica in `ctl replicas` — a line rather than a column, because most replicas are not downloading anything and a column would be empty for all of them. It is deliberately not a state transition. Treating each chunk as one would re-mark the shard as starting on every report and reset the timer that decides when a slow start has become a failed one, so an hour-long download would keep a doomed replica alive indefinitely. It arrives on its own method for that reason, and a real transition clears it — otherwise a finished download sits in the listing at ninety-nine percent under a shard that has been serving for an hour. Watchers are keyed by source rather than by shard, because a fetch is single-flight: eight shards starting together produce one download and all eight are waiting on it. No percentage is invented when the server sent no length. One that stops moving is worse than a byte count that keeps rising, so the field is absent and the listing says "so far". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
100f10ac17 |
Generate the configuration reference, and fail when it goes stale
CI / test (push) Successful in 1m41s
There was no reference. Four narrative documents explain how to install and operate llmgw, and the only way to learn that `lease_ttl` exists was to read the one that happened to mention it — or the struct. Written by hand it would have been wrong within a fortnight. Six settings arrived in the last two weeks alone: store, servers, lease_ttl, the two retention windows, min_disk_gb. A reference nobody maintains is worse than none, because it is the same list with the setting you came to look up quietly missing from it. So it is derived from the structs, with the field comments as the descriptions — which means the only way for it to be wrong is for the code to be wrong, and it makes the comments worth writing twice over. 1163 lines, 24 blocks, 87 keys. Two tests keep that true after everyone has forgotten this exists. One regenerates and compares, so adding a setting without regenerating is a failure rather than an omission — checked by adding one and watching it fail. The other walks the tree for koanf tags and fails on a configuration file that is in neither the source list nor the short list of structs an operator never writes, because a list is a thing to forget. The generator finds the repository from its own path rather than the working directory, so `go generate ./...` produces the same document from anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
8e57171c4f |
Let ctl ask a question instead of printing the whole fleet
CI / test (push) Successful in 1m42s
Five nodes fit in a table. Fifty do not, and this is built for people who already have racks of GPUs — so the flags are how a question gets asked: which nodes on this fabric, which of this GPU class, which are not ready, which replicas have a shard on this machine. Filtered here rather than in the API, so --json is narrowed the same way the table is. A server-side filter would be two implementations of one predicate, and the JSON one is the one nobody would notice had drifted. An empty result says which case it is. A filter that matched nothing and an empty fleet call for opposite next steps — one is a cluster to look at, the other is a word to correct — and printing "no nodes" for both makes a mistyped fabric name read as an outage. A state matches on prefix because states are few and distinct and somebody scanning a fleet types quickly. A pool matches either half, because it is fabric/class and both are worth narrowing by. A replica is found from any of its nodes, since one spanning two machines is on both. --label takes a bare key as "carries this at all", which is how you ask for every node with a role without caring which. The flag's help said so before the parser did; the test caught them disagreeing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
3a4260c8c5 |
Cut v0.8.0
The store stops being a file on one machine, which is what stood between this and more than one control plane. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>v0.8.0 |
||
|
Ilker ManapandClaude Opus 5
|
b9c2be6cc2 |
Record what failover actually did
CI / test (push) Successful in 1m36s
Three seconds to move leadership, six for every agent to arrive, and every replica adopted rather than restarted. Also what it did not do: the public address returned 502 while the standby served correctly on its own port, because nothing in front of the two moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |