bf2d171e8941bbff7cbbfba078a5ceb18c388f9e
31
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
5aa5675a9c |
Show a model loading in the chat, and send once it is up
A message to a model that was not running waited 45 seconds and came back as "the model is loading; retry shortly". The page now asks for the model with /chat/wake, shows where it stands from /chat/model-status - waiting for a node, downloading with bytes and a percentage, starting the engine - and sends the message when it answers. Also passes the nodes' llama.cpp build to the pages, which the mirror's checks for loadable weights needed and were never given. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
098886e0e4 |
Let a person belong to many places, or to none
A person was in exactly one project, chosen at approval, and could not use the gateway before that. People belong to several teams and to none, so: - memberships: a person is a member of any number of organizations, teams and projects, or of none; a member of a team or organization may use every project in it. users join/leave, and the people page adds and takes out from either side. Approval no longer needs a project. - personal keys: a key may have an owner and no project. /profile asks what a new key spends against; keys create --for does the same. auth.personal sets default limits for personal use or turns it off. - limits for people: --scope user holds a person across every key, project and the chat page; --per-member on an org, team or project gives each member their own share there. Resolution folds the tree, each per-member limit for the key's owner, and the owner's own, each in its own pool; the quotas page shows and sets both. - the chat page asks what to spend against and checks it per message. - usage records who spent it, so a person's budget survives a restart, personal use and chat included; restore no longer drops rows with no project. Leaving a project revokes the keys made there unless another membership still reaches it. A personal key keeps no threads or notes, which are per project, and is refused rather than dropped into the anonymous pool. Migration 0010 turns each person's project into a membership, adds usage.user_id and backfills it, and makes api_keys.project_id nullable. SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign keys off, which only works outside a transaction, so migrations marked for it run on one connection with foreign_key_check before commit. Tested on a populated SQLite store with claims, and on a copy of the cluster's PostgreSQL store: 5 people became members, 17 keys and 3 claims intact, 145 of 577 usage rows and all 69 chat rows attributed. The PostgreSQL store tests, including new membership cases, pass on 16. users move is gone; join and leave replace it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
c6df0659af |
Put the cluster's own charts on its own admin page
The dashboard charted what the cluster had been billed for — requests, tokens, spend, all from llmgw's own store — and nothing about whether it was healthy. Saturation, queue depth, cache occupancy and replica counts were numbers in a table at the instant the page loaded, with no way to see that one of them had been climbing for an hour. Four line charts now, read back from wherever the metrics were scraped to. Read back rather than kept twice. llmgw publishes these already; what a gauge cannot do is remember, and a ring buffer in this process would be a rival copy of the same numbers with its own retention, lost on every restart and disagreeing with the one an operator has on screen elsewhere. monitoring.prometheus is empty by default and the panels are then absent rather than empty. One axis per chart, always. Two measures of different scale belong in two charts: a second y-axis lets any two lines be made to cross wherever the author likes and the reader cannot tell it happened. Four colours, handed out in fixed order, never cycled. A fifth series is summed into "other" and drawn in grey — dropping it would make a chart that does not add up, and inventing a hue for it would put two lines on the page that nobody can tell apart. The fold is by timestamp, not by position: the tail's series do not share sample times, and adding them by index slides points sideways onto each other's clock. A legend for every chart with more than one line, because colour alone is not an encoding a reader can rely on. A flat zero is drawn along the bottom rather than left blank: nothing happening is a real answer, and an empty frame reads as a broken chart. A single sample is not a line and says so instead of putting a dot in the corner. One failed query fails all four panels. Three drawn and one blank invites reading the blank one as "nothing is happening there". NaN is dropped rather than plotted. Prometheus sends it as a string for a sample it has no value for, Go's ParseFloat accepts it happily, and one NaN coordinate is an SVG path that does not draw at all — taking the rest of the line with it. Axis maxima are rounded to a number a person would pick. An axis topping out at 0.8347 makes every gridline unreadable, and the shape is what a chart is for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e034862b79 |
Serve a mirrored model from the list, instead of writing YAML for it
Fetching a model put it on the mirror and left the operator to hand-write a catalogue entry and run `ctl model add` on the control plane — for a source the page was already displaying, with an engine decided by the file extension it had just shown them. A present row now offers to serve it. The form opens on that row with the name, the mirror's own URL and the engine filled in, and asks for the two things nobody can infer: the context pool and how many replicas. What it refuses is the point. A version of several files is refused with the reason. A catalogue entry names one source, so an entry for a sharded model is an entry for one shard, which loads as a broken model rather than failing to load. Nothing said so before — the mirror happily holds a four-shard model that no catalogue entry can describe. Safetensors on a fleet with no GPU is warned about before anything is added. This is the failure we hit today: fifteen gigabytes of shards arrived, and only then did looking at a node show no card, no vLLM and no SGLang. The control plane is not told which engines a node has installed, so the check is on cards, which it does know — a warning rather than a refusal, because a card can be added tomorrow. A name already declared in configuration is refused here too, matching what the model service would say anyway, but before the form is filled in rather than after. Without a store there is no serve action at all: a model added at runtime would not survive a restart, and offering it then refusing is worse than not offering. The mirror page's dependencies became a struct. Three are optional interfaces and two are the same shape, which at a call site is a swap nobody notices until a page shows the wrong thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
ed530133b6 |
Find a model in the registry instead of pasting a URL into a box
Mirroring anything meant knowing a repository existed, knowing which of its thirty files were the weights, and assembling a resolve URL by hand. The mirror page had a text box for that URL and no way to find one. It now searches Hugging Face and lists a repository's weight files with their sizes and published checksums, each with a button that fills the fetch form. Only weight files. A repository is mostly a README and a tokenizer, and listing those buries the four entries somebody came for. The size shown is the LFS size, not the plain one. For an LFS-stored file — every weights file — the plain field is the size of the pointer, so the other one reports a gigabyte model as 133 bytes, and that is the number somebody would then plan disk against. The source URL is pinned to the commit the listing came from rather than to a branch, so what the mirror fetches is what was on screen. A repository re-uploaded under the same branch otherwise leaves two different sets of weights with identical provenance and no way to tell them apart. The version defaults to that revision for the same reason. The registry's own failures are reported on their own line. It is somebody else's server on the other side of the internet, and it being down says nothing about the mirror — which is what the page is for and has to keep working regardless. With no registry configured there is no search box at all, rather than one that can only time out. Gating decodes from a bool or a string, because the field is documented as the first and is the second in practice: strict decoding fails the whole response, so one gated model would hide the other nineteen in a page of results. And a bug this made obvious: downloads were keyed by model and version, so the second shard of a sharded model was refused as "already fetching" until the first had finished. A fifteen-shard model had to be pulled one file at a time, each waiting on the last. Keyed by file now — which is what the single-flight was ever protecting, since the race is two writers on one .part. A version made of several files reports as the worst of them: it is not present until every shard is, and one shard failing makes the version unusable however many others arrived. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
ebce41e22d |
Stop calling the PostgreSQL store SQLite, and stop guessing where it is
Two things that were only ever confusing, and one of them stopped a command working on the cluster. The store type was named SQLite and kept the name when PostgreSQL was added, so OpenPostgres returned a *SQLite and all seventy-odd methods on the production store were spelled after the database it was not running on. Nothing broke; people did. Earlier today I added three methods to *SQLite for a table that on this cluster only ever lives in PostgreSQL, and the name did not once look wrong. It is a field, which is all it ever was. `llmgw users` and `llmgw keys` defaulted --store to a SQLite path. On a deployment whose store is PostgreSQL every one of them failed with "unable to open database file" while the right answer sat in auth.store in the config file beside them; the conclusion an operator draws is that the tool is broken, and they are not far off. They now ask the server's own configuration before falling back, and the error names which of the three sources chose the address — the failure was never the path, it was which one picked it. The single-machine default still applies last, so a deployment with no config file keeps working with no flags. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
cd7eedf09e |
Give the mirror a page, so managing it is not a curl command
/admin/mirror, behind the same admin session as the other admin pages. What is held, what is arriving with its byte count, what failed and why; start a fetch, retry a failed one, remove a version. Not mounted at all without a mirror configured. The right page for a deployment with no mirror is no page, rather than an empty one pointing at nothing. It reloads itself only while something is downloading. A reload at any other moment takes with it whatever is half-typed into the fetch form, and there is nothing to watch when nothing is moving. A failed row offers Retry, not Remove. A fetch that failed left nothing on disk, so the remove could only answer "the mirror holds nothing for this" — a button whose single outcome is a confusing error. Starting the same download again is what somebody looking at that row wants, and the mirror allows it: only a download still running is refused. An unreachable mirror is said outright instead of rendering as an empty table. The two look nearly identical and mean opposite things: one is a mirror waiting to be filled, the other is a machine to go and look at. Dashboard takes NavExtras rather than a second bool. Two adjacent booleans at a call site get swapped silently and are discovered by somebody clicking the wrong link. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
a76ebdb79a |
Run the store on PostgreSQL, chosen by what the address looks like
A path is still SQLite and a postgres:// URL is PostgreSQL, so every existing configuration keeps working untouched and moving is one line changed rather than a new setting to find. An address of the form @/path/to/file reads the real one out of that file, which is how a DSN carrying a password stays out of a configuration that gets copied, diffed and pasted into issues. The driver is pgx, which is pure Go. That is not a preference: the whole binary is built with CGO_ENABLED=0 so it runs on a node whose libc is older than the builder's, and a cgo driver would take that away from every deployment, including the ones still on SQLite. Tested against a real PostgreSQL, which found two things nothing else could. The schema_migrations table is created in Go rather than in a migration file, so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL, and a millisecond timestamp does not fit, so the very first migration failed to record that it had run. And the memory migration uses an FTS5 virtual table with three triggers keeping it in step, which has no counterpart: PostgreSQL derives a tsvector from the row, so there is nothing to keep in step and no trigger to forget. That leaves exactly one query that is not the same sentence in both dialects — the note search, MATCH against @@ — and it is written out twice rather than hidden behind a shared spelling that would mean something subtly different in each. Everything else is one set of queries. The text search uses the 'simple' configuration rather than English. These are an agent's own notes, in whatever language it chose, and stemming English rules over Turkish would quietly stop finding things. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
2eb79f3400 | agenta memory eklendi | ||
|
|
d26a30471e |
Let people forget a conversation, and make retention actually expire things
Two halves of keeping the promise that storing makes.
DELETE /v1/threads/{id} is a client endpoint, not an admin one: the person who
said the thing should be able to unsay it, and making them ask an operator
would mean the operator reads it first. Deleting a thread in another project
answers exactly as deleting one that never existed does — two answers would
let a caller map which ids are real in projects they cannot read, and this one
would do it by trying to destroy them.
And something now runs the pruner. Without it "we keep conversations for 24
hours" was a comment rather than a promise: the rows would accumulate and the
number a project chose would describe nothing. Hourly, and once at startup,
because a gateway that has been down for a week should not wait another hour
before honouring a window that expired while it was off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
9eb321ae3e |
Let a conversation continue on a different model than it started on
The request path half. A caller sets x-llmgw-thread and sends only the new turn; llmgw puts the stored history in front of it, sends that to whichever model is serving, and stores the exchange afterwards. A header rather than a body field because some SDKs drop fields they do not know and all of them let a caller set a header. Capturing the reply is opt-in and bounded. This is the hottest path in the system, so a request without a thread accumulates nothing at all, and one with a thread stops accumulating at a cap rather than growing to it — a gateway that kept every reply in memory would be one long answer away from trouble. A reply is truncated on a rune boundary rather than dropped: a shortened turn replays as a shortened turn, while nothing at all replays as the assistant having said nothing, and invalid UTF-8 would break the next turn's body for good rather than for one turn. The streamed path sets the captured reply on every exit, not only the one that looks like the end. A stream can also finish with the client gone or the engine stalled, and what the assistant said up to that point is still what it said. A thread on a project that keeps no conversations is refused. The e2e test asserted that and got a cheerful 200: the request was served, nothing was stored, and the caller would have discovered it only when the model had no idea what they were referring to — which reads as the model being broken. It now asks whether storage is on before doing anything else. History is spliced after the model is finally settled, because a budget fallback may have changed it and trimming is measured against the context of the model that will actually serve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
28967edab3 |
Serve a spent budget something it can afford
The accounting layer has always exempted a model that costs nothing, with a comment saying it is "what lets a key with no budget left still be served a cheap fallback". That described a caller nobody had written: the budget was checked once, against the requested model, and a refusal was the end of it, so the exemption could never fire. A caller that has run out is now offered the first model in its fallbacks it can still afford. The allowlist still applies — running out of money is not a way to reach a model a key was never allowed — and a model with no fallback is refused exactly as before, or a budget would stop meaning anything. Only a spent budget falls back: a rate limit is answered by waiting a second, and moving the request elsewhere would spend someone else's capacity to avoid a delay nobody asked to avoid. Which uncovered a header that has never worked. x-llmgw-served-model was set in finish, and finish runs after the proxy has written the response — so the one mechanism for telling a client that something else answered has been announcing it to nobody since it was written. Nothing failed, because the only thing asserting it was a comment. It is now set before dispatch, for the capacity fallback as well as this one. Finding that took bisecting a test I had written wrong: it polled for a 429 that never came, because the budget had run out and the fallback had served the request — the feature working, reported as the feature failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
1ea067d08e |
Give metrics a door that does not also open onto pprof
The metrics existed and nothing could record them. They shared a listener with pprof, the management API and the dashboard, and pprof can dump the process heap where API keys sit in plaintext — so a fleet could be monitored or keep its heap private, and not both. That is not a choice anyone should have to make, and it is why every previous answer to "how do I scrape this" was a tunnel. metrics_listen serves metrics and health and nothing else, so it can be reached from wherever Prometheus lives. Metrics stay on the admin listener too: adding a door is not the same as moving one, and anything already scraping keeps working. While there: admin_listen defaulted to :9090, every interface, with a comment beside it saying not to expose it. A default that contradicts its own documentation is a footgun, so it is loopback now and someone who wants otherwise says so. The handler is nil until the registry is built, and says so rather than serving an empty page. Ordering has already caught this exact area once — a collector registered after AdminHandler published nothing at all, silently — and an empty scrape target looks like a working one. deploy/monitoring/prometheus.yml has a starting configuration and the alerts worth having: a node that has stopped reporting, a model that should be serving and is not, a card kept out of placement by a fault, and a fleet holding models nobody is asking for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
7fe59bd5fb |
Load a model when someone asks for it, and stop it when nobody does
The feature the idle tracking was for. A cluster with more models than GPUs can now hold every model in its catalog and only the wanted ones in its cards: min 0 with a max above it means load on demand, and once nobody has asked for idle_timeout the replicas stop and the cards go back. No new field, because min 0 max 1 already reads correctly — keep none running, allow up to one. Which made min 0 max 0 meaningful for the first time: configured and deliberately not running. normalize() used to fill a zero max in, on the reasonable ground that a model with no maximum cannot run; that would now silently turn every switched-off model into an on-demand one, including the ones already written that way in this cluster's own configuration. A model with no scaling stanza stays off for the same reason — defaulting it to on-demand would let any client request start a 70B nobody intended to run. The one real decision is what happens to the request that triggers a load. Refusing makes the feature useless, since the first request for every model fails; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds holds a connection for ten minutes. So it waits briefly and then says what is happening — 503 with Retry-After and "the model is loading", which is a different answer from "no endpoint serves this" and produces different client behaviour. An unknown model still gets the unknown-model error, or on-demand would turn every typo into "loading" and the useful message would disappear. Use keeps a model alive rather than only the request that started it, without which the timer measures time since the load and a busy model is evicted on schedule however hard it is working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
42fd0aa31e |
Export what the cluster is made of, not only what it is serving
The router published twenty-two metric families about requests, endpoints, admission and scoring — everything about traffic and nothing about the machines carrying it. There was no way to graph GPU usage, watch a node go unhealthy, see a fabric fill up, or answer "was the cluster full at the time", which is the first question anyone asks after an incident. The agents exported nothing at all. Exported by the control plane rather than by each agent. It already holds every number: hardware arrives at registration and state on each heartbeat, so this is a map read away. One scrape target for a fleet of hundreds, no Prometheus configuration per node, no firewall rule per machine, and nothing new running on the nodes. On the same /metrics as the traffic figures, because correlating "the cluster was full" with "requests were queueing" should not need two systems. A collector rather than gauges kept current by a ticker: the numbers are derived from state that already exists, so computing them when asked is simpler than maintaining a copy that can drift, and nothing runs between scrapes. GPU utilisation is new, and is the number people actually watch. Memory used says a model is loaded; this says whether it is doing anything, and a fleet at ninety percent occupancy and two percent utilisation is a fleet holding models nobody is asking for. Adding it caught two things. utilization.gpu goes last in the nvidia-smi query and is optional, because a driver too old to know it returns a shorter row — requiring it made an llmgw upgrade silently stop discovering GPUs on an older node, which is far worse than not knowing how busy a card is. And I gave the new protobuf field number 14, which ecc_errors already had; buf lint is why that was a moment rather than a corrupted wire format. Absent is kept distinct from zero throughout, for the same reason as everywhere else here: an unreported utilisation shown as a flat zero reads as an idle fleet rather than a missing measurement, and a replica nothing has reported would be the most idle thing in the cluster. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
1c57a4e378 |
Let the catalog change while the cluster is running
Adding a model was an operator procedure with downtime: edit a file on the control plane, restart it, wait out the settling window, watch replicas churn while adoption catches up. I did that eight times in a day, and one of the restarts cost two minutes of outage while stalled replicas timed out. None of it was necessary. The reconciler already compares desired against actual on every pass, so a new catalog is simply a different desired. A Catalog stays immutable — New is the only way to build one and it refuses an invalid one — and what changes is which immutable catalog is current. Readers take the pointer and finish their operation against it, so a swap mid-request is invisible rather than a race. The router is told rather than asked, the same shape as the endpoint list and for the same reason: a push happens once per change, a pull would happen once per request. The router still seeds from configuration as well as being pushed to. A router with a static endpoint list and no control plane is a supported mode and has nothing to push it a catalog; without seeding it would resolve no model at all. Which turned up a bug that was latent while catalogs were fixed. reconcileModel only ever visits models that are in the catalog, so a replica whose model had been removed was visited by nothing: it kept its GPUs, kept counting, and kept running an engine for a name no client could ask for. The endpoint publisher skipped such a replica with the comment "the reconciler will tear it down", which was not true. It is now, and the comment says what actually happens instead of what was assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
9d8cd5e284 |
Track which replicas are actually being used, and summarise the fleet
Two things a cluster with many nodes and many GPUs needs that were missing, and they are the same thing seen from two distances. Idle tracking. The control plane knew what was placed and whether it was healthy, never whether anyone wanted it — so on a fleet with more models than cards there was no way to answer the first question anyone asks, which is what can I stop. The router is the only thing that knows, so it now reports it: a timestamp written in Acquire, where a request already meets an endpoint under a lock already held, and collected once a tick as a map. That shape is the point. Per-request notification would be a firehose carrying one bit — the newest timestamp wins — and this costs the same on four replicas as on four hundred. It travels through an interface in the direction opposite EndpointSink, for the same reason that one exists: the control plane decides what should run, the router knows whether anyone wants it, and neither should need to be in the other's process to say so. Unknown is kept distinct from idle throughout. A router that has just restarted knows nothing, and a zero timestamp would be the oldest idle replica in every comparison — so one restart would nominate an entire fleet for eviction. Absent from the map, "-" in the table, negative in the JSON, and IdleFor returns a second value rather than a duration anything could act on by mistake. ctl status. A per-node listing is right for four nodes and wrong for two hundred: how full am I, what is broken, what is holding cards nobody is using are aggregates, and reading them off a long table is a chore, which means it gets skipped. Capacity per GPU class with usage bars, free whole nodes per fabric — the number that decides whether a multi-node replica can land, since free cards on the wrong fabric are not capacity for a model that spans nodes — replica and model counts, models that cannot be placed, and what has gone unasked-for. Derived from the same state the tables show, so it cannot disagree with them. Unhealthy GPUs count as neither used nor free, because counting them free is how a cluster looks like it has room it does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c115681259 |
Meter a project nobody set limits on
Unlimited was the default. Creating a project without limits is the easy path and the common one — the "default" project on this cluster is one, and the person using it is metered by nothing at all. The first sign of that is the bill or the outage. auth.default_rpm and default_tpm now apply to a key no scope gives a limit to. Zero keeps the old behaviour, which is still a reasonable choice; the point is that it becomes a choice rather than an omission. Applied after the chain rather than folded in as its widest scope, because a default is a fallback and not a ceiling: a project deliberately given 600 rpm must keep it, and tightening would silently cap it at the default. An explicit limit wins in both directions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
db448a9e87 |
Stop using a liveness timeout as a generation budget
response_header sounds like a check that the endpoint is alive, and for a streaming request it is: headers arrive before generation starts. For a non-streaming request the engine sends nothing until the completion is finished, so the same setting is the total generation budget wearing a liveness timeout's name. At 60s that was invisible on GPUs and a hard cap of roughly a hundred tokens on qwen2.5-coder-14b at 1.7 tokens a second. A request for 120 tokens failed at exactly one minute, and neither the setting's name nor the error said why. Two transports now: the short header timeout for streaming, where it still catches a sick endpoint, and a generation budget for non-streaming, defaulting to ten minutes. One number could not mean both things. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
082d52bc97 |
Stop shipping a placeholder that looks like a value, and say what is served
A user copied the curl from the profile page and sent it as it stood. Every other field in that snippet is filled in for them — the base URL, the key they had just been issued — so "model": "MODEL" reads as a value rather than a blank, and they got: no endpoint serves "MODEL" which is true and useless. It reads the same whether the name was a typo, a placeholder left in from the documentation, or a real outage, and it never says what would have worked. The message now lists what is being served, capped at ten with a count for the rest, and says plainly when nothing is. That list is what /v1/models already returns to the same caller, so it discloses nothing new. The profile page's example now names a model that is actually running, real ones before simulated — the point of an example someone can run is that it runs, and sending a first request to the sim engine would answer in Greek letters. The landing and claim pages keep a placeholder, because they are served without a key and deliberately say nothing about what is here; theirs is <model-name>, which cannot be mistaken for a value, with a line saying where to get the real one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
a075ff4120 |
Add approximate prefix routing, and find out it did nothing
Tier 1 of plan §7.5: a radix tree of request prefixes that estimates which replica has already prefilled one. Bytes, not tokens — needing the engine's exact tokenizer is what makes tier 2 a separate project. It went in inert, and the decision metric is what caught it: 144 lookups, zero hits. The cause was a rule I had written and tested for. A node's label is the first request that created it, so the second request for the same system prompt diverges in its last few bytes — a new question after the same preamble, the single most common shape there is. Requiring the whole label to match threw away the 700 bytes they shared. What matters is the shared head, because that is what the engine cached, and a partial label match now counts. Two unit tests asserted the wrong rule and have been inverted. With that fixed, measured end to end against engines whose caches are too small for the working set: 22.5% cache hits without, 81.7% with, and the breakdown shows the misses are exactly the first round, when the tree is empty. The scenario matters as much as the number. With a handful of prompts and a large cache, scattering is free — every replica ends up holding everything — and prefix routing can only add constraint. The first version of the test measured that case and showed a loss, which was a true statement about a situation the feature is not for. Two things keep it from being the disaster prefix routing usually is. The imbalance guard, which needs both an absolute and a relative gap before it fires, because either alone misfires — absolute on an idle cluster, relative on a busy one. And it is a scorer rather than a filter: affinity is a vote the queue and inflight scorers can outvote, not a rule. The tree records the replica the picker actually chose, not the one the scorer proposed, and a node prefers the replica it has seen most rather than most recently — a count is stable under a randomizing picker, where recency just chases whatever the picker last did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e519dc587c |
Serve the cluster dashboard over the web, without serving pprof with it
Reaching it meant an SSH tunnel, because the page lived on the admin listener and that listener is bound to loopback. Opening the port was never the way to fix that: it also serves pprof, and pprof can dump the process heap, where every API key sits in plaintext. Exposing the port would have been exposing the key store. So the page moves rather than the port. The same dashboard is now mounted at /admin on the client-facing port, behind the session and CSRF machinery that already guards /profile, gated on a flag on the account. The admin listener keeps pprof and metrics and stays on loopback. One bit, not a role system: it grants looking. Everything that changes the cluster or issues a credential stays on the CLI, where it needs a shell rather than a session — which keeps the worst case of a stolen admin cookie to "saw the cluster" rather than "drained your fleet". Someone without the flag gets 404, not 403. A 403 confirms the page is there and that admin accounts exist here, which is free reconnaissance; 404 is what every other unknown path returns. The profile page does not show a link to a door the visitor cannot open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
216fc9bb03 |
Let people sign up, be approved, and issue their own keys
Every key needed an operator with a shell. This adds the login system that made that necessary, which is the thing I argued against building until it was asked for — so it is built as if it will be attacked, because a public login form is. Signing up produces an account that can do nothing. Approval is what makes it work and stays on the CLI: having a shell on the control plane is what makes someone an administrator, and that is a smaller thing to get right than roles and an admin UI that mints credentials. The details that matter: Passwords are argon2id, where a slow hash is correct — the input is chosen by a human and has a dictionary, which is the opposite of an API key. Both kinds of authentication now live in this codebase and they are in separate packages so neither set of assumptions leaks into the other. Length is the only rule: composition rules push people towards Password1! and cost more entropy than they buy. Signup never says whether an address already has an account, and sign-in answers the same way for a wrong password and a missing account — verifying against a dummy hash when there is no user, so a stopwatch cannot enumerate accounts. Account state is checked after the password, not before, so guessing an address reveals nothing about it. Guessing is bounded per address rather than per IP: an attacker has many addresses to come from and one to guess against, and limiting by IP locks out an office behind one NAT. Sessions are server-side, so signing out and suspending take effect immediately rather than whenever a token expires; suspension is checked in the same query that resolves the session, because a second round trip is a check somebody eventually forgets. Every state-changing form carries a CSRF token verified against a hash in the session — a cookie authenticates the browser, not the page that asked. Key ownership is enforced in the UPDATE rather than by reading first, so a guessed key id from another account changes nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
8d35faf6f8 |
Give the landing page somewhere to send people
The page explained the API, said keys come from an operator, and left the reader there. Someone who reaches the address without a key had no route to getting one, which is where onboarding stopped. `landing.contact` is that route, because the gateway cannot guess who to ask. An address becomes a mailto link, an http(s) URL stays a link, and anything else — a channel, a queue, a person — is shown as text. Other schemes are never linked: this string comes from configuration, configuration is not always written by whoever runs the gateway, and a javascript: href on a page served to every visitor is a short path to a stored XSS. Escaping the text does not make a scheme safe. Unconfigured, it now says something rather than nothing, and the page explains that a key arrives as a one-time link so the reader knows what to expect before it lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
0b93fb2883 |
Enforce budgets and rate limits, and write usage off the request path
The second half of phase 5. A key now has a cost, a quota and a bill. Rate limits use a sliding window rather than a fixed one. A fixed window lets a caller send its whole minute's allowance in the last second of one window and again in the first second of the next, so the observed rate is twice the configured limit exactly when the system is least able to take it. Checking and charging are one operation, or two concurrent requests both see room and both take it. Token limits are checked against an estimate, because the real count only exists once the request has run; the difference is charged afterwards, so a caller who consistently underestimates is still bounded, one window late. A request with no max_tokens gets a modest assumption rather than zero — assuming zero would let an unbounded request through every token limit. Limits are checked before admission, not after: a request over its quota should be refused rather than queued, where it would occupy a band's capacity to no purpose. All refusals are 429, but a spent budget carries no Retry-After because waiting does not help until the period rolls over, and saying otherwise turns one client into a retry loop. Usage goes to a buffered channel and is written in batches. Record never blocks: if the buffer fills — a database stalled long enough to lose eight thousand records — records are dropped and counted, because losing accounting is bad and turning a stalled database into an outage is worse. A clean shutdown drains what is queued; those are requests that were already served. Failures are logged at error, since silently losing billing data is the kind of thing discovered at the end of a month. Requests are accounted whatever their outcome. A request that used a GPU and then failed still used it, and a key that could spend nothing by failing would be a way around a budget. Budgets are counted in memory and restored from the store at startup, so a restart does not hand every key a fresh allowance. The budget period travels with the reservation rather than being re-derived, because a monthly budget credited to a daily counter is a budget that never binds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
45c8cd23a3 |
Close the gateway: API keys, tenancy and model allowlists
Until now anything that could reach the client port could use every model in the catalog. A key now resolves to an identity, and that identity decides what it may do. Authentication wraps the whole mux rather than checking inside each handler, so adding a route cannot add an unauthenticated one by omission. Health stays open, because a load balancer has no key and a health check that needs one reports the gateway down. Nothing on the request path reads the database. A key is resolved once and answered from memory until the entry expires; rejections are cached too, so a client looping with a bad key cannot turn into a database read per request. The price is that a revocation takes up to cache_ttl to take effect, which is the trade stated in the configuration next to the knob. A store failure is not an authentication answer: it is never cached as one and returns 503, not 401 — sending someone hunting for a key problem that does not exist wastes the one thing an incident is short of. A revoked key is refused with the same message as an unknown one, because saying otherwise confirms the key was real. With auth on the band comes from the key and the band header is ignored. The header was trusted only while there was nothing to consult, and it let any client promote itself into the top priority band. Tenant fairness within a band is now per key rather than per session, so one caller cannot take a band by opening many sessions. Allowlists are decided per model, not per string: a model's canonical name and every alias name the same thing, so either form works in an allowlist and no alias can reach a model that was not allowed. The first version of that test passed without the resolution being there at all, which is why it now checks all four combinations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
501db3d3e8 |
Add the llama.cpp engine, and let a profile ask for no GPU
llama.cpp is what makes a cluster with no cards useful: it serves the same OpenAI API on CPU, so the whole path — placement, supervision, scraping, scheduling, admission — can be exercised end to end on ordinary machines. Three things had to give way for it. A profile may now omit its GPU class, and that is how it says it runs on the CPU. Only engines that can actually serve without a card may omit it; for vLLM an omitted class stays an error, because otherwise a typo becomes a replica placed on a machine with no hardware and diagnosed minutes later during model loading. Placement spreads CPU replicas instead of bin packing them, since unlike GPUs there is no allocation to stop two of them sharing a machine's cores. Metric mappings are now per engine rather than per cluster. llama.cpp calls the same three numbers requests_processing, requests_deferred and kv_cache_usage_ratio, and a single global mapping would leave half a mixed fleet unmeasurable. An empty mapping still defaults to vLLM's names; one that names its own Running metric is taken as a complete statement, so the fields it leaves blank stay blank rather than being filled with names that engine cannot have. The adapter refuses multi-node and tensor-parallel profiles. llama.cpp splits across local GPUs by a different mechanism than this adapter drives, and accepting either would produce a replica that starts and then serves from one machine while the catalog believes it has several. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
2373726efd |
Agent stream: connect the fleet to the control plane
One bidirectional connect-go stream per node, carrying registration, heartbeats and shard status up, and start, stop and drain down. The agent dials out: GPU nodes sit behind firewalls far more often than a control plane does, and an outbound connection needs no inbound rule on every node in the fleet. - registration carries the whole hardware inventory, so there is no separate discovery call and a reconnecting agent refreshes what the scheduler knows. The fabric labels travel with it; they cannot be inferred from the wire, so losing them in transit would silently disable multi-node placement. - agentsvc is both the stream handler and the reconciler's dispatcher, because it holds the open streams and is therefore the only thing that can deliver a command to a node. - the agent validates a profile before launching anything, since an engine that refuses a profile does so after minutes of loading, then supervises the process and reports every transition with the engine's output tail on failure. - reconnects forever with jittered backoff. A control plane restart must not require touching every GPU node, and a fleet must not reconnect in lockstep and knock it over again. Engines keep serving throughout, because the router talks to them directly. llmgw agent is real now, and llmgw server runs a control plane whenever any model declares a profile. The static endpoint list stays for router-only runs. Verified with real binaries as separate processes: the agent registers, a replica is placed, an engine launches through the vLLM adapter's actual argument construction, streaming traffic is served, and killing the agent moves the node through unreachable to lost with placement explaining why nothing can run. config.Model and catalog.Model were the same thing described twice, so the router now reads catalog entries directly and the validation rules live in one place. Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted network until that lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c1b9ea1bc2 |
Phase 3 (3/3): wire the control plane to the router
Ready replicas become router endpoints with no static configuration involved. Endpoints that survive an update keep their live state — inflight counts, breaker state, drain rate — so a replica that was already serving does not start from cold because another one moved. The sink is an interface, so when the router and control plane split into separate processes it becomes a watch stream and nothing above it changes. Phase 3's exit criterion now holds end to end. A node hosting a live replica is killed while four clients stream continuously; the control plane marks it unreachable, then lost, rebuilds the replica on a survivor and republishes. Across three runs ~370 requests completed with zero failures and no replica was left on the dead node. This proves the control logic, not the transport: the agent fleet is simulated and the real path still needs the stream. The race detector caught a genuine bug while this went in. startReplica read the replica's fields after publishing it to the map, and scaleUp walked replica pointers outside the lock; both raced with agent status reports, which arrive from another goroutine and mutate exactly those fields. Against the simulated fleet the reports came back fast enough to hit the window on every run. State is snapshotted under the lock now and those functions work from values. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
dc38dd6096 |
Phase 1.5: admission control
Inserted between quota checks and scheduling: admission answers when a request may proceed, the scheduler still answers where it goes. Keeping them apart is the point — an engine queue is FIFO and knows nothing about tenants, so once a batch job is in it, no endpoint selection rescues the request behind it. - saturation detector: one number per pool, the worse of queue depth and KV occupancy, since either alone misleads. - priority bands with per-band saturation ceilings, so the cluster degrades from the bottom up. - tenant round-robin within a band: one caller sending a thousand requests takes one slot in the rotation, not a thousand. - bounded queues with capacity (503) and TTL (504); cancelled requests leave the queue instead of holding capacity. - metrics for queue depth, wait time, rejections and pool saturation. Exit criterion, measured in test/e2e: with 500 concurrent bronze requests running continuously, interactive p99 in the platinum band stays at 1.04-1.24x idle against a 1.5x budget. With admission off the same load pushes it to 55-60x. Both directions are asserted, so the test fails if admission regresses and also if the load generator stops being a real load. Two lags had to be closed, and they needed different answers. Engine metrics are scraped and lag an interval; router inflight does not, but it counts running requests, and treating those as queue makes a healthy engine look overloaded every interval. What is actually queued is inflight minus batch capacity, and capacity is not in the metric contract, so it is learned as the high-water mark of observed running. Separately, between admission granting a request and the router dispatching it the request is invisible everywhere, so a simultaneous burst passed the check en masse; the controller now counts its own outstanding leases against dispatched inflight. One plan correction: a saturation ceiling of 1.0 was documented as "never held back". It is not — it means "flow up to the healthy limit". A band that must never wait needs a ceiling above 1. Plan §7.2.2 updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
a3f71aa5ee |
Phase 1: the router serves traffic
llmgw server is real now. It fronts a static set of engine replicas with an OpenAI-compatible API, scoring endpoints through the pipeline added in the previous commit. - proxy: SSE relayed event by event with a flush after each, cancellation propagated upstream, and separate deadlines for dial, headers, first token, inter-token idle and total. The idle watchdog is what distinguishes a wedged engine from a slow one. - router: dispatch with retry and model fallback. Retry only happens while nothing has reached the client; after that, splicing two responses together would be worse than the short stream the client already sees. - config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides. Validation rejects duplicate endpoint ids and aliases that shadow a model, because both misroute silently at request time rather than failing loudly. - metrics: requests, duration, TTFT, tokens, plus per-scorer values and per-filter drops on a separate admin listener with health and pprof. Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across 3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV blocks on every engine; router overhead p99 is about 0.9 ms against the plan's 2 ms budget. The handlers live in internal/router rather than an internal/router/openai package as the plan's tree sketched: they need the dispatch loop, and splitting them apart would have meant an interface with exactly one implementation. Plan §4 updated to match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |