bf2d171e8941bbff7cbbfba078a5ceb18c388f9e
31
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
f68fbf7c49 |
Check llmgw's engine flags against real vLLM and llama.cpp
The launch arguments had been read from documentation and never from an
engine. Now they have been:
vllm 0.29.0 --host --port --tensor-parallel-size --served-model-name
--distributed-executor-backend all accepted
llama.cpp b10991 --model --host --port --metrics --alias --parallel
--n-gpu-layers all accepted
Getting there found three faults in the checker rather than in llmgw.
It checked every command's flags against the engine's help, but a shard can be
several programs — vLLM under Ray is `ray start` and then `vllm serve` — and
they have different vocabularies. It reported --head and --num-gpus as
missing, which is true of vllm and irrelevant, because ray is what receives
them. Each command is now checked against its own help, and one that cannot be
run is reported as skipped rather than failed.
vLLM does not give up its whole help for --help: it prints a summary of
section names and expects --help=all. Checked against the summary, all five
flags looked missing. The warning about truncated help did not fire, because
the summary happens to mention --help= while listing how to ask for more.
And vLLM refuses to print anything at all on a machine with no accelerator
unless VLLM_TARGET_DEVICE says what to assume. Asking an engine about its
flags should not require the hardware it runs on.
Two things remain unchecked and are written down as such: the Ray sidecar,
because vLLM 0.29 does not install Ray, and SGLang, because it is not
installed anywhere. `launch_mode: mp` avoids Ray entirely and is worth
preferring until someone checks it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
5b425c32e5 |
Ship binaries with releases, and ask engines whether our flags still exist
A release was a git tag. Installing llmgw on a node meant having a Go toolchain, and the onboarding script copied a binary from wherever it was run — which works and quietly makes whoever runs it the build server, with the fleet ending up on whatever was in their working tree. Tags now build static linux/amd64, linux/arm64 and darwin binaries and attach them. CGO_ENABLED=0 is the point: it has to run on a node whose libc is older than the builder's, and llmgw has no C dependency because modernc.org/sqlite is pure Go precisely so this stays true. Verified: "not a dynamic executable", runs on a node, reports its version. `ctl engine-check` closes the other half of the phase 4 gap. llmgw's launch arguments are read from each engine's documentation, and engines rename flags between versions; when that happens the failure arrives minutes into a model load, in the engine's own error message, on a node nobody was watching. It builds the widest command llmgw would ever produce — multi-node, since a flag that only appears there is the one nobody notices has gone — and checks each flag against the engine's own --help. No GPU: a CLI prints its help anywhere. Verified against llama.cpp b10991, all seven flags accepted. Flags are matched on a word boundary, so --port is not satisfied by --port-range. Without that, a renamed flag would sail through the check that exists to catch renamed flags. It also warns when the help text looks truncated, which is how the first run went: llama.cpp's help cut to 400 of its 725 lines reported five flags as missing that the running cluster uses. TestSidecarRunsBeforeEngine was flaky under a full run, and worse than flaky — its wait loop fell through on timeout, so a slow machine turned the ordering assertion into a no-op and the failure surfaced later as something else. It now asserts, with a deadline that suits a machine running every other package at the same time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c97b671ef9 |
Prove the multi-node path on a simulated cluster
Phase 4 had been written and never run. The hardware is not coming, so the cluster is simulated instead — four nodes across two fabrics, eight GPUs each, built with the onboarding scripts from the previous commit. `sim` is now an engine adapter, not just a standalone command. Its leader binds a rendezvous port and does not serve until every worker has joined; its workers serve nothing at all. That is the part stub processes could never exercise, and it is why this path survived four phases untested: the command line was checked, the sequence was not. Measured on the cluster: - a 2-node replica on each fabric, node01+node02 on ib-a and node03+node04 on ib-b, four GPUs from each - a request served through one of them - only the leader listening — node02 had no port open at all, which is ServesHTTP working end to end rather than in a unit test - node02 killed; at t+60s it was declared lost, the ib-a replica failed and was reaped; at t+80s a replacement was ready on ib-b, the fabric that could still host it, while node01 sat idle rather than being paired across fabrics The invariant held under the one condition that tests it: when a replica could not be rebuilt where it was, it moved to a fabric that could hold all of it rather than spanning two. What this does not prove is written into the plan next to what it does. The GPUs are an inventory, the fabric is a sysfs-shaped directory, and the engine speaks no NCCL — so the launch sequence is now exercised while the launch arguments for vLLM and SGLang are still read from documentation rather than from a running engine. docs/node-onboarding.md is the procedure, including how to run all of this on machines that have neither GPUs nor InfiniBand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
a075ff4120 |
Add approximate prefix routing, and find out it did nothing
Tier 1 of plan §7.5: a radix tree of request prefixes that estimates which replica has already prefilled one. Bytes, not tokens — needing the engine's exact tokenizer is what makes tier 2 a separate project. It went in inert, and the decision metric is what caught it: 144 lookups, zero hits. The cause was a rule I had written and tested for. A node's label is the first request that created it, so the second request for the same system prompt diverges in its last few bytes — a new question after the same preamble, the single most common shape there is. Requiring the whole label to match threw away the 700 bytes they shared. What matters is the shared head, because that is what the engine cached, and a partial label match now counts. Two unit tests asserted the wrong rule and have been inverted. With that fixed, measured end to end against engines whose caches are too small for the working set: 22.5% cache hits without, 81.7% with, and the breakdown shows the misses are exactly the first round, when the tree is empty. The scenario matters as much as the number. With a handful of prompts and a large cache, scattering is free — every replica ends up holding everything — and prefix routing can only add constraint. The first version of the test measured that case and showed a loss, which was a true statement about a situation the feature is not for. Two things keep it from being the disaster prefix routing usually is. The imbalance guard, which needs both an absolute and a relative gap before it fires, because either alone misfires — absolute on an idle cluster, relative on a busy one. And it is a scorer rather than a filter: affinity is a vote the queue and inflight scorers can outvote, not a rule. The tree records the replica the picker actually chose, not the one the scorer proposed, and a node prefers the replica it has seen most rather than most recently — a count is stable under a randomizing picker, where recency just chases whatever the picker last did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
de21649ad6 |
Add an in-browser chat, without putting a key in the browser
So someone can try a model without setting up a client, and — more often — tell whether a problem is the gateway or their code. The page carries a session, and the server turns that into the identity an API key would have produced, resolved from the person's project. The request then goes down the ordinary path by being handed to the same handler /v1 uses: admission, model allowlist, rate limits, budget, accounting. There is exactly one request path, so there is no second place for its rules to drift — and without that, a playground is a hole in every limit the rest of the gateway enforces. The allowlist test posts a forbidden model and expects 403 for that reason. The body is rebuilt rather than forwarded, so the page cannot set fields it has no business setting: a stream:false that would hang the reader, a max_tokens large enough to occupy a replica for an hour. One turn is capped at 512 tokens. The picker offers what is actually served and allowed, not what is catalogued, because offering a model with no replica produces a 503 the reader cannot interpret. Inline vanilla JavaScript, for the same reason the charts are hand-drawn SVG. It streams, because on CPU inference ten seconds is normal and a chat that hangs and then dumps a paragraph feels broken. Replies are written with textContent throughout: model output and user input have no business being parsed as markup. Its POSTs carry the CSRF token in a header, which a cross-site form cannot set at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e519dc587c |
Serve the cluster dashboard over the web, without serving pprof with it
Reaching it meant an SSH tunnel, because the page lived on the admin listener and that listener is bound to loopback. Opening the port was never the way to fix that: it also serves pprof, and pprof can dump the process heap, where every API key sits in plaintext. Exposing the port would have been exposing the key store. So the page moves rather than the port. The same dashboard is now mounted at /admin on the client-facing port, behind the session and CSRF machinery that already guards /profile, gated on a flag on the account. The admin listener keeps pprof and metrics and stays on loopback. One bit, not a role system: it grants looking. Everything that changes the cluster or issues a credential stays on the CLI, where it needs a shell rather than a session — which keeps the worst case of a stolen admin cookie to "saw the cluster" rather than "drained your fleet". Someone without the flag gets 404, not 403. A 403 confirms the page is there and that admin accounts exist here, which is free reconnaissance; 404 is what every other unknown path returns. The profile page does not show a link to a door the visitor cannot open. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
555e4678cb |
Ship the systemd units, and stop the plan lying about what is done
Phase 3 had a block duplicated into itself, including an unchecked 'EndpointSource becomes dynamic' sitting directly under the checked line saying it was wired. Several phase 5 items had shipped without being ticked. The units were only ever on the machines and in a scratch directory, which makes them undocumented operational knowledge rather than part of the project. KillMode=process in particular is not obvious and not optional: without it systemd takes the engines down with the agent, turning a thirty-second upgrade into a model reload on every node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
79e459d9ff |
Let people change their own password, and operators reset a lost one
The account system shipped without either, which made the first forgotten password an unanswerable support request. Changing it requires the current one. A session is what an attacker gets from a borrowed laptop or a stolen cookie; the password is what they do not have, and without the check the first thing they would do is take the account. Every session is closed, including the browser doing the changing — leaving the others open would mean a password change does nothing about the reason someone is changing it. A new session is then opened for the browser in front of us, so the person who just proved their password is not thrown back to the sign-in page. That response also carries a new CSRF cookie, so the forms it renders take the new token rather than the one the request arrived with; there is a test that posts a form from that exact page. Guessing the current password is bounded the same way sign-in is, because someone holding a stolen session should not get unlimited attempts at the password that would let them keep it. There is no self-service reset: this gateway sends no email, and a reset link with nowhere to send it is not a reset. `llmgw users reset-password` is the operator's answer, and it closes their sessions too — if someone lost their password, the reason might be that somebody else has it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
216fc9bb03 |
Let people sign up, be approved, and issue their own keys
Every key needed an operator with a shell. This adds the login system that made that necessary, which is the thing I argued against building until it was asked for — so it is built as if it will be attacked, because a public login form is. Signing up produces an account that can do nothing. Approval is what makes it work and stays on the CLI: having a shell on the control plane is what makes someone an administrator, and that is a smaller thing to get right than roles and an admin UI that mints credentials. The details that matter: Passwords are argon2id, where a slow hash is correct — the input is chosen by a human and has a dictionary, which is the opposite of an API key. Both kinds of authentication now live in this codebase and they are in separate packages so neither set of assumptions leaks into the other. Length is the only rule: composition rules push people towards Password1! and cost more entropy than they buy. Signup never says whether an address already has an account, and sign-in answers the same way for a wrong password and a missing account — verifying against a dummy hash when there is no user, so a stopwatch cannot enumerate accounts. Account state is checked after the password, not before, so guessing an address reveals nothing about it. Guessing is bounded per address rather than per IP: an attacker has many addresses to come from and one to guess against, and limiting by IP locks out an office behind one NAT. Sessions are server-side, so signing out and suspending take effect immediately rather than whenever a token expires; suspension is checked in the same query that resolves the session, because a second round trip is a check somebody eventually forgets. Every state-changing form carries a CSRF token verified against a hash in the session — a cookie authenticates the browser, not the page that asked. Key ownership is enforced in the UPDATE rather than by reading first, so a guessed key id from another account changes nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
3ed211b2f4 |
Onboard users with one-time links instead of pasted keys
Issuing a key worked; getting it to the person it was issued for did not. The operator had a secret on their terminal and no way to send it that did not leave a bearer credential in a chat log forever. `keys create --claim` prints a link instead. GET shows a button and consumes nothing; POST reveals the key and burns the link. That split is not politeness — chat clients, mail scanners and link previewers fetch URLs the moment they are posted, so a GET that consumed the claim would mean the first thing to unfurl the link destroys the key before anyone reads it. A pending claim has to hold the key somewhere, which would otherwise put a working credential in the database and undo what hashing bought. So the key is sealed with a key derived from the link's own token, and the token is never stored — only its hash. A database leak yields two useless hashes and a blob nothing in the database can open. HKDF rather than the raw token, so the value used for lookup can never become the value that decrypts. Consuming is one UPDATE moving claimed_at off NULL, so two people opening the same link produce one key rather than two copies. A used link, an expired one and a token that was never real all render identically, because saying which confirms that a token was once valid. The page is no-store, no-referrer and noindex: the URL is itself the credential. `GET /me` reports a key's own limits and spend, so nobody has to ask an operator to read a dashboard they cannot reach. Outside /v1, because it is not part of the OpenAI API. docs/onboarding.md is written for whoever operates this, including the parts that are deliberately absent — no self-service, no SSO, and why. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
0b93fb2883 |
Enforce budgets and rate limits, and write usage off the request path
The second half of phase 5. A key now has a cost, a quota and a bill. Rate limits use a sliding window rather than a fixed one. A fixed window lets a caller send its whole minute's allowance in the last second of one window and again in the first second of the next, so the observed rate is twice the configured limit exactly when the system is least able to take it. Checking and charging are one operation, or two concurrent requests both see room and both take it. Token limits are checked against an estimate, because the real count only exists once the request has run; the difference is charged afterwards, so a caller who consistently underestimates is still bounded, one window late. A request with no max_tokens gets a modest assumption rather than zero — assuming zero would let an unbounded request through every token limit. Limits are checked before admission, not after: a request over its quota should be refused rather than queued, where it would occupy a band's capacity to no purpose. All refusals are 429, but a spent budget carries no Retry-After because waiting does not help until the period rolls over, and saying otherwise turns one client into a retry loop. Usage goes to a buffered channel and is written in batches. Record never blocks: if the buffer fills — a database stalled long enough to lose eight thousand records — records are dropped and counted, because losing accounting is bad and turning a stalled database into an outage is worse. A clean shutdown drains what is queued; those are requests that were already served. Failures are logged at error, since silently losing billing data is the kind of thing discovered at the end of a month. Requests are accounted whatever their outcome. A request that used a GPU and then failed still used it, and a key that could spend nothing by failing would be a way around a budget. Budgets are counted in memory and restored from the store at startup, so a restart does not hand every key a fresh allowance. The budget period travels with the reservation rather than being re-derived, because a monthly budget credited to a daily counter is a budget that never binds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
45c8cd23a3 |
Close the gateway: API keys, tenancy and model allowlists
Until now anything that could reach the client port could use every model in the catalog. A key now resolves to an identity, and that identity decides what it may do. Authentication wraps the whole mux rather than checking inside each handler, so adding a route cannot add an unauthenticated one by omission. Health stays open, because a load balancer has no key and a health check that needs one reports the gateway down. Nothing on the request path reads the database. A key is resolved once and answered from memory until the entry expires; rejections are cached too, so a client looping with a bad key cannot turn into a database read per request. The price is that a revocation takes up to cache_ttl to take effect, which is the trade stated in the configuration next to the knob. A store failure is not an authentication answer: it is never cached as one and returns 503, not 401 — sending someone hunting for a key problem that does not exist wastes the one thing an incident is short of. A revoked key is refused with the same message as an unknown one, because saying otherwise confirms the key was real. With auth on the band comes from the key and the band header is ignored. The header was trusted only while there was nothing to consult, and it let any client promote itself into the top priority band. Tenant fairness within a band is now per key rather than per session, so one caller cannot take a band by opening many sessions. Allowlists are decided per model, not per string: a model's canonical name and every alias name the same thing, so either form works in an allowlist and no alias can reach a model that was not allowed. The first version of that test passed without the resolution being there at all, which is why it now checks all four combinations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
29617f8cb1 |
Add the store: tenancy, keys and usage on SQLite
Phase 5 begins. This is the persistence layer for the things the cluster cannot reconstruct — who may call, what they are allowed, what they used. Everything else llmgw knows is still derived from the fleet, which cannot be wrong about itself the way a database can (§6.5). SQLite behind an interface. The plan justified that by write volume, which was the wrong argument: the write path is batched anyway, and a 500-row transaction every 200ms is nothing for either engine. The real argument is the failure mode. Postgres is a second service, and a gateway that stops serving because its accounting database is unreachable is worse than one that buffers some accounting. Embedded means the store shares the gateway's fate instead of adding an independent way to be down. That reasoning stops holding at two instances, where budgets have to be counted somewhere shared — so the interface is here from the start, the SQL is portable, and migrations are per dialect with shared version numbers. Keys are stored as sha256 and nothing else, so a database leak hands over no working credential. The fast hash is deliberate: the input is 256 random bits with no dictionary to attack, so argon2 would buy nothing and charge tens of milliseconds per request. That assumption is written down next to the code that depends on it. Two things the schema now says that it did not: budgets are integer micro-units, because repeated addition of floats drifts; and an empty model allowlist forbids everything while an absent one permits everything, which the resolver has to keep distinct. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
41cf894c72 |
Tick the phase 1 and 1.5 boxes their releases already earned
The code shipped in v0.2.0 and v0.3.0 and both exit criteria were measured; only the checkboxes were left behind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
f84e3b642a |
Record the llama.cpp work in the plan and changelog
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
f19db242ae |
Stop naming a metric llama.cpp does not have, and say so when it happens
Deployed against real llama.cpp, every scrape failed. The mapping named llamacpp:kv_cache_usage_ratio; build b10991 serves fifteen metric families and none is a cache ratio. Because KV usage was mandatory, that one absent name threw away the two metrics the engine does publish, and the router scheduled to all three replicas blind while the cluster looked healthy. A mapping that leaves a metric unnamed now means the engine does not publish it, and the field stays zero. Naming one the engine does not serve is still a violation, so this does not become a way to tolerate a typo. Running and waiting can never be omitted: an engine that reports neither what it is doing nor what is queued cannot be scheduled against at all. For llama.cpp this costs the KV half of the saturation signal, which is a real loss softened by its fixed slot count — queue depth already says when it is full. The second half is why it took a deployment to notice. A contract violation was logged per scrape at debug level, so at the default level nothing said anything. It is now a warning, once, with a matching line when it clears. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
ec1b6b04df |
Fix an agent that could neither explain itself nor stop
Two bugs found while verifying mTLS against real machines. session cancelled its own derived context and then asked that context whether the agent was shutting down. The answer was always yes, so every stream error read as a clean close: the log said err=<nil>, the backoff reset, and an agent whose certificate the control plane refused hammered it forever at the base interval with nothing to diagnose. It now keeps the parent context to answer that question, and a refusal reaches the operator naming both the certificate and the claim. Cancelling the context also did not stop the agent. HTTP/2 watches the context while the request body is being written, and a bidirectional stream spends its life blocked on the read side instead, so the read never returned. Measured on a deployed node: `systemctl stop llmgw-agent` took 90 seconds and ended in SIGKILL, on every restart. Closing the response when the context ends is what actually unblocks it. TestAgentStopsWhenContextEnds hangs reliably without the second fix, which is the only reason to trust that it tests it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e1eb1d4149 |
Phase 3: ctl, and make draining a rolling move
ctl talks to a management API on the admin listener, not the client-facing one. The two have different audiences: one serves inference to whoever holds an API key, the other lets a caller cordon nodes and read the cluster's shape. - nodes surfaces unhealthy GPUs and why they are excluded. That is usually the answer when a node looks idle but nothing will schedule onto it, and it was invisible before. - explain-placement prints every candidate considered and why it was rejected. - replicas shows the shards a replica is made of, with per-shard errors. - cordon, uncordon and drain are one endpoint with different targets, because the difference between cordon and drain is exactly what an operator needs to be sure of before running either. Draining had to be fixed first to make ctl drain worth having. A drained replica was stopped on the next pass, which cuts in-flight requests — the thing draining exists to avoid. Now a node marked Draining has its replicas drained, the model falls below its desired count, a replacement is placed elsewhere, and the original is stopped only once the replacement is serving. A grace period bounds it, because draining the only node that can host a model would otherwise never finish and the node could never be taken out of service. Verified against a two-node cluster of real binaries: draining moved a replica to the surviving node and traffic kept returning 200 throughout. ctl logs is deliberately absent. It needs a request/response pair the agent protocol does not have, and the failure path already ships an engine log tail with every status report — so the case that matters most is covered, and reading a healthy shard's output on demand is not worth a protocol change yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
1d1a6aebb0 |
Phase 4: make a multi-node replica able to start
Placement and environment generation had landed early, which made multi-node look further along than it was. A replica spanning nodes could not come up at all, and all three reasons came down to assumptions that only hold on one node. Not every rank serves HTTP. The agent health-checked every shard at its own port, but only the leader answers; workers timed out, reported Failed and took the replica with them. Engine.ServesHTTP now says which ranks serve, and a worker is judged by staying alive. That is deliberately weaker evidence: the replica's real readiness is the leader answering, which only happens once every worker has joined. A shard is not always one process. Engine.Commands returns a sequence — every command but the last is a sidecar kept alive for the shard's lifetime and torn down with it, and the last one's exit is the shard's. vLLM under Ray is why: the leader runs a Ray head and then the server against it, while a worker runs only the join and no vLLM process at all. RayCommand had been written and tested but never called, so launch_mode: ray would have run vllm serve against a cluster that did not exist. Sidecars are killed with the shard, because a Ray head left behind holds the port the next replica needs. Start order was backwards. Workers went first, on the theory that the head should wait for them. Both engines put the rendezvous point on rank 0, so a worker started first has nothing to connect to. vLLM's server blocks until the Ray cluster has the GPUs it needs, so the leader can start immediately and waits on its own. A command can now also require that rendezvous point to be listening before the next starts, bounded by a timeout so a head that never appears fails the shard instead of hanging it. Stop was also killing only the engine, leaving sidecars behind. None of this is verified against real engines — it is tested with stub processes. The Ray sequencing, and which launch mode a given vLLM version wants, can only be confirmed on hardware. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
75a8e76706 |
Survive a control plane restart without orphaning live engines
Not persisting replica state left one hole: restarting the control plane made it forget every replica while the engines kept running. They held GPUs the inventory believed were free, so the next placement failed to allocate, with an error naming the new process rather than the old one. The fix belongs on the agent, not in a database. A row can say "running" about a process that died; the fleet cannot. - the supervisor writes the full shard spec, not just a pid and command line. A pid identifies a process; the spec is what lets it be described back to the control plane and matched to a replica. - Adopt takes over an engine a previous agent left running. It is not this process's child, so it cannot be waited on: it is watched by polling and stopped by signalling its group directly. Its earlier output is gone and the log says so rather than presenting emptiness as silence. - every Register reports what the agent is running, not only the first, because a reconnect may be a control plane restart and has to recover the same way. - the reconciler rebuilds replicas from those reports and reclaims their GPUs. Anything whose model or profile left the catalog is stopped; nothing will route to it and it is holding cards. - an engine the agent cannot identify is killed. A process holding GPUs that nothing can describe is worse than no process, because placement hands those GPUs out again. Stop was also wrong for adopted shards: it checked for an exec.Cmd, which they do not have, and reported them stopped without killing anything. TestAgentReconnects had been passing for no reason. httptest's CloseClientConnections does nothing to an h2c stream — prior-knowledge h2c hijacks the connection and httptest stops tracking hijacked ones — so the agent never reconnected, and the assertion on the connection map passed on a stale entry. Tests now sever connections through their own listener and count opened streams. Plan §11 also records what multi-node still needs, since placement landing early makes it look further along than it is: workers never become Ready because the agent health-checks every shard while only rank 0 serves HTTP; RayCommand is written and tested but never called; and startup is fire-and-forget. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
40427b2705 |
plan: correct the store design and specify how tenancy is actually stored
Two problems with what v0.1 said about persistence. First, it listed eight tables, most of which were a mistake. A database that describes the cluster can be wrong; the cluster cannot. An engine that dies during a restart leaves a row saying "running", and the scheduler believes it. Nodes, replicas, shards and the catalog are all rebuilt from the fleet or from configuration on every reconnect, so they need no rows at all. What genuinely needs persisting — API keys, usage, capacity profiles — is entirely phase 5 work, so the store moves there and gets a schema shaped by real requirements instead of guesses. Not persisting replica state has one consequence that must be closed first: restarting the control plane currently orphans live engines. They keep holding GPUs the inventory believes are free, and the next placement fails to allocate with an error pointing at the new process rather than the old one. The fix belongs on the agent, not in a database — Register carries the shards the agent is supervising, and the supervisor's orphan scan (written, tested, never called) kills whatever nothing claims. That cannot disagree with reality, and it covers agent restarts and control plane restarts with one mechanism. Added to phase 3. Second, §7.9 described tenancy behaviour but never said how any of it is stored, which left the most security-sensitive part of the system unspecified. It now covers the schema for keys and the tenancy tree, why a fast hash is the correct choice for a 256-bit random key and what assumption that rests on, how limits resolve to the most restrictive value along the chain, why authentication is a map lookup rather than four queries, the batching pipeline that keeps 86 million rows a day off the request path, what "units" mean when self-hosted models have no per-token price, and how budgets survive a restart. Two tradeoffs are now stated rather than left implicit: usage buffered for the batch writer is lost on a hard kill, which is fine for budgets and not fine for external billing; and rate limit counters are per instance, so N routers allow N times the limit. Both are in the risk table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
2373726efd |
Agent stream: connect the fleet to the control plane
One bidirectional connect-go stream per node, carrying registration, heartbeats and shard status up, and start, stop and drain down. The agent dials out: GPU nodes sit behind firewalls far more often than a control plane does, and an outbound connection needs no inbound rule on every node in the fleet. - registration carries the whole hardware inventory, so there is no separate discovery call and a reconnecting agent refreshes what the scheduler knows. The fabric labels travel with it; they cannot be inferred from the wire, so losing them in transit would silently disable multi-node placement. - agentsvc is both the stream handler and the reconciler's dispatcher, because it holds the open streams and is therefore the only thing that can deliver a command to a node. - the agent validates a profile before launching anything, since an engine that refuses a profile does so after minutes of loading, then supervises the process and reports every transition with the engine's output tail on failure. - reconnects forever with jittered backoff. A control plane restart must not require touching every GPU node, and a fleet must not reconnect in lockstep and knock it over again. Engines keep serving throughout, because the router talks to them directly. llmgw agent is real now, and llmgw server runs a control plane whenever any model declares a profile. The static endpoint list stays for router-only runs. Verified with real binaries as separate processes: the agent registers, a replica is placed, an engine launches through the vLLM adapter's actual argument construction, streaming traffic is served, and killing the agent moves the node through unreachable to lost with placement explaining why nothing can run. config.Model and catalog.Model were the same thing described twice, so the router now reads catalog entries directly and the validation rules live in one place. Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted network until that lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
c1b9ea1bc2 |
Phase 3 (3/3): wire the control plane to the router
Ready replicas become router endpoints with no static configuration involved. Endpoints that survive an update keep their live state — inflight counts, breaker state, drain rate — so a replica that was already serving does not start from cold because another one moved. The sink is an interface, so when the router and control plane split into separate processes it becomes a watch stream and nothing above it changes. Phase 3's exit criterion now holds end to end. A node hosting a live replica is killed while four clients stream continuously; the control plane marks it unreachable, then lost, rebuilds the replica on a survivor and republishes. Across three runs ~370 requests completed with zero failures and no replica was left on the dead node. This proves the control logic, not the transport: the agent fleet is simulated and the real path still needs the stream. The race detector caught a genuine bug while this went in. startReplica read the replica's fields after publishing it to the map, and scaleUp walked replica pointers outside the lock; both raced with agent status reports, which arrive from another goroutine and mutate exactly those fields. Against the simulated fleet the reports came back fast enough to hit the window on every run. State is snapshotted under the lock now and those functions work from values. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
f0134a2f95 |
Phase 3 (2/3): catalog and reconciler
The desired-versus-actual loop, with a Dispatcher interface where the agent stream will go. The loop's logic is where the interesting failures live, not the transport, so it is tested against a fake dispatcher and a synthetic clock. - catalog: models, profiles, aliases, fallbacks, scaling bounds. Aliases and fallbacks resolve at load rather than at request time, because a typo in a fallback should stop a deployment instead of surfacing during the outage when the fallback is finally needed. The node/GPU/TP arithmetic is checked here too, so a bad profile fails at config load rather than minutes into loading. - reconciler: one replica placed per pass, since placement depends on what is already allocated and several from one snapshot would be handed the same GPUs. Workers start before rank 0: SGLang's ranks rendezvous and vLLM under Ray needs its workers attached before the head serves. - a failed shard fails the whole replica. The others are waiting on collectives that will never complete while holding GPUs, so every shard is stopped and the replica rebuilt, with a jittered exponential backoff carried on the model rather than on the replica that was just destroyed. - node expiry has two stages, Unreachable then Lost, deliberately far apart: a network blip must not tear down replicas that are still serving, and rebuilding a multi-node replica costs tens of minutes of model loading. - scale-down drains before stopping, so in-flight requests finish. The phase 3 exit criterion is half met. TestLostNodeMovesReplicas covers the reconciliation half end to end. The router continuing to serve throughout needs the EndpointSource wiring and the agent stream. Writing the tests caught a real property of the design worth recording: advancing the clock makes every node look silent, so placement correctly stops. The test harness now heartbeats nodes as it advances time, except where a node going silent is the point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
06672d1b85 |
Phase 3 (1/3): cluster inventory and placement
The two hard constraints from plan §1.1 are enforced structurally rather than checked after the fact: there is no code path that can combine nodes from two fabrics, and only GPUs of the profile's class are ever considered. Getting either wrong produces a replica that starts, passes health checks, and then hangs or crawls under load — which looks like success until traffic arrives. - inventory: nodes, hardware, fabric attachments, allocations. A re-registering agent cannot clear an operator's cordon, since a cordon is usually in place because something is wrong with the node. A GPU with an ECC fault or another process on it is not free in any useful sense. Replicas are released whole, because they fail whole. - placement: highest-priority profile that fits, with every rejected candidate explained. "Unschedulable" on its own tells an operator nothing about which of six constraints bit. - bin packing fills partly used nodes first so whole nodes stay free for large models that cannot be split; multi-node replicas get the emptiest machines instead, being the least movable thing in the cluster. - NVLink-aware GPU selection keeps a tensor-parallel group inside one domain where the topology is known, falling back to lowest free indices otherwise. Property-based tests generate 400 random clusters and assert the invariants on every placement produced: one fabric per multi-node replica, only that fabric's HCAs per shard, one GPU class, no unhealthy or already-allocated GPU. The table-driven tests cover the cases someone thought of; these cover the rest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
55417546fc |
Phase 2 (2/2): NVML backend behind a build tag
NVML is the only source of NVLink topology, and that topology is what keeps a tensor-parallel group off the slow path inside a node. It needs cgo, so it sits behind the nvml tag and the default build keeps using nvidia-smi. Discover() picks the backend and falls back to nvidia-smi when NVML fails, logging the fallback rather than performing it silently: on a node that is meant to have working NVML, a quiet downgrade hides a broken driver installation and costs the scheduler its topology. CI now compiles the tagged build. That immediately paid for itself: ErrNoNVML was declared only in the untagged file, so the tests did not compile under -tags nvml. It is now declared where both builds can see it. Not verified: the NVML path has never run against a driver, because there is no GPU machine available. The NVLink domains it reports should be treated as unverified until someone runs `llmgw agent inventory` on a real node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
6a13e3dddf |
Phase 2 (1/2): node discovery, engine adapters, process supervisor
Everything here works without GPUs and is covered by tests. The control plane connection is not built yet, so llmgw agent itself is still a stub and the phase exit criterion is not met. - discovery: IB from /sys/class/infiniband, parsing the formats sysfs actually uses and telling native InfiniBand from RoCE; GPUs via nvidia-smi, the cgo-free path, with health flags for uncorrected ECC and memory already held by another process. A node without IB is not an error, it is a single-node island. - engines: vLLM and SGLang command and environment construction, kept free of process handling so it stays testable. The NCCL environment is the part the plan marks critical: NCCL_IB_HCA lists only the chosen fabric's adapters, because leaving it open lets NCCL pick an adapter on a fabric where the peer is unreachable, and the job then hangs instead of failing. - validation before launch: no mixed GPU types in a tensor-parallel group, the node/GPU/TP arithmetic has to add up, and fp8 is refused below compute capability 8.9 rather than silently falling back to a slower path. - supervisor: process groups, because engines fork workers that hold GPU memory and outlive a signal sent only to the leader; a bounded log tail, since engine output is the only diagnosis when loading fails; Starting separate from Loading, because a 70B model legitimately takes minutes; and orphan adoption through pid files fingerprinted with the command line, since pids get reused and adopting an unrelated process is worse than leaking the original. - readiness is two questions: /health says the process is up, /v1/models says the weights loaded. Routing on /health alone sends traffic into a loading model. - llmgw agent inventory reports what the scheduler would see on a node. One plan correction: GPU class sizes come from the driver in GiB, which does not always match the marketing name. An L40S is sold as 48GB and reports 44.99 GiB, so its class is l40s-45g, not l40s-48g. Consistency matters more than matching the box, because that number decides whether a model fits. Plan §2 and §5.1 updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
dc38dd6096 |
Phase 1.5: admission control
Inserted between quota checks and scheduling: admission answers when a request may proceed, the scheduler still answers where it goes. Keeping them apart is the point — an engine queue is FIFO and knows nothing about tenants, so once a batch job is in it, no endpoint selection rescues the request behind it. - saturation detector: one number per pool, the worse of queue depth and KV occupancy, since either alone misleads. - priority bands with per-band saturation ceilings, so the cluster degrades from the bottom up. - tenant round-robin within a band: one caller sending a thousand requests takes one slot in the rotation, not a thousand. - bounded queues with capacity (503) and TTL (504); cancelled requests leave the queue instead of holding capacity. - metrics for queue depth, wait time, rejections and pool saturation. Exit criterion, measured in test/e2e: with 500 concurrent bronze requests running continuously, interactive p99 in the platinum band stays at 1.04-1.24x idle against a 1.5x budget. With admission off the same load pushes it to 55-60x. Both directions are asserted, so the test fails if admission regresses and also if the load generator stops being a real load. Two lags had to be closed, and they needed different answers. Engine metrics are scraped and lag an interval; router inflight does not, but it counts running requests, and treating those as queue makes a healthy engine look overloaded every interval. What is actually queued is inflight minus batch capacity, and capacity is not in the metric contract, so it is learned as the high-water mark of observed running. Separately, between admission granting a request and the router dispatching it the request is invisible everywhere, so a simultaneous burst passed the check en masse; the controller now counts its own outstanding leases against dispatched inflight. One plan correction: a saturation ceiling of 1.0 was documented as "never held back". It is not — it means "flow up to the healthy limit". A band that must never wait needs a ceiling above 1. Plan §7.2.2 updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
a3f71aa5ee |
Phase 1: the router serves traffic
llmgw server is real now. It fronts a static set of engine replicas with an OpenAI-compatible API, scoring endpoints through the pipeline added in the previous commit. - proxy: SSE relayed event by event with a flush after each, cancellation propagated upstream, and separate deadlines for dial, headers, first token, inter-token idle and total. The idle watchdog is what distinguishes a wedged engine from a slow one. - router: dispatch with retry and model fallback. Retry only happens while nothing has reached the client; after that, splicing two responses together would be worse than the short stream the client already sees. - config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides. Validation rejects duplicate endpoint ids and aliases that shadow a model, because both misroute silently at request time rather than failing loudly. - metrics: requests, duration, TTFT, tokens, plus per-scorer values and per-filter drops on a separate admin listener with health and pprof. Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across 3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV blocks on every engine; router overhead p99 is about 0.9 ms against the plan's 2 ms budget. The handlers live in internal/router rather than an internal/router/openai package as the plan's tree sketched: they need the dispatch loop, and splitting them apart would have meant an interface with exactly one implementation. Plan §4 updated to match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
28029eb43b |
Switch the project language to English
Comments, doc comments, test names, error and log messages, CLI help, README, Makefile, CI step names and the design document are now all in English. No behaviour changes; the full suite passes under the race detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e07bc46c57 |
Faz 0: iskelet ve sim-engine
Plan v0.2'nin Faz 0 çıktısı. Yazılan sıraya göre:
- cobra tabanlı CLI iskeleti; server/agent/ctl komutları henüz stub.
- internal/sim: OpenAI uyumlu sahte inference engine. Router'ı GPU olmadan
gerçekçi koşullarda test edebilmek için gerçek bir engine'in router
açısından önemli davranışlarını modeller:
* continuous batching ve KV blok muhasebesi (head-of-line blocking dahil)
* zincirlenmiş blok hash'leri + LRU ile otomatik prefix cache
* batch büyüdükçe artan token gecikmesi
* istemci koptuğunda isteğin düşürülüp blokların bırakılması
* vLLM metrik adlarıyla /metrics (plan §7.6 engine kontratı)
- Testler: prefix hash zincirlemesi, LRU tahliyesi, iptal propagasyonu,
kuyruklama, SSE formatı, usage/cached_tokens, metriklerin gerçek
Prometheus parser'ıyla okunabilirliği.
- Gitea Actions CI: gofmt, vet, race detector.
Modül yolu gitea.mynodes.xyz/ilker/llmgw; `make rename` ile değiştirilebilir.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|